From: Carlos O'Donell via Libc-alpha <libc-alpha@sourceware.org>
To: Florian Weimer <fweimer@redhat.com>
Cc: libc-alpha@sourceware.org
Subject: Re: C.UTF-8 review
Date: Tue, 20 Jul 2021 23:11:59 -0400 [thread overview]
Message-ID: <45400f89-62ed-5449-bbe2-9cb47d27e24e@redhat.com> (raw)
In-Reply-To: <87o8b5ds5q.fsf@oldenburg.str.redhat.com>
On 7/14/21 7:57 AM, Florian Weimer wrote:
> Carlos,
>
> I reviewed your changes on the codonell/c-utf8 branch and have some
> comments.
>
> I believe the replacement character should U+FFFD REPLACEMENT CHARACTER,
> not U+003F QUESTION MARK:
Unfortunately we can't use U+FFFD.
During the final transliteration (__gconv_transliterate()) we use
_NL_CTYPE_TRANSLIT_DEFAULT_MISSING to replace characters that have no
transliteration.
When you convert from UTF-8 to ASCII, even with TRANSLIT,IGNORE the framework
attempts to use U+FFFD for the ASCII output and that immediately fails with
a conversion error.
So "default_missing" must be valid in all possible output encodings, which
means U+FFFD is not usable (today), and likely why U+003F is being used.
To fix this we would have to attempt a final transliteration of default_missing
within __gconv_transliterate, but I feel like that's a follow-on change we
could do in the future.
> +% Include the neutral transliterations. The builtin C and
> +% POSIX locales have +1600 transliterations that are built into
> +% the locales, and these are a superset of those.
> +translit_start
> +include "translit_neutral";""
> +default_missing <U003F>
> +translit_end
>
> I guess this menas that C and C.UTF-8 LC_CTYPE diverge.
Yes, in locale/C-ctype.c we have very minimal data.
For C.UTF-8 the LC_CTYPE is much broader and I think this meets
user expectations for developers and harmonizes across what downstreams
have learned from using C.UTF-8.
> The strcmp_collation keyword is reasonably explicit. I would appreciate
> if we could produce an error if it is used along with other collation
> directives. Right now, those are silently ignored.
>
> There is a trailing newline in localedata/locales/C.
Fixed.
> But overall it's very nice. We should have shipped this many years ago.
Agreed. All my fault here.
I'll update my branch tomorrow with a new test for C.UTF-8, I found a
defect in the current branch.
--
Cheers,
Carlos.
prev parent reply other threads:[~2021-07-21 3:12 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-07-14 11:57 C.UTF-8 review Florian Weimer via Libc-alpha
2021-07-14 23:06 ` Paul Eggert
2021-07-15 8:56 ` Florian Weimer via Libc-alpha
2021-07-19 15:33 ` Carlos O'Donell via Libc-alpha
2021-07-19 15:37 ` Florian Weimer via Libc-alpha
2021-07-21 3:11 ` Carlos O'Donell via Libc-alpha [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/libc/involved.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=45400f89-62ed-5449-bbe2-9cb47d27e24e@redhat.com \
--to=libc-alpha@sourceware.org \
--cc=carlos@redhat.com \
--cc=fweimer@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).