From: "наб via Libc-alpha" <libc-alpha@sourceware.org>
To: Florian Weimer <fweimer@redhat.com>
Cc: libc-alpha@sourceware.org
Subject: Re: [PATCH] POSIX locale covers every byte [BZ# 29511]
Date: Tue, 6 Sep 2022 20:06:57 +0200 [thread overview]
Message-ID: <20220906180657.eo53xia2yqqiv4ri@tarta.nabijaczleweli.xyz> (raw)
In-Reply-To: <87a67c60x6.fsf@oldenburg.str.redhat.com>
[-- Attachment #1: Type: text/plain, Size: 5588 bytes --]
Hi!
On Tue, Sep 06, 2022 at 04:19:01PM +0200, Florian Weimer wrote:
> * наб via Libc-alpha:
>
> > This is a trivial patch, largely duplicating the extant ASCII code
> >
> > There are two user-facing changes:
> > * nl_langinfo(CODESET) is "POSIX" instead of "ANSI_X3.4-1968"
> > * mbrtowc() and friends return b if b <= 0x7F else <UDF00>+b
> >
> > Since Issue 7 TC 2/Issue 8, the C/POSIX locale, effectively:
> > (a) is 1-byte, stateless, and contains 256 characters
> > (b) which collate in byte order
> > (c) the first 128 characters are equivalent to ASCII (like previous)
> > cf. https://www.austingroupbugs.net/view.php?id=663 for a summary of
> > changes to the standard;
> > in short, this means that mbrtowc() must never fail and must return
> > b if b <= 0x7F else ab+c for all bytes b
> > where c is some constant >=0x80
> > and a is a positive integer constant
> >
> > By strategically picking c=<UDF00> we land at the tail-end of the
> > Unicode Low Surrogate Area at DC00-DFFF, described as
> > > Isolated surrogate code points have no interpretation;
> > > consequently, no character code charts or names lists
> > > are provided for this range.
> > and match musl
>
> We don't match Python and its surrogateescape encoding (PEP 838).
404?
> It
> maps invalid bytes in the 0x80…0xff range to U+DC80…U+DCFF.
(The same as musl.)
> It may make
> more sense to align with that.
With a=1 and c=<UDF00>, assuming it's as you say, we very much do?
$ printf '\x80\xff' | output/elf/ld.so --library-path output/ output/iconv/iconv_prog -fPOSIX -tUCS4 | hd
00000000 00 00 df 80 00 00 df ff |........|
00000008
> Anyway, regarding mechanics, we'll need a new localedata/charmaps/POSIX
> charmap, I think. This charmap then can be tested against the gconv
> converter.
Hm, the problem with that is tst-tables -> tst-table -> tst-table-from
(and -to) convert by constructing a UTF-8 sequence. The problem with
this approach is that glibc rejects unpaired surrogates.
The output for tst-table-from UTF-8 is:
...
0xED9FBE 0xD7FE
0xED9FBF 0xD7FF
0xEE8080 0xE000
0xEE8081 0xE001
...
i.e. there's a gap for the surrogates; and, indeed, the charmap reads
<UD7FB> /xed/x9f/xbb HANGUL JONGSEONG PHIEUPH-THIEUTH
%<UD800> /xed/xa0/x80 <Non Private Use High Surrogate, First>
%<UDB7F> /xed/xad/xbf <Non Private Use High Surrogate, Last>
%<UDB80> /xed/xae/x80 <Private Use High Surrogate, First>
%<UDBFF> /xed/xaf/xbf <Private Use High Surrogate, Last>
%<UDC00> /xed/xb0/x80 <Low Surrogate, First>
%<UDFFF> /xed/xbf/xbf <Low Surrogate, Last>
<UE000>..<UE03F> /xee/x80/x80 <Private Use>
with the surrogate range commented-out;
this dates back to the inclusion of UTF-8 generator scripts in 2015
(4a4839c94a4c93ffc0d5b95c69a08b02a57007f2), these exclusions are
deliberate (grep for surrog in localedata/unicode-gen/utf8_gen.py).
Given this limitation, expanding the charmap to
ANSI_X3.4-1968 + <UDF80>..<UDFFF> doesn't actually test much:
having them as separate codepoints will always fail tests,
and dot-notation lines are ignored when generating the comparison
tables, so this particular type of test just proves that POSIX is the
same as ANSI_X3.4-1968 for the first 128 characters.
There's already an exhaustive iconv_prog-based testsuite
(cf. additions to iconv/tst-iconv_prog.sh), though.
> You should put the new converters into a separate file (not
> iconv/gconv_simple.c), then the s390x version will use that
> automatically.
Oh, of course! Moved to iconv/gconv_posix.c.
> > diff --git a/localedata/locales/POSIX b/localedata/locales/POSIX
> > index 7ec7f1c577..fc34a6abc1 100644
> > --- a/localedata/locales/POSIX
> > +++ b/localedata/locales/POSIX
> > @@ -97,6 +97,20 @@ END LC_CTYPE
> > LC_COLLATE
> > % This is the POSIX Locale definition for the LC_COLLATE category.
>
> Isn't this just the C locale?
Yes, C is defined to be POSIX.
> We don't have a separate file for that.
Yes, we very obviously do, seeing as this patch edits it?
Nothing consumes it AFAICT, but.
> > diff --git a/wcsmbs/wcsmbsload.c b/wcsmbs/wcsmbsload.c
> > index 0f0f55f9ed..f87099bcf5 100644
> > --- a/wcsmbs/wcsmbsload.c
> > +++ b/wcsmbs/wcsmbsload.c
> > @@ -33,10 +33,10 @@ static const struct __gconv_step to_wc =
> > .__shlib_handle = NULL,
> > .__modname = NULL,
> > .__counter = INT_MAX,
> > - .__from_name = (char *) "ANSI_X3.4-1968//TRANSLIT",
> > + .__from_name = (char *) "POSIX",
> > .__to_name = (char *) "INTERNAL",
> > - .__fct = __gconv_transform_ascii_internal,
> > - .__btowc_fct = __gconv_btwoc_ascii,
> > + .__fct = __gconv_transform_posix_internal,
> > + .__btowc_fct = __gconv_btwoc_posix,
> > .__init_fct = NULL,
> > .__end_fct = NULL,
> > .__min_needed_from = 1,
> > @@ -53,8 +53,8 @@ static const struct __gconv_step to_mb =
> > .__modname = NULL,
> > .__counter = INT_MAX,
> > .__from_name = (char *) "INTERNAL",
> > - .__to_name = (char *) "ANSI_X3.4-1968//TRANSLIT",
> > - .__fct = __gconv_transform_internal_ascii,
> > + .__to_name = (char *) "POSIX",
> > + .__fct = __gconv_transform_internal_posix,
> > .__btowc_fct = NULL,
> > .__init_fct = NULL,
> > .__end_fct = NULL,
>
> This makes the comment on __wcsmbs_gconv_fcts_c in the same file
> obsolete.
Comment fixed.
> Thanks,
> Florian
New patchset in followup.
Best,
наб
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2022-09-06 18:07 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-08-30 18:19 [PATCH] POSIX locale covers every byte [BZ# 29511] наб via Libc-alpha
2022-09-06 14:06 ` [PATCH v2] " наб via Libc-alpha
2022-09-06 14:19 ` [PATCH] " Florian Weimer via Libc-alpha
2022-09-06 18:06 ` наб via Libc-alpha [this message]
2022-09-06 18:10 ` [PATCH v3 1/2] iconvdata/tst-table-charmap.sh: remove handling of old, borrowed format наб via Libc-alpha
2022-09-14 2:39 ` [PATCH v4 " наб via Libc-alpha
2022-09-21 14:01 ` [PATCH v5 " наб via Libc-alpha
2022-11-02 17:17 ` [PATCH v6 " наб via Libc-alpha
2022-11-09 12:49 ` Florian Weimer via Libc-alpha
2022-11-02 17:17 ` [PATCH v6 2/2] POSIX locale covers every byte [BZ# 29511] наб via Libc-alpha
2022-11-09 14:20 ` Florian Weimer via Libc-alpha
2022-11-09 16:14 ` [PATCH v7] " наб via Libc-alpha
2022-11-10 9:52 ` Florian Weimer via Libc-alpha
2023-01-09 15:17 ` [PATCH v8] " наб via Libc-alpha
2023-02-07 14:16 ` [PATCH v9] " наб via Libc-alpha
2023-02-13 14:52 ` Florian Weimer via Libc-alpha
2023-04-26 18:54 ` наб via Libc-alpha
2023-04-26 21:27 ` Florian Weimer via Libc-alpha
2023-04-27 0:17 ` [PATCH v10] " наб via Libc-alpha
2023-04-28 15:43 ` [PATCH v11] " наб via Libc-alpha
2023-05-07 22:53 ` [PATCH v12] " наб via Libc-alpha
2023-05-29 13:54 ` [PATCH v13] " наб via Libc-alpha
2022-11-10 8:10 ` [PATCH v6 2/2] " Florian Weimer via Libc-alpha
2022-11-28 16:24 ` наб via Libc-alpha
2022-12-02 17:36 ` Florian Weimer via Libc-alpha
2022-12-02 18:42 ` наб via Libc-alpha
2022-09-21 14:01 ` [PATCH v5 " наб via Libc-alpha
2022-09-14 2:39 ` [PATCH v4 " наб via Libc-alpha
2022-09-06 18:11 ` [PATCH v3 " наб via Libc-alpha
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/libc/involved.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220906180657.eo53xia2yqqiv4ri@tarta.nabijaczleweli.xyz \
--to=libc-alpha@sourceware.org \
--cc=fweimer@redhat.com \
--cc=nabijaczleweli@nabijaczleweli.xyz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).