Re: [PATCH v9] POSIX locale covers every byte [BZ# 29511]

[Florian Weimer via Libc-alpha <libc-alpha@sourceware.org>]

* наб:

> This largely duplicates the ASCII code with the error path changed
>
> There are two user-facing changes:
>   * nl_langinfo(CODESET) is "POSIX" instead of "ANSI_X3.4-1968"
>   * mbrtowc() and friends return b if b <= 0x7F else <UDF00>+b
>
> Since Issue 7 TC 2/Issue 8, the C/POSIX locale, effectively,
>   (a) is 1-byte, stateless, and contains 256 characters
>   (b) they collate in byte order
>   (c) the first 128 characters are equivalent to ASCII (like previous)
> cf. https://www.austingroupbugs.net/view.php?id=663 for a summary of
> changes to the standard;
> in short, this means that mbrtowc() must never fail and must return
>   b if b <= 0x7F else ab+c for all bytes b
>   where c is some constant >=0x80
>     and a is a positive integer constant
>
> By strategically picking c=<UDF00> we land at the tail-end of the
> Unicode Low Surrogate Area at DC00-DFFF, described as
>   > Isolated surrogate code points have no interpretation;
>   > consequently, no character code charts or names lists
>   > are provided for this range.
> and match musl

I've thought about this some more, and I don't think this is the
direction we should be going in.

* Add a UTF-8SE charset to glibc: it's UTF-8 with surrogate encoding (in
  the Python style).  It should have the property that it can encode
  every byte string as a string of wchar_t characters, and convert the
  result back.  It's not entirely trivial because we need to handle
  partial UTF-8 sequences at the end of the buffer carefully.  There
  might be some warts regarding EILSEQ handling lurking there.  Like the
  Python approach, it is somewhat imperfect because it's not preserving
  identity under string concatenation, i.e. f(x) || f(y) is not always
  equal to f(x || y), but that's just unavoidable.

* Switch the charset for the default C locale to UTF-8SE.  This matches
  the POSIX requirement that every byte can be encoded.

* Work with POSIX to drop the requirement that the C locale needs to be
  a single-byte locale.

* (Optional, somewhat unrelated.) Add a generic mechanism so that UTF-8
  locales can be used as UTF-8SE without recompilation.

Thanks,
Florian