From: Rafal Luzynski <digitalfreak@lingonborough.com>
To: Egor Kobylkin <egor@kobylkin.com>,
libc-alpha@sourceware.org, libc-locales@sourceware.org
Subject: Re: [PATCH v9] Locales: Cyrillic -> ASCII transliteration table [BZ #2872]
Date: Fri, 16 Nov 2018 23:17:27 +0100 (CET) [thread overview]
Message-ID: <837001401.21346.1542406647888@poczta.nazwa.pl> (raw)
In-Reply-To: <b82fe65b-b880-a2b5-c97d-2a6aae9c1165@kobylkin.com>
Thank you for working on this, Egor.
Before I start reviewing I would like to summarize the things which
I think are blocking for this patch.
1. I think we need tests for transliteration. Currently there is only
one test program which is similar to what we need,
localedata/bug-iconv-trans.c. It is old and it is not quite clear
what bug it is trying to test. Therefore I think we need a new
framework to test transliteration. Is it a good idea to base the
test on the iconv(1) command line utility which is part of glibc?
2. I made few tests in the command line and it seems to me that the
transliteration from "З" to "Z" (+ lowercase as well) in uk_UA does
not work and has not been working for some time already because
I've checked some older systems as well and the result is always
the same. I think that the reason is that uk_UA defines multiple
transliteration rules for "З" depending on what is the letter following
it. It does not seem to work. AFAIK the reason is that the syntax of
transliteration rules says that a single non-Latin character may map
one or more Latin strings, each consisting of one or more characters.
There cannot be a rule transliterating multiple source characters into
one or multiple destination characters. Is it a bug in transliteration
implementation? Or maybe in the specification, including POSIX standard?
The definition of transliteration says that it is one-to-one mapping
of graphemes while a grapheme may be one or multiple characters.
It does not have to be always mapping one-to-one character. Should we
fix this bug first, make uk_UA transliteration work, and only then
add a generic Cyrillic transliteration? Egor's patch already contains
transliteration of "У" + combining acute accent to "Ú" which most
probably
will not work.
I still think that in the longer term all existing custom transliterations
of Cyrillic alphabets should be ported to a modification of your patch.
Egor, while at this I was thinking about your idea to transliterate letters
like "Ш" (uppercase) to "SH" (always uppercase) in order to distinguish
between "Шема" (-> "SHema") and "Схема" (-> "Shema" or "Sxema"). Also
you include a rule to transliterate "Х" to "H" or "X" depending on which
destination characters are available, which I told you already that will
not work because both "H" and "X" are always available and therefore only
the first rule will always be used. I still don't like the idea to
put two uppercase letters in a beginning of a word in titlecase only to
indicate that there was originally a single letter. What if we:
* drop the rule of transliterating "Х" to "H" and transliterate always to
"X",
* transliterate uppercase "Ш" to "Sh" (so it will work fine for titlecase
words)?
As a result the Latin letter "h" will only appear as part of a digraph and
never as a transliteration of "Х" and therefore will never cause a conflict.
Examples:
* "Шема" -> "Shema",
* "Схема" -> "Sxema".
Will this solve the problem?
Regards,
Rafal
next prev parent reply other threads:[~2018-11-16 22:17 UTC|newest]
Thread overview: 111+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <41532e13-a63d-5df1-ab37-05eb4d6c8d0a@kobylkin.com>
[not found] ` <20180412224352.GB2911@altlinux.org>
2018-07-17 19:34 ` SUBJECT: [PATCH] Locales: Cyrillic -> ASCII transliteration table [BZ #2872] Egor Kobylkin
2018-07-17 19:40 ` Carlos O'Donell
2018-07-17 19:50 ` Egor Kobylkin
2018-07-17 19:59 ` Carlos O'Donell
2018-08-06 19:00 ` [PATCH] Locales: Cyrillic -> ASCII transliteration table [BZ #2872] re-submission for 2.29 Egor Kobylkin
2018-10-03 8:26 ` Egor Kobylkin
2018-10-03 9:19 ` Keld Simonsen
2018-10-03 9:32 ` Egor Kobylkin
2018-10-05 8:43 ` Marko Myllynen
2018-10-05 9:20 ` Rafal Luzynski
2018-10-05 10:36 ` Egor Kobylkin
2018-10-08 22:04 ` Rafal Luzynski
2018-10-08 22:52 ` Egor Kobylkin
2018-10-09 21:43 ` Rafal Luzynski
2018-10-08 23:20 ` Zack Weinberg
2018-10-09 15:26 ` Carlos O'Donell
2018-10-09 21:51 ` Rafal Luzynski
2018-10-09 16:10 ` Marko Myllynen
2018-10-09 16:22 ` Egor Kobylkin
2018-10-09 16:49 ` Marko Myllynen
2018-10-09 22:08 ` Rafal Luzynski
2018-10-10 11:21 ` Marko Myllynen
2018-10-11 10:10 ` Marko Myllynen
[not found] ` <deacdf31-d0bb-a92d-1de3-934d6b4cb158@kobylkin.com>
2018-10-05 11:54 ` Marko Myllynen
2018-10-05 12:00 ` Egor Kobylkin
2018-10-05 12:21 ` Marko Myllynen
2018-10-05 20:47 ` Egor Kobylkin
2018-10-08 12:40 ` Marko Myllynen
2018-10-08 22:23 ` Rafal Luzynski
2018-10-08 23:35 ` Egor Kobylkin
2018-10-09 13:18 ` Egor Kobylkin
2018-10-09 18:34 ` Egor Kobylkin
2018-10-09 22:17 ` Rafal Luzynski
2018-10-09 22:40 ` Egor Kobylkin
2018-10-09 22:42 ` Egor Kobylkin
2018-10-10 11:22 ` Marko Myllynen
2018-10-10 12:19 ` Egor Kobylkin
2018-10-10 12:34 ` Marko Myllynen
2018-10-10 22:29 ` [PATCH] Locales: Cyrillic -> ASCII transliteration table [BZ #2872] v2 Egor Kobylkin
2018-10-11 9:59 ` Marko Myllynen
2018-10-11 11:04 ` Rafal Luzynski
2018-10-11 13:10 ` Marko Myllynen
2018-10-11 13:50 ` Volodymyr Lisivka
2018-10-11 14:59 ` Egor Kobylkin
2018-10-11 21:30 ` Egor Kobylkin
2018-10-11 15:05 ` Egor Kobylkin
2018-10-11 15:44 ` [PATCH] Locales: Cyrillic -> ASCII transliteration table [BZ #2872] v3 Egor Kobylkin
2018-10-11 21:33 ` [PATCH] Locales: Cyrillic -> ASCII transliteration table [BZ #2872] v4 Egor Kobylkin
2018-10-12 14:05 ` [PATCH v5] Locales: Cyrillic -> ASCII transliteration table [BZ #2872] Egor Kobylkin
2018-10-13 0:59 ` Rafal Luzynski
2018-10-13 16:58 ` Egor Kobylkin
2018-10-15 11:04 ` Marko Myllynen
2018-10-15 11:54 ` Egor Kobylkin
2018-10-23 23:08 ` Rafal Luzynski
2018-10-17 14:16 ` [PATCH v6] " Egor Kobylkin
2018-11-01 22:51 ` [PATCH v7] " Egor Kobylkin
2018-11-02 0:00 ` [PATCH v8] " Egor Kobylkin
2018-11-02 22:22 ` Rafal Luzynski
2018-11-02 23:27 ` Egor Kobylkin
2018-11-14 21:25 ` [PATCH v9] " Egor Kobylkin
2018-11-16 22:17 ` Rafal Luzynski [this message]
2018-11-17 18:34 ` Egor Kobylkin
2018-11-19 7:13 ` Marko Myllynen
2018-11-19 9:21 ` Egor Kobylkin
2018-11-19 19:35 ` Marko Myllynen
2018-12-01 22:07 ` Rafal Luzynski
2018-12-01 22:53 ` Egor Kobylkin
2018-12-03 22:19 ` Egor Kobylkin
2018-12-08 1:15 ` Rafal Luzynski
2018-12-10 21:20 ` Marko Myllynen
2018-12-19 22:25 ` Rafal Luzynski
2018-12-19 22:48 ` Egor Kobylkin
2018-12-19 23:50 ` Rafal Luzynski
2018-11-19 11:10 ` [PATCH v10] " Egor Kobylkin
2018-12-07 23:35 ` Rafal Luzynski
2018-12-08 21:51 ` Egor Kobylkin
2018-12-19 22:41 ` Rafal Luzynski
2018-12-19 23:02 ` Egor Kobylkin
2018-12-20 0:05 ` Rafal Luzynski
2018-12-08 22:28 ` [PATCH v11] Locales: Cyrillic -> ASCII transliteration " Egor Kobylkin
2018-12-19 23:16 ` Egor Kobylkin
2018-12-26 10:07 ` Siddhesh Poyarekar
2018-12-26 12:13 ` Egor Kobylkin
2018-12-27 1:30 ` Siddhesh Poyarekar
2018-12-27 11:28 ` Rafal Luzynski
2019-01-02 18:38 ` [PATCH v12] " Egor Kobylkin
2019-01-05 14:35 ` Rafal Luzynski
2019-01-05 21:12 ` Egor Kobylkin
2019-01-07 20:37 ` Marko Myllynen
2019-01-09 0:46 ` Egor Kobylkin
2019-01-09 20:03 ` Marko Myllynen
2019-02-04 7:14 ` [PATCH v12] Locales: Cyrillic -> ASCII transliteration [BZ #2872] ping for 2.30 Egor Kobylkin
2019-02-14 16:48 ` Marko Myllynen
2019-03-04 22:11 ` Egor Kobylkin
2019-03-11 13:59 ` PING " Egor Kobylkin
2019-03-14 19:48 ` Egor Kobylkin
2019-04-19 22:24 ` Rafal Luzynski
[not found] ` <5ELixS9SQ0DW4mlvswp96ASpLobBabU9KQ6zOTH-Udrb34mABhcqiPERpBZfPWZ9F77s8XNmiLIAq9UWu0AjLFFdjOz_FZVU5_xF-SiQkrw=@kobylkin.com>
2019-04-27 2:51 ` Siddhesh Poyarekar
2019-04-27 7:34 ` Diego (Egor) Kobylkin
2019-04-09 1:04 ` [PATCH v12] Locales: Cyrillic -> ASCII transliteration [BZ #2872] Carlos O'Donell
2019-03-19 10:39 ` ping " Egor Kobylkin
2019-03-28 16:20 ` [PING^4][PATCH " Marko Myllynen
2019-04-04 19:44 ` [PING^5][PATCH " Egor Kobylkin
2019-04-06 1:36 ` Siddhesh Poyarekar
2019-04-16 7:15 ` [PING^6][PATCH " Marko Myllynen
2019-04-16 13:17 ` Carlos O'Donell
2019-04-16 17:06 ` Egor Kobylkin
2019-04-16 17:58 ` Carlos O'Donell
2019-04-16 18:41 ` Egor Kobylkin
2019-04-16 19:06 ` Carlos O'Donell
2019-05-10 12:19 ` Marko Myllynen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/libc/involved.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=837001401.21346.1542406647888@poczta.nazwa.pl \
--to=digitalfreak@lingonborough.com \
--cc=egor@kobylkin.com \
--cc=libc-alpha@sourceware.org \
--cc=libc-locales@sourceware.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).