portability issues with unicodeio (was: [GNU Bison 3.6.90] testsuite: 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 196 220 221 228 244 245 246 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 549 555 562 567 577 failed)

portability issues with unicodeio (was: [GNU Bison 3.6.90] testsuite: 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 196 220 221 228 244 245 246 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 549 555 562 567 577 failed)

[Akim Demaille]

Hi!

Bison uses gnulib's unicodeio module to emit bullets (•) portably,
with a fallback to '.'.  It's implemented this way (src/gram.h):

> /* Fallback in case we can't print "•".  */
> static inline long
> print_dot_fallback (unsigned int code _GL_UNUSED,
>                     const char *msg _GL_UNUSED,
>                     void *callback_arg)
> {
>   FILE *out = (FILE *) callback_arg;
>   putc ('.', out);
>   return -1;
> }
> 
> /* Print "•", the symbol used to represent a point in an item (aka, a
>    dotted rule).  */
> static inline void
> print_dot (FILE *out)
> {
>   unicode_to_mb (0x2022, fwrite_success_callback, print_dot_fallback, out);
> }

Unfortunately on Kiyoshi's environment (SunOS hidden 5.11 11.3 i86pc i386 i86pc,
GCC 9.3.0) we get '?' instead of '.' in the C locale.  We get a genuine ASCII
'?', it's not some fallback from the terminal which fails to display the
character.  And we properly get the bullet with en_US.UTF-8.

Kiyoshi can reproduce the problem with GNU Coreutils' printf, where he
get's a '?', although the fallback display the escape sequence (i.e.,
it should repeat '\u2022'):

> /* Simple failure callback that displays a fallback representation in plain
>    ASCII, using the same notation as ISO C99 strings.  */
> static long
> fallback_failure_callback (unsigned int code,
>                            const char *msg _GL_UNUSED,
>                            void *callback_arg)
> {
>   FILE *stream = (FILE *) callback_arg;
> 
>   if (code < 0x10000)
>     fprintf (stream, "\\u%04X", code);
>   else
>     fprintf (stream, "\\U%08X", code);
>   return -1;
> }
> 
> /* Outputs the Unicode character CODE to the output stream STREAM.
>    Upon failure, exit if exit_on_error is true, otherwise output a fallback
>    notation.  */
> void
> print_unicode_char (FILE *stream, unsigned int code, int exit_on_error)
> {
>   unicode_to_mb (code, fwrite_success_callback,
>                  exit_on_error
>                  ? exit_failure_callback
>                  : fallback_failure_callback,
>                  stream);
> }



Kiyoshi's messages start here:

https://lists.gnu.org/r/bug-bison/2020-07/msg00001.html

The latest:

> Le 6 juil. 2020 à 22:35, Kiyoshi KANAZAWA <yoi_no_myoujou@yahoo.co.jp> a écrit :
> 
> Hi Akim,
> 
> $ LC_ALL=C $coreutilsbin/printf '\u2022\n' | od -t x1
> 0000000 3f 0a
> 0000002
> 
> $ LC_ALL=en_US.UTF-8 $coreutilsbin/printf '\u2022\n' | od -t x1
> 0000000 e2 80 a2 0a
> 0000004
> 
> 
> FYI, I have very limited locale.
> $ locale -a
> C
> POSIX
> en_US.ISO8859-1
> en_US.ISO8859-15
> en_US.ISO8859-15@euro
> en_US.UTF-8
> ja_JP.PCK
> ja_JP.UTF-8
> ja_JP.UTF-8@cldr
> ja_JP.eucJP

I'm unsure what the next steps would be from here.

Thanks in advance!

Re: portability issues with unicodeio

[Bruno Haible]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

Hi Akim, Kiyoshi-san,

> Unfortunately on Kiyoshi's environment (SunOS hidden 5.11 11.3 i86pc i386 i86pc,
> GCC 9.3.0) we get '?' instead of '.' in the C locale.  We get a genuine ASCII
> '?', it's not some fallback from the terminal which fails to display the
> character.  And we properly get the bullet with en_US.UTF-8.

I am adding a unit test for the 'unicodeio' module (see below). It contains
a test for the C locale, and this test works fine on Solaris 11.3.

Therefore I have to ask Kiyoshi two questions:

1) What is the output of the program 'locale'?
   $ locale

2) What are the link dependencies of the coreutils printf program?
   $ ldd $coreutilsbin/printf

Bruno


2020-07-08  Bruno Haible  <bruno@clisp.org>

	unicodeio: Add tests.
	* tests/test-unicodeio.c: New file.
	* tests/test-unicodeio1.sh: New file.
	* tests/test-unicodeio2.sh: New file.
	* tests/test-unicodeio3.sh: New file.
	* modules/unicodeio-tests: New file.

2020-07-08  Bruno Haible  <bruno@clisp.org>

	unicodeio: Document link requirements.
	* modules/unicodeio (Link): New section.


[-- Attachment #2: 0001-unicodeio-Document-link-requirements.patch --]
[-- Type: text/x-patch, Size: 1000 bytes --]

From 77157f813248fb7dfd4d854ab85d107a1fe026b6 Mon Sep 17 00:00:00 2001
From: Bruno Haible <bruno@clisp.org>
Date: Wed, 8 Jul 2020 20:00:00 +0200
Subject: [PATCH 1/2] unicodeio: Document link requirements.

* modules/unicodeio (Link): New section.
---
 ChangeLog         | 5 +++++
 modules/unicodeio | 3 +++
 2 files changed, 8 insertions(+)

diff --git a/ChangeLog b/ChangeLog
index 9cac500..aa36674 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,8 @@
+2020-07-08  Bruno Haible  <bruno@clisp.org>
+
+	unicodeio: Document link requirements.
+	* modules/unicodeio (Link): New section.
+
 2020-07-07  Bruno Haible  <bruno@clisp.org>
 
 	doc: Remove support for some very old platforms.
diff --git a/modules/unicodeio b/modules/unicodeio
index 8164ac4..51721db 100644
--- a/modules/unicodeio
+++ b/modules/unicodeio
@@ -24,6 +24,9 @@ lib_SOURCES += unicodeio.h unicodeio.c
 Include:
 "unicodeio.h"
 
+Link:
+$(LTLIBICONV) when linking with libtool, $(LIBICONV) otherwise
+
 License:
 GPL
 
-- 
2.7.4


[-- Attachment #3: 0002-unicodeio-Add-tests.patch --]
[-- Type: text/x-patch, Size: 5753 bytes --]

From 7f89a25089f411dce4b22b55000de75c4bffe524 Mon Sep 17 00:00:00 2001
From: Bruno Haible <bruno@clisp.org>
Date: Wed, 8 Jul 2020 20:01:36 +0200
Subject: [PATCH 2/2] unicodeio: Add tests.

* tests/test-unicodeio.c: New file.
* tests/test-unicodeio1.sh: New file.
* tests/test-unicodeio2.sh: New file.
* tests/test-unicodeio3.sh: New file.
* modules/unicodeio-tests: New file.
---
 ChangeLog                |  9 ++++++
 modules/unicodeio-tests  | 21 +++++++++++++
 tests/test-unicodeio.c   | 81 ++++++++++++++++++++++++++++++++++++++++++++++++
 tests/test-unicodeio1.sh |  5 +++
 tests/test-unicodeio2.sh | 15 +++++++++
 tests/test-unicodeio3.sh | 15 +++++++++
 6 files changed, 146 insertions(+)
 create mode 100644 modules/unicodeio-tests
 create mode 100644 tests/test-unicodeio.c
 create mode 100755 tests/test-unicodeio1.sh
 create mode 100755 tests/test-unicodeio2.sh
 create mode 100755 tests/test-unicodeio3.sh

diff --git a/ChangeLog b/ChangeLog
index aa36674..c2182f0 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,14 @@
 2020-07-08  Bruno Haible  <bruno@clisp.org>
 
+	unicodeio: Add tests.
+	* tests/test-unicodeio.c: New file.
+	* tests/test-unicodeio1.sh: New file.
+	* tests/test-unicodeio2.sh: New file.
+	* tests/test-unicodeio3.sh: New file.
+	* modules/unicodeio-tests: New file.
+
+2020-07-08  Bruno Haible  <bruno@clisp.org>
+
 	unicodeio: Document link requirements.
 	* modules/unicodeio (Link): New section.
 
diff --git a/modules/unicodeio-tests b/modules/unicodeio-tests
new file mode 100644
index 0000000..d771ff8
--- /dev/null
+++ b/modules/unicodeio-tests
@@ -0,0 +1,21 @@
+Files:
+tests/test-unicodeio1.sh
+tests/test-unicodeio2.sh
+tests/test-unicodeio3.sh
+tests/test-unicodeio.c
+tests/macros.h
+m4/locale-fr.m4
+m4/locale-zh.m4
+m4/codeset.m4
+
+Depends-on:
+setlocale
+
+configure.ac:
+gt_LOCALE_FR_UTF8
+gt_LOCALE_ZH_CN
+
+Makefile.am:
+TESTS += test-unicodeio1.sh test-unicodeio2.sh test-unicodeio3.sh
+check_PROGRAMS += test-unicodeio
+test_unicodeio_LDADD = $(LDADD) $(LIB_SETLOCALE) $(LIBICONV)
diff --git a/tests/test-unicodeio.c b/tests/test-unicodeio.c
new file mode 100644
index 0000000..be91a91
--- /dev/null
+++ b/tests/test-unicodeio.c
@@ -0,0 +1,81 @@
+/* Tests for Unicode character output.
+
+   Copyright (C) 2020 Free Software Foundation, Inc.
+
+   This program is free software: you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <https://www.gnu.org/licenses/>.  */
+
+/* Written by Bruno Haible, 2020.  */
+
+#include <config.h>
+
+/* Specification.  */
+#include "unicodeio.h"
+
+#include <locale.h>
+#include <string.h>
+
+#include "macros.h"
+
+#define TEST_CODE 0x2022
+#define TEST_CODE_AS_UTF8 "\xe2\x80\xa2"
+#define TEST_CODE_AS_GB18030 "\x81\x36\xa6\x31"
+
+static char result[64];
+
+static long
+success_callback (const char *buf, size_t buflen, void *callback_arg)
+{
+  memcpy (result, buf, buflen);
+  result[buflen] = '\0';
+  return 42;
+}
+
+static long
+failure_callback (unsigned int code, const char *msg, void *callback_arg)
+{
+  ASSERT (code == TEST_CODE);
+  strcpy (result, ".");
+  return 55;
+}
+
+int
+main (int argc, char *argv[])
+{
+  /* configure should already have checked that the locale is supported.  */
+  if (setlocale (LC_ALL, "") == NULL)
+    return 1;
+
+  switch (unicode_to_mb (TEST_CODE, success_callback, failure_callback, NULL))
+    {
+    case 42:
+      if (argc > 1)
+        switch (argv[1][0])
+          {
+          case '2':
+            ASSERT (strcmp (result, TEST_CODE_AS_UTF8) == 0);
+            break;
+          case '3':
+            ASSERT (strcmp (result, TEST_CODE_AS_GB18030) == 0);
+            break;
+          }
+      break;
+    case 55:
+      ASSERT (strcmp (result, ".") == 0);
+      break;
+    default:
+      ASSERT (0);
+    }
+
+  return 0;
+}
diff --git a/tests/test-unicodeio1.sh b/tests/test-unicodeio1.sh
new file mode 100755
index 0000000..5f5f1bd
--- /dev/null
+++ b/tests/test-unicodeio1.sh
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+# Test in the C locale.
+LC_ALL=C \
+${CHECKER} ./test-unicodeio${EXEEXT} 1
diff --git a/tests/test-unicodeio2.sh b/tests/test-unicodeio2.sh
new file mode 100755
index 0000000..f0ec8fb
--- /dev/null
+++ b/tests/test-unicodeio2.sh
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+# Test whether a specific UTF-8 locale is installed.
+: ${LOCALE_FR_UTF8=fr_FR.UTF-8}
+if test $LOCALE_FR_UTF8 = none; then
+  if test -f /usr/bin/localedef; then
+    echo "Skipping test: no french Unicode locale is installed"
+  else
+    echo "Skipping test: no french Unicode locale is supported"
+  fi
+  exit 77
+fi
+
+LC_ALL=$LOCALE_FR_UTF8 \
+${CHECKER} ./test-unicodeio${EXEEXT} 2
diff --git a/tests/test-unicodeio3.sh b/tests/test-unicodeio3.sh
new file mode 100755
index 0000000..1544c96
--- /dev/null
+++ b/tests/test-unicodeio3.sh
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+# Test whether a specific GB18030 locale is installed.
+: ${LOCALE_ZH_CN=zh_CN.GB18030}
+if test $LOCALE_ZH_CN = none; then
+  if test -f /usr/bin/localedef; then
+    echo "Skipping test: no transitional chinese locale is installed"
+  else
+    echo "Skipping test: no transitional chinese locale is supported"
+  fi
+  exit 77
+fi
+
+LC_ALL=$LOCALE_ZH_CN \
+${CHECKER} ./test-unicodeio${EXEEXT} 3
-- 
2.7.4

Re: portability issues with unicodeio

[Kiyoshi KANAZAWA]

Hi Bruno,

1)
$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=

2)
$ ldd $coreutilsbin/printf
        libc.so.1 =>     /lib/libc.so.1


Regards,

--- Kiyoshi




----- Original Message -----
> From: Bruno Haible <bruno@clisp.org>
> To: bug-gnulib@gnu.org
> Cc: Akim Demaille <akim.demaille@gmail.com>; Kiyoshi KANAZAWA <yoi_no_myoujou@yahoo.co.jp>; Bison Bugs <bug-bison@gnu.org>
> Date: 2020/7/9, Thu 03:08
> Subject: Re: portability issues with unicodeio
> 
> Hi Akim, Kiyoshi-san,
> 
>>  Unfortunately on Kiyoshi's environment (SunOS hidden 5.11 11.3 i86pc 
> i386 i86pc,
>>  GCC 9.3.0) we get '?' instead of '.' in the C locale.  We 
> get a genuine ASCII
>>  '?', it's not some fallback from the terminal which fails to 
> display the
>>  character.  And we properly get the bullet with en_US.UTF-8.
> 
> I am adding a unit test for the 'unicodeio' module (see below). It 
> contains
> a test for the C locale, and this test works fine on Solaris 11.3.
> 
> Therefore I have to ask Kiyoshi two questions:
> 
> 1) What is the output of the program 'locale'?
>    $ locale
> 
> 2) What are the link dependencies of the coreutils printf program?
>    $ ldd $coreutilsbin/printf
> 
> Bruno
> 
> 
> 2020-07-08  Bruno Haible  <bruno@clisp.org>
> 
>     unicodeio: Add tests.
>     * tests/test-unicodeio.c: New file.
>     * tests/test-unicodeio1.sh: New file.
>     * tests/test-unicodeio2.sh: New file.
>     * tests/test-unicodeio3.sh: New file.
>     * modules/unicodeio-tests: New file.
> 
> 2020-07-08  Bruno Haible  <bruno@clisp.org>
> 
>     unicodeio: Document link requirements.
>     * modules/unicodeio (Link): New section.
> 

Re: portability issues with unicodeio

[Kiyoshi KANAZAWA]

Hi Bruno, Hi Akim,

FYI, also checked on Solaris 11.4 x86/x64, and had the same result.


$ uname -a
SunOS hidden 5.11 11.4.0.15.0 i86pc i386 i86pc


$ LC_ALL=C $coreutilsbin/printf '\u2022\n' | od -t x1
0000000 3f 0a
0000002


$ LC_ALL=en_US.UTF-8 $coreutilsbin/printf '\u2022\n' | od -t x1
0000000 e2 80 a2 0a
0000004


$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=


$ ldd $coreutilsbin/printf
        libc.so.1 =>     /lib/libc.so.1


--- Kiyoshi

Re: portability issues with unicodeio

[Bruno Haible]

Hi Kiyoshi-san,

> FYI, also checked on Solaris 11.4 x86/x64, and had the same result.

Whereas I checked Solaris 11.3 (x86 and sparc) and Solaris 11.4 (x86),
and had no issue with the newly added unit test.

Can you please configure, build, and 'make check' the package
  https://haible.de/bruno/gnu/testdir-unicodeio.tar.gz ?

It is succeeds this test, then what is the difference between the
coreutils printf program and the test-unicodeio.c program? Both call
setlocale (LC_ALL, "").

If it fails the test-unicodeio1.sh test, please tell how you configure
(environment variables and configure options).

Bruno

Re: portability issues with unicodeio

[Kiyoshi KANAZAWA]

Hi Bruno,

>   https://haible.de/bruno/gnu/testdir-unicodeio.tar.gz ?

make check passed both on Solaris 11.3 & 11.4.


> It is succeeds this test, then what is the difference between the
> coreutils printf program and the test-unicodeio.c program? Both call
> setlocale (LC_ALL, "").

I do not understand.
What should I do next ?


Regards,

--- Kiyoshi

Re: portability issues with unicodeio

[Bruno Haible]

Kiyoshi KANAZAWA wrote:
> make check passed both on Solaris 11.3 & 11.4.

Oh, I see. So my unit test was not complete.

> > It is succeeds this test, then what is the difference between the
> > coreutils printf program and the test-unicodeio.c program? Both call
> > setlocale (LC_ALL, "").
> 
> I do not understand.
> What should I do next ?

I single-stepped both the coreutils printf program and the test-unicodeio.c
program, and found the issue.


2020-07-09  Bruno Haible  <bruno@clisp.org>

	unicodeio: Fix wrong result on Solaris 11.
	Reported by Kiyoshi Kanazawa <yoi_no_myoujou@yahoo.co.jp>
	via Akim Demaille <akim.demaille@gmail.com> in
	<https://lists.gnu.org/archive/html/bug-gnulib/2020-07/msg00036.html>.
	* lib/unicodeio.c (unicode_to_mb): Handle question mark fallback
	characters on Solaris.
	* tests/test-unicodeio.c (main): In the "C" locale, expect either the
	UTF-8 output or the specified fallback.

diff --git a/lib/unicodeio.c b/lib/unicodeio.c
index 8ae3308..81fe0dd 100644
--- a/lib/unicodeio.c
+++ b/lib/unicodeio.c
@@ -134,6 +134,10 @@ unicode_to_mb (unsigned int code,
 # if !defined _LIBICONV_VERSION && (defined sgi || defined __sgi)
           || (res > 0 && code != 0 && outptr - outbuf == 1 && *outbuf == '\0')
 # endif
+          /* Solaris 11 iconv() inserts a '?' if it cannot convert. */
+# if !defined _LIBICONV_VERSION && defined __sun
+          || (res > 0 && code != 0 && outptr - outbuf == 1 && *outbuf == '?')
+# endif
          )
         return failure (code, NULL, callback_arg);
 
diff --git a/tests/test-unicodeio.c b/tests/test-unicodeio.c
index be91a91..fa28e98 100644
--- a/tests/test-unicodeio.c
+++ b/tests/test-unicodeio.c
@@ -62,6 +62,7 @@ main (int argc, char *argv[])
       if (argc > 1)
         switch (argv[1][0])
           {
+          case '1': /* On some platforms, the "C" locale has UTF-8 encoding.  */
           case '2':
             ASSERT (strcmp (result, TEST_CODE_AS_UTF8) == 0);
             break;

Re: portability issues with unicodeio

[Akim Demaille]

Hi Bruno,

> Le 9 juil. 2020 à 17:31, Bruno Haible <bruno@clisp.org> a écrit :
> 
> Kiyoshi KANAZAWA wrote:
>> make check passed both on Solaris 11.3 & 11.4.
> 
> Oh, I see. So my unit test was not complete.
> 
>>> It is succeeds this test, then what is the difference between the
>>> coreutils printf program and the test-unicodeio.c program? Both call
>>> setlocale (LC_ALL, "").
>> 
>> I do not understand.
>> What should I do next ?
> 
> I single-stepped both the coreutils printf program and the test-unicodeio.c
> program, and found the issue.
> 
> 
> 2020-07-09  Bruno Haible  <bruno@clisp.org>
> 
> 	unicodeio: Fix wrong result on Solaris 11.
> 	Reported by Kiyoshi Kanazawa <yoi_no_myoujou@yahoo.co.jp>
> 	via Akim Demaille <akim.demaille@gmail.com> in
> 	<https://lists.gnu.org/archive/html/bug-gnulib/2020-07/msg00036.html>.
> 	* lib/unicodeio.c (unicode_to_mb): Handle question mark fallback
> 	characters on Solaris.
> 	* tests/test-unicodeio.c (main): In the "C" locale, expect either the
> 	UTF-8 output or the specified fallback.

Excellent!  Bruno, thanks a lot for this.

Yesterday evening I launched the process of releasing a new beta so
that Kiyoshi could try it, but did not send the announcement.  Yet
Kiyoshi managed to find out there was a new beta, and already confirmed
the issue is fixed.

Thanks to both of you.

Cheers!

Re: portability issues with unicodeio

[Bruno Haible]

> 	* tests/test-unicodeio.c (main): In the "C" locale, expect either the
> 	UTF-8 output or the specified fallback.

Now that we have a unit test, I see that the test fails on NetBSD 9.0. This
patch fixes it.


2020-07-10  Bruno Haible  <bruno@clisp.org>

	unicodeio: Fix wrong result on NetBSD.
	* lib/unicodeio.c (unicode_to_mb): Handle question mark fallback
	characters also on NetBSD.

diff --git a/lib/unicodeio.c b/lib/unicodeio.c
index 81fe0dd..b616e3d 100644
--- a/lib/unicodeio.c
+++ b/lib/unicodeio.c
@@ -134,9 +134,10 @@ unicode_to_mb (unsigned int code,
 # if !defined _LIBICONV_VERSION && (defined sgi || defined __sgi)
           || (res > 0 && code != 0 && outptr - outbuf == 1 && *outbuf == '\0')
 # endif
-          /* Solaris 11 iconv() inserts a '?' if it cannot convert. */
-# if !defined _LIBICONV_VERSION && defined __sun
-          || (res > 0 && code != 0 && outptr - outbuf == 1 && *outbuf == '?')
+          /* NetBSD iconv() and Solaris 11 iconv() insert a '?' if they cannot
+             convert.  */
+# if !defined _LIBICONV_VERSION && (defined __NetBSD__ || defined __sun)
+          || (res > 0 && outptr - outbuf == 1 && *outbuf == '?')
 # endif
          )
         return failure (code, NULL, callback_arg);