Re: z/OS, iconv, and charset aliases

[Bruno Haible <bruno@clisp.org>]

[-- Attachment #1: Type: text/plain, Size: 2498 bytes --]

Hi Daniel,

> I've attached a file with the output of "iconv -l". The names appear 
> consistent with what's in iconv_open-aix.gperf.

Thanks. From this, I think we can equate the following vendor names with
GNU canonical names:


Vendor name  Canonical name          References

00367        ASCII, ANSI_X3.4-1968   https://en.wikipedia.org/wiki/Code_page_367
                                     https://haible.de/bruno/charsets/conversion-tables/ASCII.html
ISO8859-1    ISO-8859-1
ISO8859-2    ISO-8859-2
ISO8859-4    ISO-8859-4
ISO8859-5    ISO-8859-5
ISO8859-6    ISO-8859-6
ISO8859-7    ISO-8859-7
ISO8859-8    ISO-8859-8
ISO8859-9    ISO-8859-9
ISO8859-13   ISO-8859-13
ISO8859-15   ISO-8859-15
IBM-437      CP437
IBM-850      CP850
IBM-852      CP852
IBM-855      CP855
IBM-856      CP856
IBM-861      CP861
IBM-862      CP862
IBM-864      CP864
IBM-866      CP866
IBM-869      CP869
TIS-620      CP874                   https://haible.de/bruno/charsets/conversion-tables/Thai.html
IBM-922      CP922
IBM-eucJC    CP932
IBM-943      CP943
IBM-949      CP949
IBM-1046     CP1046
IBM-1124     CP1124
IBM-1125     CP1125
IBM-1250     CP1250
IBM-1251     CP1251
IBM-1252     CP1252
IBM-1253     CP1253
IBM-1254     CP1254
IBM-1255     CP1255
IBM-1256     CP1256
IBM-eucCN    GB2312
EUCJP        EUC-JP
IBM-eucKR    EUC-KR
IBM-eucTW    EUC-TW
BIG5         BIG5
IBM-936      GBK
TIS-620      TIS-620
UTF-8        UTF-8


Fortunately, all encodings listed as locale encodings in
"Table 3. Supported language-territory names and LT codes for ASCII locales"
of https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.cbcpx01/locnamc.htm
are in this list.

Omitting identical names on both sides (e.g. BIG5 BIG5), I arrive at the two
attached patches.


2019-12-20  Bruno Haible  <bruno@clisp.org>

	iconv_open: Add support for z/OS encoding names.
	Reported by Daniel Richard G. in
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-12/msg00172.html>.
	* lib/iconv_open-zos.gperf: New file.
	* modules/iconv_open (Files): Add iconv_open-zos.gperf.
	(Makefile.am): Add rules for generating iconv_open-zos.h from it.
	* lib/iconv_open.c (ICONV_FLAVOR_ZOS): New macro.
	* m4/iconv_open.m4 (gl_FUNC_ICONV_OPEN): On z/OS, use ICONV_FLAVOR_ZOS.
	* doc/posix-functions/iconv_open.texi: Mention z/OS.

2019-12-20  Bruno Haible  <bruno@clisp.org>

	localcharset: Add support for z/OS encoding names.
	* lib/localcharset.h: Mention which encodings are used as locale
	encodings on z/OS.



[-- Attachment #2: 0001-iconv_open-Add-support-for-z-OS-encoding-names.patch --]
[-- Type: text/x-patch, Size: 8174 bytes --]

From 49e78fcade5457b00b877fa7f7309056076a9b53 Mon Sep 17 00:00:00 2001
From: Bruno Haible <bruno@clisp.org>
Date: Fri, 20 Dec 2019 09:12:37 +0100
Subject: [PATCH 1/2] iconv_open: Add support for z/OS encoding names.

Reported by Daniel Richard G. in
<https://lists.gnu.org/archive/html/bug-gnulib/2019-12/msg00172.html>.

* lib/iconv_open-zos.gperf: New file.
* modules/iconv_open (Files): Add iconv_open-zos.gperf.
(Makefile.am): Add rules for generating iconv_open-zos.h from it.
* lib/iconv_open.c (ICONV_FLAVOR_ZOS): New macro.
* m4/iconv_open.m4 (gl_FUNC_ICONV_OPEN): On z/OS, use ICONV_FLAVOR_ZOS.
* doc/posix-functions/iconv_open.texi: Mention z/OS.
---
 ChangeLog                           | 12 +++++++
 doc/posix-functions/iconv_open.texi |  2 +-
 lib/iconv_open-zos.gperf            | 68 +++++++++++++++++++++++++++++++++++++
 lib/iconv_open.c                    |  1 +
 m4/iconv_open.m4                    | 13 +++----
 modules/iconv_open                  | 12 ++++---
 6 files changed, 97 insertions(+), 11 deletions(-)
 create mode 100644 lib/iconv_open-zos.gperf

diff --git a/ChangeLog b/ChangeLog
index dece371..9d6d06b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,17 @@
 2019-12-20  Bruno Haible  <bruno@clisp.org>
 
+	iconv_open: Add support for z/OS encoding names.
+	Reported by Daniel Richard G. in
+	<https://lists.gnu.org/archive/html/bug-gnulib/2019-12/msg00172.html>.
+	* lib/iconv_open-zos.gperf: New file.
+	* modules/iconv_open (Files): Add iconv_open-zos.gperf.
+	(Makefile.am): Add rules for generating iconv_open-zos.h from it.
+	* lib/iconv_open.c (ICONV_FLAVOR_ZOS): New macro.
+	* m4/iconv_open.m4 (gl_FUNC_ICONV_OPEN): On z/OS, use ICONV_FLAVOR_ZOS.
+	* doc/posix-functions/iconv_open.texi: Mention z/OS.
+
+2019-12-20  Bruno Haible  <bruno@clisp.org>
+
 	doc: Document the problem of the per-thread locale functions on z/OS.
 	* doc/posix-functions/uselocale.texi: Document the z/OS problem.
 	* doc/posix-functions/newlocale.texi: Likewise.
diff --git a/doc/posix-functions/iconv_open.texi b/doc/posix-functions/iconv_open.texi
index a70e1f5..d5f05ee 100644
--- a/doc/posix-functions/iconv_open.texi
+++ b/doc/posix-functions/iconv_open.texi
@@ -20,7 +20,7 @@ Portability problems fixed by Gnulib module @code{iconv_open}:
 @item
 This function recognizes only non-standard aliases for many encodings (not
 the IANA registered encoding names) on many platforms:
-AIX 5.1, HP-UX 11, IRIX 6.5, Solaris 11 2010-11.
+AIX 5.1, HP-UX 11, IRIX 6.5, Solaris 11 2010-11, z/OS.
 @end itemize
 
 Portability problems fixed by Gnulib module @code{iconv_open-utf}:
diff --git a/lib/iconv_open-zos.gperf b/lib/iconv_open-zos.gperf
new file mode 100644
index 0000000..d44b5d7
--- /dev/null
+++ b/lib/iconv_open-zos.gperf
@@ -0,0 +1,68 @@
+/* Character set conversion.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation; either version 2, or (at your option)
+   any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License along
+   with this program; if not, see <https://www.gnu.org/licenses/>.  */
+
+struct mapping { int standard_name; const char vendor_name[10 + 1]; };
+%struct-type
+%language=ANSI-C
+%define slot-name standard_name
+%define hash-function-name mapping_hash
+%define lookup-function-name mapping_lookup
+%readonly-tables
+%global-table
+%define word-array-name mappings
+%pic
+%%
+ASCII, "00367"
+ISO-8859-1, "ISO8859-1"
+ISO-8859-2, "ISO8859-2"
+ISO-8859-4, "ISO8859-4"
+ISO-8859-5, "ISO8859-5"
+ISO-8859-6, "ISO8859-6"
+ISO-8859-7, "ISO8859-7"
+ISO-8859-8, "ISO8859-8"
+ISO-8859-9, "ISO8859-9"
+ISO-8859-13, "ISO8859-13"
+ISO-8859-15, "ISO8859-15"
+CP437, "IBM-437"
+CP850, "IBM-850"
+CP852, "IBM-852"
+CP855, "IBM-855"
+CP856, "IBM-856"
+CP861, "IBM-861"
+CP862, "IBM-862"
+CP864, "IBM-864"
+CP866, "IBM-866"
+CP869, "IBM-869"
+CP874, "TIS-620"
+CP922, "IBM-922"
+CP932, "IBM-eucJC"
+CP943, "IBM-943"
+CP949, "IBM-949"
+CP1046, "IBM-1046"
+CP1124, "IBM-1124"
+CP1125, "IBM-1125"
+CP1250, "IBM-1250"
+CP1251, "IBM-1251"
+CP1252, "IBM-1252"
+CP1253, "IBM-1253"
+CP1254, "IBM-1254"
+CP1255, "IBM-1255"
+CP1256, "IBM-1256"
+GB2312, "IBM-eucCN"
+EUC-JP, "EUCJP"
+EUC-KR, "IBM-eucKR"
+EUC-TW, "IBM-eucTW"
+GBK, "IBM-936"
diff --git a/lib/iconv_open.c b/lib/iconv_open.c
index 928ccf2..918b89c 100644
--- a/lib/iconv_open.c
+++ b/lib/iconv_open.c
@@ -36,6 +36,7 @@
 #define ICONV_FLAVOR_IRIX "iconv_open-irix.h"
 #define ICONV_FLAVOR_OSF "iconv_open-osf.h"
 #define ICONV_FLAVOR_SOLARIS "iconv_open-solaris.h"
+#define ICONV_FLAVOR_ZOS "iconv_open-zos.h"
 
 #ifdef ICONV_FLAVOR
 # include ICONV_FLAVOR
diff --git a/m4/iconv_open.m4 b/m4/iconv_open.m4
index bfcd354..b4730a9 100644
--- a/m4/iconv_open.m4
+++ b/m4/iconv_open.m4
@@ -1,4 +1,4 @@
-# iconv_open.m4 serial 15
+# iconv_open.m4 serial 16
 dnl Copyright (C) 2007-2019 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -23,11 +23,12 @@ AC_DEFUN([gl_FUNC_ICONV_OPEN],
     if test $gl_func_iconv_gnu = no; then
       iconv_flavor=
       case "$host_os" in
-        aix*)     iconv_flavor=ICONV_FLAVOR_AIX ;;
-        irix*)    iconv_flavor=ICONV_FLAVOR_IRIX ;;
-        hpux*)    iconv_flavor=ICONV_FLAVOR_HPUX ;;
-        osf*)     iconv_flavor=ICONV_FLAVOR_OSF ;;
-        solaris*) iconv_flavor=ICONV_FLAVOR_SOLARIS ;;
+        aix*)         iconv_flavor=ICONV_FLAVOR_AIX ;;
+        irix*)        iconv_flavor=ICONV_FLAVOR_IRIX ;;
+        hpux*)        iconv_flavor=ICONV_FLAVOR_HPUX ;;
+        osf*)         iconv_flavor=ICONV_FLAVOR_OSF ;;
+        solaris*)     iconv_flavor=ICONV_FLAVOR_SOLARIS ;;
+        openedition*) iconv_flavor=ICONV_FLAVOR_ZOS ;;
       esac
       if test -n "$iconv_flavor"; then
         AC_DEFINE_UNQUOTED([ICONV_FLAVOR], [$iconv_flavor],
diff --git a/modules/iconv_open b/modules/iconv_open
index 7032dca..3486901 100644
--- a/modules/iconv_open
+++ b/modules/iconv_open
@@ -8,6 +8,7 @@ lib/iconv_open-hpux.gperf
 lib/iconv_open-irix.gperf
 lib/iconv_open-osf.gperf
 lib/iconv_open-solaris.gperf
+lib/iconv_open-zos.gperf
 lib/iconv.c
 lib/iconv_close.c
 m4/iconv_open.m4
@@ -48,10 +49,13 @@ $(srcdir)/iconv_open-osf.h: $(srcdir)/iconv_open-osf.gperf
 $(srcdir)/iconv_open-solaris.h: $(srcdir)/iconv_open-solaris.gperf
 	$(V_GPERF)$(GPERF) -m 10 $(srcdir)/iconv_open-solaris.gperf > $(srcdir)/iconv_open-solaris.h-t && \
 	mv $(srcdir)/iconv_open-solaris.h-t $(srcdir)/iconv_open-solaris.h
-BUILT_SOURCES        += iconv_open-aix.h iconv_open-hpux.h iconv_open-irix.h iconv_open-osf.h iconv_open-solaris.h
-MOSTLYCLEANFILES     += iconv_open-aix.h-t iconv_open-hpux.h-t iconv_open-irix.h-t iconv_open-osf.h-t iconv_open-solaris.h-t
-MAINTAINERCLEANFILES += iconv_open-aix.h iconv_open-hpux.h iconv_open-irix.h iconv_open-osf.h iconv_open-solaris.h
-EXTRA_DIST           += iconv_open-aix.h iconv_open-hpux.h iconv_open-irix.h iconv_open-osf.h iconv_open-solaris.h
+$(srcdir)/iconv_open-zos.h: $(srcdir)/iconv_open-zos.gperf
+	$(V_GPERF)$(GPERF) -m 10 $(srcdir)/iconv_open-zos.gperf > $(srcdir)/iconv_open-zos.h-t && \
+	mv $(srcdir)/iconv_open-zos.h-t $(srcdir)/iconv_open-zos.h
+BUILT_SOURCES        += iconv_open-aix.h iconv_open-hpux.h iconv_open-irix.h iconv_open-osf.h iconv_open-solaris.h iconv_open-zos.h
+MOSTLYCLEANFILES     += iconv_open-aix.h-t iconv_open-hpux.h-t iconv_open-irix.h-t iconv_open-osf.h-t iconv_open-solaris.h-t iconv_open-zos.h-t
+MAINTAINERCLEANFILES += iconv_open-aix.h iconv_open-hpux.h iconv_open-irix.h iconv_open-osf.h iconv_open-solaris.h iconv_open-zos.h
+EXTRA_DIST           += iconv_open-aix.h iconv_open-hpux.h iconv_open-irix.h iconv_open-osf.h iconv_open-solaris.h iconv_open-zos.h
 
 Include:
 <iconv.h>
-- 
2.7.4


[-- Attachment #3: 0002-localcharset-Add-support-for-z-OS-encoding-names.patch --]
[-- Type: text/x-patch, Size: 5308 bytes --]

From 3f7d8da2ee9e513a9db318dc9c4aa91ca6ed8b3b Mon Sep 17 00:00:00 2001
From: Bruno Haible <bruno@clisp.org>
Date: Fri, 20 Dec 2019 09:17:20 +0100
Subject: [PATCH 2/2] localcharset: Add support for z/OS encoding names.

* lib/localcharset.h: Mention which encodings are used as locale
encodings on z/OS.
---
 ChangeLog          |  6 ++++++
 lib/localcharset.h | 24 ++++++++++++------------
 2 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 9d6d06b..9a47dbc 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,11 @@
 2019-12-20  Bruno Haible  <bruno@clisp.org>
 
+	localcharset: Add support for z/OS encoding names.
+	* lib/localcharset.h: Mention which encodings are used as locale
+	encodings on z/OS.
+
+2019-12-20  Bruno Haible  <bruno@clisp.org>
+
 	iconv_open: Add support for z/OS encoding names.
 	Reported by Daniel Richard G. in
 	<https://lists.gnu.org/archive/html/bug-gnulib/2019-12/msg00172.html>.
diff --git a/lib/localcharset.h b/lib/localcharset.h
index 5897140..81ebfae 100644
--- a/lib/localcharset.h
+++ b/lib/localcharset.h
@@ -48,15 +48,15 @@ extern const char * locale_charset (void);
                                     (darwin = Mac OS X, windows = native Windows)
 
    ASCII, ANSI_X3.4-1968       glibc solaris freebsd netbsd darwin minix cygwin
-   ISO-8859-1              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin
-   ISO-8859-2              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin
+   ISO-8859-1              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin zos
+   ISO-8859-2              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin zos
    ISO-8859-3              Y   glibc solaris cygwin
    ISO-8859-4              Y   hpux osf solaris freebsd netbsd openbsd darwin
-   ISO-8859-5              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin
+   ISO-8859-5              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin zos
    ISO-8859-6              Y   glibc aix hpux solaris cygwin
-   ISO-8859-7              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin
-   ISO-8859-8              Y   glibc aix hpux osf solaris cygwin
-   ISO-8859-9              Y   glibc aix hpux irix osf solaris freebsd darwin cygwin
+   ISO-8859-7              Y   glibc aix hpux irix osf solaris freebsd netbsd openbsd darwin cygwin zos
+   ISO-8859-8              Y   glibc aix hpux osf solaris cygwin zos
+   ISO-8859-9              Y   glibc aix hpux irix osf solaris freebsd darwin cygwin zos
    ISO-8859-13                 glibc hpux solaris freebsd netbsd openbsd darwin cygwin
    ISO-8859-14                 glibc cygwin
    ISO-8859-15                 glibc aix irix osf solaris freebsd netbsd openbsd darwin cygwin
@@ -79,7 +79,7 @@ extern const char * locale_charset (void);
    CP874                       windows dos
    CP922                       aix
    CP932                       aix cygwin windows dos
-   CP943                       aix
+   CP943                       aix zos
    CP949                       osf darwin windows dos
    CP950                       windows dos
    CP1046                      aix
@@ -95,17 +95,17 @@ extern const char * locale_charset (void);
    CP1255                      glibc windows
    CP1256                      windows
    CP1257                      windows
-   GB2312                  Y   glibc aix hpux irix solaris freebsd netbsd darwin cygwin
+   GB2312                  Y   glibc aix hpux irix solaris freebsd netbsd darwin cygwin zos
    EUC-JP                  Y   glibc aix hpux irix osf solaris freebsd netbsd darwin cygwin
-   EUC-KR                  Y   glibc aix hpux irix osf solaris freebsd netbsd darwin cygwin
+   EUC-KR                  Y   glibc aix hpux irix osf solaris freebsd netbsd darwin cygwin zos
    EUC-TW                      glibc aix hpux irix osf solaris netbsd
-   BIG5                    Y   glibc aix hpux osf solaris freebsd netbsd darwin cygwin
+   BIG5                    Y   glibc aix hpux osf solaris freebsd netbsd darwin cygwin zos
    BIG5-HKSCS                  glibc hpux solaris netbsd darwin
    GBK                         glibc aix osf solaris freebsd darwin cygwin windows dos
    GB18030                     glibc hpux solaris freebsd netbsd darwin
    SHIFT_JIS               Y   hpux osf solaris freebsd netbsd darwin
    JOHAB                       glibc solaris windows
-   TIS-620                     glibc aix hpux osf solaris cygwin
+   TIS-620                     glibc aix hpux osf solaris cygwin zos
    VISCII                  Y   glibc
    TCVN5712-1                  glibc
    ARMSCII-8                   glibc freebsd netbsd darwin
@@ -119,7 +119,7 @@ extern const char * locale_charset (void);
    HP-KANA8                    hpux
    DEC-KANJI                   osf
    DEC-HANYU                   osf
-   UTF-8                   Y   glibc aix hpux osf solaris netbsd darwin cygwin
+   UTF-8                   Y   glibc aix hpux osf solaris netbsd darwin cygwin zos
 
    Note: Names which are not marked as being a MIME name should not be used in
    Internet protocols for information interchange (mail, news, etc.).
-- 
2.7.4