IBM z/OS compatibility issues

IBM z/OS compatibility issues

[Daniel Richard G.]

Hello list,

I have been testing gnulib on IBM z/OS in recent days, and have found
a few compatibility issues (some old, some new) that I'd like to
address here.

This is intended to be a comprehensive review, so it will be a little
long; please bear with me!


BROKEN LOCALE FUNCTIONS

The configure script finds the following:

    checking for xlocale.h... no
    checking for duplocale... yes
    checking for uselocale... yes
    checking for newlocale... yes
    checking for freelocale... yes

    checking whether locale.h conforms to POSIX:2001... yes
    checking whether uselocale works... no

    checking whether duplocale(LC_GLOBAL_LOCALE) works... no

    checking whether setlocale supports the C locale... yes

The problem is, those duplocale(), newlocale(), and freelocale()
functions are not usable. Not only are they not declared in locale.h,
not only does the runtime crash if you call them, the locale_t type
isn't even defined. Here is a portion of the config.log output for the
"duplocale works" test:

    configure:37076: checking whether duplocale(LC_GLOBAL_LOCALE) works
    configure:37166: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  conftest.c  >&5
    ERROR CCN3275 ./conftest.c:405   Unexpected text loc encountered.
    ERROR CCN3045 ./conftest.c:405   Undeclared identifier locale_t.
    ERROR CCN3045 ./conftest.c:415   Undeclared identifier LC_GLOBAL_LOCALE.
    ERROR CCN3045 ./conftest.c:415   Undeclared identifier loc.
    CCN0793(I) Compilation failed for file ./conftest.c.  Object file not created.
    configure:37166: $? = 12
    configure: program exited with status 12

And the "uselocale works" test, which fails in the crashing way:

    configure:24730: checking whether uselocale works
    configure:24757: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  conftest.c  >&5
    configure:24757: $? = 0
    configure:24757: ./conftest
    CEE3728S The use of a function, which is not supported by this release of Language Environment was detected.
             From compile unit /tmp/gnulib-build/conftest.c at entry point main at statement 275 at compile unit offset
              +00000074 at entry offset +00000074 at address 2190A9BC.
    configure:24757: $? = 137
    configure: program exited with status 137

Currently, gnulib reads this as "Oh, the system duplocale() et al. are
broken, let me replace them." Unfortunately, this results in build
errors at the replacement function prototype, due to the missing
locale_t type:

    xlc-wrap -DHAVE_CONFIG_H -I. -I/u/username/testdir/gllib -I.. -DGNULIB_STRICT_CHECKING=1 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o hard-locale.o /u/username/testdir/gllib/hard-locale.c
    ERROR CCN3166 ./locale.h:702   Definition of function locale_t requires parentheses.
    ERROR CCN3276 ./locale.h:702   Syntax error: possible missing '{'?

In order to get a successful build, I have to set

    ac_cv_func_duplocale=no
    ac_cv_func_newlocale=no

before configuring. Could these functions (perhaps freelocale() too) be
treated as non-existent if there is no usable locale_t type?
Alternately, checking for their (missing) declarations should also work.


PTHREAD ENVIRONMENT

z/OS has effectively two major pthread interfaces: _OPEN_THREADS, and
_UNIX95_THREADS. The latter is the one you want to use, because its API
is compatible with other systems. (_OPEN_THREADS uses e.g. a different
signature for pthread_getspecific() that comes from a draft POSIX
specification; refer to

    https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ptgetsp.htm

for the gory details. pthread_detach() is different, too, and
pthread_cond_timedwait() can return EAGAIN.)

Additional information on z/OS feature test macros, if desired:

    https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ftms.htm

More specifically, to get a suitable pthreads interface, you need to
#define _UNIX95_THREADS and _XOPEN_SOURCE=600. (This could perhaps be
added to AC_USE_SYSTEM_EXTENSIONS for this system.)

However, one annoyance of _UNIX95_THREADS is that it does not define
PTHREAD_RWLOCK_INITIALIZER, supposedly because the SUSv3 standard (which
is what that feature test macro requests) does not specify it. However,
IBM does provide the "implementation-defined"
PTHREAD_RWLOCK_INITIALIZER_NP, which can be used in its place. See the
note here:

    https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/rp0r0i.htm

I would thus suggest adding something like the following to
lib/pthread.in.h:

    #if defined(__MVS__) && !defined(PTHREAD_RWLOCK_INITIALIZER)
    # define PTHREAD_RWLOCK_INITIALIZER PTHREAD_RWLOCK_INITIALIZER_NP
    #endif


SHELL SCRIPTING SNAFUS

In the z/OS environment, if you want to use a proper Bash shell, you
typically get the build provided by Rocket Software:

    https://www.rocketsoftware.com/product-categories/mainframe/bash-zos

The problem, however, is that shell scripting with this version of Bash
can be a little tricky, because it doesn't fully embrace EBCDIC. A brief
demonstration:

    bash-4.3$ echo $BASH_VERSION
    4.3.46(51)-release

    bash-4.3$ printf ABC | od -t x1
    0000000000    41  42  43
    0000000003

    bash-4.3$ /bin/printf ABC | od -t x1
    0000000000    C1  C2  C3
    0000000003

    bash-4.3$ echo ABC | grep ABC
    (no output)

    bash-4.3$ /bin/echo ABC | grep ABC
    ABC

    bash-4.3$ echo ABC >test.txt
    bash-4.3$ cat test.txt
    ????

    bash-4.3$ od -t x1 <test.txt
    0000000000    41  42  43  0A
    0000000004

    bash-4.3$ echo `cat test.txt`
    ABC

It's very confusing and annoying, and when I first tested gnulib using
this version of Bash, it ended up looping infinitely on running
test-atexit.sh. The loop occurred in mktempd_(), in gltests/init.sh,
because the sheer brokenness of the shell environment resulted in the
MAX_TRIES_ logic not working.

In order to get the shell to work sanely, you have to set a bevy of
environment variables discussed in

    /usr/lpp/ported/bash-4.3/share/doc/bash/4.3/README.ZOS

(I cannot find a public copy to link to, unfortunately). These are the
settings I used which allow things to work:

    _ENCODE_FILE_NEW=IBM-1047
    _ENCODE_FILE_EXISTING=IBM-1047
    _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
    _BPXK_AUTOCVT=ON
    _TAG_REDIR_ERR=txt
    _TAG_REDIR_IN=txt
    _TAG_REDIR_OUT=txt

(Note: This may not be a minimal set)

While I would not recommend giving to init.sh knowledge of the above
variables, I think it would be helpful to do some basic sanity checking
(like the echo|grep invocation above) to avoid more pathological
breakage later in the script. The failure message could include a hint
to the user about what's wrong with the shell, and what needs to be done
to fix it.

Note: The Rocket Software version of Bash is not the only one that
exists on this platform. My org has an older version, apparently
compiled by us long ago, that has given me a lot fewer headaches.
Therefore, no assumption can be made about Bash on z/OS being the Rocket
version. Unfortunately, as far as I can tell, there is no good way of
uniquely identifying the Rocket version, either (as you can see, the
BASH_VERSION string has not been tweaked).


ENVIRONMENT VARIABLES

One annoyance on this platform is that the default behavior of things
can sometimes be weirdly different from other platforms, in a way which
breaks programs expecting normal Unix/POSIX behavior. Unfortunately,
IBM's response to this is often not "Let us fix that so it works like
other Unix systems," but "That's too bad. We can't change the default
behavior because mumblemumble, but we can provide an environment
variable that, if set, with cause that thing to behave in the manner
you expect."

At present, the only such variable worth mentioning here is
_EDC_SIG_DFLT, which when set to 1, causes certain default signal
handlers *not* to print out messages:

    https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.cbcpx01/edc_sig_dflt.htm

This is something that should at least be set in the gnulib test
environment so that test-sigpipe.sh doesn't break, but may be worth
adding to the library itself so that programs can continue to expect
"normal" signal semantics.

(There are other environment variables documented at the link above that
may be of interest, but I have not found any additional ones to be
necessary to fix issues in gnulib's test suite.)


AUTOCONF ISSUES

A few issues in this environment relate to compiler quirks, which I
intend to bring to the attention of the Autoconf folks:

* -qfloat=ieee is needed to use "normal" IEEE754 floating-point format
  (the default is IBM's proprietary "hexadecimal" format)

* -qhaltonmsg=CCN3296 is needed so that the compiler treats missing
  header files as an error instead of a warning (!)

* Some extra rigmarole is needed so that a system foo.h is not favored
  over a foo.h file in some -I<dir> location; the "xlc-wrap" script I am
  using addresses this, and is as follows:

    #!/bin/sh
    exec xlc -qnosearch "$@" -qsearch=/usr/include

* Other compiler frontends are available on z/OS: c89, c99, cc, xlC,
  xlc++, xlclang, xlclang++. Not all of them are usable, and some of
  them take a strange option syntax; generally the ones starting with
  "xl" are easier to deal with

  (Some work previously done on this can be seen in GNU Gawk's
  m4/arch.m4 file)


MISCELLANEOUS BUGS

There are a handful of additional issues that I would like to mention
here for completeness, but am taking up primarily with IBM, rather than
this list. If anyone would like me to elaborate on any of these, please
feel free to ask:

* [sys/]signal.h does not #define NSIG (nor SIGMAX)

* File descriptor passed to fdopendir() no longer behaves like a normal
  fd; in particular, dup2(fd,fd) and fcntl(fd,F_GETFL) both fail (but
  close(fd) does not), causing test-fdopendir to fail spuriously (breaks
  the "fdopendir should not close fd" assertion)

* select() on /dev/null always returns 0, even though reading or writing
  to it never blocks

* IBM XLC compiler does not support the C11 _Alignas() specifier, nor
  UTF-8 literal strings, despite ostensibly supporting C11 (other
  features are mostly there, save for the next point)

* C11 _Thread_local support is utterly broken (test-thread_local fails)

* The z/OS Make utility is severely broken: does not support VPATH, nor
  recipe lines preceded with a hyphen, nor $(VAR:=.suffix) (can change a
  suffix but not add a new one); thankfully I have GNU Make here


That is basically everything I have. I'd like to get this knowledge out
into the open, and hopefully integrated into gnulib and elsewhere, so
that less manual intervention is needed when building GNU software on
this platform. I am happy to answer questions, and help with testing any
proposed changes.


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

That's quite a list of compatibility issues, and I don't envy you the 
job of porting to that platform. One little area I may be able to help 
out is here:

On 11/5/19 10:03 AM, Daniel Richard G. wrote:

> * IBM XLC compiler does not support the C11 _Alignas() specifier,

I take it that the compiler defines __STDC_VERSION__ to be 201112 
without actually supporting C11? That's annoying. Would the attached 
Gnulib patch work around the problem for C? What would be needed for C++?


[-- Attachment #2: zos.diff --]
[-- Type: text/x-patch, Size: 454 bytes --]

diff --git a/lib/stdalign.in.h b/lib/stdalign.in.h
index 5c2c72e2b..9d7321c31 100644
--- a/lib/stdalign.in.h
+++ b/lib/stdalign.in.h
@@ -97,7 +97,7 @@
 
    */
 
-#if !defined __STDC_VERSION__ || __STDC_VERSION__ < 201112
+#if !defined __STDC_VERSION__ || __STDC_VERSION__ < 201112 || defined __xlC__
 # if defined __cplusplus && 201103 <= __cplusplus
 #  define _Alignas(a) alignas (a)
 # elif ((defined __APPLE__ && defined __MACH__                  \

Re: IBM z/OS compatibility issues

[Daniel Richard G.]

On Tue, 2019 Nov  5 17:23-05:00, Paul Eggert wrote:
> That's quite a list of compatibility issues, and I don't envy you the
> job of porting to that platform.

The necessary detective work is done; I only need help with integrating
the know-how into Gnulib. I attempted to prepare a patch to address the
duplocale() issue, for example, but could not achieve the desired
result without brute-forcing it. There appear to be subtleties that I'm
not aware of.

> One little area I may be able to help out is here:
> 
> On 11/5/19 10:03 AM, Daniel Richard G. wrote:
> 
> > * IBM XLC compiler does not support the C11 _Alignas() specifier,
> 
> I take it that the compiler defines __STDC_VERSION__ to be 201112 
> without actually supporting C11? That's annoying. Would the attached 
> Gnulib patch work around the problem for C? What would be needed for C++?

Confirmed that the compiler sets __STDC_VERSION__ == 201112 without
actually supporting that level. Note that this in fact doesn't cause a
problem for Gnulib, because _Alignas() is part of its "checking for $CC
option to enable C11 features" check, thereby causing it to fail with
"unsupported".

I experimented a bit with this, nevertheless. alignas() does not work,
but alignof() does. What leaves me scratching my head is that in
/usr/include/stdalign.h, I see the following:

    #if defined(__C1X)
      #define alignas _Alignas
      #define alignof _Alignof
      #define __alignas_is_defined 1
      #define __alignof_is_defined 1
    #endif

One aside: This kind of thing can't be conditionalized on a compiler def
alone; XLC on z/OS is AFAICT a completely different implementation from
XLC on AIX. There are many commonalities, like option syntax, but also
many points of difference that would make little sense if they came from
the same codebase. I've often had to point IBM to AIX behavior to give
them an example of "the Right(tm) way" that can't be dismissed as NIH.

If a compiler conditional is needed, I would advise

    #if defined(__MVS__) && defined(__IBMC__)

(XLC on z/OS does not appear to define __xlC__).


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues

[Paul Eggert]

On 11/6/19 6:57 AM, Daniel Richard G. wrote:

> Confirmed that the compiler sets __STDC_VERSION__ == 201112 without
> actually supporting that level.

IBM's user guide for the compiler 
<https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R3sc147307/$file/cbcux01_v2r3.pdf>, 
dated 2019-06-26, states the following on page 153:

"Until IBM's implementation of all the features of the C11 standard is 
complete,... IBM makes no attempt to maintain compatibility ... with 
earlier releases of IBM's implementation of the new features of the C11 
standard and therefore they should not be relied on as a stable 
programming interface."

With this in mind, perhaps it'd be better to use that compiler in its 
C99 mode, to avoid having 'configure' attempt to use C11 features before 
IBM gets its C11 act together. That should be good enough for now, and 
we can worry about getting Gnulib to work with IBM's C11 support once 
that support is ready.

> Note that this in fact doesn't cause a
> problem for Gnulib, because _Alignas() is part of its "checking for $CC
> option to enable C11 features" check, thereby causing it to fail with
> "unsupported".
Although 'configure' does the right thing and says that _Alignas is 
unsupported, 'make' then goes ahead and builds a stdalign.h that assumes 
that _Alignas works (because the compiler defines __STDC_VERSION__ to be 
201112), and the generated stdalign.h therefore defines alignas and 
__alignas_is_defined, which is wrong.

> What leaves me scratching my head is that in
> /usr/include/stdalign.h, I see the following:
> 
>      #if defined(__C1X)
>        #define alignas _Alignas
>        #define alignof _Alignof
>        #define __alignas_is_defined 1
>        #define __alignof_is_defined 1
>      #endif

So /usr/include/stdalign.h has the same bug as the Gnulib stdalign.h 
replacement, in that it also defines alignas and __alignas_is_defined 
even though _Alignas does not work. Presumably /usr/include/stdalign.h 
is relying on the statement in the manual that C11 support is still 
unstable and can't be relied upon.

So it sounds like we should let this particular sleeping dog lie, as far 
as Gnulib is concerned.

Re: IBM z/OS compatibility issues

[Daniel Richard G.]

(Apologies for the delayed response)

On Wed, 2019 Nov  6 14:32-05:00, Paul Eggert wrote:
> 
> IBM's user guide for the compiler 
> <https://www-01.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R3sc147307/$file/cbcux01_v2r3.pdf>, 
> dated 2019-06-26, states the following on page 153:
> 
> "Until IBM's implementation of all the features of the C11 standard is
> complete,... IBM makes no attempt to maintain compatibility ... with
> earlier releases of IBM's implementation of the new features of the
> C11 standard and therefore they should not be relied on as a stable
> programming interface."

Ah, good find. I suspected that the compiler is in a transitional period
w.r.t. C11.

> With this in mind, perhaps it'd be better to use that compiler in its
> C99 mode, to avoid having 'configure' attempt to use C11 features
> before IBM gets its C11 act together. That should be good enough for
> now, and we can worry about getting Gnulib to work with IBM's C11
> support once that support is ready.

The C99 mode (-qlanglvl=extc99) works without issue. That's what I've
been using for all my testing.

> Although 'configure' does the right thing and says that _Alignas is 
> unsupported, 'make' then goes ahead and builds a stdalign.h that assumes 
> that _Alignas works (because the compiler defines __STDC_VERSION__ to be 
> 201112), and the generated stdalign.h therefore defines alignas and 
> __alignas_is_defined, which is wrong.

Hmm, I see what you mean. FWIW, the "checking for working stdalign.h"
check does end with "no".

> > What leaves me scratching my head is that in
> > /usr/include/stdalign.h, I see the following:
> > 
> >      #if defined(__C1X)
> >        #define alignas _Alignas
> >        #define alignof _Alignof
> >        #define __alignas_is_defined 1
> >        #define __alignof_is_defined 1
> >      #endif
> 
> So /usr/include/stdalign.h has the same bug as the Gnulib stdalign.h
> replacement, in that it also defines alignas and __alignas_is_defined
> even though _Alignas does not work. Presumably /usr/include/stdalign.h
> is relying on the statement in the manual that C11 support is still
> unstable and can't be relied upon.
>
> So it sounds like we should let this particular sleeping dog lie, as
> far as Gnulib is concerned.

I do get the feeling that the C11 brokenness of this compiler is not
worth working around. IBM is normally a stickler for standards, which
makes this an unusual scenario, but I think it's a good enough solution
for users who are bitten by this to press IBM to get it right. I
certainly plan on doing so.


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues

[Bruno Haible]

Hi Daniel,

> The C99 mode (-qlanglvl=extc99) works without issue. That's what I've
> been using for all my testing.

Do you have a set of configure settings that you would recommend on z/OS?
So that we can add this info to the Gnow-How wiki [1]...

Bruno

[1] https://gitlab.com/ghwiki/gnow-how/wikis/Platforms/Configuration

Re: IBM z/OS compatibility issues - per-thread locale functions

[Bruno Haible]

Hello Daniel,

> BROKEN LOCALE FUNCTIONS
> 
> The configure script finds the following:
> 
>     checking for xlocale.h... no
>     checking for duplocale... yes
>     checking for uselocale... yes
>     checking for newlocale... yes
>     checking for freelocale... yes
> 
>     checking whether locale.h conforms to POSIX:2001... yes
>     checking whether uselocale works... no
> 
>     checking whether duplocale(LC_GLOBAL_LOCALE) works... no
> 
>     checking whether setlocale supports the C locale... yes
> 
> The problem is, those duplocale(), newlocale(), and freelocale()
> functions are not usable. Not only are they not declared in locale.h,
> not only does the runtime crash if you call them, the locale_t type
> isn't even defined. Here is a portion of the config.log output for the
> "duplocale works" test:
> 
>     configure:37076: checking whether duplocale(LC_GLOBAL_LOCALE) works
>     configure:37166: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  conftest.c  >&5
>     ERROR CCN3275 ./conftest.c:405   Unexpected text loc encountered.
>     ERROR CCN3045 ./conftest.c:405   Undeclared identifier locale_t.
>     ERROR CCN3045 ./conftest.c:415   Undeclared identifier LC_GLOBAL_LOCALE.
>     ERROR CCN3045 ./conftest.c:415   Undeclared identifier loc.
>     CCN0793(I) Compilation failed for file ./conftest.c.  Object file not created.
>     configure:37166: $? = 12
>     configure: program exited with status 12
> 
> And the "uselocale works" test, which fails in the crashing way:
> 
>     configure:24730: checking whether uselocale works
>     configure:24757: xlc-wrap -o conftest -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  conftest.c  >&5
>     configure:24757: $? = 0
>     configure:24757: ./conftest
>     CEE3728S The use of a function, which is not supported by this release of Language Environment was detected.
>              From compile unit /tmp/gnulib-build/conftest.c at entry point main at statement 275 at compile unit offset
>               +00000074 at entry offset +00000074 at address 2190A9BC.
>     configure:24757: $? = 137
>     configure: program exited with status 137
> 
> Currently, gnulib reads this as "Oh, the system duplocale() et al. are
> broken, let me replace them." Unfortunately, this results in build
> errors at the replacement function prototype, due to the missing
> locale_t type:
> 
>     xlc-wrap -DHAVE_CONFIG_H -I. -I/u/username/testdir/gllib -I.. -DGNULIB_STRICT_CHECKING=1 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o hard-locale.o /u/username/testdir/gllib/hard-locale.c
>     ERROR CCN3166 ./locale.h:702   Definition of function locale_t requires parentheses.
>     ERROR CCN3276 ./locale.h:702   Syntax error: possible missing '{'?
> 
> In order to get a successful build, I have to set
> 
>     ac_cv_func_duplocale=no
>     ac_cv_func_newlocale=no
> 
> before configuring. Could these functions (perhaps freelocale() too) be
> treated as non-existent if there is no usable locale_t type?
> Alternately, checking for their (missing) declarations should also work.

This patch should do it.


2019-11-17  Bruno Haible  <bruno@clisp.org>

	locale, localename: Improve z/OS support.
	Reported by Daniel Richard G. in
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-11/msg00001.html>.
	* m4/locale_h.m4 (gl_LOCALE_T): New macro, partially extracted from
	gl_LOCALE_H.
	(gl_LOCALE_H): Require it.
	* m4/localename.m4 (gl_LOCALENAME): Likewise. If locale_t is not
	defined, don't even check for newlocale, duplocale, freelocale.
	* m4/intl-thread-locale.m4 (gt_FUNC_USELOCALE): Make the test fail when
	locale_t is not defined.

diff --git a/m4/intl-thread-locale.m4 b/m4/intl-thread-locale.m4
index 3ecba86..e43f602 100644
--- a/m4/intl-thread-locale.m4
+++ b/m4/intl-thread-locale.m4
@@ -1,4 +1,4 @@
-# intl-thread-locale.m4 serial 4
+# intl-thread-locale.m4 serial 5
 dnl Copyright (C) 2015-2019 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -139,13 +139,15 @@ AC_DEFUN([gt_FUNC_USELOCALE],
 [
   AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles
 
-  dnl Persuade Solaris <locale.h> to define 'locale_t'.
+  dnl Persuade glibc and Solaris <locale.h> to define 'locale_t'.
   AC_REQUIRE([AC_USE_SYSTEM_EXTENSIONS])
 
   AC_CHECK_FUNCS_ONCE([uselocale])
 
   dnl On AIX 7.2, the uselocale() function is not documented and leads to
   dnl crashes in subsequent setlocale() invocations.
+  dnl In 2019, some versions of z/OS lack the locale_t type and have a broken
+  dnl uselocale function.
   if test $ac_cv_func_uselocale = yes; then
     AC_CHECK_HEADERS_ONCE([xlocale.h])
     AC_CACHE_CHECK([whether uselocale works],
@@ -156,6 +158,7 @@ AC_DEFUN([gt_FUNC_USELOCALE],
 #if HAVE_XLOCALE_H
 # include <xlocale.h>
 #endif
+locale_t loc1;
 int main ()
 {
   uselocale (NULL);
@@ -164,10 +167,10 @@ int main ()
 }]])],
          [gt_cv_func_uselocale_works=yes],
          [gt_cv_func_uselocale_works=no],
-         [# Guess no on AIX, yes otherwise.
+         [# Guess no on AIX and z/OS, yes otherwise.
           case "$host_os" in
-            aix*) gt_cv_func_uselocale_works="guessing no" ;;
-            *)    gt_cv_func_uselocale_works="guessing yes" ;;
+            aix* | mvs*) gt_cv_func_uselocale_works="guessing no" ;;
+            *)           gt_cv_func_uselocale_works="guessing yes" ;;
           esac
          ])
       ])
diff --git a/m4/locale_h.m4 b/m4/locale_h.m4
index d28ba0c..380e40b 100644
--- a/m4/locale_h.m4
+++ b/m4/locale_h.m4
@@ -1,4 +1,4 @@
-# locale_h.m4 serial 21
+# locale_h.m4 serial 22
 dnl Copyright (C) 2007, 2009-2019 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -17,6 +17,8 @@ AC_DEFUN([gl_LOCALE_H],
   dnl If <stddef.h> is replaced, then <locale.h> must also be replaced.
   AC_REQUIRE([gl_STDDEF_H])
 
+  AC_REQUIRE([gl_LOCALE_T])
+
   dnl Solaris 11.0 defines the int_p_*, int_n_* members of 'struct lconv'
   dnl only if _LCONV_C99 is defined.
   AC_REQUIRE([AC_CANONICAL_HOST])
@@ -37,34 +39,6 @@ AC_DEFUN([gl_LOCALE_H],
        [gl_cv_header_locale_h_posix2001=yes],
        [gl_cv_header_locale_h_posix2001=no])])
 
-  dnl Check for <xlocale.h>.
-  AC_CHECK_HEADERS_ONCE([xlocale.h])
-  if test $ac_cv_header_xlocale_h = yes; then
-    HAVE_XLOCALE_H=1
-    dnl Check whether use of locale_t requires inclusion of <xlocale.h>,
-    dnl e.g. on Mac OS X 10.5. If <locale.h> does not define locale_t by
-    dnl itself, we assume that <xlocale.h> will do so.
-    AC_CACHE_CHECK([whether locale.h defines locale_t],
-      [gl_cv_header_locale_has_locale_t],
-      [AC_COMPILE_IFELSE(
-         [AC_LANG_PROGRAM(
-            [[#include <locale.h>
-              locale_t x;]],
-            [[]])],
-         [gl_cv_header_locale_has_locale_t=yes],
-         [gl_cv_header_locale_has_locale_t=no])
-      ])
-    if test $gl_cv_header_locale_has_locale_t = yes; then
-      gl_cv_header_locale_h_needs_xlocale_h=no
-    else
-      gl_cv_header_locale_h_needs_xlocale_h=yes
-    fi
-  else
-    HAVE_XLOCALE_H=0
-    gl_cv_header_locale_h_needs_xlocale_h=no
-  fi
-  AC_SUBST([HAVE_XLOCALE_H])
-
   dnl Check whether 'struct lconv' is complete.
   dnl Bionic libc's 'struct lconv' is just a dummy.
   dnl On OpenBSD 4.9, HP-UX 11, IRIX 6.5, OSF/1 5.1, Solaris 9, Cygwin 1.5.x,
@@ -99,6 +73,49 @@ AC_DEFUN([gl_LOCALE_H],
     [setlocale newlocale duplocale freelocale])
 ])
 
+dnl Checks to determine whether the system has the locale_t type,
+dnl and how to obtain it.
+AC_DEFUN([gl_LOCALE_T],
+[
+  dnl Persuade glibc and Solaris <locale.h> to define locale_t.
+  AC_REQUIRE([gl_USE_SYSTEM_EXTENSIONS])
+
+  dnl Check whether use of locale_t requires inclusion of <xlocale.h>,
+  dnl e.g. on Mac OS X 10.5. If <locale.h> does not define locale_t by
+  dnl itself, we assume that <xlocale.h> will do so.
+  AC_CACHE_CHECK([whether locale.h defines locale_t],
+    [gl_cv_header_locale_has_locale_t],
+    [AC_COMPILE_IFELSE(
+       [AC_LANG_PROGRAM(
+          [[#include <locale.h>
+            locale_t x;]],
+          [[]])],
+       [gl_cv_header_locale_has_locale_t=yes],
+       [gl_cv_header_locale_has_locale_t=no])
+    ])
+
+  dnl Check for <xlocale.h>.
+  AC_CHECK_HEADERS_ONCE([xlocale.h])
+  if test $ac_cv_header_xlocale_h = yes; then
+    HAVE_XLOCALE_H=1
+    if test $gl_cv_header_locale_has_locale_t = yes; then
+      gl_cv_header_locale_h_needs_xlocale_h=no
+    else
+      gl_cv_header_locale_h_needs_xlocale_h=yes
+    fi
+    HAVE_LOCALE_T=1
+  else
+    HAVE_XLOCALE_H=0
+    gl_cv_header_locale_h_needs_xlocale_h=no
+    if test $gl_cv_header_locale_has_locale_t = yes; then
+      HAVE_LOCALE_T=1
+    else
+      HAVE_LOCALE_T=0
+    fi
+  fi
+  AC_SUBST([HAVE_XLOCALE_H])
+])
+
 AC_DEFUN([gl_LOCALE_MODULE_INDICATOR],
 [
   dnl Use AC_REQUIRE here, so that the default settings are expanded once only.
diff --git a/m4/localename.m4 b/m4/localename.m4
index a9f629d..8e8d249 100644
--- a/m4/localename.m4
+++ b/m4/localename.m4
@@ -1,4 +1,4 @@
-# localename.m4 serial 6
+# localename.m4 serial 7
 dnl Copyright (C) 2007, 2009-2019 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -7,11 +7,20 @@ dnl with or without modifications, as long as this notice is preserved.
 AC_DEFUN([gl_LOCALENAME],
 [
   AC_REQUIRE([gl_LOCALE_H_DEFAULTS])
+  AC_REQUIRE([gl_LOCALE_T])
   AC_REQUIRE([gt_LC_MESSAGES])
   AC_REQUIRE([gt_INTL_THREAD_LOCALE_NAME])
   AC_REQUIRE([gt_INTL_MACOSX])
   AC_CHECK_HEADERS_ONCE([langinfo.h])
-  AC_CHECK_FUNCS_ONCE([newlocale duplocale freelocale])
+  if test $HAVE_LOCALE_T = 1; then
+    AC_CHECK_FUNCS_ONCE([newlocale duplocale freelocale])
+  else
+    dnl In 2019, some versions of z/OS lack the locale_t type and have broken
+    dnl newlocale, duplocale, freelocale functions.
+    ac_cv_func_newlocale=no
+    ac_cv_func_duplocale=no
+    ac_cv_func_freelocale=no
+  fi
   if test $ac_cv_func_newlocale != yes; then
     HAVE_NEWLOCALE=0
   fi

Re: IBM z/OS compatibility issues - pthread

[Bruno Haible]

Hello Daniel,

> PTHREAD ENVIRONMENT
> 
> z/OS has effectively two major pthread interfaces: _OPEN_THREADS, and
> _UNIX95_THREADS. The latter is the one you want to use, because its API
> is compatible with other systems. (_OPEN_THREADS uses e.g. a different
> signature for pthread_getspecific() that comes from a draft POSIX
> specification; refer to
> 
>     https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ptgetsp.htm
> 
> for the gory details. pthread_detach() is different, too, and
> pthread_cond_timedwait() can return EAGAIN.)
> 
> Additional information on z/OS feature test macros, if desired:
> 
>     https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/ftms.htm
> 
> More specifically, to get a suitable pthreads interface, you need to
> #define _UNIX95_THREADS and _XOPEN_SOURCE=600.

I'd suggest to add this to the recommended values of CC and CXX in
https://gitlab.com/ghwiki/gnow-how/wikis/Platforms/Configuration

> (This could perhaps be
> added to AC_USE_SYSTEM_EXTENSIONS for this system.)

I'm not sure adding these flags in AC_USE_SYSTEM_EXTENSIONS or
gl_USE_SYSTEM_EXTENSIONS would work for all packages. For the majority,
probably yes, but for all of them? Not sure.

> However, one annoyance of _UNIX95_THREADS is that it does not define
> PTHREAD_RWLOCK_INITIALIZER, supposedly because the SUSv3 standard (which
> is what that feature test macro requests) does not specify it. However,
> IBM does provide the "implementation-defined"
> PTHREAD_RWLOCK_INITIALIZER_NP, which can be used in its place. See the
> note here:
> 
>     https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/rp0r0i.htm
> 
> I would thus suggest adding something like the following to
> lib/pthread.in.h:
> 
>     #if defined(__MVS__) && !defined(PTHREAD_RWLOCK_INITIALIZER)
>     # define PTHREAD_RWLOCK_INITIALIZER PTHREAD_RWLOCK_INITIALIZER_NP
>     #endif

Tell the package maintainers to import the gnulib module 'pthread-rwlock'.
It will deal with the missing PTHREAD_RWLOCK_INITIALIZER; this issue occurs
also on some older Solaris versions.

Bruno

Re: IBM z/OS compatibility issues - shell environment

[Bruno Haible]

Hello Daniel,

> SHELL SCRIPTING SNAFUS
> 
> In the z/OS environment, if you want to use a proper Bash shell, you
> typically get the build provided by Rocket Software:
> 
>     https://www.rocketsoftware.com/product-categories/mainframe/bash-zos
> 
> The problem, however, is that shell scripting with this version of Bash
> can be a little tricky, because it doesn't fully embrace EBCDIC. ...
> These are the
> settings I used which allow things to work:
> 
>     _ENCODE_FILE_NEW=IBM-1047
>     _ENCODE_FILE_EXISTING=IBM-1047
>     _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
>     _BPXK_AUTOCVT=ON
>     _TAG_REDIR_ERR=txt
>     _TAG_REDIR_IN=txt
>     _TAG_REDIR_OUT=txt

Can we have this documented in the column "Other variables" of
https://gitlab.com/ghwiki/gnow-how/wikis/Platforms/Configuration ?

> While I would not recommend giving to init.sh knowledge of the above
> variables, I think it would be helpful to do some basic sanity checking
> (like the echo|grep invocation above) to avoid more pathological
> breakage later in the script. The failure message could include a hint
> to the user about what's wrong with the shell, and what needs to be done
> to fix it.

The 'echo ABC | grep ABC > /dev/null' command is indeed something that we
could check in a number of shell scripts. In which shell scripts do you wish
it to be added?

Bruno

Re: IBM z/OS compatibility issues - environment variables

[Bruno Haible]

Hi Daniel,

> ENVIRONMENT VARIABLES
> 
> One annoyance on this platform is that the default behavior of things
> can sometimes be weirdly different from other platforms, in a way which
> breaks programs expecting normal Unix/POSIX behavior. Unfortunately,
> IBM's response to this is often not "Let us fix that so it works like
> other Unix systems," but "That's too bad. We can't change the default
> behavior because mumblemumble, but we can provide an environment
> variable that, if set, with cause that thing to behave in the manner
> you expect."
> 
> At present, the only such variable worth mentioning here is
> _EDC_SIG_DFLT, which when set to 1, causes certain default signal
> handlers *not* to print out messages:
> 
>     https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.cbcpx01/edc_sig_dflt.htm
> 
> This is something that should at least be set in the gnulib test
> environment so that test-sigpipe.sh doesn't break, but may be worth
> adding to the library itself so that programs can continue to expect
> "normal" signal semantics.
> 
> (There are other environment variables documented at the link above that
> may be of interest, but I have not found any additional ones to be
> necessary to fix issues in gnulib's test suite.)

So, the gnulib tests for the math functions and *printf work fine without
_EDC_SUSV3? And the ones for *printf work fine without _EDC_C99_NAN?

I think many programs that are built with gnulib invoke other programs;
therefore it is risky to add setenv calls for such environment variables
to gnulib itself. But we can do so
  - in the documentation,
  - in particular tests. You mentioned test-sigpipe.sh; do you have a
    list of tests which fail by default and succeed with one of these
    environment variables.

Bruno

Re: IBM z/OS compatibility issues - miscellaneous bugs

[Bruno Haible]

Hi Daniel,

> MISCELLANEOUS BUGS
> 
> There are a handful of additional issues that I would like to mention
> here for completeness, but am taking up primarily with IBM, rather than
> this list. If anyone would like me to elaborate on any of these, please
> feel free to ask:
> 
> * [sys/]signal.h does not #define NSIG (nor SIGMAX)
> 
> * File descriptor passed to fdopendir() no longer behaves like a normal
>   fd; in particular, dup2(fd,fd) and fcntl(fd,F_GETFL) both fail (but
>   close(fd) does not), causing test-fdopendir to fail spuriously (breaks
>   the "fdopendir should not close fd" assertion)
> 
> * select() on /dev/null always returns 0, even though reading or writing
>   to it never blocks
> ...
> * C11 _Thread_local support is utterly broken (test-thread_local fails)

I agree that these are bugs that are best discussed with the vendor.

Should we add a workaround to the "check whether _Thread_local works"
autoconf test, hardwiring the 'defined __MVS__' condition?

Bruno

Re: IBM z/OS compatibility issues - per-thread locale functions

[Daniel Richard G.]

Hi Bruno,

On Sun, 2019 Nov 17 19:17-05:00, Bruno Haible wrote:
> 
> This patch should do it.
> 
> 2019-11-17  Bruno Haible  <bruno@clisp.org>
> 
> 	locale, localename: Improve z/OS support.
> 	Reported by Daniel Richard G. in
> 	<https://lists.gnu.org/archive/html/bug-gnulib/2019-11/msg00001.html>.
> 	* m4/locale_h.m4 (gl_LOCALE_T): New macro, partially extracted from
> 	gl_LOCALE_H.
> 	(gl_LOCALE_H): Require it.
> 	* m4/localename.m4 (gl_LOCALENAME): Likewise. If locale_t is not
> 	defined, don't even check for newlocale, duplocale, freelocale.
> 	* m4/intl-thread-locale.m4 (gt_FUNC_USELOCALE): Make the test fail when
> 	locale_t is not defined.

Thanks so much for looking at this, and the other facets. I will be
following up on all of them.

I tested Git commit 5bd825f0.

Unfortunately, I am still seeing build breakage related to attempted
duplocale() replacement:

    source='/tmp/testdir/gllib/hard-locale.c' object='hard-locale.o' libtool=no \
    DEPDIR=.deps depmode=aix /bin/sh /tmp/testdir/build-aux/depcomp \
    xlc-wrap -qlanglvl=extc1x -DHAVE_CONFIG_H -I. -I/tmp/testdir/gllib -I..  -DGNULIB_STRICT_CHECKING=1 -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -q64 -qfloat=ieee -qenumsize=4  -c -o hard-locale.o /tmp/testdir/gllib/hard-locale.c
    ERROR CCN3166 ./locale.h:702   Definition of function locale_t requires parentheses.
    ERROR CCN3276 ./locale.h:702   Syntax error: possible missing '{'?
    ERROR CCN3273 /usr/include/stdlib.h:64    Missing type in declaration of div_t.
    ERROR CCN3166 /usr/include/stdlib.h:544   Definition of function div_t requires parentheses.
    ERROR CCN3276 /usr/include/stdlib.h:544   Syntax error: possible missing '{'?
    [...]

(Line 702 is "_GL_FUNCDECL_RPL (duplocale, locale_t, ...")

This was the very problem I encountered when I tried my hand at a
patch. I could pick up on the missing locale_t, but I could not figure
out how to (cleanly) negate the "duplocale() et al. are present but
broken" outcomes.

Locale-related configure findings:

    bash-2.03$ grep locale configure.out | fgrep -v '(cached)'
    checking for xlocale.h... no
    checking for duplocale... yes
    checking for uselocale... yes
    checking for newlocale... yes
    checking for freelocale... yes
    checking for a traditional french locale... fr_FR
    checking whether locale.h defines locale_t... no
    checking whether locale.h conforms to POSIX:2001... yes
    checking whether uselocale works... no
    checking for a traditional japanese locale... none
    checking for a transitional chinese locale... none
    checking for a french Unicode locale... none
    checking whether the C locale is free of encoding errors... yes
    checking whether wcwidth works reasonably in UTF-8 locales... yes
    checking whether duplocale(LC_GLOBAL_LOCALE) works... no
    checking whether setlocale supports the C locale... yes
    checking whether wcrtomb works in the C locale... yes
    checking for a turkish Unicode locale... none

Please let me know if there is any additional reconnaissance I can
provide that may be helpful.


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - per-thread locale functions

[Bruno Haible]

Hi Daniel,

> Locale-related configure findings:
> 
>     bash-2.03$ grep locale configure.out | fgrep -v '(cached)'
>     checking for xlocale.h... no
>     checking for duplocale... yes
>     checking for uselocale... yes
>     checking for newlocale... yes
>     checking for freelocale... yes
>     checking for a traditional french locale... fr_FR
>     checking whether locale.h defines locale_t... no
>     checking whether locale.h conforms to POSIX:2001... yes
>     checking whether uselocale works... no
>     checking for a traditional japanese locale... none
>     checking for a transitional chinese locale... none
>     checking for a french Unicode locale... none
>     checking whether the C locale is free of encoding errors... yes
>     checking whether wcwidth works reasonably in UTF-8 locales... yes
>     checking whether duplocale(LC_GLOBAL_LOCALE) works... no
>     checking whether setlocale supports the C locale... yes
>     checking whether wcrtomb works in the C locale... yes
>     checking for a turkish Unicode locale... none

I don't understand this.

> Please let me know if there is any additional reconnaissance I can
> provide that may be helpful.

Yes, I need more details here.

  1) Please make sure you build in a fresh directory ('rm -rf testdir'
     before you generate a new testdir), and have no global config cache.
  2) Run configure with option '-C'.
  3) Please provide all three of the following:
     - grep locale configure.out
     - grep -i locale config.cache
     - grep -i locale config.status

Bruno

Re: IBM z/OS compatibility issues - per-thread locale functions

[Daniel Richard G.]

On Mon, 2019 Nov 18 06:41-05:00, Bruno Haible wrote:
> Hi Daniel,
> 
> > Locale-related configure findings:
> > 
> >     bash-2.03$ grep locale configure.out | fgrep -v '(cached)'
> >     checking for xlocale.h... no
> >     checking for duplocale... yes
[...]
> >     checking whether wcrtomb works in the C locale... yes
> >     checking for a turkish Unicode locale... none
> 
> I don't understand this.

At least I'm not alone :]

> Yes, I need more details here.
> 
>   1) Please make sure you build in a fresh directory ('rm -rf testdir'
>      before you generate a new testdir), and have no global config
>      cache.

Because I've been running the test suite repeatedly, I've scripted most
of the process to eliminate casual errors. As a matter of course it
begins with an empty build directory.

I've hardly ever used config caches, but in any event...

    bash-2.03$ env | grep CONFIG
    (no output)

>   2) Run configure with option '-C'.

I've added this, and re-run the build. The following information is from
that new run. (Same Git commit.)

>   3) Please provide all three of the following:
>      - grep locale configure.out

Including the "(cached)" lines, ah? It's a bit more verbiage:

    bash-2.03$ grep locale configure.out
    checking for xlocale.h... no
    checking for duplocale... yes
    checking for uselocale... yes
    checking for newlocale... yes
    checking for freelocale... yes
    checking for a traditional french locale... fr_FR
    checking whether locale.h defines locale_t... no
    checking whether locale.h conforms to POSIX:2001... yes
    checking whether uselocale works... no
    checking for a traditional japanese locale... none
    checking for a transitional chinese locale... none
    checking for a french Unicode locale... none
    checking whether the C locale is free of encoding errors... yes
    checking whether wcwidth works reasonably in UTF-8 locales... yes
    checking whether duplocale(LC_GLOBAL_LOCALE) works... no
    checking whether locale.h conforms to POSIX:2001... (cached) yes
    checking whether the C locale is free of encoding errors... (cached) yes
    checking whether setlocale supports the C locale... yes
    checking whether wcrtomb works in the C locale... yes
    checking whether wcwidth works reasonably in UTF-8 locales... (cached) yes
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a turkish Unicode locale... none
    checking whether uselocale works... (cached) no
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional japanese locale... (cached) none
    checking for a transitional chinese locale... (cached) none
    checking for a french Unicode locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional japanese locale... (cached) none
    checking for a transitional chinese locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional japanese locale... (cached) none
    checking for a transitional chinese locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional japanese locale... (cached) none
    checking for a transitional chinese locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional japanese locale... (cached) none
    checking for a transitional chinese locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional japanese locale... (cached) none
    checking for a transitional chinese locale... (cached) none
    checking for a traditional french locale... (cached) fr_FR
    checking for a french Unicode locale... (cached) none
    checking for a traditional japanese locale... (cached) none
    checking for a transitional chinese locale... (cached) none

>      - grep -i locale config.cache

    bash-2.03$ grep -i locale config.cache 
    ac_cv_func_duplocale=${ac_cv_func_duplocale="yes"}
    ac_cv_func_freelocale=${ac_cv_func_freelocale="yes"}
    ac_cv_func_newlocale=${ac_cv_func_newlocale="yes"}
    ac_cv_func_uselocale=${ac_cv_func_uselocale="yes"}
    ac_cv_header_xlocale_h=${ac_cv_header_xlocale_h="no"}
    gl_cv_C_locale_sans_EILSEQ=${gl_cv_C_locale_sans_EILSEQ="yes"}
    gl_cv_func_duplocale_works=${gl_cv_func_duplocale_works="no"}
    gl_cv_func_setlocale_works=${gl_cv_func_setlocale_works="yes"}
    gl_cv_header_locale_h_needs_xlocale_h=${gl_cv_header_locale_h_needs_xlocale_h="}
    gl_cv_header_locale_h_posix2001=${gl_cv_header_locale_h_posix2001="yes"}
    gl_cv_header_locale_has_locale_t=${gl_cv_header_locale_has_locale_t="no"}
    gl_cv_next_locale_h=${gl_cv_next_locale_h="<locale.h>"}
    gt_cv_func_CFLocaleCopyPreferredLanguages=${gt_cv_func_CFLocaleCopyPreferredLan}
    gt_cv_func_uselocale_works=${gt_cv_func_uselocale_works="no"}
    gt_cv_locale_fake=${gt_cv_locale_fake="no"}
    gt_cv_locale_fr=${gt_cv_locale_fr="fr_FR"}
    gt_cv_locale_fr_utf8=${gt_cv_locale_fr_utf8="none"}
    gt_cv_locale_ja=${gt_cv_locale_ja="none"}
    gt_cv_locale_solaris114=${gt_cv_locale_solaris114="no"}
    gt_cv_locale_tr_utf8=${gt_cv_locale_tr_utf8="none"}
    gt_cv_locale_zh_CN=${gt_cv_locale_zh_CN="none"}

>      - grep -i locale config.status

    bash-2.03$ grep -i locale config.status
    S["gl_LTLIBOBJS"]=" alphasort.lo asnprintf.lo at-func2.lo canonicalize-lgpl.lo cnd.lo dirfd.lo dprintf.lo duplocale.lo error.lo euidaccess.lo faccessat.lo fchmodat.lo "\
    S["gl_LIBOBJS"]=" alphasort.o asnprintf.o at-func2.o canonicalize-lgpl.o cnd.o dirfd.o dprintf.o duplocale.o error.o euidaccess.o faccessat.o fchmodat.o fchownat.o f"\
    S["LOCALE_TR_UTF8"]="none"
    S["LOCALE_FR_UTF8"]="none"
    S["LOCALE_ZH_CN"]="none"
    S["LOCALE_JA"]="none"
    S["NEXT_AS_FIRST_DIRECTIVE_LOCALE_H"]="<locale.h>"
    S["NEXT_LOCALE_H"]="<locale.h>"
    S["HAVE_XLOCALE_H"]="0"
    S["REPLACE_FREELOCALE"]="0"
    S["REPLACE_DUPLOCALE"]="1"
    S["REPLACE_NEWLOCALE"]="0"
    S["REPLACE_SETLOCALE"]="0"
    S["REPLACE_LOCALECONV"]="0"
    S["HAVE_FREELOCALE"]="0"
    S["HAVE_DUPLOCALE"]="0"
    S["HAVE_NEWLOCALE"]="0"
    S["GNULIB_LOCALENAME"]="1"
    S["GNULIB_DUPLOCALE"]="1"
    S["GNULIB_SETLOCALE"]="1"
    S["GNULIB_LOCALECONV"]="1"
    S["LOCALE_FR"]="fr_FR"
    S["localedir"]="${datarootdir}/locale"
    D["HAVE_DUPLOCALE"]=" 1"
    D["HAVE_USELOCALE"]=" 1"
    D["HAVE_NEWLOCALE"]=" 1"
    D["HAVE_FREELOCALE"]=" 1"
    D["GNULIB_TEST_DUPLOCALE"]=" 1"
    D["GNULIB_TEST_LOCALECONV"]=" 1"
    D["GNULIB_TEST_LOCALENAME"]=" 1"
    D["GNULIB_TEST_SETLOCALE"]=" 1"
    /@localedir@/p
    *@datadir@*|*@docdir@*|*@infodir@*|*@localedir@*|*@mandir@*)
      s&@localedir@&${datarootdir}/locale&g

Is it feasible to test the configure logic on your end by setting e.g.
gl_cv_header_locale_has_locale_t=no ?


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues

[Daniel Richard G.]

On Sun, 2019 Nov 17 18:21-05:00, Bruno Haible wrote:
> Hi Daniel,
> 
> > The C99 mode (-qlanglvl=extc99) works without issue. That's what I've
> > been using for all my testing.
> 
> Do you have a set of configure settings that you would recommend on z/OS?
> So that we can add this info to the Gnow-How wiki [1]...

There are quite a few, unfortunately. Before getting into them, however,
there is something I am not clear on:

How should platform-specific compiler flags, feature-test macros, etc.
be split up among the following?

  1. Flags set in Autoconf (common system/compiler checks)

  2. Flags set in Automake ("compile" script)

  3. Flags set in Gnulib's AC_USE_SYSTEM_EXTENSIONS()

  4. Flags set in other Gnulib modules

  5. Flags set by the user in the environment or configure command line
     (obtained from online documentation, discussions, etc.)

z/OS has a number of idiosyncrasies that are not shared even by the
other strange Unix platforms, and putting a lot of weight in bucket #5
seems to me like it will make the system needlessly tricky to navigate
for GNU users unfamiliar with it.

Here are examples of what I currently think should go in each bucket:

  1. -qhaltonmsg=CCN3296, so that missing header files are treated as
     errors, not warnings (thus allowing header checks to return
     meaningful results);

  2. The -qnosearch/-qsearch logic previously described for xlc-wrap,
     so that it is possible for a project to replace an existing
     system header;

  3 or 4. Feature-test macros needed to obtain a recognizable
     pthreads API;

  5. CC=xlc (there are a few usable XLC compiler frontends, and the user
     might prefer one or another for whatever reason)


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - pthread

[Daniel Richard G.]

On Sun, 2019 Nov 17 22:18-05:00, Bruno Haible wrote:
>
> > More specifically, to get a suitable pthreads interface, you need to
> > #define _UNIX95_THREADS and _XOPEN_SOURCE=600.
> 
> I'd suggest to add this to the recommended values of CC and CXX in
> https://gitlab.com/ghwiki/gnow-how/wikis/Platforms/Configuration

For archive readers: There is a parallel thread discussing what bits of
system knowledge should go into online docs, and which ones might be
better integrated into the build framework.

> > (This could perhaps be added to AC_USE_SYSTEM_EXTENSIONS for this
> > system.)
>
> I'm not sure adding these flags in AC_USE_SYSTEM_EXTENSIONS or
> gl_USE_SYSTEM_EXTENSIONS would work for all packages. For the
> majority, probably yes, but for all of them? Not sure.

I am wondering how inclusive "all" is meant to be.

One could, in theory, have a package that wants/requires the alternate
form of the pthreads API that z/OS provides. But such a package would
only be usable on z/OS, and that is not a likely design decision for a
GNU project.

Would not most packages be suited by a build environment that provides
the standard pthreads API, as well as others? The above feature macros
are similar in spirit to _ALL_SOURCE / _GNU_SOURCE, in that way (i.e.
"why wouldn't you want this?").

Admittedly, there may be some features that are blocked out by those. If
this is occurring in my Gnulib test runs, and it is not producing an
error/failure (e.g. tests skipped due to missing functionality), then I
am not aware of it. The flags (and bug reports!) I've compiled have been
driven primarily by the desire to allow Gnulib to build everything it
wants to build---which presumably spans a broad enough array of
functionality to cover the needs of most, if not all packages.

> > I would thus suggest adding something like the following to
> > lib/pthread.in.h:
> > 
> >     #if defined(__MVS__) && !defined(PTHREAD_RWLOCK_INITIALIZER)
> >     # define PTHREAD_RWLOCK_INITIALIZER PTHREAD_RWLOCK_INITIALIZER_NP
> >     #endif
> 
> Tell the package maintainers to import the gnulib module 'pthread-
> rwlock'. It will deal with the missing PTHREAD_RWLOCK_INITIALIZER;
> this issue occurs also on some older Solaris versions.

If I omit that fragment, I get this build error:

    source='/tmp/testdir/gltests/test-pthread.c' object='test-pthread.o' libtool=no \
    DEPDIR=.deps depmode=aix /bin/sh /tmp/testdir/build-aux/depcomp \
    xlc-wrap -DHAVE_CONFIG_H -DEXEEXT=\"\" -I. -I/tmp/testdir/gltests -I..  -DGNULIB_STRICT_CHECKING=1 -DIN_GNULIB_TESTS=1 -I. -I/tmp/testdir/gltests -I.. -I/tmp/testdir/gltests/.. -I../gllib -I/tmp/testdir/gltests/../gllib -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -q64 -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o test-pthread.o /tmp/testdir/gltests/test-pthread.c
    ERROR CCN3221 /tmp/testdir/gltests/test-pthread.c:35    Initializer must be a valid constant expression.
    WARNING CCN3196 /tmp/testdir/gltests/test-pthread.c:56    Initialization between types "void*" and "unsigned long" is not allowed.
    CCN0793(I) Compilation failed for file /tmp/testdir/gltests/test-pthread.c.  Object file not created.
    make[4]: *** [test-pthread.o] Error 12

I don't see "PTHREAD_RWLOCK_INITIALIZER_NP" anywhere in a clean Gnulib
tree, so I'm not sure how this could already be addressed...


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - shell environment

[Daniel Richard G.]

On Sun, 2019 Nov 17 22:24-05:00, Bruno Haible wrote:
> > 
> > The problem, however, is that shell scripting with this version
> > of Bash can be a little tricky, because it doesn't fully
> > embrace EBCDIC. ... These are the settings I used which allow
> > things to work:
> > 
> >     _ENCODE_FILE_NEW=IBM-1047
> >     _ENCODE_FILE_EXISTING=IBM-1047
> >     _CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
> >     _BPXK_AUTOCVT=ON
> >     _TAG_REDIR_ERR=txt
> >     _TAG_REDIR_IN=txt
> >     _TAG_REDIR_OUT=txt
> 
> Can we have this documented in the column "Other variables" of
> https://gitlab.com/ghwiki/gnow-how/wikis/Platforms/Configuration ?

It's quite a bit of verbiage to stick into a table cell! Might a
separate page be appropriate? Scripts could give the URL to it in
lieu of a more descriptive error message if they detect breakage
related to this.

> > While I would not recommend giving to init.sh knowledge of the above
> > variables, I think it would be helpful to do some basic sanity
> > checking (like the echo|grep invocation above) to avoid more
> > pathological breakage later in the script. The failure message could
> > include a hint to the user about what's wrong with the shell, and
> > what needs to be done to fix it.
>
> The 'echo ABC | grep ABC > /dev/null' command is indeed something that
> we could check in a number of shell scripts. In which shell scripts do
> you wish it to be added?

When /bin/sh is used to build Gnulib, everything works great. The
top-level configure run and most other incidental shell scripts are
covered by this.

It's only when a shell script decides "/bin/sh is junk, I want bash"
that trouble arises. And when I do a standard "./configure && make &&
make check" run, this doesn't happen until it gets to test-atexit.sh.
What is notable about that one is that it appears to be the first
shell-script test that sources init.sh.

Here is what appears on the console when I attempt to run this test
manually, with the Rocket build of bash in my PATH:

    $ env srcdir=/tmp/testdir/gltests /tmp/testdir/gltests/test-atexit.sh 
    expr: non-numeric argument "@@@@@??@"
    /tmp/testdir/gltests/init.sh: line 366: test: ?: integer expression expecd
    expr: non-numeric argument "?"
    /tmp/testdir/gltests/init.sh: line 366: test: 4: unary operator expected
    Usage: expr expression
    /tmp/testdir/gltests/init.sh: line 366: test: 4: unary operator expected
    Usage: expr expression
    /tmp/testdir/gltests/init.sh: line 366: test: 4: unary operator expected
    Usage: expr expression
    /tmp/testdir/gltests/init.sh: line 366: test: 4: unary operator expected
    Usage: expr expression
    /tmp/testdir/gltests/init.sh: line 366: test: 4: unary operator expected
    Usage: expr expression
    [continues on forever]

The init.sh script starts off in /bin/sh but looks for and executes
bash, presumably to obtain more advanced shell functionality.

Even before trying to fix this, I'm wondering: Is this really necessary?
The place where it gets stuck above is in the mktempd_() shell function,
which provides the functionality of a missing mktemp(1). Given that this
is running in a build tree, wouldn't something like prefix$$ be enough?

If that can't be helped, then at least init.sh should perform this
sanity check. I see that there is already some shell-checking in place
via gl_shell_test_script_; perhaps this could be an added check. I hope
a helpful error message can be part of it too.


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - environment variables

[Daniel Richard G.]

On Sun, 2019 Nov 17 22:34-05:00, Bruno Haible wrote:
>
> > (There are other environment variables documented at the link above
> > that may be of interest, but I have not found any additional ones to
> > be necessary to fix issues in gnulib's test suite.)
>
> So, the gnulib tests for the math functions and *printf work fine
> without _EDC_SUSV3? And the ones for *printf work fine without
> _EDC_C99_NAN?

I have not seen any test failures related to the functionality affected
by those two variables. But are the behaviors at issue covered by the
tests, thorough as they otherwise are?

For example, if I look at the log10*() family of functions, I see this
in the IBM doc:

    If x is 0, the function returns -HUGE_VAL and errno remains
    unchanged.

    Note: When environment variable _EDC_SUSV3 is set to 2, and if x is
    0, the function returns -HUGE_VAL and sets errno to ERANGE.

    https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxbd00/log10.htm

So it sets ERANGE where it otherwise wouldn't have before. But there is
no mention of ERANGE in tests/test-log* ... should there be?

> I think many programs that are built with gnulib invoke other programs;
> therefore it is risky to add setenv calls for such environment variables
> to gnulib itself. But we can do so
>   - in the documentation,
>   - in particular tests. You mentioned test-sigpipe.sh; do you have a
>     list of tests which fail by default and succeed with one of these
>     environment variables.

That is the only one I've found, of all the tests. It fails because the
output file t-sigpipeA.tmp is not empty; it contains the text

    CEE5213S The signal SIGPIPE was received.

z/OS often takes the opportunity to do things slightly differently from
other systems, if the letter of the standard allows it :/


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - miscellaneous bugs

[Daniel Richard G.]

On Sun, 2019 Nov 17 22:41-05:00, Bruno Haible wrote:
>
> > * C11 _Thread_local support is utterly broken (test-thread_local
> >   fails)
>
> I agree that these are bugs that are best discussed with the vendor.
>
> Should we add a workaround to the "check whether _Thread_local works"
> autoconf test, hardwiring the 'defined __MVS__' condition?

I do trust that this will start working at some point (IBM can't leave
it like this forever), so testing the functionality without assuming
anything for the system seems like a good status quo.

There is one bit that left me scratching my head, however. If I drop
-qlanglvl=extc99 from my CFLAGS (on the notion that the configure script
should find this flag itself), I see this result:

    checking for xlc-wrap option to enable C11 features... unsupported
    checking for xlc-wrap option to enable C99 features... -qlanglvl=extc1x

Is it proper for _AC_PROG_CC_C99() to try a C11 flag to get C99 mode?
This may yield C99 functionality, but __STDC_VERSION__ still indicates
C11, and applications taking this at face value will run into problems
if they attempt to use certain aspects of C11.

As far as I've seen, the C99 mode on this compiler (per my original
flag) is complete and functional.


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - per-thread locale functions

[Bruno Haible]

Hi Daniel,

> >   3) Please provide all three of the following:
> >      - grep locale configure.out
> ...
> >      - grep -i locale config.status
> 
>     bash-2.03$ grep -i locale config.status
>     S["gl_LTLIBOBJS"]=" alphasort.lo asnprintf.lo at-func2.lo canonicalize-lgpl.lo cnd.lo dirfd.lo dprintf.lo duplocale.lo error.lo euidaccess.lo faccessat.lo fchmodat.lo "\
>     S["gl_LIBOBJS"]=" alphasort.o asnprintf.o at-func2.o canonicalize-lgpl.o cnd.o dirfd.o dprintf.o duplocale.o error.o euidaccess.o faccessat.o fchmodat.o fchownat.o f"\
> ...
>     S["REPLACE_DUPLOCALE"]="1"
> ...
>     S["HAVE_DUPLOCALE"]="0"

The REPLACE_DUPLOCALE=1 is wrong. HAVE_DUPLOCALE should be 0, not only
when set by localename.m4 but also when set by duplocale.m4. This patch
should fix it.


2019-12-12  Bruno Haible  <bruno@clisp.org>

	duplocale: Don't attempt to override if locale_t does not exist.
	Reported by Daniel Richard G. in
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-11/msg00039.html>.
	* m4/duplocale.m4 (gl_FUNC_DUPLOCALE): If locale_t does not exist, set
	HAVE_DUPLOCALE to 0.

diff --git -w a/m4/duplocale.m4 b/m4/duplocale.m4
index c2c95dd..fb5e602 100644
--- a/m4/duplocale.m4
+++ b/m4/duplocale.m4
@@ -1,4 +1,4 @@
-# duplocale.m4 serial 10
+# duplocale.m4 serial 11
 dnl Copyright (C) 2009-2019 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -17,6 +17,7 @@ AC_DEFUN([gl_FUNC_DUPLOCALE],
     dnl Also, on NetBSD 7.0, duplocale(LC_GLOBAL_LOCALE) returns a locale that
     dnl corresponds to the C locale.
     AC_REQUIRE([gl_LOCALE_H])
+    if test $HAVE_LOCALE_T = 1; then
       AC_CHECK_FUNCS_ONCE([snprintf_l nl_langinfo_l])
       AC_CACHE_CHECK([whether duplocale(LC_GLOBAL_LOCALE) works],
         [gl_cv_func_duplocale_works],
@@ -100,6 +101,12 @@ int main ()
         *no) REPLACE_DUPLOCALE=1 ;;
       esac
     else
+      dnl In 2019, some versions of z/OS lack the locale_t type and have broken
+      dnl newlocale, duplocale, freelocale functions. In this situation, we
+      dnl cannot use nor override duplocale.
+      HAVE_DUPLOCALE=0
+    fi
+  else
     HAVE_DUPLOCALE=0
   fi
 ])

Re: IBM z/OS compatibility issues - per-thread locale functions

[Daniel Richard G.]

Hi Bruno,

On Thu, 2019 Dec 12 07:56-05:00, Bruno Haible wrote:
>
> The REPLACE_DUPLOCALE=1 is wrong. HAVE_DUPLOCALE should be 0, not only
> when set by localename.m4 but also when set by duplocale.m4. This
> patch should fix it.

Tested with Git ea54538a.

I can now get past the library build, but then get errors compiling
test-duplocale.c and test-locale.c. The former fails at line 26:

    SIGNATURE_CHECK (duplocale, locale_t, (locale_t));

The latter fails at line 49:

    locale_t b = LC_GLOBAL_LOCALE;

These tests probably don't make sense without the locale_t type.

In any event, there's a bit of change in the output variables, but
HAVE_DUPLOCALE et al. are the same as before:

    $ grep -i locale config.status
    S["LOCALE_TR_UTF8"]="none"
    S["LOCALE_ZH_CN"]="none"
    S["LOCALE_FR_UTF8"]="none"
    S["LOCALE_JA"]="none"
    S["NEXT_AS_FIRST_DIRECTIVE_LOCALE_H"]="<locale.h>"
    S["NEXT_LOCALE_H"]="<locale.h>"
    S["HAVE_XLOCALE_H"]="0"
    S["REPLACE_FREELOCALE"]="0"
    S["REPLACE_DUPLOCALE"]="0"
    S["REPLACE_NEWLOCALE"]="0"
    S["REPLACE_SETLOCALE"]="0"
    S["REPLACE_LOCALECONV"]="0"
    S["HAVE_FREELOCALE"]="0"
    S["HAVE_DUPLOCALE"]="0"
    S["HAVE_NEWLOCALE"]="0"
    S["GNULIB_LOCALENAME"]="1"
    S["GNULIB_DUPLOCALE"]="1"
    S["GNULIB_SETLOCALE"]="1"
    S["GNULIB_LOCALECONV"]="1"
    S["LOCALE_FR"]="fr_FR"
    S["localedir"]="${datarootdir}/locale"
    D["HAVE_DUPLOCALE"]=" 1"
    D["HAVE_USELOCALE"]=" 1"
    D["HAVE_NEWLOCALE"]=" 1"
    D["HAVE_FREELOCALE"]=" 1"
    D["GNULIB_TEST_DUPLOCALE"]=" 1"
    D["GNULIB_TEST_LOCALECONV"]=" 1"
    D["GNULIB_TEST_LOCALENAME"]=" 1"
    D["GNULIB_TEST_SETLOCALE"]=" 1"
    /@localedir@/p
    *@datadir@*|*@docdir@*|*@infodir@*|*@localedir@*|*@mandir@*)
      s&@localedir@&${datarootdir}/locale&g


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - per-thread locale functions

[Bruno Haible]

Daniel Richard G. wrote:
> I can now get past the library build, but then get errors compiling
> test-duplocale.c and test-locale.c. The former fails at line 26:
> 
>     SIGNATURE_CHECK (duplocale, locale_t, (locale_t));
> 
> The latter fails at line 49:
> 
>     locale_t b = LC_GLOBAL_LOCALE;
> 
> These tests probably don't make sense without the locale_t type.

This should fix it. Thanks for the feedback.


2019-12-13  Bruno Haible  <bruno@clisp.org>

	locale, duplocale, localename: Fix errors if locale_t does not exist.
	Reported by Daniel Richard G. in
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-12/msg00078.html>.
	* lib/locale.in.h (HAVE_WORKING_NEWLOCALE, HAVE_WORKING_DUPLOCALE): New
	macros.
	* tests/test-locale.c: Test HAVE_WORKING_NEWLOCALE instead of
	HAVE_NEWLOCALE.
	* tests/test-localename.c: Likewise.
	* tests/test-duplocale.c: Test HAVE_WORKING_DUPLOCALE instead of
	HAVE_DUPLOCALE.
	* tests/test-locale-c++.cc: Likewise.

diff --git a/lib/locale.in.h b/lib/locale.in.h
index 77b8b3b..c0915a5 100644
--- a/lib/locale.in.h
+++ b/lib/locale.in.h
@@ -227,6 +227,9 @@ _GL_CXXALIAS_SYS (newlocale, locale_t,
 # if @HAVE_NEWLOCALE@
 _GL_CXXALIASWARN (newlocale);
 # endif
+# ifndef HAVE_WORKING_NEWLOCALE
+#  define HAVE_WORKING_NEWLOCALE 1
+# endif
 #elif defined GNULIB_POSIXCHECK
 # undef newlocale
 # if HAVE_RAW_DECL_NEWLOCALE
@@ -251,6 +254,9 @@ _GL_CXXALIAS_SYS (duplocale, locale_t, (locale_t locale));
 # if @HAVE_DUPLOCALE@
 _GL_CXXALIASWARN (duplocale);
 # endif
+# ifndef HAVE_WORKING_DUPLOCALE
+#  define HAVE_WORKING_DUPLOCALE 1
+# endif
 #elif defined GNULIB_POSIXCHECK
 # undef duplocale
 # if HAVE_RAW_DECL_DUPLOCALE
diff --git a/tests/test-duplocale.c b/tests/test-duplocale.c
index 3313c83..a0f699e 100644
--- a/tests/test-duplocale.c
+++ b/tests/test-duplocale.c
@@ -20,7 +20,7 @@
 
 #include <locale.h>
 
-#if HAVE_DUPLOCALE
+#if HAVE_WORKING_DUPLOCALE
 
 #include "signature.h"
 SIGNATURE_CHECK (duplocale, locale_t, (locale_t));
diff --git a/tests/test-locale-c++.cc b/tests/test-locale-c++.cc
index ae24705..84abe9d 100644
--- a/tests/test-locale-c++.cc
+++ b/tests/test-locale-c++.cc
@@ -36,7 +36,7 @@ SIGNATURE_CHECK (GNULIB_NAMESPACE::setlocale, char *, (int, const char *));
 SIGNATURE_CHECK (GNULIB_NAMESPACE::newlocale, locale_t, (int, const char *, locale_t));
 #endif
 
-#if GNULIB_TEST_DUPLOCALE && HAVE_DUPLOCALE
+#if GNULIB_TEST_DUPLOCALE && HAVE_WORKING_DUPLOCALE
 SIGNATURE_CHECK (GNULIB_NAMESPACE::duplocale, locale_t, (locale_t));
 #endif
 
diff --git a/tests/test-locale.c b/tests/test-locale.c
index 91deeed..80ca924 100644
--- a/tests/test-locale.c
+++ b/tests/test-locale.c
@@ -44,7 +44,7 @@ verify (sizeof NULL == sizeof (void *));
 int
 main ()
 {
-#if HAVE_NEWLOCALE
+#if HAVE_WORKING_NEWLOCALE
   /* Check that the locale_t type and the LC_GLOBAL_LOCALE macro are defined.  */
   locale_t b = LC_GLOBAL_LOCALE;
   (void) b;
diff --git a/tests/test-localename.c b/tests/test-localename.c
index 77699d9..5a52748 100644
--- a/tests/test-localename.c
+++ b/tests/test-localename.c
@@ -26,7 +26,7 @@
 
 #include "macros.h"
 
-#if HAVE_NEWLOCALE && HAVE_WORKING_USELOCALE && !HAVE_FAKE_LOCALES
+#if HAVE_WORKING_NEWLOCALE && HAVE_WORKING_USELOCALE && !HAVE_FAKE_LOCALES
 # define HAVE_GOOD_USELOCALE 1
 #endif
 

Re: IBM z/OS compatibility issues - pthread

[Bruno Haible]

Hi Daniel,

> > Tell the package maintainers to import the gnulib module 'pthread-
> > rwlock'. It will deal with the missing PTHREAD_RWLOCK_INITIALIZER;
> > this issue occurs also on some older Solaris versions.
> 
> If I omit that fragment, I get this build error:
> 
>     source='/tmp/testdir/gltests/test-pthread.c' object='test-pthread.o' libtool=no \
>     DEPDIR=.deps depmode=aix /bin/sh /tmp/testdir/build-aux/depcomp \
>     xlc-wrap -DHAVE_CONFIG_H -DEXEEXT=\"\" -I. -I/tmp/testdir/gltests -I..  -DGNULIB_STRICT_CHECKING=1 -DIN_GNULIB_TESTS=1 -I. -I/tmp/testdir/gltests -I.. -I/tmp/testdir/gltests/.. -I../gllib -I/tmp/testdir/gltests/../gllib -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -q64 -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o test-pthread.o /tmp/testdir/gltests/test-pthread.c
>     ERROR CCN3221 /tmp/testdir/gltests/test-pthread.c:35    Initializer must be a valid constant expression.
>     WARNING CCN3196 /tmp/testdir/gltests/test-pthread.c:56    Initialization between types "void*" and "unsigned long" is not allowed.
>     CCN0793(I) Compilation failed for file /tmp/testdir/gltests/test-pthread.c.  Object file not created.
>     make[4]: *** [test-pthread.o] Error 12
> 
> I don't see "PTHREAD_RWLOCK_INITIALIZER_NP" anywhere in a clean Gnulib
> tree, so I'm not sure how this could already be addressed...

The gnulib module 'pthread-rwlock' fixes this issue in a platform-independent
way (by overriding pthread_rwlock_t). That's the reason why you don't see
PTHREAD_RWLOCK_INITIALIZER_NP in the gnulib source code.

Bruno

Re: IBM z/OS compatibility issues

[Bruno Haible]

Hi Daniel,

> > Do you have a set of configure settings that you would recommend on z/OS?
> > So that we can add this info to the Gnow-How wiki [1]...
> 
> There are quite a few, unfortunately. Before getting into them, however,
> there is something I am not clear on:
> 
> How should platform-specific compiler flags, feature-test macros, etc.
> be split up among the following?
> 
>   1. Flags set in Autoconf (common system/compiler checks)
> 
>   2. Flags set in Automake ("compile" script)
> 
>   3. Flags set in Gnulib's AC_USE_SYSTEM_EXTENSIONS()
> 
>   4. Flags set in other Gnulib modules
> 
>   5. Flags set by the user in the environment or configure command line
>      (obtained from online documentation, discussions, etc.)
> 
> z/OS has a number of idiosyncrasies that are not shared even by the
> other strange Unix platforms, and putting a lot of weight in bucket #5
> seems to me like it will make the system needlessly tricky to navigate
> for GNU users unfamiliar with it.
> 
> Here are examples of what I currently think should go in each bucket:
> 
>   1. -qhaltonmsg=CCN3296, so that missing header files are treated as
>      errors, not warnings (thus allowing header checks to return
>      meaningful results);
> 
>   2. The -qnosearch/-qsearch logic previously described for xlc-wrap,
>      so that it is possible for a project to replace an existing
>      system header;
> 
>   3 or 4. Feature-test macros needed to obtain a recognizable
>      pthreads API;
> 
>   5. CC=xlc (there are a few usable XLC compiler frontends, and the user
>      might prefer one or another for whatever reason)

You're asking good questions. However, I would like to suggest to
start simple and get the more complex things done afterwards. By that, I mean:

  - Put the xlc-wrap logic into the 'compile' script. We should not have
    two different compile scripts, one for MSVC and one for z/OS. A single
    script should do it.

  - Other than that, use documentation. I have added the most important
    information (from this mail thread) in
    <https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration>
    and would like to invite you to create a new wiki page
    <https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration/zOS>.

Reason:
  - There are few people trying to build GNU software for z/OS; therefore
    communication among you and them should be fairly simple.
  - Autoconf and Automake releases are not frequent these days; therefore
    you would have to wait a long time until modified macros AC_PROG_CC
    etc. become universally available.
  - What you have written about this environment so far indicates that there
    are a lot of complexities; therefore it is likely that you will have to
    tweak your recommendations regarding compiler options and environment
    variables. (Just like I tweaked the recommended options for building
    on HP-UX today. And HP-UX is _way_ simpler than z/OS!)

Bruno

Re: IBM z/OS compatibility issues - per-thread locale functions

[Daniel Richard G.]

On Fri, 2019 Dec 13 05:32-05:00, Bruno Haible wrote:
> 
> This should fix it. Thanks for the feedback.

Arrrgh, not quite there yet  >_<

Tested Git 83710ffa. test-locale now builds, but test-duplocale still
does not, failing in the same place. HAVE_WORKING_DUPLOCALE appears to
be 1 (true).

Isn't it possible to get a final result of HAVE_DUPLOCALE=0,
REPLACE_DUPLOCALE=0, possibly GNULIB_DUPLOCALE=0 and
GNULIB_TEST_DUPLOCALE=0 ? Officially, per IBM documentation, duplocale()
doesn't even exist on z/OS...

    $ grep -i locale config.status
    S["LOCALE_TR_UTF8"]="none"
    S["LOCALE_ZH_CN"]="none"
    S["LOCALE_FR_UTF8"]="none"
    S["LOCALE_JA"]="none"
    S["NEXT_AS_FIRST_DIRECTIVE_LOCALE_H"]="<locale.h>"
    S["NEXT_LOCALE_H"]="<locale.h>"
    S["HAVE_XLOCALE_H"]="0"
    S["REPLACE_FREELOCALE"]="0"
    S["REPLACE_DUPLOCALE"]="0"
    S["REPLACE_NEWLOCALE"]="0"
    S["REPLACE_SETLOCALE"]="0"
    S["REPLACE_LOCALECONV"]="0"
    S["HAVE_FREELOCALE"]="0"
    S["HAVE_DUPLOCALE"]="0"
    S["HAVE_NEWLOCALE"]="0"
    S["GNULIB_LOCALENAME"]="1"
    S["GNULIB_DUPLOCALE"]="1"
    S["GNULIB_SETLOCALE"]="1"
    S["GNULIB_LOCALECONV"]="1"
    S["LOCALE_FR"]="fr_FR"
    S["localedir"]="${datarootdir}/locale"
    D["HAVE_DUPLOCALE"]=" 1"
    D["HAVE_USELOCALE"]=" 1"
    D["HAVE_NEWLOCALE"]=" 1"
    D["HAVE_FREELOCALE"]=" 1"
    D["GNULIB_TEST_DUPLOCALE"]=" 1"
    D["GNULIB_TEST_LOCALECONV"]=" 1"
    D["GNULIB_TEST_LOCALENAME"]=" 1"
    D["GNULIB_TEST_SETLOCALE"]=" 1"
    /@localedir@/p
    *@datadir@*|*@docdir@*|*@infodir@*|*@localedir@*|*@mandir@*)
      s&@localedir@&${datarootdir}/locale&g


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - pthread

[Daniel Richard G.]

Hi Bruno,

On Fri, 2019 Dec 13 07:43-05:00, Bruno Haible wrote:
>
> > I don't see "PTHREAD_RWLOCK_INITIALIZER_NP" anywhere in a clean Gnulib
> > tree, so I'm not sure how this could already be addressed...
> 
> The gnulib module 'pthread-rwlock' fixes this issue in a platform-independent
> way (by overriding pthread_rwlock_t). That's the reason why you don't see
> PTHREAD_RWLOCK_INITIALIZER_NP in the gnulib source code.

Something must be missing, because if I remove my conditional definition
of PTHREAD_RWLOCK_INITIALIZER from pthread.in.h, test-pthread-rwlock is
no longer in the set of tests built by default. If I then attempt to
build it manually, I get the following error:

source='/tmp/testdir/gltests/test-pthread-rwlock.c' object='test-pthread-rwlock.o' libtool=no \
DEPDIR=.deps depmode=aix /bin/sh /tmp/testdir/build-aux/depcomp \
xlc-wrap -DHAVE_CONFIG_H -DEXEEXT=\"\" -I. -I/tmp/testdir/gltests -I..  -DGNULIB_STRICT_CHECKING=1 -DIN_GNULIB_TESTS=1 -I. -I/tmp/testdir/gltests -I.. -I/tmp/testdir/gltests/.. -I../gllib -I/tmp/testdir/gltests/../gllib -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -q64 -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o test-pthread-rwlock.o /tmp/testdir/gltests/test-pthread-rwlock.c
ERROR CCN3221 /tmp/testdir/gltests/test-pthread-rwlock.c:271   Initializer must be a valid constant expression.
CCN0793(I) Compilation failed for file /tmp/testdir/gltests/test-pthread-rwlock.c.  Object file not created.
make: *** [test-pthread-rwlock.o] Error 12

(Note that all my Gnulib work is done in a testdir tree created using
gnulib-tool; my presumption is that this should set up everything
correctly from the get-go.)


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues

[Daniel Richard G.]

On Fri, 2019 Dec 13 07:58-05:00, Bruno Haible wrote:
> 
> You're asking good questions. However, I would like to suggest to
> start simple and get the more complex things done afterwards. By
> that, I mean:
> 
>   - Put the xlc-wrap logic into the 'compile' script. We should not
>     have two different compile scripts, one for MSVC and one for z/OS.
>     A single script should do it.

This sounds good. It would imply broadening the scope of the script
beyond just working around the -c -o limitation, but then it was always
a curiosity that that has been the *only* issue this script was intended
to work around (until now).

I'll e-mail the Automake list about this soon.

Also: I see that AC_USE_SYSTEM_EXTENSIONS() actually comes from
Autoconf. Would that be a good place to put e.g. the -D_UNIX95_THREADS
flag needed to get a proper pthreads API? I see that as similar in
spirit to -D__EXTENSIONS__.

>   - Other than that, use documentation. I have added the most important
>     information (from this mail thread) in
>     <https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration>
>     and would like to invite you to create a new wiki page
>     <https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration/zOS>.

I can add some information, but am concerned about discoverability. I
myself was not aware of this wiki until you mentioned it to me, and
I'm already more informed than most people in my org about
GNU-related matters.

> Reason:
>   - There are few people trying to build GNU software for z/OS;
>     therefore communication among you and them should be fairly
>     simple.

True, though the goal should be for the process to be simple/robust
enough that e-mail correspondence isn't needed :)

>   - Autoconf and Automake releases are not frequent these days;
>     therefore you would have to wait a long time until modified macros
>     AC_PROG_CC etc. become universally available.

This is a good point. Perhaps the wiki can serve as a stopgap
measure (it's certainly better than nothing) until these projects
release updates.

>   - What you have written about this environment so far indicates that
>     there are a lot of complexities; therefore it is likely that you
>     will have to tweak your recommendations regarding compiler options
>     and environment variables. (Just like I tweaked the recommended
>     options for building on HP-UX today. And HP-UX is _way_ simpler
>     than z/OS!)

I'll be happy to tweak what's there if I can get feedback from other
users. To date, I've been effectively working alone on this platform...


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - per-thread locale functions

[Bruno Haible]

On Freitag, 13. Dezember 2019 15:33:42 CET Daniel Richard G. wrote:
> On Fri, 2019 Dec 13 05:32-05:00, Bruno Haible wrote:
> > 
> > This should fix it. Thanks for the feedback.
> 
> Arrrgh, not quite there yet  >_<
> 
> Tested Git 83710ffa. test-locale now builds, but test-duplocale still
> does not, failing in the same place. HAVE_WORKING_DUPLOCALE appears to
> be 1 (true).

Oops. The patch below should fix that (hopefully).

> Isn't it possible to get a final result of HAVE_DUPLOCALE=0,
> REPLACE_DUPLOCALE=0,

We have that result already, as you showed through
'grep -i locale config.status'.


2019-12-14  Bruno Haible  <bruno@clisp.org>

	locale, duplocale, localename: Fix last patch.
	Reported by Daniel Richard G. in
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-12/msg00093.html>.
	* lib/locale.in.h (HAVE_WORKING_NEWLOCALE, HAVE_WORKING_DUPLOCALE):
	Don't define if locale_t does not exist.

diff --git -i a/lib/locale.in.h b/lib/locale.in.h
index c0915a5..9e897a3 100644
--- a/lib/locale.in.h
+++ b/lib/locale.in.h
@@ -227,9 +227,11 @@ _GL_CXXALIAS_SYS (newlocale, locale_t,
 # if @HAVE_NEWLOCALE@
 _GL_CXXALIASWARN (newlocale);
 # endif
+# if @HAVE_NEWLOCALE@ || @REPLACE_NEWLOCALE@
 #  ifndef HAVE_WORKING_NEWLOCALE
 #   define HAVE_WORKING_NEWLOCALE 1
 #  endif
+# endif
 #elif defined GNULIB_POSIXCHECK
 # undef newlocale
 # if HAVE_RAW_DECL_NEWLOCALE
@@ -254,9 +256,11 @@ _GL_CXXALIAS_SYS (duplocale, locale_t, (locale_t locale));
 # if @HAVE_DUPLOCALE@
 _GL_CXXALIASWARN (duplocale);
 # endif
+# if @HAVE_DUPLOCALE@ || @REPLACE_DUPLOCALE@
 #  ifndef HAVE_WORKING_DUPLOCALE
 #   define HAVE_WORKING_DUPLOCALE 1
 #  endif
+# endif
 #elif defined GNULIB_POSIXCHECK
 # undef duplocale
 # if HAVE_RAW_DECL_DUPLOCALE

Re: IBM z/OS compatibility issues - pthread

[Bruno Haible]

On Freitag, 13. Dezember 2019 16:10:21 CET Daniel Richard G. wrote:
> Hi Bruno,
> 
> On Fri, 2019 Dec 13 07:43-05:00, Bruno Haible wrote:
> >
> > > I don't see "PTHREAD_RWLOCK_INITIALIZER_NP" anywhere in a clean Gnulib
> > > tree, so I'm not sure how this could already be addressed...
> > 
> > The gnulib module 'pthread-rwlock' fixes this issue in a platform-independent
> > way (by overriding pthread_rwlock_t). That's the reason why you don't see
> > PTHREAD_RWLOCK_INITIALIZER_NP in the gnulib source code.
> 
> Something must be missing, because if I remove my conditional definition
> of PTHREAD_RWLOCK_INITIALIZER from pthread.in.h, test-pthread-rwlock is
> no longer in the set of tests built by default.

I don't understand how that can be, because none of the Autoconf tests
references PTHREAD_RWLOCK_INITIALIZER.

Can you please do a fresh start on this?
  - Create a testdir for module 'pthread-rwlock'.
  - Build it with -D_UNIX95_THREADS -D_XOPEN_SOURCE=600.

Bruno

Re: IBM z/OS compatibility issues

[Bruno Haible]

Hi Daniel,

> >   - Put the xlc-wrap logic into the 'compile' script. We should not
> >     have two different compile scripts, one for MSVC and one for z/OS.
> >     A single script should do it.
> 
> This sounds good. It would imply broadening the scope of the script
> beyond just working around the -c -o limitation, but then it was always
> a curiosity that that has been the *only* issue this script was intended
> to work around (until now).

In fact, the other issue this script works around are the invocation
conventions of the MSVC compiler (slash options instead of dash options).

> Also: I see that AC_USE_SYSTEM_EXTENSIONS() actually comes from
> Autoconf. Would that be a good place to put e.g. the -D_UNIX95_THREADS
> flag needed to get a proper pthreads API? I see that as similar in
> spirit to -D__EXTENSIONS__.

Yes, I agree.

> >   - Other than that, use documentation. I have added the most important
> >     information (from this mail thread) in
> >     <https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration>
> >     and would like to invite you to create a new wiki page
> >     <https://gitlab.com/ghwiki/gnow-how/-/wikis/Platforms/Configuration/zOS>.
> 
> I can add some information, but am concerned about discoverability. I
> myself was not aware of this wiki until you mentioned it to me

This wiki is very recent (just a month old). The more useful content we add,
the more known it will become. Gnulib did not become widely known within a
month either :-)

Bruno

Re: IBM z/OS compatibility issues - per-thread locale functions

[Daniel Richard G.]

On Sat, 2019 Dec 14 08:36-05:00, Bruno Haible wrote:
> On Freitag, 13. Dezember 2019 15:33:42 CET Daniel Richard G. wrote:
> > On Fri, 2019 Dec 13 05:32-05:00, Bruno Haible wrote:
> > > 
> > > This should fix it. Thanks for the feedback.
> > 
> > Arrrgh, not quite there yet  >_<
> > 
> > Tested Git 83710ffa. test-locale now builds, but test-duplocale 
> > still does not, failing in the same place. HAVE_WORKING_DUPLOCALE 
> > appears to be 1 (true).
> 
> Oops. The patch below should fix that (hopefully).

I'm happy to report that the build now completes without issue. The 
test-duplocale test exits with status 77, giving the message "Skipping 
test: function duplocale not available".

> > Isn't it possible to get a final result of HAVE_DUPLOCALE=0,
> > REPLACE_DUPLOCALE=0,
> 
> We have that result already, as you showed through
> 'grep -i locale config.status'.

config.h still has a

    #define HAVE_DUPLOCALE 1

line, however. This will be misleading for application code.


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - pthread

[Daniel Richard G.]

On Sat, 2019 Dec 14 08:42-05:00, Bruno Haible wrote:
>
> > Something must be missing, because if I remove my conditional 
> > definition of PTHREAD_RWLOCK_INITIALIZER from pthread.in.h, 
> > test-pthread-rwlock is no longer in the set of tests built by 
> > default.
> 
> I don't understand how that can be, because none of the Autoconf tests 
> references PTHREAD_RWLOCK_INITIALIZER.

Hm. I'm no longer able to reproduce this result, so I may be 
misremembering that detail. At present, with my definition removed, I 
get build failures for test-pthread and test-pthread-rwlock (using 
"make -k").

> Can you please do a fresh start on this?
>   - Create a testdir for module 'pthread-rwlock'.
>   - Build it with -D_UNIX95_THREADS -D_XOPEN_SOURCE=600.

That gives me the same compile error as a full build:

source='/tmp/testdir/gltests/test-pthread.c' object='test-pthread.o' libtool=no \
DEPDIR=.deps depmode=aix /bin/sh /tmp/testdir/build-aux/depcomp \
xlc-wrap -DHAVE_CONFIG_H -I. -I/tmp/testdir/gltests -I..  -DGNULIB_STRICT_CHECKING=1 -DIN_GNULIB_TESTS=1 -I. -I/tmp/testdir/gltests -I.. -I/tmp/testdir/gltests/.. -I../gllib -I/tmp/testdir/gltests/../gllib -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -q64 -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o test-pthread.o /tmp/testdir/gltests/test-pthread.c
ERROR CCN3221 /tmp/testdir/gltests/test-pthread.c:35    Initializer must be a valid constant expression.
WARNING CCN3196 /tmp/testdir/gltests/test-pthread.c:56    Initialization between types "void*" and "unsigned long" is not allowed.
CCN0793(I) Compilation failed for file /tmp/testdir/gltests/test-pthread.c.
  Object file not created.
make[4]: *** [test-pthread.o] Error 12
make[4]: Leaving directory `/tmp/gnulib-build/gltests'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/tmp/gnulib-build/gltests'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/tmp/gnulib-build/gltests'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/gnulib-build'
make: *** [all] Error 2

Whatever magic exists to work around a missing PTHREAD_RWLOCK_INITIALIZER,
I'm not seeing it...


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.

Re: IBM z/OS compatibility issues - per-thread locale functions

[Bruno Haible]

Hi Daniel,

> config.h still has a
> 
>     #define HAVE_DUPLOCALE 1
> 
> line, however. This will be misleading for application code.

Yes, there is a potential for application code to be misled. But
- Any application code can do AC_CHECK_FUNCS_ONCE([duplocale]),
  and this would conflict with gnulib if gnulib was to assign a
  different meaning to HAVE_DUPLOCALE than AC_CHECK_FUNCS gives it.
- We mitigate the problem through documentation.


2019-12-20  Bruno Haible  <bruno@clisp.org>

	doc: Document the problem of the per-thread locale functions on z/OS.
	* doc/posix-functions/uselocale.texi: Document the z/OS problem.
	* doc/posix-functions/newlocale.texi: Likewise.
	* doc/posix-functions/duplocale.texi: Likewise.
	* doc/posix-functions/freelocale.texi: Likewise.

diff --git a/doc/posix-functions/duplocale.texi b/doc/posix-functions/duplocale.texi
index 4b2ac3e..2fc9841 100644
--- a/doc/posix-functions/duplocale.texi
+++ b/doc/posix-functions/duplocale.texi
@@ -22,4 +22,8 @@ Portability problems not fixed by Gnulib:
 @item
 This function is missing on many platforms:
 FreeBSD 9.0, NetBSD 5.0, OpenBSD 6.1, Minix 3.1.8, AIX 6.1, HP-UX 11, IRIX 6.5, Solaris 11.3, Cygwin 2.5.x, mingw, MSVC 14, Android 4.4.
+@item
+This function is useless because the @code{locale_t} type is not defined
+on some platforms:
+z/OS.
 @end itemize
diff --git a/doc/posix-functions/freelocale.texi b/doc/posix-functions/freelocale.texi
index 04b9038..a3267dc 100644
--- a/doc/posix-functions/freelocale.texi
+++ b/doc/posix-functions/freelocale.texi
@@ -15,4 +15,8 @@ Portability problems not fixed by Gnulib:
 @item
 This function is missing on many platforms:
 FreeBSD 9.0, NetBSD 5.0, OpenBSD 6.1, Minix 3.1.8, AIX 6.1, HP-UX 11, IRIX 6.5, Solaris 11.3, Cygwin 2.5.x, mingw, MSVC 14, Android 4.4.
+@item
+This function is useless because the @code{locale_t} type is not defined
+on some platforms:
+z/OS.
 @end itemize
diff --git a/doc/posix-functions/newlocale.texi b/doc/posix-functions/newlocale.texi
index 8fc2d4f..f9b623b 100644
--- a/doc/posix-functions/newlocale.texi
+++ b/doc/posix-functions/newlocale.texi
@@ -16,6 +16,10 @@ Portability problems not fixed by Gnulib:
 This function is missing on many platforms:
 FreeBSD 9.0, NetBSD 5.0, OpenBSD 6.1, Minix 3.1.8, AIX 6.1, HP-UX 11, IRIX 6.5, Solaris 11.3, Cygwin 2.5.x, mingw, MSVC 14, Android 4.4.
 @item
+This function is useless because the @code{locale_t} type is not defined
+on some platforms:
+z/OS.
+@item
 This function is useless because the @code{locale_t} type contains basically
 no information on some platforms:
 OpenBSD 6.3.
diff --git a/doc/posix-functions/uselocale.texi b/doc/posix-functions/uselocale.texi
index 809055d..ad11dc7 100644
--- a/doc/posix-functions/uselocale.texi
+++ b/doc/posix-functions/uselocale.texi
@@ -20,6 +20,10 @@ This function is not documented and leads to crashes in subsequent
 @code{setlocale} invocations on some platforms:
 AIX 7.2.
 @item
+This function is useless because the @code{locale_t} type is not defined
+on some platforms:
+z/OS.
+@item
 This function is useless because the @code{locale_t} type contains basically
 no information on some platforms:
 OpenBSD 6.3.

Re: IBM z/OS compatibility issues - pthread

[Bruno Haible]

Hi Daniel,

> > Can you please do a fresh start on this?
> >   - Create a testdir for module 'pthread-rwlock'.
> >   - Build it with -D_UNIX95_THREADS -D_XOPEN_SOURCE=600.
> 
> That gives me the same compile error as a full build:
> 
> source='/tmp/testdir/gltests/test-pthread.c' object='test-pthread.o' libtool=no \
> DEPDIR=.deps depmode=aix /bin/sh /tmp/testdir/build-aux/depcomp \
> xlc-wrap -DHAVE_CONFIG_H -I. -I/tmp/testdir/gltests -I..  -DGNULIB_STRICT_CHECKING=1 -DIN_GNULIB_TESTS=1 -I. -I/tmp/testdir/gltests -I.. -I/tmp/testdir/gltests/.. -I../gllib -I/tmp/testdir/gltests/../gllib -D_UNIX95_THREADS -D_XOPEN_SOURCE=600 -DNSIG=39 -qhaltonmsg=CCN3296  -g -q64 -qfloat=ieee -qlanglvl=extc99 -qenumsize=4  -c -o test-pthread.o /tmp/testdir/gltests/test-pthread.c
> ERROR CCN3221 /tmp/testdir/gltests/test-pthread.c:35    Initializer must be a valid constant expression.
> WARNING CCN3196 /tmp/testdir/gltests/test-pthread.c:56    Initialization between types "void*" and "unsigned long" is not allowed.
> CCN0793(I) Compilation failed for file /tmp/testdir/gltests/test-pthread.c.
>   Object file not created.
> make[4]: *** [test-pthread.o] Error 12
> make[4]: Leaving directory `/tmp/gnulib-build/gltests'
> make[3]: *** [all-recursive] Error 1
> make[3]: Leaving directory `/tmp/gnulib-build/gltests'
> make[2]: *** [all] Error 2
> make[2]: Leaving directory `/tmp/gnulib-build/gltests'
> make[1]: *** [all-recursive] Error 1
> make[1]: Leaving directory `/tmp/gnulib-build'
> make: *** [all] Error 2
> 
> Whatever magic exists to work around a missing PTHREAD_RWLOCK_INITIALIZER,
> I'm not seeing it...

Oh, I see now. We do have a workaround for this situation in the 'lock'
module, but not in the 'pthread-rwlock' module.

Since PTHREAD_RWLOCK_INITIALIZER_NP is documented [1], we can use it, as you
suggest.

[1] https://www.ibm.com/support/knowledgecenter/cs/SSLTBW_2.2.0/com.ibm.zos.v2r2.bpxbd00/pthrdh.htm


2019-12-21  Bruno Haible  <bruno@clisp.org>

	pthread-thread, lock: On z/OS, use PTHREAD_RWLOCK_INITIALIZER_NP.
	Reported by Daniel Richard G. in
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-11/msg00001.html>
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-12/msg00167.html>
	* lib/pthread.in.h (PTHREAD_RWLOCK_INITIALIZER): Define to
	PTHREAD_RWLOCK_INITIALIZER_NP when possible.
	* lib/glthread/lock.h: Allow PTHREAD_RWLOCK_INITIALIZER_NP as an
	alternative to PTHREAD_RWLOCK_INITIALIZER.
	* lib/glthread/lock.c: Likewise.

diff --git a/lib/glthread/lock.c b/lib/glthread/lock.c
index 2537839..9d84090 100644
--- a/lib/glthread/lock.c
+++ b/lib/glthread/lock.c
@@ -254,7 +254,7 @@ glthread_recursive_lock_destroy (gl_recursive_lock_t *lock)
 
 # if HAVE_PTHREAD_RWLOCK && (HAVE_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER || (defined PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP && (__GNU_LIBRARY__ > 1)))
 
-#  ifdef PTHREAD_RWLOCK_INITIALIZER
+#  if defined PTHREAD_RWLOCK_INITIALIZER || defined PTHREAD_RWLOCK_INITIALIZER_NP
 
 #   if !HAVE_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER
      /* glibc with bug https://sourceware.org/bugzilla/show_bug.cgi?id=13701 */
diff --git a/lib/glthread/lock.h b/lib/glthread/lock.h
index ffbec15..890c459 100644
--- a/lib/glthread/lock.h
+++ b/lib/glthread/lock.h
@@ -308,7 +308,7 @@ typedef pthread_mutex_t gl_lock_t;
 
 # if HAVE_PTHREAD_RWLOCK && (HAVE_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER || (defined PTHREAD_RWLOCK_WRITER_NONRECURSIVE_INITIALIZER_NP && (__GNU_LIBRARY__ > 1)))
 
-#  ifdef PTHREAD_RWLOCK_INITIALIZER
+#  if defined PTHREAD_RWLOCK_INITIALIZER || defined PTHREAD_RWLOCK_INITIALIZER_NP
 
 typedef pthread_rwlock_t gl_rwlock_t;
 #   define gl_rwlock_define(STORAGECLASS, NAME) \
@@ -316,8 +316,13 @@ typedef pthread_rwlock_t gl_rwlock_t;
 #   define gl_rwlock_define_initialized(STORAGECLASS, NAME) \
       STORAGECLASS pthread_rwlock_t NAME = gl_rwlock_initializer;
 #   if HAVE_PTHREAD_RWLOCK_RDLOCK_PREFER_WRITER
-#    define gl_rwlock_initializer \
-       PTHREAD_RWLOCK_INITIALIZER
+#    if defined PTHREAD_RWLOCK_INITIALIZER
+#     define gl_rwlock_initializer \
+        PTHREAD_RWLOCK_INITIALIZER
+#    else
+#     define gl_rwlock_initializer \
+        PTHREAD_RWLOCK_INITIALIZER_NP
+#    endif
 #    define glthread_rwlock_init(LOCK) \
        (pthread_in_use () ? pthread_rwlock_init (LOCK, NULL) : 0)
 #   else /* glibc with bug https://sourceware.org/bugzilla/show_bug.cgi?id=13701 */
diff --git a/lib/pthread.in.h b/lib/pthread.in.h
index a6fb3d9..1baa938 100644
--- a/lib/pthread.in.h
+++ b/lib/pthread.in.h
@@ -281,7 +281,11 @@ typedef unsigned int pthread_rwlockattr_t;
 # define PTHREAD_RWLOCK_INITIALIZER \
    { PTHREAD_MUTEX_INITIALIZER, PTHREAD_COND_INITIALIZER, PTHREAD_COND_INITIALIZER, 0, 0 }
 #else
-# if !@HAVE_PTHREAD_T@
+# if @HAVE_PTHREAD_T@
+#  if !defined PTHREAD_RWLOCK_INITIALIZER && defined PTHREAD_RWLOCK_INITIALIZER_NP /* z/OS */
+#   define PTHREAD_RWLOCK_INITIALIZER PTHREAD_RWLOCK_INITIALIZER_NP
+#  endif
+# else
 #  if !GNULIB_defined_pthread_rwlock_types
 typedef int pthread_rwlock_t;
 typedef unsigned int pthread_rwlockattr_t;

Re: IBM z/OS compatibility issues

[Daniel Richard G.]

Apologies for the delay; holiday travel kept me occupied!

On Sat, 2019 Dec 14 08:51-05:00, Bruno Haible wrote:
>
> > This sounds good. It would imply broadening the scope of the script
> > beyond just working around the -c -o limitation, but then it was
> > always a curiosity that that has been the *only* issue this script
> > was intended to work around (until now).
>
> In fact, the other issue this script works around are the invocation
> conventions of the MSVC compiler (slash options instead of dash
> options).

I had previously glanced over lib/compile in the Automake Git repository
to refresh my memory. Someone needs to update the description at the top
of the file :]

> > Also: I see that AC_USE_SYSTEM_EXTENSIONS() actually comes from
> > Autoconf. Would that be a good place to put e.g. the
> > -D_UNIX95_THREADS flag needed to get a proper pthreads API? I see
> > that as similar in spirit to -D__EXTENSIONS__.
>
> Yes, I agree.

Ah, good. I'll forward this to them once I've caught up with my
correspondence.

> > I can add some information, but am concerned about discoverability.
> > I myself was not aware of this wiki until you mentioned it to me
>
> This wiki is very recent (just a month old). The more useful content
> we add, the more known it will become. Gnulib did not become widely
> known within a month either :-)

Understood; I'll update that when I have the opportunity.


--Daniel


-- 
Daniel Richard G. || skunk@iSKUNK.ORG
My ASCII-art .sig got a bad case of Times New Roman.