glob-h: Avoid conflict with preprocessor macros owned by the system

glob-h: Avoid conflict with preprocessor macros owned by the system

[Bruno Haible]

Building a testdir created with

  ./gnulib-tool --create-testdir --dir=../testdir --with-c++-tests \
                --without-privileged-tests --single-configure \
                dynarray glob scratch_buffer

on DragonFly BSD 6.0, I see this error:

In file included from ../../gltests/../gllib/libc-config.h:158,
                 from ../gllib/glob.h:581,
                 from ../../gltests/test-glob-h-c++.cc:22:
/usr/include/machine/endian.h:152:17: error: duplicate 'inline'
 static __inline __always_inline __uint64_t
                 ^~~~~~~~~~~~~~~
/usr/include/machine/endian.h:162:17: error: duplicate 'inline'
 static __inline __always_inline __uint32_t
                 ^~~~~~~~~~~~~~~
/usr/include/machine/endian.h:172:17: error: duplicate 'inline'
 static __inline __always_inline __uint16_t
                 ^~~~~~~~~~~~~~~

What happens here is a conflict between the definition of
'__always_inline' in gnulib's <cdefs.h> (included from
glob.h -> libc-config.h -> cdefs.h) and the definition of
'__always_inline' in DragonFly's <sys/cdefs.h>.

In C++ it is invalid to write
  static __inline __inline ... some_function (...) { ... }

The DragonFly headers are consistent: They define
  #define __always_inline __attribute__((__always_inline__))
and use it like this in <machine/endian.h>:
  static __inline __always_inline __uint64_t __bswap64 ...

Gnulib's <cdefs.h> definition of __always_inline is tailored for
glibc sources, which do
  static __always_inline bool ...
Therefore it defines it as
  #define __always_inline __inline __attribute__ ((__always_inline__))

But gnulib's <cdefs.h> *overrides* the system's definition. And it
is visible after any of these header files has been included:

  lib/dynarray.h
  lib/glob.in.h
  lib/scratch_buffer.h

After one of these files has been included, the application can
include any system include file, and these system include files will
typically rely on the system's definition of __* macros from <sys/cdefs.h>.

It's a different thing to include <cdefs.h> for the compilation of
a Gnulib .c file than to expose it through a .h file that the application
can include. In the first case, the Gnulib .c file includes a small set
of system's .h files, and therefore we can hope to have resolved the
possible conflicts in a reasonable amount of time. In the second case,
there is an uncountable number of conflicts; so, this problem will
haunt us for years if we don't fix it.

My fix here is to process the glob-libc.h file so that it does not
rely on __* macros (that belong to the system's namespace), only on
Gnulib macros, which we haven't seen conflict with system headers in
17 years.

When glob-libc.h changes (following merges from glibc), the processing
steps in modules/glob-h might need changes, but they will typically
be small and straightforward.


2021-06-06  Bruno Haible  <bruno@clisp.org>

	glob-h: Avoid conflict with preprocessor macros owned by the system.
	This fixes a compilation error on DragonFly BSD 6.0.
	* lib/glob.in.h: Don't include <libc-config.h>. Don't define __USE_GNU.
	Include glob-libc.gl.h instead of glob-libc.h.
	* modules/glob-h (Makefile.am): Arrange to create glob-libc.gl.h from
	glob-libc.h.
	* lib/libc-config.h: Add comment.

diff --git a/lib/glob.in.h b/lib/glob.in.h
index 4952666..dbc5b63 100644
--- a/lib/glob.in.h
+++ b/lib/glob.in.h
@@ -70,10 +70,6 @@ typedef int (*_gl_glob_errfunc_fn) (const char *, int);
 
 /* Preparations for including the standard GNU C Library header.  */
 
-# ifndef __attribute_maybe_unused__
-#  include <libc-config.h>
-# endif
-
 # include <stddef.h>
 
 /* On some systems, such as AIX 5.1, <sys/stat.h> does a "#define stat stat64".
@@ -81,10 +77,6 @@ typedef int (*_gl_glob_errfunc_fn) (const char *, int);
    rely on 'struct stat'.  */
 # include <sys/stat.h>
 
-# ifndef __USE_GNU
-#  define __USE_GNU    1
-# endif
-
 # if @REPLACE_GLOB@
 #  if !(defined __cplusplus && defined GNULIB_NAMESPACE)
 #   define glob rpl_glob
@@ -102,7 +94,7 @@ typedef int (*_gl_glob_errfunc_fn) (const char *, int);
 
 
 /* Now the standard GNU C Library header should work.  */
-# include "glob-libc.h"
+# include "glob-libc.gl.h"
 
 
 #endif
diff --git a/lib/libc-config.h b/lib/libc-config.h
index fabca2f..f68749f 100644
--- a/lib/libc-config.h
+++ b/lib/libc-config.h
@@ -28,7 +28,10 @@
 
    When compiled as part of glibc this is a no-op; when compiled as
    part of Gnulib this includes Gnulib's <config.h> and defines macros
-   that glibc library code would normally assume.  */
+   that glibc library code would normally assume.
+
+   Note: This header file MUST NOT be included by public header files
+   of Gnulib.  */
 
 #include <config.h>
 
diff --git a/modules/glob-h b/modules/glob-h
index 40df48f..0b71fd2 100644
--- a/modules/glob-h
+++ b/modules/glob-h
@@ -51,6 +51,23 @@ glob.h: $(top_builddir)/config.status
 endif
 MOSTLYCLEANFILES += glob.h glob.h-t
 
+BUILT_SOURCES += glob-libc.gl.h
+
+glob-libc.gl.h: glob-libc.h
+	$(AM_V_GEN)rm -f $@-t $@ && \
+	{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
+	  sed -e '/__BEGIN_DECLS/{ s/__BEGIN_DECLS/#ifdef __cplusplus%extern "C" {%#endif/; y/%/\n/; }' \
+	      -e '/__END_DECLS/{ s/__END_DECLS/#ifdef __cplusplus%}%#endif/; y/%/\n/; }' \
+	      -e 's|__THROW||g' \
+	      -e 's|defined __USE_MISC|1|' \
+	      -e 's|ifdef __USE_GNU|if 1|' \
+	      -e 's|ifdef __USE_LARGEFILE64|if 0|' \
+	      < $(srcdir)/glob-libc.h; \
+	} > $@-t && \
+	mv $@-t $@
+MOSTLYCLEANFILES += glob-libc.gl.h glob-libc.gl.h-t
+
+
 Include:
 <glob.h>
 

dynarray, scratch_buffer: Avoid conflict with preprocessor macros owned by the system

[Bruno Haible]

[-- Attachment #1: Type: text/plain, Size: 2051 bytes --]

> But gnulib's <cdefs.h> *overrides* the system's definition. And it
> is visible after any of these header files has been included:
> 
>   lib/dynarray.h
>   lib/glob.in.h
>   lib/scratch_buffer.h
> 
> After one of these files has been included, the application can
> include any system include file, and these system include files will
> typically rely on the system's definition of __* macros from <sys/cdefs.h>.
> 
> It's a different thing to include <cdefs.h> for the compilation of
> a Gnulib .c file than to expose it through a .h file that the application
> can include. In the first case, the Gnulib .c file includes a small set
> of system's .h files, and therefore we can hope to have resolved the
> possible conflicts in a reasonable amount of time. In the second case,
> there is an uncountable number of conflicts; so, this problem will
> haunt us for years if we don't fix it.
> 
> My fix here is to process the glob-libc.h file so that it does not
> rely on __* macros (that belong to the system's namespace)

And likewise for dynarray.h and scratch_buffer.h.


2021-06-06  Bruno Haible  <bruno@clisp.org>

	scratch_buffer: Avoid conflict with prepr. macros owned by the system.
	* lib/scratch_buffer.h: Don't include <libc-config.h>. Define
	_GL_LIKELY, _GL_UNLIKELY. Include malloc/scratch_buffer.gl.h instead of
	malloc/scratch_buffer.h.
	* modules/scratch_buffer (Depends-on): Add builtin-expect.
	(Makefile.am): Arrange to create malloc/scratch_buffer.gl.h from
	malloc/scratch_buffer.h.

2021-06-06  Bruno Haible  <bruno@clisp.org>

	dynarray: Avoid conflict with preprocessor macros owned by the system.
	* lib/dynarray.h: Don't include <libc-config.h>. Define _GL_LIKELY,
	_GL_UNLIKELY. Include malloc/dynarray.gl.h instead of malloc/dynarray.h.
	Include malloc/dynarray-skeleton.gl.h instead of
	malloc/dynarray-skeleton.c.
	* modules/dynarray (Depends-on): Add builtin-expect.
	(Makefile.am): Arrange to create malloc/dynarray.gl.h from
	malloc/dynarray.h and malloc/dynarray-skeleton.gl.h from
	malloc/dynarray-skeleton.c.


[-- Attachment #2: 0001-dynarray-Avoid-conflict-with-preprocessor-macros-own.patch --]
[-- Type: text/x-patch, Size: 4155 bytes --]

From 0c907f7da13232908f05c415b8cec56024071906 Mon Sep 17 00:00:00 2001
From: Bruno Haible <bruno@clisp.org>
Date: Mon, 7 Jun 2021 00:54:25 +0200
Subject: [PATCH 1/3] dynarray: Avoid conflict with preprocessor macros owned
 by the system.

* lib/dynarray.h: Don't include <libc-config.h>. Define _GL_LIKELY,
_GL_UNLIKELY. Include malloc/dynarray.gl.h instead of malloc/dynarray.h.
Include malloc/dynarray-skeleton.gl.h instead of
malloc/dynarray-skeleton.c.
* modules/dynarray (Depends-on): Add builtin-expect.
(Makefile.am): Arrange to create malloc/dynarray.gl.h from
malloc/dynarray.h and malloc/dynarray-skeleton.gl.h from
malloc/dynarray-skeleton.c.
---
 ChangeLog        | 12 ++++++++++++
 lib/dynarray.h   | 10 +++++++---
 modules/dynarray | 25 +++++++++++++++++++++++++
 3 files changed, 44 insertions(+), 3 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 79fc6ff..b354ff0 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,17 @@
 2021-06-06  Bruno Haible  <bruno@clisp.org>
 
+	dynarray: Avoid conflict with preprocessor macros owned by the system.
+	* lib/dynarray.h: Don't include <libc-config.h>. Define _GL_LIKELY,
+	_GL_UNLIKELY. Include malloc/dynarray.gl.h instead of malloc/dynarray.h.
+	Include malloc/dynarray-skeleton.gl.h instead of
+	malloc/dynarray-skeleton.c.
+	* modules/dynarray (Depends-on): Add builtin-expect.
+	(Makefile.am): Arrange to create malloc/dynarray.gl.h from
+	malloc/dynarray.h and malloc/dynarray-skeleton.gl.h from
+	malloc/dynarray-skeleton.c.
+
+2021-06-06  Bruno Haible  <bruno@clisp.org>
+
 	glob-h: Avoid conflict with preprocessor macros owned by the system.
 	This fixes a compilation error on DragonFly BSD 6.0.
 	* lib/glob.in.h: Don't include <libc-config.h>. Don't define __USE_GNU.
diff --git a/lib/dynarray.h b/lib/dynarray.h
index 375d3da..ec64273 100644
--- a/lib/dynarray.h
+++ b/lib/dynarray.h
@@ -257,18 +257,22 @@ static DYNARRAY_ELEMENT *
 
 #if defined DYNARRAY_STRUCT || defined DYNARRAY_ELEMENT || defined DYNARRAY_PREFIX
 
-# include <libc-config.h>
+# ifndef _GL_LIKELY
+/* Rely on __builtin_expect, as provided by the module 'builtin-expect'.  */
+#  define _GL_LIKELY(cond) __builtin_expect ((cond), 1)
+#  define _GL_UNLIKELY(cond) __builtin_expect ((cond), 0)
+# endif
 
 /* Define auxiliary structs and declare auxiliary functions, common to all
    instantiations of dynarray.  */
-# include <malloc/dynarray.h>
+# include <malloc/dynarray.gl.h>
 
 /* Define the instantiation, specified through
      DYNARRAY_STRUCT
      DYNARRAY_ELEMENT
      DYNARRAY_PREFIX
    etc.  */
-# include <malloc/dynarray-skeleton.c>
+# include <malloc/dynarray-skeleton.gl.h>
 
 #else
 
diff --git a/modules/dynarray b/modules/dynarray
index dcdcba4..7fd444f 100644
--- a/modules/dynarray
+++ b/modules/dynarray
@@ -13,6 +13,7 @@ lib/malloc/dynarray_resize_clear.c
 
 Depends-on:
 c99
+builtin-expect
 libc-config
 stdbool
 stddef
@@ -21,6 +22,30 @@ intprops
 configure.ac:
 
 Makefile.am:
+BUILT_SOURCES += malloc/dynarray.gl.h malloc/dynarray-skeleton.gl.h
+
+malloc/dynarray.gl.h: malloc/dynarray.h
+	$(AM_V_GEN)rm -f $@-t $@ && \
+	{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
+	  sed -e '/libc_hidden_proto/d' < $(srcdir)/malloc/dynarray.h; \
+	} > $@-t && \
+	mv $@-t $@
+MOSTLYCLEANFILES += malloc/dynarray.gl.h malloc/dynarray.gl.h-t
+
+malloc/dynarray-skeleton.gl.h: malloc/dynarray-skeleton.c
+	$(AM_V_GEN)rm -f $@-t $@ && \
+	{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
+	  sed -e 's|<malloc/dynarray\.h>|<malloc/dynarray.gl.h>|g' \
+	      -e 's|__attribute_maybe_unused__|_GL_ATTRIBUTE_MAYBE_UNUSED|g' \
+	      -e 's|__attribute_nonnull__|_GL_ATTRIBUTE_NONNULL|g' \
+	      -e 's|__attribute_warn_unused_result__|_GL_ATTRIBUTE_NODISCARD|g' \
+	      -e 's|__glibc_likely|_GL_LIKELY|g' \
+	      -e 's|__glibc_unlikely|_GL_UNLIKELY|g' \
+	      < $(srcdir)/malloc/dynarray-skeleton.c; \
+	} > $@-t && \
+	mv $@-t $@
+MOSTLYCLEANFILES += malloc/dynarray-skeleton.gl.h malloc/dynarray-skeleton.gl.h-t
+
 lib_SOURCES += malloc/dynarray_at_failure.c \
                malloc/dynarray_emplace_enlarge.c \
                malloc/dynarray_finalize.c \
-- 
2.7.4


[-- Attachment #3: 0002-scratch_buffer-Avoid-conflict-with-prepr.-macros-own.patch --]
[-- Type: text/x-patch, Size: 3597 bytes --]

From 469f9f3685c4026e79ccc8f3fb35c3eb25fd3716 Mon Sep 17 00:00:00 2001
From: Bruno Haible <bruno@clisp.org>
Date: Mon, 7 Jun 2021 00:56:50 +0200
Subject: [PATCH 2/3] scratch_buffer: Avoid conflict with prepr. macros owned
 by the system.

* lib/scratch_buffer.h: Don't include <libc-config.h>. Define
_GL_LIKELY, _GL_UNLIKELY. Include malloc/scratch_buffer.gl.h instead of
malloc/scratch_buffer.h.
* modules/scratch_buffer (Depends-on): Add builtin-expect.
(Makefile.am): Arrange to create malloc/scratch_buffer.gl.h from
malloc/scratch_buffer.h.
---
 ChangeLog              | 10 ++++++++++
 lib/scratch_buffer.h   | 10 +++++++---
 modules/scratch_buffer | 15 +++++++++++++++
 3 files changed, 32 insertions(+), 3 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index b354ff0..16132b1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,15 @@
 2021-06-06  Bruno Haible  <bruno@clisp.org>
 
+	scratch_buffer: Avoid conflict with prepr. macros owned by the system.
+	* lib/scratch_buffer.h: Don't include <libc-config.h>. Define
+	_GL_LIKELY, _GL_UNLIKELY. Include malloc/scratch_buffer.gl.h instead of
+	malloc/scratch_buffer.h.
+	* modules/scratch_buffer (Depends-on): Add builtin-expect.
+	(Makefile.am): Arrange to create malloc/scratch_buffer.gl.h from
+	malloc/scratch_buffer.h.
+
+2021-06-06  Bruno Haible  <bruno@clisp.org>
+
 	dynarray: Avoid conflict with preprocessor macros owned by the system.
 	* lib/dynarray.h: Don't include <libc-config.h>. Define _GL_LIKELY,
 	_GL_UNLIKELY. Include malloc/dynarray.gl.h instead of malloc/dynarray.h.
diff --git a/lib/scratch_buffer.h b/lib/scratch_buffer.h
index 7bb6fe8..8873577 100644
--- a/lib/scratch_buffer.h
+++ b/lib/scratch_buffer.h
@@ -110,14 +110,18 @@ extern void *scratch_buffer_dupfree (struct scratch_buffer *buffer,
 
 /* The implementation is imported from glibc.  */
 
-#include <libc-config.h>
-
 /* Avoid possible conflicts with symbols exported by the GNU libc.  */
 #define __libc_scratch_buffer_dupfree gl_scratch_buffer_dupfree
 #define __libc_scratch_buffer_grow gl_scratch_buffer_grow
 #define __libc_scratch_buffer_grow_preserve gl_scratch_buffer_grow_preserve
 #define __libc_scratch_buffer_set_array_size gl_scratch_buffer_set_array_size
 
-#include <malloc/scratch_buffer.h>
+#ifndef _GL_LIKELY
+/* Rely on __builtin_expect, as provided by the module 'builtin-expect'.  */
+# define _GL_LIKELY(cond) __builtin_expect ((cond), 1)
+# define _GL_UNLIKELY(cond) __builtin_expect ((cond), 0)
+#endif
+
+#include <malloc/scratch_buffer.gl.h>
 
 #endif /* _GL_SCRATCH_BUFFER_H */
diff --git a/modules/scratch_buffer b/modules/scratch_buffer
index cf83ab5..ede77a9 100644
--- a/modules/scratch_buffer
+++ b/modules/scratch_buffer
@@ -11,6 +11,7 @@ lib/malloc/scratch_buffer_set_array_size.c
 
 Depends-on:
 c99
+builtin-expect
 libc-config
 stdbool
 stddef
@@ -21,6 +22,20 @@ free-posix
 configure.ac:
 
 Makefile.am:
+BUILT_SOURCES += malloc/scratch_buffer.gl.h
+
+malloc/scratch_buffer.gl.h: malloc/scratch_buffer.h
+	$(AM_V_GEN)rm -f $@-t $@ && \
+	{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
+	  sed -e 's|__always_inline|inline _GL_ATTRIBUTE_ALWAYS_INLINE|g' \
+	      -e 's|__glibc_likely|_GL_LIKELY|g' \
+	      -e 's|__glibc_unlikely|_GL_UNLIKELY|g' \
+	      -e '/libc_hidden_proto/d' \
+	      < $(srcdir)/malloc/scratch_buffer.h; \
+	} > $@-t && \
+	mv $@-t $@
+MOSTLYCLEANFILES += malloc/scratch_buffer.gl.h malloc/scratch_buffer.gl.h-t
+
 lib_SOURCES += malloc/scratch_buffer_dupfree.c \
                malloc/scratch_buffer_grow.c \
                malloc/scratch_buffer_grow_preserve.c \
-- 
2.7.4

Re: dynarray, scratch_buffer: Avoid conflict with preprocessor macros owned by the system

[Bruno Haible]

Oops, I forgot to 'mkdir -p malloc' in the build directory. In some
environments, the Makefile's 'dirstamp' handling does it automatically;
in others, it doesn't.


2021-06-07  Bruno Haible  <bruno@clisp.org>

	dynarray, scratch_buffer: Fix VPATH builds (regression from yesterday).
	* modules/dynarray (configure.ac): Invoke AC_PROG_MKDIR_P.
	(Makefile.am): Create malloc/ directory before creating
	malloc/dynarray.gl.h and malloc/dynarray-skeleton.gl.h.
	* modules/scratch_buffer (configure.ac): Invoke AC_PROG_MKDIR_P.
	(Makefile.am): Create malloc/ directory before creating
	malloc/scratch_buffer.gl.h.

diff --git a/modules/dynarray b/modules/dynarray
index 7fd444f..fa608de 100644
--- a/modules/dynarray
+++ b/modules/dynarray
@@ -20,11 +20,13 @@ stddef
 intprops
 
 configure.ac:
+AC_PROG_MKDIR_P
 
 Makefile.am:
 BUILT_SOURCES += malloc/dynarray.gl.h malloc/dynarray-skeleton.gl.h
 
 malloc/dynarray.gl.h: malloc/dynarray.h
+	$(AM_V_at)$(MKDIR_P) malloc
 	$(AM_V_GEN)rm -f $@-t $@ && \
 	{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
 	  sed -e '/libc_hidden_proto/d' < $(srcdir)/malloc/dynarray.h; \
@@ -33,6 +35,7 @@ malloc/dynarray.gl.h: malloc/dynarray.h
 MOSTLYCLEANFILES += malloc/dynarray.gl.h malloc/dynarray.gl.h-t
 
 malloc/dynarray-skeleton.gl.h: malloc/dynarray-skeleton.c
+	$(AM_V_at)$(MKDIR_P) malloc
 	$(AM_V_GEN)rm -f $@-t $@ && \
 	{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
 	  sed -e 's|<malloc/dynarray\.h>|<malloc/dynarray.gl.h>|g' \
diff --git a/modules/scratch_buffer b/modules/scratch_buffer
index ede77a9..184f0b6 100644
--- a/modules/scratch_buffer
+++ b/modules/scratch_buffer
@@ -20,11 +20,13 @@ realloc-posix
 free-posix
 
 configure.ac:
+AC_PROG_MKDIR_P
 
 Makefile.am:
 BUILT_SOURCES += malloc/scratch_buffer.gl.h
 
 malloc/scratch_buffer.gl.h: malloc/scratch_buffer.h
+	$(AM_V_at)$(MKDIR_P) malloc
 	$(AM_V_GEN)rm -f $@-t $@ && \
 	{ echo '/* DO NOT EDIT! GENERATED AUTOMATICALLY! */'; \
 	  sed -e 's|__always_inline|inline _GL_ATTRIBUTE_ALWAYS_INLINE|g' \