bug-gnulib@gnu.org mirror (unofficial)
 help / color / mirror / Atom feed
* ptrdiff_t overflow checks for malloc-posix etc.
@ 2021-04-18  2:02 Paul Eggert
  2021-04-18 11:59 ` Bruno Haible
                   ` (4 more replies)
  0 siblings, 5 replies; 17+ messages in thread
From: Paul Eggert @ 2021-04-18  2:02 UTC (permalink / raw)
  To: Gnulib bugs

[-- Attachment #1: Type: text/plain, Size: 1680 bytes --]

I installed the attached patches into Gnulib to make its malloc 
replacements ptrdiff_t safe. This should help us move in a direction 
where we can use idx_t (which is signed and therefore safer) for sizes 
and indexes, instead of using size_t.

In creating these patches I found a reasonable amount of cruft of which 
I tried to clean up some (see ChangeLog entry). If I went too far please 
let me know and I’ll work to unclean it.

I initially attempted to come up with new modules malloc-ptrdiff_t, etc. 
but ran into complexity issues with all the possible combinations the 
various malloc modules. So instead, I simply added the fixes to 
malloc-posix, realloc-posix, and realloc-posix, where they will 
automatically percolate into malloc-gnu etc.

Come to think of it, why do we have both malloc-gnu and malloc-posix 
modules (and similarly for calloc and realloc)?  Was it because GNU 
realloc was incompatible with C99 realloc, so we needed realloc-gnu vs 
realloc-posix modules?  If so, I suggest that we stop worrying about it, 
as that worry is now obsolete - C17 allows the GNU behavior.

In other words, I suggest that we remove malloc-posix, realloc-posix and 
calloc-posix, or failing that simply make them obsolete compatibility 
aliases for malloc-gnu etc. This would simplify the configuration of 
malloc-using code, and any runtime cost would surely be insignificant 
(and would occur only on older or non-GNU hosts).

The first attached patch does the heavy lifting; the second shows how 
the xalloc module can be simplified because of the malloc etc. fixes. 
Other simplifications are possible elsewhere; one step at a time.

[-- Attachment #2: 0001-malloc-etc.-check-for-ptrdiff_t-overflow.patch --]
[-- Type: text/x-patch, Size: 31145 bytes --]

From 1ca8ecafc9066c9c0fb4b1030088969a11dce3af Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 17 Apr 2021 18:44:25 -0700
Subject: [PATCH 1/2] malloc, etc.: check for ptrdiff_t overflow
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

In glibc 2.30 and later, malloc, realloc and calloc reject
attempts to create objects larger than PTRDIFF_MAX bytes.
This patch changes malloc-gnu etc. to support this behavior
on non-GNU hosts.  It also makes this change for malloc-posix etc.
since it’s a safety measure that ought to be in POSIX (perhaps
we can talk them into that...).

In writing this patch I found a complicated set of code that had
accumulated over the years, some written by yours truly.  I got
rid of the code I couldn’t see the need for nowadays.  Among other
things, the GNU realloc behavior is no longer incompatible with
the C standard, because in C17 the latter was relaxed to allow the
former.  If I went too far in cleaning up, the old stuff can be
resurrected.

This change is mostly for 32-bit platforms, since practical 64-bit
platforms cannot create objects larger than PTRDIFF_MAX bytes anyway.
* doc/posix-functions/calloc.texi:
* doc/posix-functions/malloc.texi:
* doc/posix-functions/realloc.texi:
Mention ptrdiff_t issues, and go into more detail about what
the gnu extension module does.
* doc/posix-functions/realloc.texi: Fix now-obsolete commentary
about C99 vs glibc, as C17 allows the glibc behavior and POSIX
will follow suit when it gets around to it.
* lib/calloc.c, lib/malloc.c, lib/realloc.c:
Simplify by always supplying a GNU-compatible version,
as that suffices for correctness and is good enough for performance.
Include xalloc-oversized.h, and use xalloc_oversized to
check for ptrdiff_t overflow.
(NEED_CALLOC_GNU, NEED_MALLOC_GNU, NEED_REALLOC_GNU): Remove.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF):
* m4/malloc.m4 (_AC_FUNC_MALLOC_IF):
* m4/realloc.m4 (_AC_FUNC_REALLOC_IF):
Don’t start with a newline.  Fix message to match behavior.
* m4/calloc.m4 (_AC_FUNC_CALLOC_IF): Don’t test for size_t overflow,
as the ptrdiff_t test is good enough.
* m4/calloc.m4 (gl_FUNC_CALLOC_GNU):
* m4/malloc.m4 (gl_FUNC_MALLOC_GNU):
* m4/realloc.m4 (gl_FUNC_REALLOC_GNU):
Do not define HAVE_CALLOC_GNU, HAVE_MALLOC_GNU, HAVE_REALLOC_GNU.
It’s not worth the aggravation of maintaining these, as they
are confusing (they don’t really mean GNU-compatible anyway).
Don’t bother testing for GNU behavior if we have already decided
to replace the function, since the replacement is always GNUish.
* m4/calloc.m4 (gl_FUNC_CALLOC_POSIX):
* m4/realloc.m4 (gl_FUNC_REALLOC_POSIX):
Defer to gl_FUNC_MALLOC_POSIX.
* m4/malloc.m4 (gl_FUNC_MALLOC_PTRDIFF, gl_CHECK_MALLOC_PTRDIFF):
New macros.
(gl_FUNC_MALLOC_POSIX): Use them to check for ptrdiff_t overflow.
* modules/calloc-gnu, modules/malloc-gnu, modules/realloc-gnu:
Remove no-longer-needed module indicators.
* modules/calloc-posix, modules/malloc-posix, modules/realloc-posix:
Depend on xalloc-oversized.
* modules/malloc-posix: Require gl_FUNC_MALLOC_POSIX instead of
calling it directly, so that other code can require it.
* modules/realloc-posix: Depend on free-posix and malloc-posix.
---
 ChangeLog                        | 62 +++++++++++++++++++++++++
 doc/posix-functions/calloc.texi  | 17 +++++--
 doc/posix-functions/malloc.texi  | 17 ++++---
 doc/posix-functions/realloc.texi | 37 ++++++++++-----
 lib/calloc.c                     | 37 +++++----------
 lib/malloc.c                     | 32 +++++--------
 lib/realloc.c                    | 55 +++++++---------------
 m4/calloc.m4                     | 67 +++++++--------------------
 m4/malloc.m4                     | 79 ++++++++++++++++++++++++--------
 m4/realloc.m4                    | 32 ++++---------
 modules/calloc-gnu               |  1 -
 modules/calloc-posix             |  1 +
 modules/malloc-gnu               |  1 -
 modules/malloc-posix             |  4 +-
 modules/realloc-gnu              |  1 -
 modules/realloc-posix            |  4 +-
 16 files changed, 243 insertions(+), 204 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 1e4c95524..825201fe2 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,65 @@
+2021-04-17  Paul Eggert  <eggert@cs.ucla.edu>
+
+	malloc, etc.: check for ptrdiff_t overflow
+	In glibc 2.30 and later, malloc, realloc and calloc reject
+	attempts to create objects larger than PTRDIFF_MAX bytes.
+	This patch changes malloc-gnu etc. to support this behavior
+	on non-GNU hosts.  It also makes this change for malloc-posix etc.
+	since it’s a safety measure that ought to be in POSIX (perhaps
+	we can talk them into that...).
+
+	In writing this patch I found a complicated set of code that had
+	accumulated over the years, some written by yours truly.  I got
+	rid of the code I couldn’t see the need for nowadays.  Among other
+	things, the GNU realloc behavior is no longer incompatible with
+	the C standard, because in C17 the latter was relaxed to allow the
+	former.  If I went too far in cleaning up, the old stuff can be
+	resurrected.
+
+	This change is mostly for 32-bit platforms, since practical 64-bit
+	platforms cannot create objects larger than PTRDIFF_MAX bytes anyway.
+	* doc/posix-functions/calloc.texi:
+	* doc/posix-functions/malloc.texi:
+	* doc/posix-functions/realloc.texi:
+	Mention ptrdiff_t issues, and go into more detail about what
+	the gnu extension module does.
+	* doc/posix-functions/realloc.texi: Fix now-obsolete commentary
+	about C99 vs glibc, as C17 allows the glibc behavior and POSIX
+	will follow suit when it gets around to it.
+	* lib/calloc.c, lib/malloc.c, lib/realloc.c:
+	Simplify by always supplying a GNU-compatible version,
+	as that suffices for correctness and is good enough for performance.
+	Include xalloc-oversized.h, and use xalloc_oversized to
+	check for ptrdiff_t overflow.
+	(NEED_CALLOC_GNU, NEED_MALLOC_GNU, NEED_REALLOC_GNU): Remove.
+	* m4/calloc.m4 (_AC_FUNC_CALLOC_IF):
+	* m4/malloc.m4 (_AC_FUNC_MALLOC_IF):
+	* m4/realloc.m4 (_AC_FUNC_REALLOC_IF):
+	Don’t start with a newline.  Fix message to match behavior.
+	* m4/calloc.m4 (_AC_FUNC_CALLOC_IF): Don’t test for size_t overflow,
+	as the ptrdiff_t test is good enough.
+	* m4/calloc.m4 (gl_FUNC_CALLOC_GNU):
+	* m4/malloc.m4 (gl_FUNC_MALLOC_GNU):
+	* m4/realloc.m4 (gl_FUNC_REALLOC_GNU):
+	Do not define HAVE_CALLOC_GNU, HAVE_MALLOC_GNU, HAVE_REALLOC_GNU.
+	It’s not worth the aggravation of maintaining these, as they
+	are confusing (they don’t really mean GNU-compatible anyway).
+	Don’t bother testing for GNU behavior if we have already decided
+	to replace the function, since the replacement is always GNUish.
+	* m4/calloc.m4 (gl_FUNC_CALLOC_POSIX):
+	* m4/realloc.m4 (gl_FUNC_REALLOC_POSIX):
+	Defer to gl_FUNC_MALLOC_POSIX.
+	* m4/malloc.m4 (gl_FUNC_MALLOC_PTRDIFF, gl_CHECK_MALLOC_PTRDIFF):
+	New macros.
+	(gl_FUNC_MALLOC_POSIX): Use them to check for ptrdiff_t overflow.
+	* modules/calloc-gnu, modules/malloc-gnu, modules/realloc-gnu:
+	Remove no-longer-needed module indicators.
+	* modules/calloc-posix, modules/malloc-posix, modules/realloc-posix:
+	Depend on xalloc-oversized.
+	* modules/malloc-posix: Require gl_FUNC_MALLOC_POSIX instead of
+	calling it directly, so that other code can require it.
+	* modules/realloc-posix: Depend on free-posix and malloc-posix.
+
 2021-04-17  Bruno Haible  <bruno@clisp.org>
 
 	stdio: Fix build error in some configurations (regression 2021-04-11).
diff --git a/doc/posix-functions/calloc.texi b/doc/posix-functions/calloc.texi
index 22c51be02..9ba40c0d4 100644
--- a/doc/posix-functions/calloc.texi
+++ b/doc/posix-functions/calloc.texi
@@ -12,11 +12,22 @@ Portability problems fixed by Gnulib:
 Upon failure, the function does not set @code{errno} to @code{ENOMEM} on
 some platforms:
 mingw, MSVC 14.
-@end itemize
 
-Portability problems not fixed by Gnulib:
-@itemize
+@item
+On some platforms, @code{calloc (n, s)} can succeed even if
+multiplying @code{n} by @code{s} would exceed @code{PTRDIFF_MAX} or
+@code{SIZE_MAX}.  Although failing to check for exceeding
+@code{PTRDIFF_MAX} is arguably allowed by POSIX it can lead to
+undefined behavior later, so @code{calloc-posix} does not allow
+going over the limit.
 @end itemize
 
 Extension: Gnulib provides a module @samp{calloc-gnu} that substitutes a
 @code{calloc} implementation that behaves more like the glibc implementation.
+It fixes this portability problem:
+
+@itemize
+@item
+On some platforms, @code{calloc (0, s)} and @code{calloc (n, 0)}
+return @code{NULL} on success.
+@end itemize
diff --git a/doc/posix-functions/malloc.texi b/doc/posix-functions/malloc.texi
index 6d5995078..829517311 100644
--- a/doc/posix-functions/malloc.texi
+++ b/doc/posix-functions/malloc.texi
@@ -12,14 +12,19 @@ Portability problems fixed by Gnulib:
 Upon failure, the function does not set @code{errno} to @code{ENOMEM} on
 some platforms:
 mingw, MSVC 14.
-@end itemize
 
-Portability problems not fixed by Gnulib:
-@itemize
-@item @code{malloc (0)} always returns a NULL pointer on some platforms:
-AIX 5.1.
+@item
+On some platforms, @code{malloc (n)} can succeed even if @code{n}
+exceeds @code{PTRDIFF_MAX}.  Although this behavior is arguably
+allowed by POSIX it can lead to behavior not defined by POSIX later,
+so @code{malloc-posix} does not allow going over the limit.
 @end itemize
 
 Extension: Gnulib provides a module @samp{malloc-gnu} that substitutes a
 @code{malloc} implementation that behaves more like the glibc implementation,
-regarding the result of @code{malloc (0)}.
+by fixing this portability problem:
+
+@itemize
+@item
+On some platforms, @code{malloc (0)} returns @code{NULL} on success.
+@end itemize
diff --git a/doc/posix-functions/realloc.texi b/doc/posix-functions/realloc.texi
index 1f0b18e95..282e36051 100644
--- a/doc/posix-functions/realloc.texi
+++ b/doc/posix-functions/realloc.texi
@@ -12,24 +12,37 @@ Portability problems fixed by Gnulib:
 Upon failure, the function does not set @code{errno} to @code{ENOMEM} on
 some platforms:
 mingw, MSVC 14.
-@end itemize
 
-Portability problems not fixed by Gnulib:
-@itemize
 @item
-It is not portable to call @code{realloc} with a size of 0.  With a
+On some platforms, @code{realloc (p, n)} can succeed even if @code{n}
+exceeds @code{PTRDIFF_MAX}.  Although this behavior is arguably
+allowed by POSIX it can lead to behavior not defined by POSIX later,
+so @code{realloc-posix} does not allow going over the limit.
+@end itemize
+
+Without the @samp{realloc-gnu} module described below, it is not portable
+to call @code{realloc} with a size of 0.  With a
 NULL pointer argument, this is the same ambiguity as @code{malloc (0)}
 on whether a unique zero-size object is created.  With a non-NULL
-pointer argument, C99 requires that if @code{realloc (p, 0)} returns
-@code{NULL} then @code{p} is still valid.  Among implementations that
-obey C99, behavior varies on whether @code{realloc (p, 0)} always
+pointer argument @code{p}, C17 says that it is implementation-defined
+whether @code{realloc (p, 0)} frees @code{p}.
+Behavior varies on whether @code{realloc (p, 0)} always frees @code{p}
+and successfully returns a null pointer, or always
 fails and leaves @code{p} valid, or usually succeeds and returns a
-unique zero-size object; either way, a program not suspecting these
+unique zero-size object; a program not suspecting these variations in
 semantics will leak memory (either the still-valid @code{p}, or the
-non-NULL return value).  Meanwhile, several implementations violate
-C99, by always calling @code{free (p)} but returning NULL:
-glibc, Cygwin
-@end itemize
+non-NULL return value).
 
 Extension: Gnulib provides a module @samp{realloc-gnu} that substitutes a
 @code{realloc} implementation that behaves more like the glibc implementation.
+It fixes these portability problems:
+
+@itemize
+@item
+On some platforms, @code{realloc (NULL, 0)} returns @code{NULL} on success.
+
+@item
+On some platforms, @code{realloc (p, 0)} with non-null @code{p}
+might not free @code{p}, or might clobber @code{errno},
+or might not return @code{NULL}.
+@end itemize
diff --git a/lib/calloc.c b/lib/calloc.c
index 5ab1975cf..c9da08047 100644
--- a/lib/calloc.c
+++ b/lib/calloc.c
@@ -19,49 +19,34 @@
 
 #include <config.h>
 
-/* The gnulib module 'calloc-gnu' defines HAVE_CALLOC_GNU.  */
-#if GNULIB_CALLOC_GNU && !HAVE_CALLOC_GNU
-# define NEED_CALLOC_GNU 1
-#endif
-
 /* Specification.  */
 #include <stdlib.h>
 
 #include <errno.h>
 
+#include "xalloc-oversized.h"
+
 /* Call the system's calloc below.  */
 #undef calloc
 
-/* Allocate and zero-fill an NxS-byte block of memory from the heap.
-   If N or S is zero, allocate and zero-fill a 1-byte block.  */
+/* Allocate and zero-fill an NxS-byte block of memory from the heap,
+   even if N or S is zero.  */
 
 void *
 rpl_calloc (size_t n, size_t s)
 {
-  void *result;
-
-#if NEED_CALLOC_GNU
   if (n == 0 || s == 0)
+    n = s = 1;
+
+  if (xalloc_oversized (n, s))
     {
-      n = 1;
-      s = 1;
-    }
-  else
-    {
-      /* Defend against buggy calloc implementations that mishandle
-         size_t overflow.  */
-      size_t bytes = n * s;
-      if (bytes / s != n)
-        {
-          errno = ENOMEM;
-          return NULL;
-        }
+      errno = ENOMEM;
+      return NULL;
     }
-#endif
 
-  result = calloc (n, s);
+  void *result = calloc (n, s);
 
-#if !HAVE_CALLOC_POSIX
+#if !HAVE_MALLOC_POSIX
   if (result == NULL)
     errno = ENOMEM;
 #endif
diff --git a/lib/malloc.c b/lib/malloc.c
index 272b8b9f4..a900046ec 100644
--- a/lib/malloc.c
+++ b/lib/malloc.c
@@ -20,43 +20,35 @@
 #define _GL_USE_STDLIB_ALLOC 1
 #include <config.h>
 
-/* The gnulib module 'malloc-gnu' defines HAVE_MALLOC_GNU.  */
-#if GNULIB_MALLOC_GNU && !HAVE_MALLOC_GNU
-# define NEED_MALLOC_GNU 1
-#endif
-
 #include <stdlib.h>
 
-/* A function definition is only needed if NEED_MALLOC_GNU is defined above
-   or if the module 'malloc-posix' requests it.  */
-#if NEED_MALLOC_GNU || (GNULIB_MALLOC_POSIX && !HAVE_MALLOC_POSIX)
+#include <errno.h>
 
-# include <errno.h>
+#include "xalloc-oversized.h"
 
 /* Call the system's malloc below.  */
 # undef malloc
 
-/* Allocate an N-byte block of memory from the heap.
-   If N is zero, allocate a 1-byte block.  */
+/* Allocate an N-byte block of memory from the heap, even if N is 0.  */
 
 void *
 rpl_malloc (size_t n)
 {
-  void *result;
-
-# if NEED_MALLOC_GNU
   if (n == 0)
     n = 1;
-# endif
 
-  result = malloc (n);
+  if (xalloc_oversized (n, 1))
+    {
+      errno = ENOMEM;
+      return NULL;
+    }
+
+  void *result = malloc (n);
 
-# if !HAVE_MALLOC_POSIX
+#if !HAVE_MALLOC_POSIX
   if (result == NULL)
     errno = ENOMEM;
-# endif
+#endif
 
   return result;
 }
-
-#endif
diff --git a/lib/realloc.c b/lib/realloc.c
index c3e3cdfc5..c0d94b439 100644
--- a/lib/realloc.c
+++ b/lib/realloc.c
@@ -21,66 +21,43 @@
 #define _GL_USE_STDLIB_ALLOC 1
 #include <config.h>
 
-/* The gnulib module 'realloc-gnu' defines HAVE_REALLOC_GNU.  */
-#if GNULIB_REALLOC_GNU && !HAVE_REALLOC_GNU
-# define NEED_REALLOC_GNU 1
-#endif
-
-/* Infer the properties of the system's malloc function.
-   The gnulib module 'malloc-gnu' defines HAVE_MALLOC_GNU.  */
-#if GNULIB_MALLOC_GNU && HAVE_MALLOC_GNU
-# define SYSTEM_MALLOC_GLIBC_COMPATIBLE 1
-#endif
-
 #include <stdlib.h>
 
-/* A function definition is only needed if NEED_REALLOC_GNU is defined above
-   or if the module 'realloc-posix' requests it.  */
-#if NEED_REALLOC_GNU || (GNULIB_REALLOC_POSIX && !HAVE_REALLOC_POSIX)
+#include <errno.h>
 
-# include <errno.h>
+#include "xalloc-oversized.h"
 
-/* Call the system's malloc and realloc below.  */
-# undef malloc
-# undef realloc
+/* Call the system's realloc below.  */
+#undef realloc
 
 /* Change the size of an allocated block of memory P to N bytes,
-   with error checking.  If N is zero, change it to 1.  If P is NULL,
-   use malloc.  */
+   with error checking.  If P is NULL, use malloc.  Otherwise if N is zero,
+   free P and return NULL.  */
 
 void *
 rpl_realloc (void *p, size_t n)
 {
-  void *result;
+  if (p == NULL)
+    return malloc (n);
 
-# if NEED_REALLOC_GNU
   if (n == 0)
     {
-      n = 1;
-
-      /* In theory realloc might fail, so don't rely on it to free.  */
       free (p);
-      p = NULL;
+      return NULL;
     }
-# endif
 
-  if (p == NULL)
+  if (xalloc_oversized (n, 1))
     {
-# if GNULIB_REALLOC_GNU && !NEED_REALLOC_GNU && !SYSTEM_MALLOC_GLIBC_COMPATIBLE
-      if (n == 0)
-        n = 1;
-# endif
-      result = malloc (n);
+      errno = ENOMEM;
+      return NULL;
     }
-  else
-    result = realloc (p, n);
 
-# if !HAVE_REALLOC_POSIX
+  void *result = realloc (p, n);
+
+#if !HAVE_MALLOC_POSIX
   if (result == NULL)
     errno = ENOMEM;
-# endif
+#endif
 
   return result;
 }
-
-#endif
diff --git a/m4/calloc.m4 b/m4/calloc.m4
index eeeb033d8..143f3bc3f 100644
--- a/m4/calloc.m4
+++ b/m4/calloc.m4
@@ -1,4 +1,4 @@
-# calloc.m4 serial 23
+# calloc.m4 serial 24
 
 # Copyright (C) 2004-2021 Free Software Foundation, Inc.
 # This file is free software; the Free Software Foundation
@@ -14,12 +14,10 @@
 
 # _AC_FUNC_CALLOC_IF([IF-WORKS], [IF-NOT])
 # -------------------------------------
-# If 'calloc (0, 0)' is properly handled, run IF-WORKS, otherwise, IF-NOT.
+# If calloc is compatible with GNU calloc, run IF-WORKS, otherwise, IF-NOT.
 AC_DEFUN([_AC_FUNC_CALLOC_IF],
-[
-  AC_REQUIRE([AC_TYPE_SIZE_T])dnl
-  AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles
-  AC_CACHE_CHECK([for GNU libc compatible calloc],
+[ AC_REQUIRE([AC_CANONICAL_HOST]) dnl for cross-compiles
+  AC_CACHE_CHECK([whether calloc (0, n) and calloc (n, 0) return nonnull],
     [ac_cv_func_calloc_0_nonnull],
     [if test $cross_compiling != yes; then
        ac_cv_func_calloc_0_nonnull=yes
@@ -35,32 +33,6 @@ AC_DEFUN([_AC_FUNC_CALLOC_IF],
             ]])],
          [],
          [ac_cv_func_calloc_0_nonnull=no])
-       AC_RUN_IFELSE(
-         [AC_LANG_PROGRAM(
-            [AC_INCLUDES_DEFAULT],
-            [[int result;
-              typedef struct { char c[8]; } S8;
-              size_t n = (size_t) -1 / sizeof (S8) + 2;
-              S8 * volatile s = calloc (n, sizeof (S8));
-              if (s)
-                {
-                  s[0].c[0] = 1;
-                  if (s[n - 1].c[0])
-                    result = 0;
-                  else
-                    result = 2;
-                }
-              else
-                result = 3;
-              free (s);
-              return result;
-            ]])],
-         dnl The exit code of this program is 0 if calloc() succeeded with a
-         dnl wrap-around bug (which it shouldn't), 2 if calloc() succeeded in
-         dnl a non-flat address space, 3 if calloc() failed, or 1 if some leak
-         dnl sanitizer terminated the program as a result of the calloc() call.
-         [ac_cv_func_calloc_0_nonnull=no],
-         [])
      else
        case "$host_os" in
                         # Guess yes on glibc systems.
@@ -82,38 +54,31 @@ AC_DEFUN([_AC_FUNC_CALLOC_IF],
       $2
       ;;
   esac
-])# AC_FUNC_CALLOC
+])
 
 
 # gl_FUNC_CALLOC_GNU
 # ------------------
-# Report whether 'calloc (0, 0)' is properly handled, and replace calloc if
-# needed.
+# Replace calloc if it is not compatible with GNU libc.
 AC_DEFUN([gl_FUNC_CALLOC_GNU],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  _AC_FUNC_CALLOC_IF(
-    [AC_DEFINE([HAVE_CALLOC_GNU], [1],
-               [Define to 1 if your system has a GNU libc compatible 'calloc'
-                function, and to 0 otherwise.])],
-    [AC_DEFINE([HAVE_CALLOC_GNU], [0])
-     REPLACE_CALLOC=1
-    ])
+  AC_REQUIRE([gl_FUNC_CALLOC_POSIX])
+  test $REPLACE_CALLOC = 1 || _AC_FUNC_CALLOC_IF([], [REPLACE_CALLOC=1])
 ])# gl_FUNC_CALLOC_GNU
 
-
 # gl_FUNC_CALLOC_POSIX
 # --------------------
 # Test whether 'calloc' is POSIX compliant (sets errno to ENOMEM when it
-# fails), and replace calloc if it is not.
+# fails, and doesn't mess up with ptrdiff_t or size_t overflow),
+# and replace calloc if it is not.
 AC_DEFUN([gl_FUNC_CALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  AC_REQUIRE([gl_CHECK_MALLOC_POSIX])
-  if test $gl_cv_func_malloc_posix = yes; then
-    AC_DEFINE([HAVE_CALLOC_POSIX], [1],
-      [Define if the 'calloc' function is POSIX compliant.])
-  else
-    REPLACE_CALLOC=1
-  fi
+  AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
+  REPLACE_CALLOC=$REPLACE_MALLOC
+  dnl Although in theory we should also test for size_t overflow,
+  dnl in practice testing for ptrdiff_t overflow suffices
+  dnl since PTRDIFF_MAX <= SIZE_MAX on all known Gnulib porting targets.
+  dnl A separate size_t test would slow down 'configure'.
 ])
diff --git a/m4/malloc.m4 b/m4/malloc.m4
index 32ab42ec0..503da2cf8 100644
--- a/m4/malloc.m4
+++ b/m4/malloc.m4
@@ -1,4 +1,4 @@
-# malloc.m4 serial 22
+# malloc.m4 serial 23
 dnl Copyright (C) 2007, 2009-2021 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -7,9 +7,8 @@ dnl with or without modifications, as long as this notice is preserved.
 # This is adapted with modifications from upstream Autoconf here:
 # https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=04be2b7a29d65d9a08e64e8e56e594c91749598c
 AC_DEFUN([_AC_FUNC_MALLOC_IF],
-[
-  AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
-  AC_CACHE_CHECK([for GNU libc compatible malloc],
+[ AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
+  AC_CACHE_CHECK([whether malloc (0) returns nonnull],
     [ac_cv_func_malloc_0_nonnull],
     [AC_RUN_IFELSE(
        [AC_LANG_PROGRAM(
@@ -44,47 +43,89 @@ AC_DEFUN([_AC_FUNC_MALLOC_IF],
 
 # gl_FUNC_MALLOC_GNU
 # ------------------
-# Test whether 'malloc (0)' is handled like in GNU libc, and replace malloc if
-# it is not.
+# Replace malloc if it is not compatible with GNU libc.
 AC_DEFUN([gl_FUNC_MALLOC_GNU],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  dnl _AC_FUNC_MALLOC_IF is defined in Autoconf.
-  _AC_FUNC_MALLOC_IF(
-    [AC_DEFINE([HAVE_MALLOC_GNU], [1],
-               [Define to 1 if your system has a GNU libc compatible 'malloc'
-                function, and to 0 otherwise.])],
-    [AC_DEFINE([HAVE_MALLOC_GNU], [0])
-     REPLACE_MALLOC=1
+  AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
+  test $REPLACE_MALLOC = 1 || _AC_FUNC_MALLOC_IF([], [REPLACE_MALLOC=1])
+])
+
+# gl_FUNC_MALLOC_PTRDIFF
+# ----------------------
+# Test whether malloc (N) reliably fails when N exceeds PTRDIFF_MAX,
+# and replace malloc otherwise.
+AC_DEFUN([gl_FUNC_MALLOC_PTRDIFF],
+[
+  AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
+  AC_REQUIRE([gl_CHECK_MALLOC_PTRDIFF])
+  test "$gl_cv_malloc_ptrdiff" = yes || REPLACE_MALLOC=1
+])
+
+# Test whether malloc, realloc, calloc refuse to create objects
+# larger than what can be expressed in ptrdiff_t.
+# Set gl_cv_func_malloc_gnu to yes or no accordingly.
+AC_DEFUN([gl_CHECK_MALLOC_PTRDIFF],
+[
+  AC_CACHE_CHECK([whether malloc is ptrdiff_t safe],
+    [gl_cv_malloc_ptrdiff],
+    [AC_COMPILE_IFELSE(
+       [AC_LANG_PROGRAM(
+          [[#include <stdint.h>
+          ]],
+          [[/* 64-bit ptrdiff_t is so wide that no practical platform
+               can exceed it.  */
+            #define WIDE_PTRDIFF (PTRDIFF_MAX >> 31 >> 31 != 0)
+
+            /* On rare machines where size_t fits in ptrdiff_t there
+               is no problem.  */
+            #define NARROW_SIZE (SIZE_MAX <= PTRDIFF_MAX)
+
+            /* glibc 2.30 and later malloc refuses to exceed ptrdiff_t
+               bounds even on 32-bit platforms.  We don't know which
+               non-glibc systems are safe.  */
+            #define KNOWN_SAFE (2 < __GLIBC__ + (30 <= __GLIBC_MINOR__))
+
+            #if WIDE_PTRDIFF || NARROW_SIZE || KNOWN_SAFE
+              return 0;
+            #else
+              #error "malloc might not be ptrdiff_t safe"
+              syntax error
+            #endif
+          ]])],
+       [gl_cv_malloc_ptrdiff=yes],
+       [gl_cv_malloc_ptrdiff=no])
     ])
 ])
 
 # gl_FUNC_MALLOC_POSIX
 # --------------------
 # Test whether 'malloc' is POSIX compliant (sets errno to ENOMEM when it
-# fails), and replace malloc if it is not.
+# fails, and doesn't mess up with ptrdiff_t overflow), and replace
+# malloc if it is not.
 AC_DEFUN([gl_FUNC_MALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
+  AC_REQUIRE([gl_FUNC_MALLOC_PTRDIFF])
   AC_REQUIRE([gl_CHECK_MALLOC_POSIX])
-  if test $gl_cv_func_malloc_posix = yes; then
+  if test "$gl_cv_func_malloc_posix" = yes; then
     AC_DEFINE([HAVE_MALLOC_POSIX], [1],
-      [Define if the 'malloc' function is POSIX compliant.])
+      [Define if malloc, realloc, and calloc set errno on allocation failure.])
   else
     REPLACE_MALLOC=1
   fi
 ])
 
-# Test whether malloc, realloc, calloc are POSIX compliant,
+# Test whether malloc, realloc, calloc set errno on failure.
 # Set gl_cv_func_malloc_posix to yes or no accordingly.
 AC_DEFUN([gl_CHECK_MALLOC_POSIX],
 [
-  AC_CACHE_CHECK([whether malloc, realloc, calloc are POSIX compliant],
+  AC_CACHE_CHECK([whether malloc, realloc, calloc set errno on failure],
     [gl_cv_func_malloc_posix],
     [
       dnl It is too dangerous to try to allocate a large amount of memory:
       dnl some systems go to their knees when you do that. So assume that
-      dnl all Unix implementations of the function are POSIX compliant.
+      dnl all Unix implementations of the function set errno on failure.
       AC_COMPILE_IFELSE(
         [AC_LANG_PROGRAM(
            [[]],
diff --git a/m4/realloc.m4 b/m4/realloc.m4
index a80a02a6b..4939516a0 100644
--- a/m4/realloc.m4
+++ b/m4/realloc.m4
@@ -1,4 +1,4 @@
-# realloc.m4 serial 20
+# realloc.m4 serial 21
 dnl Copyright (C) 2007, 2009-2021 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -7,9 +7,8 @@ dnl with or without modifications, as long as this notice is preserved.
 # This is adapted with modifications from upstream Autoconf here:
 # https://git.savannah.gnu.org/cgit/autoconf.git/commit/?id=04be2b7a29d65d9a08e64e8e56e594c91749598c
 AC_DEFUN([_AC_FUNC_REALLOC_IF],
-[
-  AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
-  AC_CACHE_CHECK([for GNU libc compatible realloc],
+[ AC_REQUIRE([AC_CANONICAL_HOST])dnl for cross-compiles
+  AC_CACHE_CHECK([whether realloc (0, 0) returns nonnull],
     [ac_cv_func_realloc_0_nonnull],
     [AC_RUN_IFELSE(
        [AC_LANG_PROGRAM(
@@ -44,33 +43,22 @@ AC_DEFUN([_AC_FUNC_REALLOC_IF],
 
 # gl_FUNC_REALLOC_GNU
 # -------------------
-# Test whether 'realloc (0, 0)' is handled like in GNU libc, and replace
-# realloc if it is not.
+# Replace realloc if it is not compatible with GNU libc.
 AC_DEFUN([gl_FUNC_REALLOC_GNU],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  dnl _AC_FUNC_REALLOC_IF is defined in Autoconf.
-  _AC_FUNC_REALLOC_IF(
-    [AC_DEFINE([HAVE_REALLOC_GNU], [1],
-               [Define to 1 if your system has a GNU libc compatible 'realloc'
-                function, and to 0 otherwise.])],
-    [AC_DEFINE([HAVE_REALLOC_GNU], [0])
-     REPLACE_REALLOC=1
-    ])
+  AC_REQUIRE([gl_FUNC_REALLOC_POSIX])
+  test $REPLACE_REALLOC = 1 || _AC_FUNC_REALLOC_IF([], [REPLACE_REALLOC=1])
 ])# gl_FUNC_REALLOC_GNU
 
 # gl_FUNC_REALLOC_POSIX
 # ---------------------
 # Test whether 'realloc' is POSIX compliant (sets errno to ENOMEM when it
-# fails), and replace realloc if it is not.
+# fails, and doesn't mess up with ptrdiff_t overflow),
+# and replace realloc if it is not.
 AC_DEFUN([gl_FUNC_REALLOC_POSIX],
 [
   AC_REQUIRE([gl_STDLIB_H_DEFAULTS])
-  AC_REQUIRE([gl_CHECK_MALLOC_POSIX])
-  if test $gl_cv_func_malloc_posix = yes; then
-    AC_DEFINE([HAVE_REALLOC_POSIX], [1],
-      [Define if the 'realloc' function is POSIX compliant.])
-  else
-    REPLACE_REALLOC=1
-  fi
+  AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
+  REPLACE_REALLOC=$REPLACE_MALLOC
 ])
diff --git a/modules/calloc-gnu b/modules/calloc-gnu
index ffc8b50ef..2aa2dd1c0 100644
--- a/modules/calloc-gnu
+++ b/modules/calloc-gnu
@@ -13,7 +13,6 @@ gl_FUNC_CALLOC_GNU
 if test $REPLACE_CALLOC = 1; then
   AC_LIBOBJ([calloc])
 fi
-gl_MODULE_INDICATOR([calloc-gnu])
 
 Makefile.am:
 
diff --git a/modules/calloc-posix b/modules/calloc-posix
index cead843ae..dc9cadd5c 100644
--- a/modules/calloc-posix
+++ b/modules/calloc-posix
@@ -8,6 +8,7 @@ m4/malloc.m4
 
 Depends-on:
 stdlib
+xalloc-oversized     [test $REPLACE_REALLOC = 1]
 
 configure.ac:
 gl_FUNC_CALLOC_POSIX
diff --git a/modules/malloc-gnu b/modules/malloc-gnu
index 0bfa92d75..f8bcdae55 100644
--- a/modules/malloc-gnu
+++ b/modules/malloc-gnu
@@ -17,7 +17,6 @@ gl_FUNC_MALLOC_GNU
 if test $REPLACE_MALLOC = 1; then
   AC_LIBOBJ([malloc])
 fi
-gl_MODULE_INDICATOR([malloc-gnu])
 
 Makefile.am:
 
diff --git a/modules/malloc-posix b/modules/malloc-posix
index 1a2d72c5a..bafcf3801 100644
--- a/modules/malloc-posix
+++ b/modules/malloc-posix
@@ -7,14 +7,14 @@ m4/malloc.m4
 
 Depends-on:
 stdlib
+xalloc-oversized     [test $REPLACE_MALLOC = 1]
 
 configure.ac:
-gl_FUNC_MALLOC_POSIX
+AC_REQUIRE([gl_FUNC_MALLOC_POSIX])
 if test $REPLACE_MALLOC = 1; then
   AC_LIBOBJ([malloc])
 fi
 gl_STDLIB_MODULE_INDICATOR([malloc-posix])
-gl_MODULE_INDICATOR([malloc-posix])
 
 Makefile.am:
 
diff --git a/modules/realloc-gnu b/modules/realloc-gnu
index 760edcda1..7752d8c6a 100644
--- a/modules/realloc-gnu
+++ b/modules/realloc-gnu
@@ -17,7 +17,6 @@ gl_FUNC_REALLOC_GNU
 if test $REPLACE_REALLOC = 1; then
   AC_LIBOBJ([realloc])
 fi
-gl_MODULE_INDICATOR([realloc-gnu])
 
 Makefile.am:
 
diff --git a/modules/realloc-posix b/modules/realloc-posix
index 7f481102b..a30356f31 100644
--- a/modules/realloc-posix
+++ b/modules/realloc-posix
@@ -8,6 +8,9 @@ m4/malloc.m4
 
 Depends-on:
 stdlib
+free-posix           [test $REPLACE_REALLOC = 1]
+malloc-posix         [test $REPLACE_REALLOC = 1]
+xalloc-oversized     [test $REPLACE_REALLOC = 1]
 
 configure.ac:
 gl_FUNC_REALLOC_POSIX
@@ -15,7 +18,6 @@ if test $REPLACE_REALLOC = 1; then
   AC_LIBOBJ([realloc])
 fi
 gl_STDLIB_MODULE_INDICATOR([realloc-posix])
-gl_MODULE_INDICATOR([realloc-posix])
 
 Makefile.am:
 
-- 
2.27.0


[-- Attachment #3: 0002-xalloc-adjust-to-malloc-ptrdiff_t-change.patch --]
[-- Type: text/x-patch, Size: 3657 bytes --]

From 7415a0fa4abc1b8e8af37afd5b5408c1d5dfff2e Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Sat, 17 Apr 2021 18:48:38 -0700
Subject: [PATCH 2/2] xalloc: adjust to malloc ptrdiff_t change

* lib/xmalloc.c (HAVE_GNU_CALLOC, HAVE_GNU_MALLOC, HAVE_GNU_REALLOC):
Remove.
(xmalloc, xrealloc, xcalloc): Simplify by assuming GNU behavior.
* modules/xalloc (Depends-on): Add calloc-gnu, malloc-gnu,
realloc-gnu.
---
 ChangeLog      |  7 +++++++
 lib/xmalloc.c  | 41 ++++-------------------------------------
 modules/xalloc |  3 +++
 3 files changed, 14 insertions(+), 37 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 825201fe2..03221a292 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,12 @@
 2021-04-17  Paul Eggert  <eggert@cs.ucla.edu>
 
+	xalloc: adjust to malloc ptrdiff_t change
+	* lib/xmalloc.c (HAVE_GNU_CALLOC, HAVE_GNU_MALLOC, HAVE_GNU_REALLOC):
+	Remove.
+	(xmalloc, xrealloc, xcalloc): Simplify by assuming GNU behavior.
+	* modules/xalloc (Depends-on): Add calloc-gnu, malloc-gnu,
+	realloc-gnu.
+
 	malloc, etc.: check for ptrdiff_t overflow
 	In glibc 2.30 and later, malloc, realloc and calloc reject
 	attempts to create objects larger than PTRDIFF_MAX bytes.
diff --git a/lib/xmalloc.c b/lib/xmalloc.c
index 4a6589571..39ce893ad 100644
--- a/lib/xmalloc.c
+++ b/lib/xmalloc.c
@@ -27,34 +27,13 @@
 #include <stdlib.h>
 #include <string.h>
 
-/* 1 if calloc, malloc and realloc are known to be compatible with GNU.
-   This matters if we are not also using the calloc-gnu, malloc-gnu
-   and realloc-gnu modules, which define HAVE_CALLOC_GNU,
-   HAVE_MALLOC_GNU and HAVE_REALLOC_GNU and support the GNU API even
-   on non-GNU platforms.  */
-#if defined HAVE_CALLOC_GNU || (defined __GLIBC__ && !defined __UCLIBC__)
-enum { HAVE_GNU_CALLOC = 1 };
-#else
-enum { HAVE_GNU_CALLOC = 0 };
-#endif
-#if defined HAVE_MALLOC_GNU || (defined __GLIBC__ && !defined __UCLIBC__)
-enum { HAVE_GNU_MALLOC = 1 };
-#else
-enum { HAVE_GNU_MALLOC = 0 };
-#endif
-#if defined HAVE_REALLOC_GNU || (defined __GLIBC__ && !defined __UCLIBC__)
-enum { HAVE_GNU_REALLOC = 1 };
-#else
-enum { HAVE_GNU_REALLOC = 0 };
-#endif
-
 /* Allocate N bytes of memory dynamically, with error checking.  */
 
 void *
 xmalloc (size_t n)
 {
   void *p = malloc (n);
-  if (!p && (HAVE_GNU_MALLOC || n))
+  if (!p)
     xalloc_die ();
   return p;
 }
@@ -65,15 +44,8 @@ xmalloc (size_t n)
 void *
 xrealloc (void *p, size_t n)
 {
-  if (!HAVE_GNU_REALLOC && !n && p)
-    {
-      /* The GNU and C99 realloc behaviors disagree here.  Act like GNU.  */
-      free (p);
-      return NULL;
-    }
-
   void *r = realloc (p, n);
-  if (!r && (n || (HAVE_GNU_REALLOC && !p)))
+  if (!r && (!p || n))
     xalloc_die ();
   return r;
 }
@@ -175,13 +147,8 @@ xzalloc (size_t n)
 void *
 xcalloc (size_t n, size_t s)
 {
-  void *p;
-  /* Test for overflow, since objects with size greater than
-     PTRDIFF_MAX cause pointer subtraction to go awry.  Omit size-zero
-     tests if HAVE_GNU_CALLOC, since GNU calloc never returns NULL if
-     successful.  */
-  if (xalloc_oversized (n, s)
-      || (! (p = calloc (n, s)) && (HAVE_GNU_CALLOC || n != 0)))
+  void *p = calloc (n, s);
+  if (!p)
     xalloc_die ();
   return p;
 }
diff --git a/modules/xalloc b/modules/xalloc
index 5fa386a5d..000933e94 100644
--- a/modules/xalloc
+++ b/modules/xalloc
@@ -8,10 +8,13 @@ m4/xalloc.m4
 
 Depends-on:
 c99
+calloc-gnu
 extern-inline
 idx
 intprops
+malloc-gnu
 minmax
+realloc-gnu
 stdint
 xalloc-die
 xalloc-oversized
-- 
2.27.0


^ permalink raw reply related	[flat|nested] 17+ messages in thread
end of thread, other threads:[~2021-05-10  8:45 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-18  2:02 ptrdiff_t overflow checks for malloc-posix etc Paul Eggert
2021-04-18 11:59 ` Bruno Haible
2021-04-18 12:11 ` Bruno Haible
2021-04-18 20:24   ` Paul Eggert
2021-04-18 20:32     ` Bruno Haible
2021-04-18 12:13 ` Bruno Haible
2021-04-18 19:47   ` Paul Eggert
2021-04-18 20:23     ` Bruno Haible
2021-04-18 22:33       ` Paul Eggert
2021-04-19  0:04         ` Bruno Haible
2021-04-19  4:20           ` Paul Eggert
2021-04-18 12:13 ` Bruno Haible
2021-04-18 18:37   ` Paul Eggert
2021-05-09 20:19     ` Bruno Haible
2021-05-09 23:00       ` Bruno Haible
2021-05-10  8:45         ` Paul Eggert
2021-05-09 16:46 ` Bruno Haible

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).