[PATCH v3 0/7] Fix getdents{64} regression on some FS

[PATCH v3 0/7] Fix getdents{64} regression on some FS

[Adhemerval Zanella via Libc-alpha]

This is updated version [1], rebased against master add with some
suggestion from previous review.  The main changes are:

  - The translation buffer is allocated as an auxiliarry one (instead
    of a reserved spacefrom the dirstream) and readdir might reallocate
    it if it finds a large entry.

  - Failures in non-LFS readdir (such as d_ino/d_off overflow or failure
    in expand the translation buffer) is ignored.  This allows code that
    does not handle possible failure in readdir to successful read the
    rest of entries in the directory.

  - So minor fixes in dirent/tst-seekdir2.c

  - Added __attribute_deprecated_msg__ on getdirentries to redirect to
    getdents64.

I have checked on x86_64-linux-gnu, powerpc64-linux-gnu and on some
32-bit ABIs arm-linux-gnueabihf, powerpc-linux-gnu, and sparcv9-linux-gnu.


[1] https://sourceware.org/pipermail/libc-alpha/2020-October/118274.html

Adhemerval Zanella (7):
  linux: Do not skip entries with zero d_ino values [BZ #12165]
  linux: Use getdents64 on non-LFS readdir
  linux: Set internal DIR filepos as off64_t [BZ #23960, BZ #24050]
  linux: Add __readdir64_unlocked
  linux: Add __old_readdir64_unlocked
  linux: Use getdents64 on readdir64 compat implementation
  dirent: Deprecate getdirentries

 NEWS                                  |   3 +
 dirent/Makefile                       |   2 +-
 dirent/dirent.h                       |  14 +-
 dirent/tst-seekdir2.c                 | 158 +++++++++++++++++++++
 include/dirent.h                      |   1 +
 sysdeps/unix/sysv/linux/Makefile      |   3 +
 sysdeps/unix/sysv/linux/closedir.c    |   8 ++
 sysdeps/unix/sysv/linux/dirstream.h   |  11 +-
 sysdeps/unix/sysv/linux/getdents64.c  |  93 ------------
 sysdeps/unix/sysv/linux/olddirent.h   |   4 +-
 sysdeps/unix/sysv/linux/opendir.c     |  37 +++++
 sysdeps/unix/sysv/linux/readdir.c     |  83 +++++++----
 sysdeps/unix/sysv/linux/readdir64.c   | 196 +++++++++++++++-----------
 sysdeps/unix/sysv/linux/readdir64_r.c | 159 +++++----------------
 sysdeps/unix/sysv/linux/rewinddir.c   |   5 +
 sysdeps/unix/sysv/linux/seekdir.c     |  36 ++++-
 sysdeps/unix/sysv/linux/telldir.c     |  47 +++++-
 sysdeps/unix/sysv/linux/telldir.h     |  64 +++++++++
 18 files changed, 583 insertions(+), 341 deletions(-)
 create mode 100644 dirent/tst-seekdir2.c
 create mode 100644 sysdeps/unix/sysv/linux/telldir.h

-- 
2.25.1

[PATCH v3 1/7] linux: Do not skip entries with zero d_ino values [BZ #12165]

[Adhemerval Zanella via Libc-alpha]

According to Linux commit 2adc376c55194 (vfs: avoid creation of inode
number 0 in get_next_ino) Linux did not treat d_ino == 0 as a special
case (it is a valid inode number).

This patch fixes readdir{64} by not ignoring entried with d_ino being
0.

Checked on x86_64-linux-gnu and i686-linux-gnu.
---
 sysdeps/unix/sysv/linux/readdir.c   | 59 +++++++++++------------------
 sysdeps/unix/sysv/linux/readdir64.c | 59 +++++++++++------------------
 2 files changed, 44 insertions(+), 74 deletions(-)

diff --git a/sysdeps/unix/sysv/linux/readdir.c b/sysdeps/unix/sysv/linux/readdir.c
index ca2a8964e9..1a9820e475 100644
--- a/sysdeps/unix/sysv/linux/readdir.c
+++ b/sysdeps/unix/sysv/linux/readdir.c
@@ -25,51 +25,36 @@
 struct dirent *
 __readdir_unlocked (DIR *dirp)
 {
-  struct dirent *dp;
-  int saved_errno = errno;
+  const int saved_errno = errno;
 
-  do
+  if (dirp->offset >= dirp->size)
     {
-      size_t reclen;
-
-      if (dirp->offset >= dirp->size)
+      /* We've emptied out our buffer.  Refill it.  */
+      ssize_t bytes = __getdents (dirp->fd, dirp->data, dirp->allocation);
+      if (bytes <= 0)
 	{
-	  /* We've emptied out our buffer.  Refill it.  */
-
-	  size_t maxread = dirp->allocation;
-	  ssize_t bytes;
-
-	  bytes = __getdents (dirp->fd, dirp->data, maxread);
-	  if (bytes <= 0)
-	    {
-	      /* On some systems getdents fails with ENOENT when the
-		 open directory has been rmdir'd already.  POSIX.1
-		 requires that we treat this condition like normal EOF.  */
-	      if (bytes < 0 && errno == ENOENT)
-		bytes = 0;
-
-	      /* Don't modifiy errno when reaching EOF.  */
-	      if (bytes == 0)
-		__set_errno (saved_errno);
-	      dp = NULL;
-	      break;
-	    }
-	  dirp->size = (size_t) bytes;
-
-	  /* Reset the offset into the buffer.  */
-	  dirp->offset = 0;
+	  /* On some systems getdents fails with ENOENT when the
+	     open directory has been rmdir'd already.  POSIX.1
+	     requires that we treat this condition like normal EOF.  */
+	  if (bytes < 0 && errno == ENOENT)
+	    bytes = 0;
+
+	  /* Don't modifiy errno when reaching EOF.  */
+	  if (bytes == 0)
+	    __set_errno (saved_errno);
+	  return NULL;
 	}
+      dirp->size = bytes;
 
-      dp = (struct dirent *) &dirp->data[dirp->offset];
-
-      reclen = dp->d_reclen;
+      /* Reset the offset into the buffer.  */
+      dirp->offset = 0;
+    }
 
-      dirp->offset += reclen;
+  struct dirent *dp = (struct dirent *) &dirp->data[dirp->offset];
 
-      dirp->filepos = dp->d_off;
+  dirp->offset += dp->d_reclen;
 
-      /* Skip deleted files.  */
-    } while (dp->d_ino == 0);
+  dirp->filepos = dp->d_off;
 
   return dp;
 }
diff --git a/sysdeps/unix/sysv/linux/readdir64.c b/sysdeps/unix/sysv/linux/readdir64.c
index 1aa6e2664f..d35a4595f6 100644
--- a/sysdeps/unix/sysv/linux/readdir64.c
+++ b/sysdeps/unix/sysv/linux/readdir64.c
@@ -30,55 +30,40 @@
 struct dirent64 *
 __readdir64 (DIR *dirp)
 {
-  struct dirent64 *dp;
-  int saved_errno = errno;
+  const int saved_errno = errno;
 
 #if IS_IN (libc)
   __libc_lock_lock (dirp->lock);
 #endif
 
-  do
+  if (dirp->offset >= dirp->size)
     {
-      size_t reclen;
-
-      if (dirp->offset >= dirp->size)
+      /* We've emptied out our buffer.  Refill it.  */
+      ssize_t bytes = __getdents64 (dirp->fd, dirp->data, dirp->allocation);
+      if (bytes <= 0)
 	{
-	  /* We've emptied out our buffer.  Refill it.  */
-
-	  size_t maxread = dirp->allocation;
-	  ssize_t bytes;
-
-	  bytes = __getdents64 (dirp->fd, dirp->data, maxread);
-	  if (bytes <= 0)
-	    {
-	      /* On some systems getdents fails with ENOENT when the
-		 open directory has been rmdir'd already.  POSIX.1
-		 requires that we treat this condition like normal EOF.  */
-	      if (bytes < 0 && errno == ENOENT)
-		bytes = 0;
-
-	      /* Don't modifiy errno when reaching EOF.  */
-	      if (bytes == 0)
-		__set_errno (saved_errno);
-	      dp = NULL;
-	      break;
-	    }
-	  dirp->size = (size_t) bytes;
-
-	  /* Reset the offset into the buffer.  */
-	  dirp->offset = 0;
+	  /* On some systems getdents fails with ENOENT when the
+	     open directory has been rmdir'd already.  POSIX.1
+	     requires that we treat this condition like normal EOF.  */
+	  if (bytes < 0 && errno == ENOENT)
+	    bytes = 0;
+
+	  /* Don't modifiy errno when reaching EOF.  */
+	  if (bytes == 0)
+	    __set_errno (saved_errno);
+	  return NULL;
 	}
+      dirp->size = bytes;
 
-      dp = (struct dirent64 *) &dirp->data[dirp->offset];
-
-      reclen = dp->d_reclen;
+      /* Reset the offset into the buffer.  */
+      dirp->offset = 0;
+   }
 
-      dirp->offset += reclen;
+  struct dirent64 *dp = (struct dirent64 *) &dirp->data[dirp->offset];
 
-      dirp->filepos = dp->d_off;
+  dirp->offset += dp->d_reclen;
 
-      /* Skip deleted files.  */
-    } while (dp->d_ino == 0);
+  dirp->filepos = dp->d_off;
 
 #if IS_IN (libc)
   __libc_lock_unlock (dirp->lock);
-- 
2.25.1

[PATCH v3 2/7] linux: Use getdents64 on non-LFS readdir

[Adhemerval Zanella via Libc-alpha]

The opendir allocates a translation buffer to be used to return the
non-LFS readdir entry.  The obtained dirent64 struct is translated
to the temporary buffer on each readdir call.

Entries that overflow d_off/d_ino and the buffer reallocation failure
(in case of large d_name) are ignored.

Checked on x86_64-linux-gnu and i686-linux-gnu.
---
 sysdeps/unix/sysv/linux/closedir.c  |  4 ++
 sysdeps/unix/sysv/linux/dirstream.h |  5 ++
 sysdeps/unix/sysv/linux/opendir.c   | 21 +++++++
 sysdeps/unix/sysv/linux/readdir.c   | 97 +++++++++++++++++++++--------
 4 files changed, 101 insertions(+), 26 deletions(-)

diff --git a/sysdeps/unix/sysv/linux/closedir.c b/sysdeps/unix/sysv/linux/closedir.c
index ccc19eefcd..d67b590d5f 100644
--- a/sysdeps/unix/sysv/linux/closedir.c
+++ b/sysdeps/unix/sysv/linux/closedir.c
@@ -47,6 +47,10 @@ __closedir (DIR *dirp)
   __libc_lock_fini (dirp->lock);
 #endif
 
+#if !_DIRENT_MATCHES_DIRENT64
+  free (dirp->tbuffer);
+#endif
+
   free ((void *) dirp);
 
   return __close_nocancel (fd);
diff --git a/sysdeps/unix/sysv/linux/dirstream.h b/sysdeps/unix/sysv/linux/dirstream.h
index a3ea2b7197..da9315d353 100644
--- a/sysdeps/unix/sysv/linux/dirstream.h
+++ b/sysdeps/unix/sysv/linux/dirstream.h
@@ -41,6 +41,11 @@ struct __dirstream
 
     int errcode;		/* Delayed error code.  */
 
+#if !defined __OFF_T_MATCHES_OFF64_T || !defined __INO_T_MATCHES_INO64_T
+    char *tbuffer;		/* Translation buffer for non-LFS calls.  */
+    size_t tbuffer_size;	/* Size of translation buffer.  */
+#endif
+
     /* Directory block.  We must make sure that this block starts
        at an address that is aligned adequately enough to store
        dirent entries.  Using the alignment of "void *" is not
diff --git a/sysdeps/unix/sysv/linux/opendir.c b/sysdeps/unix/sysv/linux/opendir.c
index 2198224588..329575e809 100644
--- a/sysdeps/unix/sysv/linux/opendir.c
+++ b/sysdeps/unix/sysv/linux/opendir.c
@@ -119,6 +119,27 @@ __alloc_dir (int fd, bool close_fd, int flags, const struct stat64 *statp)
       return NULL;
     }
 
+#if !_DIRENT_MATCHES_DIRENT64
+  /* Allocates a translation buffer to use as the returned 'struct direct'
+     for non-LFS 'readdir' calls.
+
+     The initial NAME_MAX size should handle most cases, while readdir might
+     expand the buffer if required.  */
+  enum
+    {
+      tbuffer_size = sizeof (struct dirent) + NAME_MAX + 1
+    };
+  dirp->tbuffer = malloc (tbuffer_size);
+  if (dirp->tbuffer == NULL)
+    {
+      free (dirp);
+      if (close_fd)
+	__close_nocancel_nostatus (fd);
+      return NULL;
+    }
+  dirp->tbuffer_size = tbuffer_size;
+#endif
+
   dirp->fd = fd;
 #if IS_IN (libc)
   __libc_lock_init (dirp->lock);
diff --git a/sysdeps/unix/sysv/linux/readdir.c b/sysdeps/unix/sysv/linux/readdir.c
index 1a9820e475..9af4561a8c 100644
--- a/sysdeps/unix/sysv/linux/readdir.c
+++ b/sysdeps/unix/sysv/linux/readdir.c
@@ -21,42 +21,87 @@
 #if !_DIRENT_MATCHES_DIRENT64
 #include <dirstream.h>
 
+/* Translate the DP64 entry to the non-LFS one in the translation buffer
+   at dirstream DS.  Return true is the translation was possible or
+   false if either an internal fields can be represented in the non-LFS
+   entry or if the translation can not be resized.  */
+static bool
+dirstream_entry (struct __dirstream *ds, const struct dirent64 *dp64)
+{
+  off_t d_off = dp64->d_off;
+  if (d_off != dp64->d_off)
+    return false;
+  ino_t d_ino = dp64->d_ino;
+  if (d_ino != dp64->d_ino)
+    return false;
+
+  /* Expand the translation buffer to hold the new name size.  */
+  size_t new_reclen = sizeof (struct dirent)
+		    + dp64->d_reclen - offsetof (struct dirent64, d_name);
+  if (new_reclen > ds->tbuffer_size)
+    {
+      char *newbuffer = realloc (ds->tbuffer, new_reclen);
+      if (newbuffer == NULL)
+	return false;
+      ds->tbuffer = newbuffer;
+      ds->tbuffer_size = new_reclen;
+    }
+
+  struct dirent *dp = (struct dirent *) ds->tbuffer;
+
+  dp->d_off = d_off;
+  dp->d_ino = d_ino;
+  dp->d_reclen = new_reclen;
+  dp->d_type = dp64->d_type;
+  memcpy (dp->d_name, dp64->d_name,
+	  dp64->d_reclen - offsetof (struct dirent64, d_name));
+
+  return true;
+}
+
 /* Read a directory entry from DIRP.  */
 struct dirent *
 __readdir_unlocked (DIR *dirp)
 {
   const int saved_errno = errno;
 
-  if (dirp->offset >= dirp->size)
+  while (1)
     {
-      /* We've emptied out our buffer.  Refill it.  */
-      ssize_t bytes = __getdents (dirp->fd, dirp->data, dirp->allocation);
-      if (bytes <= 0)
+      if (dirp->offset >= dirp->size)
 	{
-	  /* On some systems getdents fails with ENOENT when the
-	     open directory has been rmdir'd already.  POSIX.1
-	     requires that we treat this condition like normal EOF.  */
-	  if (bytes < 0 && errno == ENOENT)
-	    bytes = 0;
-
-	  /* Don't modifiy errno when reaching EOF.  */
-	  if (bytes == 0)
-	    __set_errno (saved_errno);
-	  return NULL;
+	  /* We've emptied out our buffer.  Refill it.  */
+	  ssize_t bytes = __getdents64 (dirp->fd, dirp->data,
+					dirp->allocation);
+	  if (bytes <= 0)
+	    {
+	      /* On some systems getdents fails with ENOENT when the
+		 open directory has been rmdir'd already.  POSIX.1
+		 requires that we treat this condition like normal EOF.  */
+	      if (bytes < 0 && errno == ENOENT)
+		bytes = 0;
+
+	      /* Don't modifiy errno when reaching EOF.  */
+	      if (bytes == 0)
+		__set_errno (saved_errno);
+	      return NULL;
+	    }
+	  dirp->size = bytes;
+
+ 	  /* Reset the offset into the buffer.  */
+	  dirp->offset = 0;
+ 	}
+
+      struct dirent64 *dp64 = (struct dirent64 *) &dirp->data[dirp->offset];
+      dirp->offset += dp64->d_reclen;
+
+      /* Skip entries which might overflow d_off/d_ino or if the translation
+	 buffer can't be resized.  */
+      if (dirstream_entry (dirp, dp64))
+	{
+          dirp->filepos = dp64->d_off;
+	  return (struct dirent *) dirp->tbuffer;
 	}
-      dirp->size = bytes;
-
-      /* Reset the offset into the buffer.  */
-      dirp->offset = 0;
     }
-
-  struct dirent *dp = (struct dirent *) &dirp->data[dirp->offset];
-
-  dirp->offset += dp->d_reclen;
-
-  dirp->filepos = dp->d_off;
-
-  return dp;
 }
 
 struct dirent *
-- 
2.25.1

[PATCH v3 3/7] linux: Set internal DIR filepos as off64_t [BZ #23960, BZ #24050]

[Adhemerval Zanella via Libc-alpha]

It allows to obtain the expected entry offset on telldir and set
it correctly on seekdir on platforms where long int is smaller
than off64_t.

On such cases telldir will mantain an internal list that maps the
DIR object off64_t offsets to the returned long int (the function
return value).  The seekdir will then set the correct offset from
the internal list using the telldir as the list key.

It also removes the overflow check on readdir and the returned value
will be truncated by the non-LFS off_t size.  As Joseph has noted
in BZ #23960 comment #22, d_off is an opaque value and since
telldir/seekdir works regardless of the returned dirent d_off value.

Finally it removed the requirement to check for overflow values on
telldir (BZ #24050).

Checked on x86_64-linux-gnu, i686-linux-gnu, powerpc-linux-gnu,
and arm-linux-gnueabihf.
---
 dirent/Makefile                     |   2 +-
 dirent/tst-seekdir2.c               | 158 ++++++++++++++++++++++++++++
 sysdeps/unix/sysv/linux/closedir.c  |   4 +
 sysdeps/unix/sysv/linux/dirstream.h |   6 +-
 sysdeps/unix/sysv/linux/opendir.c   |   3 +
 sysdeps/unix/sysv/linux/readdir.c   |   1 +
 sysdeps/unix/sysv/linux/rewinddir.c |   5 +
 sysdeps/unix/sysv/linux/seekdir.c   |  36 ++++++-
 sysdeps/unix/sysv/linux/telldir.c   |  47 ++++++++-
 sysdeps/unix/sysv/linux/telldir.h   |  64 +++++++++++
 10 files changed, 317 insertions(+), 9 deletions(-)
 create mode 100644 dirent/tst-seekdir2.c
 create mode 100644 sysdeps/unix/sysv/linux/telldir.h

diff --git a/dirent/Makefile b/dirent/Makefile
index e917d5ceab..ba9755da64 100644
--- a/dirent/Makefile
+++ b/dirent/Makefile
@@ -31,7 +31,7 @@ routines	:= opendir closedir readdir readdir_r rewinddir \
 		   scandir-cancel scandir-tail scandir64-tail
 
 tests	   := list tst-seekdir opendir-tst1 bug-readdir1 tst-fdopendir \
-	      tst-fdopendir2 tst-scandir tst-scandir64
+	      tst-fdopendir2 tst-scandir tst-scandir64 tst-seekdir2
 
 CFLAGS-scandir.c += $(uses-callbacks)
 CFLAGS-scandir64.c += $(uses-callbacks)
diff --git a/dirent/tst-seekdir2.c b/dirent/tst-seekdir2.c
new file mode 100644
index 0000000000..3e01b361e5
--- /dev/null
+++ b/dirent/tst-seekdir2.c
@@ -0,0 +1,158 @@
+/* Check multiple telldir and seekdir.
+   Copyright (C) 2020 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <https://www.gnu.org/licenses/>.  */
+
+#include <dirent.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <support/temp_file.h>
+#include <support/support.h>
+#include <support/check.h>
+
+/* Some filesystems returns a arbitrary value for d_off direnty entry (ext4
+   for instance, where the value is an internal hash key).  The idea of
+   create a large number of file is to try trigger a overflow d_off value
+   in a entry to check if telldir/seekdir does work corretly in such
+   case.  */
+static const char *dirname;
+static const size_t nfiles = 10240;
+
+static void
+do_prepare (int argc, char *argv[])
+{
+  dirname = support_create_temp_directory ("tst-seekdir2-");
+
+  for (size_t i = 0; i < nfiles; i++)
+    {
+      int fd = create_temp_file_in_dir ("tempfile.", dirname, NULL);
+      TEST_VERIFY_EXIT (fd > 0);
+      close (fd);
+    }
+}
+#define PREPARE do_prepare
+
+/* Check for old non Large File Support (LFS).  */
+static int
+do_test_not_lfs (void)
+{
+  DIR *dirp = opendir (dirname);
+  TEST_VERIFY_EXIT (dirp != NULL);
+
+  size_t dirp_count = 0;
+  for (struct dirent *dp = readdir (dirp);
+       dp != NULL;
+       dp = readdir (dirp))
+    dirp_count++;
+
+  /* The 2 extra files are '.' and '..'.  */
+  TEST_COMPARE (dirp_count, nfiles + 2);
+
+  rewinddir (dirp);
+
+  long *tdirp = xmalloc (dirp_count * sizeof (long));
+  struct dirent **ddirp = xmalloc (dirp_count * sizeof (struct dirent *));
+
+  size_t i = 0;
+  do
+    {
+      tdirp[i] = telldir (dirp);
+      struct dirent *dp = readdir (dirp);
+      TEST_VERIFY_EXIT (dp != NULL);
+      ddirp[i] = xmalloc (dp->d_reclen);
+      memcpy (ddirp[i], dp, dp->d_reclen);
+    } while (++i < dirp_count);
+
+  for (i = 0; i < dirp_count - 1; i++)
+    {
+      seekdir (dirp, tdirp[i]);
+      struct dirent *dp = readdir (dirp);
+      TEST_COMPARE (strcmp (dp->d_name, ddirp[i]->d_name), 0);
+      TEST_COMPARE (dp->d_ino, ddirp[i]->d_ino);
+      TEST_COMPARE (dp->d_off, ddirp[i]->d_off);
+    }
+
+  closedir (dirp);
+  free (tdirp);
+  for (i = 0; i < dirp_count; i++)
+    free (ddirp[i]);
+  free (ddirp);
+
+  return 0;
+}
+
+/* Same as before but with LFS support.  */
+static int
+do_test_lfs (void)
+{
+  DIR *dirp = opendir (dirname);
+  TEST_VERIFY_EXIT (dirp != NULL);
+
+  size_t dirp_count = 0;
+  for (struct dirent64 * dp = readdir64 (dirp);
+       dp != NULL;
+       dp = readdir64 (dirp))
+    dirp_count++;
+
+  /* The 2 extra files are '.' and '..'.  */
+  TEST_COMPARE (dirp_count, nfiles + 2);
+
+  rewinddir (dirp);
+
+  long *tdirp = xmalloc (dirp_count * sizeof (long));
+  struct dirent64 **ddirp = xmalloc (dirp_count * sizeof (struct dirent64 *));
+
+  size_t i = 0;
+  do
+    {
+      tdirp[i] = telldir (dirp);
+      struct dirent64 *dp = readdir64 (dirp);
+      TEST_VERIFY_EXIT (dp != NULL);
+      ddirp[i] = xmalloc (dp->d_reclen);
+      memcpy (ddirp[i], dp, dp->d_reclen);
+    } while (++i < dirp_count);
+
+  for (i = 0; i < dirp_count - 1; i++)
+    {
+      seekdir (dirp, tdirp[i]);
+      struct dirent64 *dp = readdir64 (dirp);
+      TEST_COMPARE (strcmp (dp->d_name, ddirp[i]->d_name), 0);
+      TEST_COMPARE (dp->d_ino, ddirp[i]->d_ino);
+      TEST_COMPARE (dp->d_off, ddirp[i]->d_off);
+    }
+
+  closedir (dirp);
+  free (tdirp);
+  for (i = 0; i < dirp_count; i++)
+    free (ddirp[i]);
+  free (ddirp);
+
+  return 0;
+}
+
+static int
+do_test (void)
+{
+  do_test_not_lfs ();
+  do_test_lfs ();
+
+  return 0;
+}
+
+#include <support/test-driver.c>
diff --git a/sysdeps/unix/sysv/linux/closedir.c b/sysdeps/unix/sysv/linux/closedir.c
index d67b590d5f..3436780519 100644
--- a/sysdeps/unix/sysv/linux/closedir.c
+++ b/sysdeps/unix/sysv/linux/closedir.c
@@ -43,6 +43,10 @@ __closedir (DIR *dirp)
 
   fd = dirp->fd;
 
+#ifndef __LP64__
+  dirstream_loc_clear (&dirp->locs);
+#endif
+
 #if IS_IN (libc)
   __libc_lock_fini (dirp->lock);
 #endif
diff --git a/sysdeps/unix/sysv/linux/dirstream.h b/sysdeps/unix/sysv/linux/dirstream.h
index da9315d353..369d0a697c 100644
--- a/sysdeps/unix/sysv/linux/dirstream.h
+++ b/sysdeps/unix/sysv/linux/dirstream.h
@@ -21,6 +21,7 @@
 #include <sys/types.h>
 
 #include <libc-lock.h>
+#include <telldir.h>
 
 /* Directory stream type.
 
@@ -37,7 +38,7 @@ struct __dirstream
     size_t size;		/* Total valid data in the block.  */
     size_t offset;		/* Current offset into the block.  */
 
-    off_t filepos;		/* Position of next entry to read.  */
+    off64_t filepos;		/* Position of next entry to read.  */
 
     int errcode;		/* Delayed error code.  */
 
@@ -45,6 +46,9 @@ struct __dirstream
     char *tbuffer;		/* Translation buffer for non-LFS calls.  */
     size_t tbuffer_size;	/* Size of translation buffer.  */
 #endif
+#ifndef __LP64__
+    struct dirstream_loc_t locs; /* off64_t to long int map for telldir.  */
+#endif
 
     /* Directory block.  We must make sure that this block starts
        at an address that is aligned adequately enough to store
diff --git a/sysdeps/unix/sysv/linux/opendir.c b/sysdeps/unix/sysv/linux/opendir.c
index 329575e809..28fd55341e 100644
--- a/sysdeps/unix/sysv/linux/opendir.c
+++ b/sysdeps/unix/sysv/linux/opendir.c
@@ -149,6 +149,9 @@ __alloc_dir (int fd, bool close_fd, int flags, const struct stat64 *statp)
   dirp->offset = 0;
   dirp->filepos = 0;
   dirp->errcode = 0;
+#ifndef __LP64__
+  dirstream_loc_init (&dirp->locs);
+#endif
 
   return dirp;
 }
diff --git a/sysdeps/unix/sysv/linux/readdir.c b/sysdeps/unix/sysv/linux/readdir.c
index 9af4561a8c..656672c706 100644
--- a/sysdeps/unix/sysv/linux/readdir.c
+++ b/sysdeps/unix/sysv/linux/readdir.c
@@ -17,6 +17,7 @@
    <https://www.gnu.org/licenses/>.  */
 
 #include <dirent.h>
+#include <unistd.h>
 
 #if !_DIRENT_MATCHES_DIRENT64
 #include <dirstream.h>
diff --git a/sysdeps/unix/sysv/linux/rewinddir.c b/sysdeps/unix/sysv/linux/rewinddir.c
index 860bfda004..8db0d0be4a 100644
--- a/sysdeps/unix/sysv/linux/rewinddir.c
+++ b/sysdeps/unix/sysv/linux/rewinddir.c
@@ -33,6 +33,11 @@ __rewinddir (DIR *dirp)
   dirp->offset = 0;
   dirp->size = 0;
   dirp->errcode = 0;
+
+#ifndef __LP64__
+  dirstream_loc_clear (&dirp->locs);
+#endif
+
 #if IS_IN (libc)
   __libc_lock_unlock (dirp->lock);
 #endif
diff --git a/sysdeps/unix/sysv/linux/seekdir.c b/sysdeps/unix/sysv/linux/seekdir.c
index 3c30520928..5f39ef9eef 100644
--- a/sysdeps/unix/sysv/linux/seekdir.c
+++ b/sysdeps/unix/sysv/linux/seekdir.c
@@ -22,14 +22,40 @@
 #include <dirstream.h>
 
 /* Seek to position POS in DIRP.  */
-/* XXX should be __seekdir ? */
 void
 seekdir (DIR *dirp, long int pos)
 {
+  off64_t filepos;
+
   __libc_lock_lock (dirp->lock);
-  (void) __lseek (dirp->fd, pos, SEEK_SET);
-  dirp->size = 0;
-  dirp->offset = 0;
-  dirp->filepos = pos;
+
+#ifndef __LP64__
+  union dirstream_packed dsp;
+
+  dsp.l = pos;
+
+  if (dsp.p.is_packed == 1)
+    filepos = dsp.p.info;
+  else
+    {
+      size_t index = dsp.p.info;
+
+      if (index >= dirstream_loc_size (&dirp->locs))
+	return;
+      struct dirstream_loc *loc = dirstream_loc_at (&dirp->locs, index);
+      filepos = loc->filepos;
+    }
+#else
+  filepos = pos;
+#endif
+
+  if (dirp->filepos != filepos)
+    {
+      __lseek64 (dirp->fd, filepos, SEEK_SET);
+      dirp->filepos = filepos;
+      dirp->offset = 0;
+      dirp->size = 0;
+    }
+
   __libc_lock_unlock (dirp->lock);
 }
diff --git a/sysdeps/unix/sysv/linux/telldir.c b/sysdeps/unix/sysv/linux/telldir.c
index 57d435ed21..bb33626fa4 100644
--- a/sysdeps/unix/sysv/linux/telldir.c
+++ b/sysdeps/unix/sysv/linux/telldir.c
@@ -18,16 +18,59 @@
 #include <dirent.h>
 
 #include <dirstream.h>
+#include <telldir.h>
 
 /* Return the current position of DIRP.  */
 long int
 telldir (DIR *dirp)
 {
-  long int ret;
+#ifndef __LP64__
+  /* If the directory position fits in the packet structure returns it.
+     Otherwise, check if the position is already been recorded in the
+     dynamic array.  If not, add the new record.  */
+
+  union dirstream_packed dsp;
+  size_t i;
 
   __libc_lock_lock (dirp->lock);
-  ret = dirp->filepos;
+
+  if (dirp->filepos < (1U << 31))
+    {
+      dsp.p.is_packed = 1;
+      dsp.p.info = dirp->filepos;
+      goto out;
+    }
+
+  dsp.l = -1;
+
+  for (i = 0; i < dirstream_loc_size (&dirp->locs); i++)
+    {
+      struct dirstream_loc *loc = dirstream_loc_at (&dirp->locs, i);
+      if (loc->filepos == dirp->filepos)
+	break;
+    }
+  if (i == dirstream_loc_size (&dirp->locs))
+    {
+      dirstream_loc_add (&dirp->locs,
+	(struct dirstream_loc) { dirp->filepos });
+      if (dirstream_loc_has_failed (&dirp->locs))
+	goto out;
+    }
+
+  dsp.p.is_packed = 0;
+  /* This assignment might overflow, however most likely ENOMEM would happen
+     long before.  */
+  dsp.p.info = i;
+
+out:
   __libc_lock_unlock (dirp->lock);
 
+  return dsp.l;
+#else
+  long int ret;
+  __libc_lock_lock (dirp->lock);
+  ret = dirp->filepos;
+  __libc_lock_unlock (dirp->lock);
   return ret;
+#endif
 }
diff --git a/sysdeps/unix/sysv/linux/telldir.h b/sysdeps/unix/sysv/linux/telldir.h
new file mode 100644
index 0000000000..7c45886341
--- /dev/null
+++ b/sysdeps/unix/sysv/linux/telldir.h
@@ -0,0 +1,64 @@
+/* Linux internal telldir definitions.
+   Copyright (C) 2020 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <https://www.gnu.org/licenses/>.  */
+
+#ifndef _TELLDIR_H
+#define _TELLDIR_H 1
+
+#ifndef __LP64__
+
+/* On platforms where long int is smaller than off64_t this is how the
+   returned value is encoded and returned by 'telldir'.  If the directory
+   offset can be enconded in 31 bits it is returned in the 'info' member
+   with 'is_packed' set to 1.
+
+   Otherwise, the 'info' member describes an index in a dynamic array at
+   'DIR' structure.  */
+
+union dirstream_packed
+{
+  long int l;
+  struct
+  {
+    unsigned long is_packed:1;
+    unsigned long info:31;
+  } p;
+};
+
+_Static_assert (sizeof (long int) == sizeof (union dirstream_packed),
+		"sizeof (long int) != sizeof (union dirstream_packed)");
+
+/* telldir will mantain a list of offsets that describe the obtained diretory
+   position if it can fit this information in the returned 'dirstream_packed'
+   struct.  */
+
+struct dirstream_loc
+{
+  off64_t filepos;
+};
+
+# define DYNARRAY_STRUCT  dirstream_loc_t
+# define DYNARRAY_ELEMENT struct dirstream_loc
+# define DYNARRAY_PREFIX  dirstream_loc_
+# include <malloc/dynarray-skeleton.c>
+#else
+
+_Static_assert (sizeof (long int) == sizeof (off64_t),
+		"sizeof (long int) != sizeof (off64_t)");
+#endif /* __LP64__  */
+
+#endif /* _TELLDIR_H  */
-- 
2.25.1

[PATCH v3 4/7] linux: Add __readdir64_unlocked

[Adhemerval Zanella via Libc-alpha]

And use it on readdir_r implementation.

Checked on i686-linux-gnu.
---
 include/dirent.h                      |  1 +
 sysdeps/unix/sysv/linux/readdir64.c   | 20 +++++--
 sysdeps/unix/sysv/linux/readdir64_r.c | 80 ++++++---------------------
 3 files changed, 33 insertions(+), 68 deletions(-)

diff --git a/include/dirent.h b/include/dirent.h
index 8325a19e5f..79c8fce969 100644
--- a/include/dirent.h
+++ b/include/dirent.h
@@ -21,6 +21,7 @@ extern DIR *__fdopendir (int __fd) attribute_hidden;
 extern int __closedir (DIR *__dirp) attribute_hidden;
 extern struct dirent *__readdir (DIR *__dirp) attribute_hidden;
 extern struct dirent *__readdir_unlocked (DIR *__dirp) attribute_hidden;
+extern struct dirent64 *__readdir64_unlocked (DIR *__dirp) attribute_hidden;
 extern struct dirent64 *__readdir64 (DIR *__dirp);
 libc_hidden_proto (__readdir64)
 extern int __readdir_r (DIR *__dirp, struct dirent *__entry,
diff --git a/sysdeps/unix/sysv/linux/readdir64.c b/sysdeps/unix/sysv/linux/readdir64.c
index d35a4595f6..ee41807b97 100644
--- a/sysdeps/unix/sysv/linux/readdir64.c
+++ b/sysdeps/unix/sysv/linux/readdir64.c
@@ -28,14 +28,10 @@
 
 /* Read a directory entry from DIRP.  */
 struct dirent64 *
-__readdir64 (DIR *dirp)
+__readdir64_unlocked (DIR *dirp)
 {
   const int saved_errno = errno;
 
-#if IS_IN (libc)
-  __libc_lock_lock (dirp->lock);
-#endif
-
   if (dirp->offset >= dirp->size)
     {
       /* We've emptied out our buffer.  Refill it.  */
@@ -65,6 +61,20 @@ __readdir64 (DIR *dirp)
 
   dirp->filepos = dp->d_off;
 
+  return dp;
+}
+
+struct dirent64 *
+__readdir64 (DIR *dirp)
+{
+  struct dirent64 *dp;
+
+#if IS_IN (libc)
+  __libc_lock_lock (dirp->lock);
+#endif
+
+  dp = __readdir64_unlocked (dirp);
+
 #if IS_IN (libc)
   __libc_lock_unlock (dirp->lock);
 #endif
diff --git a/sysdeps/unix/sysv/linux/readdir64_r.c b/sysdeps/unix/sysv/linux/readdir64_r.c
index c587787417..e5ef7be4c2 100644
--- a/sysdeps/unix/sysv/linux/readdir64_r.c
+++ b/sysdeps/unix/sysv/linux/readdir64_r.c
@@ -32,89 +32,43 @@ __readdir64_r (DIR *dirp, struct dirent64 *entry, struct dirent64 **result)
 {
   struct dirent64 *dp;
   size_t reclen;
-  const int saved_errno = errno;
-  int ret;
 
   __libc_lock_lock (dirp->lock);
-
-  do
+  while (1)
     {
-      if (dirp->offset >= dirp->size)
-	{
-	  /* We've emptied out our buffer.  Refill it.  */
-
-	  size_t maxread = dirp->allocation;
-	  ssize_t bytes;
-
-	  maxread = dirp->allocation;
-
-	  bytes = __getdents64 (dirp->fd, dirp->data, maxread);
-	  if (bytes <= 0)
-	    {
-	      /* On some systems getdents fails with ENOENT when the
-		 open directory has been rmdir'd already.  POSIX.1
-		 requires that we treat this condition like normal EOF.  */
-	      if (bytes < 0 && errno == ENOENT)
-		{
-		  bytes = 0;
-		  __set_errno (saved_errno);
-		}
-	      if (bytes < 0)
-		dirp->errcode = errno;
-
-	      dp = NULL;
-	      break;
-	    }
-	  dirp->size = (size_t) bytes;
-
-	  /* Reset the offset into the buffer.  */
-	  dirp->offset = 0;
-	}
-
-      dp = (struct dirent64 *) &dirp->data[dirp->offset];
+      dp = __readdir64_unlocked (dirp);
+      if (dp == NULL)
+	break;
 
       reclen = dp->d_reclen;
+      if (reclen <= offsetof (struct dirent64, d_name) + NAME_MAX + 1)
+	break;
 
-      dirp->offset += reclen;
-
-      dirp->filepos = dp->d_off;
-
-      if (reclen > offsetof (struct dirent64, d_name) + NAME_MAX + 1)
+      /* The record is very long.  It could still fit into the caller-supplied
+	 buffer if we can skip padding at the end.  */
+      size_t namelen = _D_EXACT_NAMLEN (dp);
+      if (namelen <= NAME_MAX)
 	{
-	  /* The record is very long.  It could still fit into the
-	     caller-supplied buffer if we can skip padding at the
-	     end.  */
-	  size_t namelen = _D_EXACT_NAMLEN (dp);
-	  if (namelen <= NAME_MAX)
-	    reclen = offsetof (struct dirent64, d_name) + namelen + 1;
-	  else
-	    {
-	      /* The name is too long.  Ignore this file.  */
-	      dirp->errcode = ENAMETOOLONG;
-	      dp->d_ino = 0;
-	      continue;
-	    }
+	  reclen = offsetof (struct dirent64, d_name) + namelen + 1;
+	  break;
 	}
 
-      /* Skip deleted and ignored files.  */
+      /* The name is too long.  Ignore this file.  */
+      dirp->errcode = ENAMETOOLONG;
+      dp->d_ino = 0;
     }
-  while (dp->d_ino == 0);
 
   if (dp != NULL)
     {
       *result = memcpy (entry, dp, reclen);
       entry->d_reclen = reclen;
-      ret = 0;
     }
   else
-    {
-      *result = NULL;
-      ret = dirp->errcode;
-    }
+    *result = NULL;
 
   __libc_lock_unlock (dirp->lock);
 
-  return ret;
+  return dp != NULL ? 0 : dirp->errcode;
 }
 
 
-- 
2.25.1

[PATCH v3 5/7] linux: Add __old_readdir64_unlocked

[Adhemerval Zanella via Libc-alpha]

And use it __old_readdir64_r.

Checked on i686-linux-gnu.
---
 sysdeps/unix/sysv/linux/olddirent.h   |  2 +
 sysdeps/unix/sysv/linux/readdir64.c   | 21 +++++--
 sysdeps/unix/sysv/linux/readdir64_r.c | 79 ++++++---------------------
 3 files changed, 35 insertions(+), 67 deletions(-)

diff --git a/sysdeps/unix/sysv/linux/olddirent.h b/sysdeps/unix/sysv/linux/olddirent.h
index 2d2559c1e9..b118b0ef96 100644
--- a/sysdeps/unix/sysv/linux/olddirent.h
+++ b/sysdeps/unix/sysv/linux/olddirent.h
@@ -32,6 +32,8 @@ struct __old_dirent64
 /* Now define the internal interfaces.  */
 extern struct __old_dirent64 *__old_readdir64 (DIR *__dirp);
 libc_hidden_proto (__old_readdir64);
+extern struct __old_dirent64 *__old_readdir64_unlocked (DIR *__dirp)
+        attribute_hidden;
 extern int __old_readdir64_r (DIR *__dirp, struct __old_dirent64 *__entry,
 			  struct __old_dirent64 **__result);
 extern __ssize_t __old_getdents64 (int __fd, char *__buf, size_t __nbytes)
diff --git a/sysdeps/unix/sysv/linux/readdir64.c b/sysdeps/unix/sysv/linux/readdir64.c
index ee41807b97..acd190733c 100644
--- a/sysdeps/unix/sysv/linux/readdir64.c
+++ b/sysdeps/unix/sysv/linux/readdir64.c
@@ -101,15 +101,11 @@ versioned_symbol (libc, __readdir64, readdir64, GLIBC_2_2);
 
 attribute_compat_text_section
 struct __old_dirent64 *
-__old_readdir64 (DIR *dirp)
+__old_readdir64_unlocked (DIR *dirp)
 {
   struct __old_dirent64 *dp;
   int saved_errno = errno;
 
-#if IS_IN (libc)
-  __libc_lock_lock (dirp->lock);
-#endif
-
   do
     {
       size_t reclen;
@@ -153,6 +149,21 @@ __old_readdir64 (DIR *dirp)
       /* Skip deleted files.  */
     } while (dp->d_ino == 0);
 
+  return dp;
+}
+
+attribute_compat_text_section
+struct __old_dirent64 *
+__old_readdir64 (DIR *dirp)
+{
+  struct __old_dirent64 *dp;
+
+#if IS_IN (libc)
+  __libc_lock_lock (dirp->lock);
+#endif
+
+  dp = __old_readdir64_unlocked (dirp);
+
 #if IS_IN (libc)
   __libc_lock_unlock (dirp->lock);
 #endif
diff --git a/sysdeps/unix/sysv/linux/readdir64_r.c b/sysdeps/unix/sysv/linux/readdir64_r.c
index e5ef7be4c2..133c534a69 100644
--- a/sysdeps/unix/sysv/linux/readdir64_r.c
+++ b/sysdeps/unix/sysv/linux/readdir64_r.c
@@ -91,89 +91,44 @@ __old_readdir64_r (DIR *dirp, struct __old_dirent64 *entry,
 {
   struct __old_dirent64 *dp;
   size_t reclen;
-  const int saved_errno = errno;
-  int ret;
 
   __libc_lock_lock (dirp->lock);
 
-  do
+  while (1)
     {
-      if (dirp->offset >= dirp->size)
-	{
-	  /* We've emptied out our buffer.  Refill it.  */
-
-	  size_t maxread = dirp->allocation;
-	  ssize_t bytes;
-
-	  maxread = dirp->allocation;
-
-	  bytes = __old_getdents64 (dirp->fd, dirp->data, maxread);
-	  if (bytes <= 0)
-	    {
-	      /* On some systems getdents fails with ENOENT when the
-		 open directory has been rmdir'd already.  POSIX.1
-		 requires that we treat this condition like normal EOF.  */
-	      if (bytes < 0 && errno == ENOENT)
-		{
-		  bytes = 0;
-		  __set_errno (saved_errno);
-		}
-	      if (bytes < 0)
-		dirp->errcode = errno;
-
-	      dp = NULL;
-	      break;
-	    }
-	  dirp->size = (size_t) bytes;
-
-	  /* Reset the offset into the buffer.  */
-	  dirp->offset = 0;
-	}
-
-      dp = (struct __old_dirent64 *) &dirp->data[dirp->offset];
+      dp = __old_readdir64_unlocked (dirp);
+      if (dp == NULL)
+	break;
 
       reclen = dp->d_reclen;
+      if (reclen <= offsetof (struct __old_dirent64, d_name) + NAME_MAX + 1)
+	break;
 
-      dirp->offset += reclen;
-
-      dirp->filepos = dp->d_off;
-
-      if (reclen > offsetof (struct __old_dirent64, d_name) + NAME_MAX + 1)
+      /* The record is very long.  It could still fit into the caller-supplied
+	 buffer if we can skip padding at the end.  */
+      size_t namelen = _D_EXACT_NAMLEN (dp);
+      if (namelen <= NAME_MAX)
 	{
-	  /* The record is very long.  It could still fit into the
-	     caller-supplied buffer if we can skip padding at the
-	     end.  */
-	  size_t namelen = _D_EXACT_NAMLEN (dp);
-	  if (namelen <= NAME_MAX)
-	    reclen = offsetof (struct __old_dirent64, d_name) + namelen + 1;
-	  else
-	    {
-	      /* The name is too long.  Ignore this file.  */
-	      dirp->errcode = ENAMETOOLONG;
-	      dp->d_ino = 0;
-	      continue;
-	    }
+	  reclen = offsetof (struct dirent64, d_name) + namelen + 1;
+	  break;
 	}
 
-      /* Skip deleted and ignored files.  */
+      /* The name is too long.  Ignore this file.  */
+      dirp->errcode = ENAMETOOLONG;
+      dp->d_ino = 0;
     }
-  while (dp->d_ino == 0);
 
   if (dp != NULL)
     {
       *result = memcpy (entry, dp, reclen);
       entry->d_reclen = reclen;
-      ret = 0;
     }
   else
-    {
-      *result = NULL;
-      ret = dirp->errcode;
-    }
+    *result = NULL;
 
   __libc_lock_unlock (dirp->lock);
 
-  return ret;
+  return dp != NULL ? 0 : dirp->errcode;
 }
 
 compat_symbol (libc, __old_readdir64_r, readdir64_r, GLIBC_2_1);
-- 
2.25.1

[PATCH v3 6/7] linux: Use getdents64 on readdir64 compat implementation

[Adhemerval Zanella via Libc-alpha]

It uses a similar strategy from the non-LFS readdir that also
uses getdents64 internally and uses a translation buffer to return
the compat readdir64 entry.

It allows to remove __old_getdents64.

Checked on i686-linux-gnu.
---
 sysdeps/unix/sysv/linux/getdents64.c |  93 ------------------------
 sysdeps/unix/sysv/linux/olddirent.h  |   2 -
 sysdeps/unix/sysv/linux/opendir.c    |  15 +++-
 sysdeps/unix/sysv/linux/readdir64.c  | 104 +++++++++++++++++----------
 4 files changed, 79 insertions(+), 135 deletions(-)

diff --git a/sysdeps/unix/sysv/linux/getdents64.c b/sysdeps/unix/sysv/linux/getdents64.c
index 75892c2823..b8682c3472 100644
--- a/sysdeps/unix/sysv/linux/getdents64.c
+++ b/sysdeps/unix/sysv/linux/getdents64.c
@@ -36,97 +36,4 @@ weak_alias (__getdents64, getdents64)
 
 #if _DIRENT_MATCHES_DIRENT64
 strong_alias (__getdents64, __getdents)
-#else
-# include <shlib-compat.h>
-
-# if SHLIB_COMPAT(libc, GLIBC_2_1, GLIBC_2_2)
-#  include <olddirent.h>
-#  include <unistd.h>
-
-static ssize_t
-handle_overflow (int fd, __off64_t offset, ssize_t count)
-{
-  /* If this is the first entry in the buffer, we can report the
-     error.  */
-  if (offset == 0)
-    {
-      __set_errno (EOVERFLOW);
-      return -1;
-    }
-
-  /* Otherwise, seek to the overflowing entry, so that the next call
-     will report the error, and return the data read so far.  */
-  if (__lseek64 (fd, offset, SEEK_SET) != 0)
-    return -1;
-  return count;
-}
-
-ssize_t
-__old_getdents64 (int fd, char *buf, size_t nbytes)
-{
-  /* We do not move the individual directory entries.  This is only
-     possible if the target type (struct __old_dirent64) is smaller
-     than the source type.  */
-  _Static_assert (offsetof (struct __old_dirent64, d_name)
-		  <= offsetof (struct dirent64, d_name),
-		  "__old_dirent64 is larger than dirent64");
-  _Static_assert (__alignof__ (struct __old_dirent64)
-		  <= __alignof__ (struct dirent64),
-		  "alignment of __old_dirent64 is larger than dirent64");
-
-  ssize_t retval = INLINE_SYSCALL_CALL (getdents64, fd, buf, nbytes);
-  if (retval > 0)
-    {
-      /* This is the marker for the first entry.  Offset 0 is reserved
-	 for the first entry (see rewinddir).  Here, we use it as a
-	 marker for the first entry in the buffer.  We never actually
-	 seek to offset 0 because handle_overflow reports the error
-	 directly, so it does not matter that the offset is incorrect
-	 if entries have been read from the descriptor before (so that
-	 the descriptor is not actually at offset 0).  */
-      __off64_t previous_offset = 0;
-
-      char *p = buf;
-      char *end = buf + retval;
-      while (p < end)
-	{
-	  struct dirent64 *source = (struct dirent64 *) p;
-
-	  /* Copy out the fixed-size data.  */
-	  __ino_t ino = source->d_ino;
-	  __off64_t offset = source->d_off;
-	  unsigned int reclen = source->d_reclen;
-	  unsigned char type = source->d_type;
-
-	  /* Check for ino_t overflow.  */
-	  if (__glibc_unlikely (ino != source->d_ino))
-	    return handle_overflow (fd, previous_offset, p - buf);
-
-	  /* Convert to the target layout.  Use a separate struct and
-	     memcpy to side-step aliasing issues.  */
-	  struct __old_dirent64 result;
-	  result.d_ino = ino;
-	  result.d_off = offset;
-	  result.d_reclen = reclen;
-	  result.d_type = type;
-
-	  /* Write the fixed-sized part of the result to the
-	     buffer.  */
-	  size_t result_name_offset = offsetof (struct __old_dirent64, d_name);
-	  memcpy (p, &result, result_name_offset);
-
-	  /* Adjust the position of the name if necessary.  Copy
-	     everything until the end of the record, including the
-	     terminating NUL byte.  */
-	  if (result_name_offset != offsetof (struct dirent64, d_name))
-	    memmove (p + result_name_offset, source->d_name,
-		     reclen - offsetof (struct dirent64, d_name));
-
-	  p += reclen;
-	  previous_offset = offset;
-	}
-     }
-  return retval;
-}
-# endif /* SHLIB_COMPAT(libc, GLIBC_2_1, GLIBC_2_2)  */
 #endif /* _DIRENT_MATCHES_DIRENT64  */
diff --git a/sysdeps/unix/sysv/linux/olddirent.h b/sysdeps/unix/sysv/linux/olddirent.h
index b118b0ef96..63d8e97868 100644
--- a/sysdeps/unix/sysv/linux/olddirent.h
+++ b/sysdeps/unix/sysv/linux/olddirent.h
@@ -36,8 +36,6 @@ extern struct __old_dirent64 *__old_readdir64_unlocked (DIR *__dirp)
         attribute_hidden;
 extern int __old_readdir64_r (DIR *__dirp, struct __old_dirent64 *__entry,
 			  struct __old_dirent64 **__result);
-extern __ssize_t __old_getdents64 (int __fd, char *__buf, size_t __nbytes)
-	attribute_hidden;
 int __old_scandir64 (const char * __dir,
 		     struct __old_dirent64 *** __namelist,
 		     int (*__selector) (const struct __old_dirent64 *),
diff --git a/sysdeps/unix/sysv/linux/opendir.c b/sysdeps/unix/sysv/linux/opendir.c
index 28fd55341e..3a08b76c0a 100644
--- a/sysdeps/unix/sysv/linux/opendir.c
+++ b/sysdeps/unix/sysv/linux/opendir.c
@@ -23,6 +23,11 @@
 
 #include <not-cancel.h>
 
+#include <shlib-compat.h>
+#if SHLIB_COMPAT(libc, GLIBC_2_1, GLIBC_2_2)
+# include <olddirent.h>
+#endif
+
 enum {
   opendir_oflags = O_RDONLY|O_NDELAY|O_DIRECTORY|O_LARGEFILE|O_CLOEXEC
 };
@@ -127,7 +132,15 @@ __alloc_dir (int fd, bool close_fd, int flags, const struct stat64 *statp)
      expand the buffer if required.  */
   enum
     {
-      tbuffer_size = sizeof (struct dirent) + NAME_MAX + 1
+      tbuffer_size =
+# if SHLIB_COMPAT(libc, GLIBC_2_1, GLIBC_2_2)
+      /* This is used on compat readdir64.  */
+		     MAX (sizeof (struct dirent),
+			  sizeof (struct __old_dirent64))
+# else
+		     sizeof (struct dirent)
+# endif
+                     + NAME_MAX + 1
     };
   dirp->tbuffer = malloc (tbuffer_size);
   if (dirp->tbuffer == NULL)
diff --git a/sysdeps/unix/sysv/linux/readdir64.c b/sysdeps/unix/sysv/linux/readdir64.c
index acd190733c..799708174a 100644
--- a/sysdeps/unix/sysv/linux/readdir64.c
+++ b/sysdeps/unix/sysv/linux/readdir64.c
@@ -99,57 +99,83 @@ versioned_symbol (libc, __readdir64, readdir64, GLIBC_2_2);
 # if SHLIB_COMPAT(libc, GLIBC_2_1, GLIBC_2_2)
 #  include <olddirent.h>
 
+/* Translate the DP64 entry to the old LFS one in the translation buffer
+   at dirstream DS.  Return true is the translation was possible or
+   false if either an internal fields can be represented in the non-LFS
+   entry or if the translation can not be resized.  */
+static bool
+dirstream_old_entry (struct __dirstream *ds, const struct dirent64 *dp64)
+{
+  /* Check for overflow.  */
+  ino_t d_ino = dp64->d_ino;
+  if (d_ino != dp64->d_ino)
+    return false;
+
+  /* Expand the translation buffer to hold the new namesize.  */
+  size_t d_reclen = sizeof (struct __old_dirent64)
+		    + dp64->d_reclen - offsetof (struct dirent64, d_name);
+  if (d_reclen > ds->tbuffer_size)
+    {
+      char *newbuffer = realloc (ds->tbuffer, d_reclen);
+      if (newbuffer == NULL)
+	return false;
+      ds->tbuffer = newbuffer;
+      ds->tbuffer_size = d_reclen;
+    }
+
+  struct __old_dirent64 *olddp64 = (struct __old_dirent64 *) ds->tbuffer;
+
+  olddp64->d_off = dp64->d_off;
+  olddp64->d_ino = dp64->d_ino;
+  olddp64->d_reclen = dp64->d_reclen;
+  olddp64->d_type = dp64->d_type;
+  memcpy (olddp64->d_name, dp64->d_name,
+	  dp64->d_reclen - offsetof (struct dirent64, d_name));
+
+  return true;
+}
+
 attribute_compat_text_section
 struct __old_dirent64 *
 __old_readdir64_unlocked (DIR *dirp)
 {
-  struct __old_dirent64 *dp;
-  int saved_errno = errno;
+  const int saved_errno = errno;
 
-  do
+  if (dirp->offset >= dirp->size)
     {
-      size_t reclen;
-
-      if (dirp->offset >= dirp->size)
+      /* We've emptied out our buffer.  Refill it.  */
+      ssize_t bytes = __getdents64 (dirp->fd, dirp->data, dirp->allocation);
+      if (bytes <= 0)
 	{
-	  /* We've emptied out our buffer.  Refill it.  */
-
-	  size_t maxread = dirp->allocation;
-	  ssize_t bytes;
-
-	  bytes = __old_getdents64 (dirp->fd, dirp->data, maxread);
-	  if (bytes <= 0)
-	    {
-	      /* On some systems getdents fails with ENOENT when the
-		 open directory has been rmdir'd already.  POSIX.1
-		 requires that we treat this condition like normal EOF.  */
-	      if (bytes < 0 && errno == ENOENT)
-		bytes = 0;
-
-	      /* Don't modifiy errno when reaching EOF.  */
-	      if (bytes == 0)
-		__set_errno (saved_errno);
-	      dp = NULL;
-	      break;
-	    }
-	  dirp->size = (size_t) bytes;
-
-	  /* Reset the offset into the buffer.  */
-	  dirp->offset = 0;
-	}
-
-      dp = (struct __old_dirent64 *) &dirp->data[dirp->offset];
+	  /* On some systems getdents fails with ENOENT when the
+	     open directory has been rmdir'd already.  POSIX.1
+	     requires that we treat this condition like normal EOF.  */
+	  if (bytes < 0 && errno == ENOENT)
+	    bytes = 0;
 
-      reclen = dp->d_reclen;
+	  /* Don't modifiy errno when reaching EOF.  */
+	  if (bytes == 0)
+	    __set_errno (saved_errno);
+	  return NULL;
+	}
+      dirp->size = bytes;
 
-      dirp->offset += reclen;
+      /* Reset the offset into the buffer.  */
+      dirp->offset = 0;
+    }
 
-      dirp->filepos = dp->d_off;
+  struct dirent64 *dp64 = (struct dirent64 *) &dirp->data[dirp->offset];
+  dirp->offset += dp64->d_reclen;
 
-      /* Skip deleted files.  */
-    } while (dp->d_ino == 0);
+  /* Skip entries which might overflow d_ino or for memory allocation failure
+     in case of large file names.  */
+  if (dirstream_old_entry (dirp, dp64))
+    {
+      dirp->filepos = dp64->d_off;
+      return (struct __old_dirent64 *) dirp->tbuffer;
+    }
 
-  return dp;
+  return NULL;
 }
 
 attribute_compat_text_section
-- 
2.25.1

[PATCH v3 7/7] dirent: Deprecate getdirentries

[Adhemerval Zanella via Libc-alpha]

The interface has some issues:

  1. It is build on top getdents on Linux and requires handling
     non-LFS call using LFS getdents.

  2. It is not wildly used and the non-LFS support is as problematic
     as non-LFS readdir.  glibc only exports the LFS getdents.

  3. It is not a direct replacement over BSD since on some plataform
     its signature has changed (FreeBSD 11, for instance, used to
     set the offset as a 'long' and changed to 'off_t' on version 12).

The idea is to eventually move the symbols to compat ones.
---
 NEWS                             |  3 +++
 dirent/dirent.h                  | 14 ++++++++++----
 sysdeps/unix/sysv/linux/Makefile |  3 +++
 3 files changed, 16 insertions(+), 4 deletions(-)

diff --git a/NEWS b/NEWS
index 3c1e509744..63623dcee2 100644
--- a/NEWS
+++ b/NEWS
@@ -47,6 +47,9 @@ Deprecated and removed features, and other changes affecting compatibility:
   as a compatibility symbol.  Applications should use the getrlimit or
   prlimit.
 
+* The function getdirentries is now deprecated, applications should use
+  either getdents64, readdir64 or readdir.
+
 Changes to build and runtime requirements:
 
 * On Linux, the system administrator needs to configure /dev/pts with
diff --git a/dirent/dirent.h b/dirent/dirent.h
index 92d0925047..62bb77b4e1 100644
--- a/dirent/dirent.h
+++ b/dirent/dirent.h
@@ -348,29 +348,35 @@ extern int alphasort64 (const struct dirent64 **__e1,
 /* Read directory entries from FD into BUF, reading at most NBYTES.
    Reading starts at offset *BASEP, and *BASEP is updated with the new
    position after reading.  Returns the number of bytes read; zero when at
-   end of directory; or -1 for errors.  */
+   end of directory; or -1 for errors.
+   This is deprecated and getdents64 or readdir should be used instead.  */
 # ifndef __USE_FILE_OFFSET64
 extern __ssize_t getdirentries (int __fd, char *__restrict __buf,
 				size_t __nbytes,
 				__off_t *__restrict __basep)
-     __THROW __nonnull ((2, 4));
+     __THROW __nonnull ((2, 4))
+     __attribute_deprecated_msg__ ("Use getdents64 instead");
 # else
 #  ifdef __REDIRECT
 extern __ssize_t __REDIRECT_NTH (getdirentries,
 				 (int __fd, char *__restrict __buf,
 				  size_t __nbytes,
 				  __off64_t *__restrict __basep),
-				 getdirentries64) __nonnull ((2, 4));
+				 getdirentries64)
+     __THROW __nonnull ((2, 4))
+     __attribute_deprecated_msg__ ("Use getdents64 instead");
 #  else
 #   define getdirentries getdirentries64
 #  endif
 # endif
 
 # ifdef __USE_LARGEFILE64
+/* This is deprecated and getdents64 or readdir64 should be used instead.  */
 extern __ssize_t getdirentries64 (int __fd, char *__restrict __buf,
 				  size_t __nbytes,
 				  __off64_t *__restrict __basep)
-     __THROW __nonnull ((2, 4));
+     __THROW __nonnull ((2, 4))
+     __attribute_deprecated_msg__ ("Use getdents64 instead");
 # endif
 #endif /* Use misc.  */
 
diff --git a/sysdeps/unix/sysv/linux/Makefile b/sysdeps/unix/sysv/linux/Makefile
index 09604e128b..242b07734f 100644
--- a/sysdeps/unix/sysv/linux/Makefile
+++ b/sysdeps/unix/sysv/linux/Makefile
@@ -262,6 +262,9 @@ ifeq ($(subdir),dirent)
 sysdep_routines += getdirentries getdirentries64
 tests += tst-getdents64
 tests-internal += tst-readdir64-compat
+
+# Avoid the warning for the weak_alias for _DIRENT_MATCHES_DIRENT64
+CFLAGS-getdirentries64.c = -Wno-deprecated-declarations
 endif
 
 ifeq ($(subdir),nis)
-- 
2.25.1

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[John Paul Adrian Glaubitz]

Hi!

On 10/21/20 4:15 PM, Adhemerval Zanella wrote:
> This is updated version [1], rebased against master add with some
> suggestion from previous review.  The main changes are:
> 
>   - The translation buffer is allocated as an auxiliarry one (instead
>     of a reserved spacefrom the dirstream) and readdir might reallocate
>     it if it finds a large entry.
> 
>   - Failures in non-LFS readdir (such as d_ino/d_off overflow or failure
>     in expand the translation buffer) is ignored.  This allows code that
>     does not handle possible failure in readdir to successful read the
>     rest of entries in the directory.
> 
>   - So minor fixes in dirent/tst-seekdir2.c
> 
>   - Added __attribute_deprecated_msg__ on getdirentries to redirect to
>     getdents64.
> 
> I have checked on x86_64-linux-gnu, powerpc64-linux-gnu and on some
> 32-bit ABIs arm-linux-gnueabihf, powerpc-linux-gnu, and sparcv9-linux-gnu.
Are there any news on this patch series? The bug is still present as far as I know.

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[John Paul Adrian Glaubitz]

Hello!

On 10/21/20 4:15 PM, Adhemerval Zanella wrote:
> This is updated version [1], rebased against master add with some
> suggestion from previous review.  The main changes are:
> 
>   - The translation buffer is allocated as an auxiliarry one (instead
>     of a reserved spacefrom the dirstream) and readdir might reallocate
>     it if it finds a large entry.
> 
>   - Failures in non-LFS readdir (such as d_ino/d_off overflow or failure
>     in expand the translation buffer) is ignored.  This allows code that
>     does not handle possible failure in readdir to successful read the
>     rest of entries in the directory.
> 
>   - So minor fixes in dirent/tst-seekdir2.c
> 
>   - Added __attribute_deprecated_msg__ on getdirentries to redirect to
>     getdents64.
> 
> I have checked on x86_64-linux-gnu, powerpc64-linux-gnu and on some
> 32-bit ABIs arm-linux-gnueabihf, powerpc-linux-gnu, and sparcv9-linux-gnu.

Are there any news on this patch series besides the v3 version? [1]

Thanks,
Adrian

> [1] https://patchwork.ozlabs.org/project/glibc/list/?series=209257

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[Adhemerval Zanella via Libc-alpha]

On 19/08/2021 13:07, John Paul Adrian Glaubitz wrote:
> Hello!
> 
> On 10/21/20 4:15 PM, Adhemerval Zanella wrote:
>> This is updated version [1], rebased against master add with some
>> suggestion from previous review.  The main changes are:
>>
>>   - The translation buffer is allocated as an auxiliarry one (instead
>>     of a reserved spacefrom the dirstream) and readdir might reallocate
>>     it if it finds a large entry.
>>
>>   - Failures in non-LFS readdir (such as d_ino/d_off overflow or failure
>>     in expand the translation buffer) is ignored.  This allows code that
>>     does not handle possible failure in readdir to successful read the
>>     rest of entries in the directory.
>>
>>   - So minor fixes in dirent/tst-seekdir2.c
>>
>>   - Added __attribute_deprecated_msg__ on getdirentries to redirect to
>>     getdents64.
>>
>> I have checked on x86_64-linux-gnu, powerpc64-linux-gnu and on some
>> 32-bit ABIs arm-linux-gnueabihf, powerpc-linux-gnu, and sparcv9-linux-gnu.
> 
> Are there any news on this patch series besides the v3 version? [1]

Unfortunately it is requires reviews. Since it only affects legacy
ABIs there is no much traction for fixing it.

> 
> Thanks,
> Adrian
> 
>> [1] https://patchwork.ozlabs.org/project/glibc/list/?series=209257
> 

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[John Paul Adrian Glaubitz]

Hi!

On 8/19/21 6:19 PM, Adhemerval Zanella wrote:
>> Are there any news on this patch series besides the v3 version? [1]
> 
> Unfortunately it is requires reviews. Since it only affects legacy
> ABIs there is no much traction for fixing it.

Would it help if I put a bounty on this bug report?

I still think this is an important fix as it breaks 32-bit qemu-user guests
on 64-bit systems and currently, I have to build the glibc package manually
for Debian whenever it's updated in the archives.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[Adhemerval Zanella via Libc-alpha]

On 19/08/2021 13:50, John Paul Adrian Glaubitz wrote:
> Hi!
> 
> On 8/19/21 6:19 PM, Adhemerval Zanella wrote:
>>> Are there any news on this patch series besides the v3 version? [1]
>>
>> Unfortunately it is requires reviews. Since it only affects legacy
>> ABIs there is no much traction for fixing it.
> 
> Would it help if I put a bounty on this bug report?

I am not sure in fact; the main problem is no one is really interested in
revise if my approach is the expected one and if the patch is what we want.

> 
> I still think this is an important fix as it breaks 32-bit qemu-user guests
> on 64-bit systems and currently, I have to build the glibc package manually
> for Debian whenever it's updated in the archives.
> 
> Adrian
> 

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[John Paul Adrian Glaubitz]

On 8/19/21 6:52 PM, Adhemerval Zanella wrote:> On 19/08/2021 13:50, John Paul Adrian Glaubitz wrote:
>> Hi!
>>
>> On 8/19/21 6:19 PM, Adhemerval Zanella wrote:
>>>> Are there any news on this patch series besides the v3 version? [1]
>>>
>>> Unfortunately it is requires reviews. Since it only affects legacy
>>> ABIs there is no much traction for fixing it.
>>
>> Would it help if I put a bounty on this bug report?
> 
> I am not sure in fact; the main problem is no one is really interested in
> revise if my approach is the expected one and if the patch is what we want.

Well, maybe a little incentive can help which is why I asked :-).

I don't have the knowledge myself to help with the issue, but I would greatly
appreciate it if it could finally get fixed since it has been a burden ever
since the 2.29 release when this API got changed.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[Florian Weimer via Libc-alpha]

* John Paul Adrian Glaubitz:

> On 8/19/21 6:52 PM, Adhemerval Zanella wrote:> On 19/08/2021 13:50, John Paul Adrian Glaubitz wrote:
>>> Hi!
>>>
>>> On 8/19/21 6:19 PM, Adhemerval Zanella wrote:
>>>>> Are there any news on this patch series besides the v3 version? [1]
>>>>
>>>> Unfortunately it is requires reviews. Since it only affects legacy
>>>> ABIs there is no much traction for fixing it.
>>>
>>> Would it help if I put a bounty on this bug report?
>> 
>> I am not sure in fact; the main problem is no one is really interested in
>> revise if my approach is the expected one and if the patch is what we want.
>
> Well, maybe a little incentive can help which is why I asked :-).

I can't speak for everyone, but for me offers of payment raise
complicated compliance issues.  It's hard to avoid the appearance of a
conflict of interest once third-party payments are involved.  That is
even true if the people involved never intended to claim any money.

Thanks,
Florian

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[John Paul Adrian Glaubitz]

On 8/19/21 7:15 PM, Florian Weimer wrote:
>> Well, maybe a little incentive can help which is why I asked :-).
> 
> I can't speak for everyone, but for me offers of payment raise
> complicated compliance issues.  It's hard to avoid the appearance of a
> conflict of interest once third-party payments are involved.  That is
> even true if the people involved never intended to claim any money.

So, what would be the best way to get this bug fixed then? I'm open to
any suggested path way.

It's an important fix for anyone using qemu-user in a common use case.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 0/7] Fix getdents{64} regression on some FS

[Dave Flogeras via Libc-alpha]

Hi all,

It seems that Peter Maydell of qemu fame thinks it could be solved in qemu
if/when the kernel people add an explicit 32bit mode to the filesystem API.

See the bug here:
https://gitlab.com/qemu-project/qemu/-/issues/263

And specifically, he mentions that Linus Walleij has submitted a
preliminary patch, which has yet to be looked at.

https://lore.kernel.org/qemu-
devel/20201117233928.255671-1-linus.walleij@linaro.org/

Not sure if this is an easier path or not, but it might be worth pinging
the kernel ML.

Best regards,
Dave

On Thu, Aug 19, 2021 at 5:30 PM John Paul Adrian Glaubitz <
glaubitz@physik.fu-berlin.de> wrote:

> On 8/19/21 7:15 PM, Florian Weimer wrote:
> >> Well, maybe a little incentive can help which is why I asked :-).
> >
> > I can't speak for everyone, but for me offers of payment raise
> > complicated compliance issues.  It's hard to avoid the appearance of a
> > conflict of interest once third-party payments are involved.  That is
> > even true if the people involved never intended to claim any money.
>
> So, what would be the best way to get this bug fixed then? I'm open to
> any suggested path way.
>
> It's an important fix for anyone using qemu-user in a common use case.
>
> Adrian
>
> --
>  .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer - glaubitz@debian.org
> `. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
>   `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913
>
>

Re: [PATCH v3 7/7] dirent: Deprecate getdirentries

[John Paul Adrian Glaubitz]

Hi Adhemerval!

On 10/21/20 16:15, Adhemerval Zanella wrote:
> The interface has some issues:
> 
>   1. It is build on top getdents on Linux and requires handling
>      non-LFS call using LFS getdents.
> 
>   2. It is not wildly used and the non-LFS support is as problematic
>      as non-LFS readdir.  glibc only exports the LFS getdents.
> 
>   3. It is not a direct replacement over BSD since on some plataform
>      its signature has changed (FreeBSD 11, for instance, used to
>      set the offset as a 'long' and changed to 'off_t' on version 12).
> 
> The idea is to eventually move the symbols to compat ones.
> ---
>  NEWS                             |  3 +++
>  dirent/dirent.h                  | 14 ++++++++++----
>  sysdeps/unix/sysv/linux/Makefile |  3 +++
>  3 files changed, 16 insertions(+), 4 deletions(-)
> 
> diff --git a/NEWS b/NEWS
> index 3c1e509744..63623dcee2 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -47,6 +47,9 @@ Deprecated and removed features, and other changes affecting compatibility:
>    as a compatibility symbol.  Applications should use the getrlimit or
>    prlimit.
>  
> +* The function getdirentries is now deprecated, applications should use
> +  either getdents64, readdir64 or readdir.
> +
>  Changes to build and runtime requirements:
>  
>  * On Linux, the system administrator needs to configure /dev/pts with
> diff --git a/dirent/dirent.h b/dirent/dirent.h
> index 92d0925047..62bb77b4e1 100644
> --- a/dirent/dirent.h
> +++ b/dirent/dirent.h
> @@ -348,29 +348,35 @@ extern int alphasort64 (const struct dirent64 **__e1,
>  /* Read directory entries from FD into BUF, reading at most NBYTES.
>     Reading starts at offset *BASEP, and *BASEP is updated with the new
>     position after reading.  Returns the number of bytes read; zero when at
> -   end of directory; or -1 for errors.  */
> +   end of directory; or -1 for errors.
> +   This is deprecated and getdents64 or readdir should be used instead.  */
>  # ifndef __USE_FILE_OFFSET64
>  extern __ssize_t getdirentries (int __fd, char *__restrict __buf,
>  				size_t __nbytes,
>  				__off_t *__restrict __basep)
> -     __THROW __nonnull ((2, 4));
> +     __THROW __nonnull ((2, 4))
> +     __attribute_deprecated_msg__ ("Use getdents64 instead");
>  # else
>  #  ifdef __REDIRECT
>  extern __ssize_t __REDIRECT_NTH (getdirentries,
>  				 (int __fd, char *__restrict __buf,
>  				  size_t __nbytes,
>  				  __off64_t *__restrict __basep),
> -				 getdirentries64) __nonnull ((2, 4));
> +				 getdirentries64)
> +     __THROW __nonnull ((2, 4))
> +     __attribute_deprecated_msg__ ("Use getdents64 instead");
>  #  else
>  #   define getdirentries getdirentries64
>  #  endif
>  # endif
>  
>  # ifdef __USE_LARGEFILE64
> +/* This is deprecated and getdents64 or readdir64 should be used instead.  */
>  extern __ssize_t getdirentries64 (int __fd, char *__restrict __buf,
>  				  size_t __nbytes,
>  				  __off64_t *__restrict __basep)
> -     __THROW __nonnull ((2, 4));
> +     __THROW __nonnull ((2, 4))
> +     __attribute_deprecated_msg__ ("Use getdents64 instead");
>  # endif
>  #endif /* Use misc.  */
>  
> diff --git a/sysdeps/unix/sysv/linux/Makefile b/sysdeps/unix/sysv/linux/Makefile
> index 09604e128b..242b07734f 100644
> --- a/sysdeps/unix/sysv/linux/Makefile
> +++ b/sysdeps/unix/sysv/linux/Makefile
> @@ -262,6 +262,9 @@ ifeq ($(subdir),dirent)
>  sysdep_routines += getdirentries getdirentries64
>  tests += tst-getdents64
>  tests-internal += tst-readdir64-compat
> +
> +# Avoid the warning for the weak_alias for _DIRENT_MATCHES_DIRENT64
> +CFLAGS-getdirentries64.c = -Wno-deprecated-declarations
>  endif
>  
>  ifeq ($(subdir),nis)

This particular patch seems to break the apt build:

FAILED: apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o 
/usr/bin/c++ -DAPT_DOMAIN=\"libapt-pkg6.0\" -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_WITH_GETLINE=1 -Dapt_pkg_EXPORTS -I/<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include -I/<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -specs=/usr/share/dpkg/pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -Wall -Wextra -Wcast-align -Wlogical-op -Wredundant-decls -Wmissing-declarations -Wunsafe-loop-optimizations -Wctor-dtor-privacy -Wdisabled-optimization -Winit-self -Wmissing-include-dirs -Wnoexcept -Wsign-promo -Wundef -Wdouble-promotion -Wsuggest-override -Werror=suggest-override -Werror=return-type -std=gnu++17 -MD -MT apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o -MF apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o.d -o apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o -c /<<PKGBUILDDIR>>/apt-pkg/acquire.cc
In file included from /usr/include/features.h:472,
                 from /usr/include/m68k-linux-gnu/c++/11/bits/os_defines.h:39,
                 from /usr/include/m68k-linux-gnu/c++/11/bits/c++config.h:586,
                 from /usr/include/c++/11/cstring:41,
                 from /<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg/hashes.h:17,
                 from /<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg/acquire.h:68,
                 from /<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg/acquire-item.h:22,
                 from /<<PKGBUILDDIR>>/apt-pkg/acquire.cc:17:
/usr/include/dirent.h:366:6: error: expected initializer before ‘noexcept’
  366 |      __THROW __nonnull ((2, 4))
      |      ^~~~~~~

See the full build log here:

> https://buildd.debian.org/status/fetch.php?pkg=apt&arch=m68k&ver=2.3.14&stamp=1641503494&raw=0

I have built and uploaded a custom version of the glibc package which includes your patch
so that the m68k and sh4 buildds based on qemu-user will not be affected by BZ #23960.

Do you have any suggestion how to fix this issue?

Thanks,
Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 7/7] dirent: Deprecate getdirentries

[Sam James via Libc-alpha]

[-- Attachment #1: Type: text/plain, Size: 2585 bytes --]



> On 6 Jan 2022, at 21:48, John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote:
> 
> Hi Adhemerval!
> 
> [snip]
> 
> FAILED: apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o
> /usr/bin/c++ -DAPT_DOMAIN=\"libapt-pkg6.0\" -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_WITH_GETLINE=1 -Dapt_pkg_EXPORTS -I/<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include -I/<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -specs=/usr/share/dpkg/pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -fvisibility=hidden -fvisibility-inlines-hidden -Wall -Wextra -Wcast-align -Wlogical-op -Wredundant-decls -Wmissing-declarations -Wunsafe-loop-optimizations -Wctor-dtor-privacy -Wdisabled-optimization -Winit-self -Wmissing-include-dirs -Wnoexcept -Wsign-promo -Wundef -Wdouble-promotion -Wsuggest-override -Werror=suggest-override -Werror=return-type -std=gnu++17 -MD -MT apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o -MF apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o.d -o apt-pkg/CMakeFiles/apt-pkg.dir/acquire.cc.o -c /<<PKGBUILDDIR>>/apt-pkg/acquire.cc
> In file included from /usr/include/features.h:472,
>                 from /usr/include/m68k-linux-gnu/c++/11/bits/os_defines.h:39,
>                 from /usr/include/m68k-linux-gnu/c++/11/bits/c++config.h:586,
>                 from /usr/include/c++/11/cstring:41,
>                 from /<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg/hashes.h:17,
>                 from /<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg/acquire.h:68,
>                 from /<<PKGBUILDDIR>>/obj-m68k-linux-gnu/include/apt-pkg/acquire-item.h:22,
>                 from /<<PKGBUILDDIR>>/apt-pkg/acquire.cc:17:
> /usr/include/dirent.h:366:6: error: expected initializer before ‘noexcept’
>  366 |      __THROW __nonnull ((2, 4))
>      |      ^~~~~~~
> 
> See the full build log here:
> 
>> https://buildd.debian.org/status/fetch.php?pkg=apt&arch=m68k&ver=2.3.14&stamp=1641503494&raw=0
> 
> I have built and uploaded a custom version of the glibc package which includes your patch
> so that the m68k and sh4 buildds based on qemu-user will not be affected by BZ #23960.
> 
> Do you have any suggestion how to fix this issue?

Try this version of the last commit: https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587 <https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587>.

We had the same problem in Gentoo and azanella pushed a newer version.

Best,
sam


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 618 bytes --]

Re: [PATCH v3 7/7] dirent: Deprecate getdirentries

[John Paul Adrian Glaubitz]

Hi Sam!

On 1/6/22 22:59, Sam James wrote:
>> Do you have any suggestion how to fix this issue?
> 
> Try this version of the last commit: https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587 <https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587>.
> 
> We had the same problem in Gentoo and azanella pushed a newer version.

Perfect, thank you! I will give it a try.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaubitz@debian.org
`. `'   Freie Universitaet Berlin - glaubitz@physik.fu-berlin.de
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 7/7] dirent: Deprecate getdirentries

[John Paul Adrian Glaubitz]

Hi!

On 1/6/22 22:59, Sam James wrote:
> Try this version of the last commit: https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587 <https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587>.
> 
> We had the same problem in Gentoo and azanella pushed a newer version.

The original problem seems to have returned. I can no longer build packages such as
boost1.74, openjdk-18 or globus-net-manager:

> https://buildd.debian.org/status/fetch.php?pkg=boost1.74&arch=m68k&ver=1.74.0-14&stamp=1651053703&raw=0
> https://buildd.debian.org/status/fetch.php?pkg=globus-net-manager&arch=m68k&ver=1.6-1%2Bb1&stamp=1651419562&raw=0
> https://buildd.debian.org/status/fetch.php?pkg=openjdk-18&arch=m68k&ver=18.0.1%2B10-1&stamp=1651537618&raw=0

I have not yet figured out what the problem is.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: [PATCH v3 7/7] dirent: Deprecate getdirentries

[Adhemerval Zanella via Libc-alpha]

On 10/05/2022 16:30, John Paul Adrian Glaubitz wrote:
> Hi!
> 
> On 1/6/22 22:59, Sam James wrote:
>> Try this version of the last commit: https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587 <https://github.com/zatrazz/glibc/commit/7b990b41ea23c0464c60e6a9f0707adfb46ae587>.
>>
>> We had the same problem in Gentoo and azanella pushed a newer version.
> 
> The original problem seems to have returned. I can no longer build packages such as
> boost1.74, openjdk-18 or globus-net-manager:
> 
>> https://buildd.debian.org/status/fetch.php?pkg=boost1.74&arch=m68k&ver=1.74.0-14&stamp=1651053703&raw=0
>> https://buildd.debian.org/status/fetch.php?pkg=globus-net-manager&arch=m68k&ver=1.6-1%2Bb1&stamp=1651419562&raw=0
>> https://buildd.debian.org/status/fetch.php?pkg=openjdk-18&arch=m68k&ver=18.0.1%2B10-1&stamp=1651537618&raw=0
> 
> I have not yet figured out what the problem is.

I can respin my patchset to fix the dirent on non-LFS, not sure if someone
will be interested to review it.