supersede: Avoid a failure when writing to /dev/null in Solaris zones

[Bruno Haible <bruno@clisp.org>]

Jörg Sonnenberger noticed that the new gnulib module 'supersede', when
writing to /dev/null in Solaris/Illumos zones, fails [1][2].

Per POSIX [3],
  open ("/dev/null", O_TRUNC | O_WRONLY)
and
  open ("/dev/null", O_CREAT | O_TRUNC | O_WRONLY)
should be equivalent. But in Solaris/Illumos zones, the second form is needed
because the first one fails with EINVAL.

Should we extend the 'open' module to cover this bug? Probably overkill,
especially since I can't see how we could test for it in an autoconf test.

Here's a workaround, specifically in the 'supersede' module.

[1] https://www.illumos.org/issues/13035
[2] https://pkgsrc.se/files.php?messageId=20200812233110.30230FB28@cvs.NetBSD.org
[3] https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/functions/open.html


2020-08-23  Bruno Haible  <bruno@clisp.org>

	supersede: Avoid a failure when writing to /dev/null in Solaris zones.
	Reported by Jörg Sonnenberger <joerg@netbsd.org>
	via Thomas Klausner <tk@giga.or.at> in
	<https://pkgsrc.se/files.php?messageId=20200812233110.30230FB28@cvs.NetBSD.org>.
	* lib/supersede.c (open_supersede): When opening an existing non-regular
	file on Solaris, use O_CREAT although it should not be necessary.

diff --git a/lib/supersede.c b/lib/supersede.c
index 92317f2..a03cc6d 100644
--- a/lib/supersede.c
+++ b/lib/supersede.c
@@ -78,6 +78,16 @@ open_supersede (const char *filename, int flags, mode_t mode,
                 struct supersede_final_action *action)
 {
   int fd;
+  /* Extra flags for existing devices.  */
+  int extra_flags =
+    #if defined __sun
+    /* open ("/dev/null", O_TRUNC | O_WRONLY) fails with error EINVAL on Solaris
+       zones.  See <https://www.illumos.org/issues/13035>.  As a workaround, add
+       the O_CREAT flag, although it ought not to be necessary.  */
+    O_CREAT;
+    #else
+    0;
+    #endif
 
   if (supersede_if_exists)
     {
@@ -89,7 +99,7 @@ open_supersede (const char *filename, int flags, mode_t mode,
               && ! S_ISREG (statbuf.st_mode)
               /* The file exists and is possibly a character device, socket, or
                  something like that.  */
-              && ((fd = open (filename, flags, mode)) >= 0
+              && ((fd = open (filename, flags | extra_flags, mode)) >= 0
                   || errno != ENOENT))
             {
               if (fd >= 0)
@@ -167,7 +177,7 @@ open_supersede (const char *filename, int flags, mode_t mode,
                         {
                           /* It is possibly a character device, socket, or
                              something like that.  */
-                          fd = open (canon_filename, flags, mode);
+                          fd = open (canon_filename, flags | extra_flags, mode);
                           if (fd >= 0)
                             {
                               free (canon_filename);
@@ -197,6 +207,28 @@ open_supersede (const char *filename, int flags, mode_t mode,
               action->final_rename_temp = NULL;
               action->final_rename_dest = NULL;
             }
+          #if defined __sun
+          /* Work around <https://www.illumos.org/issues/13035>.  */
+          else if (errno == EINVAL)
+            {
+              struct stat statbuf;
+
+              if (stat (filename, &statbuf) >= 0
+                  && ! S_ISREG (statbuf.st_mode))
+                {
+                  /* The file exists and is possibly a character device, socket,
+                     or something like that.  As a workaround, add the O_CREAT
+                     flag, although it ought not to be necessary.*/
+                  fd = open (filename, flags | extra_flags, mode);
+                  if (fd >= 0)
+                    {
+                      /* The file exists.  */
+                      action->final_rename_temp = NULL;
+                      action->final_rename_dest = NULL;
+                    }
+                }
+            }
+          #endif
           else if (errno == ENOENT)
             {
               /* The file does not exist.  Use a temporary file.  */