From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-3.7 required=3.0 tests=AWL,BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 3158C1F9FC for ; Sun, 21 Mar 2021 02:57:53 +0000 (UTC) Received: from localhost ([::1]:43294 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lNoHn-0000Cg-Uk for normalperson@yhbt.net; Sat, 20 Mar 2021 22:57:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46492) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lNoHk-0000CT-AT for bug-gnulib@gnu.org; Sat, 20 Mar 2021 22:57:48 -0400 Received: from mo4-p00-ob.smtp.rzone.de ([81.169.146.219]:15006) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lNoHh-0000PG-Dr for bug-gnulib@gnu.org; Sat, 20 Mar 2021 22:57:48 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1616295461; cv=none; d=strato.com; s=strato-dkim-0002; b=UylXCEQw0dvUM3h/GvuZsUra95gCPqSPPLeaVSTDOPpYkAT+UFWKHUMksTEXUSJB2h kMsSXhckSVKLe+u7Wo+Gqk8bzlmeZB7s+7ynTlqSnnwDUfxmD5VV/buLF+V5e2GBhzh0 +Z/WZrOqG5+Jn+LuzGu0dC65ExGf1YShtGLCSY8SwlME7BvuA0Qlhb9qsJ7CTi7og6DK fMHFv8hXiJgdX5GGsFq7ZZd06bXFqUHUosLBa22BBwv8K7ubTxE27ZsSNE/mbAK7B1QO 1YiuJ+7w/j9DVKCpJfheZQ9oBCkR4T7Z2U+q3LxWsiFnIP7W/V1e+uO8jLJXXOeN/BSj RyIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1616295461; s=strato-dkim-0002; d=strato.com; h=References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Cc:Date: From:Subject:Sender; bh=0UlCfgapFE3tFd12saWj+ZC7OyocDY3AAgNFTYviQ3k=; b=Ihxj0/n2/3++W0PALJUqQrXbv3qZDAxelCQ+I6xKWQ82hzs9mG/NHZkprSBZk7f1Wd 9wf47vjS7Y4U6fAkyc3pUxtUqFWJGJN6tphwk/fb68d4j+mnVUp6w5b5T8bvtUJ17/zn +M198aafGTarlIrAWs6712WySsMAdnMDf8x0iAlu6ZqORyb6UjtCGVacr2LULCfF7ZqA s9JQdGtzOPgWuls6/Xlle3UVKL06QIEyPneg72utPL9lk/KpBN6iKN9OUki4OToqoJAE ahrR5nzi9hACV6iHesIvX3YFdlHLJ23KK0YnCk2V+Egohi/8k+ddz+FlMxd9gb6A10t5 b8uA== ARC-Authentication-Results: i=1; strato.com; dkim=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1616295461; s=strato-dkim-0002; d=clisp.org; h=References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Cc:Date: From:Subject:Sender; bh=0UlCfgapFE3tFd12saWj+ZC7OyocDY3AAgNFTYviQ3k=; b=abD7VFYeFbAxZWbhjBHXbsVSfxMKm9okVk6nTJfVvmEcQFVUA3UM26i4f5Tync46Rd eAgGZvn81kn5RgAZCCqkuITeK9M7o2A7RxW2lPnLbUw8jwZ7aa47wNDy36pSEhUL4GFK B2+wjZAdkz26mSNCy7NMajknZSUNuNrExlLUThNSp7XdvZKiZl3hjMxjx8iaUNt9HXJI J3trkzM4uKJ34q+Lzh2zwEHG2LnYRsGi+nleV2bEmoqcgnEHygDLlgBHhEK6xqA8WXQc fhCxY7ijQY8Ro7M7bEBp9eU4DdZQeFpiBv2yrzAPzCeqKOjAVQ+CNsfQBMm/pEZLO8wW 5MXw== Authentication-Results: strato.com; dkim=none X-RZG-AUTH: ":Ln4Re0+Ic/6oZXR1YgKryK8brlshOcZlIWs+iCP5vnk6shH+AHjwLuWOHqf0y5JW" X-RZG-CLASS-ID: mo00 Received: from bruno.haible.de by smtp.strato.de (RZmta 47.21.0 DYNA|AUTH) with ESMTPSA id t05ec5x2L2vfEzi (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (curve X9_62_prime256v1 with 256 ECDH bits, eq. 3072 bits RSA)) (Client did not present a certificate); Sun, 21 Mar 2021 03:57:41 +0100 (CET) From: Bruno Haible To: bug-gnulib@gnu.org Subject: Re: Issues with posix functions on modern macOS/Xcode Date: Sun, 21 Mar 2021 03:57:39 +0100 Message-ID: <2615029.Q7US0F6XX7@omega> User-Agent: KMail/5.1.3 (Linux/4.4.0-203-generic; KDE/5.18.0; x86_64; ; ) In-Reply-To: <1827966.Sqf4WByit0@omega> References: <1748506.D3602SlINW@omega> <1827966.Sqf4WByit0@omega> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="iso-8859-1" Received-SPF: none client-ip=81.169.146.219; envelope-from=bruno@clisp.org; helo=mo4-p00-ob.smtp.rzone.de X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, NICE_REPLY_A=-0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_NONE=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: bug-gnulib@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Gnulib discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Martin =?ISO-8859-1?Q?Storsj=F6?= Errors-To: bug-gnulib-bounces+normalperson=yhbt.net@gnu.org Sender: "bug-gnulib" > > > FAIL: test-fflush2.sh > > > FAIL: test-fpurge > > > FAIL: test-ftell.sh > > > FAIL: test-ftell2.sh > > > FAIL: test-ftello.sh > > > FAIL: test-ftello2.sh >=20 > These would need debugging on a macOS 10.15 machine. Anyone willing to ta= ke > this challenge? The compilefarm now has a macOS 11.2 machine. These tests fail there as wel= l. I could debug it, and found that it is a regression introduced in macOS 10.= 15.0: =2D-- Libc-1272.250.1/stdio/FreeBSD/fflush.c 2015-02-04 23:57:51.00000= 0000 +0100 +++ Libc-1353.11.2/stdio/FreeBSD/fflush.c 2019-09-21 00:18:36.0000000= 00 +0200 =2E.. @@ -110,12 +90,52 @@ int n, t; =20 t =3D fp->_flags; =2D if ((t & __SWR) =3D=3D 0) =2D return (0); =20 if ((p =3D fp->_bf._base) =3D=3D NULL) return (0); =20 + /* + * SUSv3 requires that fflush() on a seekable input stream updates = the file + * position indicator with the underlying seek function. Use a dum= b fseek + * for this (don't attempt to preserve the buffers). + */ + if ((t & __SRD) !=3D 0) { + if (fp->_seek =3D=3D NULL) { + /* + * No way to seek this file -- just return "success= =2E" + */ + return (0); + } + + n =3D fp->_r; + + if (n > 0) { + /* + * See _fseeko's dumb path. + */ + if (_sseek(fp, (fpos_t)-n, SEEK_CUR) =3D=3D -1) { + if (errno =3D=3D ESPIPE) { + /* + * Ignore ESPIPE errors, since ther= e's no way to put the bytes + * back into the pipe. + */ + return (0); + } + return (EOF); + } + + if (HASUB(fp)) { + FREEUB(fp); + } + fp->_p =3D fp->_bf._base; + fp->_r =3D 0; + fp->_flags &=3D ~__SEOF; + memset(&fp->_mbstate, 0, sizeof(mbstate_t)); + } + return (0); + } + + if ((t & __SWR) !=3D 0) { n =3D fp->_p - p; /* write this much */ =20 /* @@ -142,6 +162,7 @@ return (EOF); } } + } return (0); } =20 They added new code to the function __sflush, which makes sense (in order to make fflush() POSIX compliant). But __sflush is being called by ftello(), and the new code in __sflush severely disturbs the functioning of ftello: - it causes the ungetc buffer to be discarded, - it causes a flush of buffered data (which no one has asked for), - it returns a broken file position. The bug was already spotted in the wild, in some non-GNU program: . This patch provides a workaround. 2021-03-20 Bruno Haible ftello: Work around bug in macOS >=3D 10.15. Reported by Martin Storsj=F6 in . * m4/ungetc.m4 (gl_FUNC_UNGETC_WORKS): On macOS, don't define FUNC_UNGETC_BROKEN. Instead, set gl_ftello_broken_after_ungetc to yes. * m4/ftello.m4 (gl_FUNC_FTELLO): Invoke gl_FUNC_UNGETC_WORKS, and arrange to provide the workaround if ftello is broken after ungetc. * lib/ftello.c: Include , intprops.h. (ftello) [FTELLO_BROKEN_AFTER_UNGETC]: Implement from scratch. * modules/ftello (Files): Add m4/ungetc.m4. (Depends-on): Add intprops. * doc/posix-functions/ftello.texi: Mention the macOS bug. diff --git a/doc/posix-functions/ftello.texi b/doc/posix-functions/ftello.t= exi index 2561c5d..eab591f 100644 =2D-- a/doc/posix-functions/ftello.texi +++ b/doc/posix-functions/ftello.texi @@ -20,6 +20,11 @@ This function produces incorrect results after @code{put= c} that followed a @code{getc} call that reached EOF on some platforms: Solaris 11 2010-11. @item +This function, when invoked after @code{ungetc}, throws away the @code{ung= etc} +buffer, changes the stream's file position, and returns the wrong position= on +some platforms: +macOS 10.15 and newer. +@item This function fails on seekable stdin, stdout, and stderr: cygwin <=3D 1.5= =2E24. @item On platforms where @code{off_t} is a 32-bit type, @code{ftello} does not w= ork diff --git a/lib/ftello.c b/lib/ftello.c index c5a4e0c..da13694 100644 =2D-- a/lib/ftello.c +++ b/lib/ftello.c @@ -19,6 +19,9 @@ /* Specification. */ #include =20 +#include +#include "intprops.h" + /* Get lseek. */ #include =20 @@ -40,13 +43,79 @@ ftello (FILE *fp) # endif #endif { =2D#if LSEEK_PIPE_BROKEN +#if FTELLO_BROKEN_AFTER_UNGETC /* macOS >=3D 10.15 */ + /* The system's ftello() is completely broken, because it calls __sflush, + which makes side effects on the stream. */ + + /* Handle non-seekable files first. */ + if (fp->_file < 0 || fp->_seek =3D=3D NULL) + { + errno =3D ESPIPE; + return -1; + } + + /* Determine the current offset, ignoring buffered and pushed-back bytes= =2E */ + off_t pos; + + if (fp->_flags & __SOFF) + pos =3D fp->_offset; + else + { + pos =3D fp->_seek (fp->_cookie, 0, SEEK_CUR); + if (pos < 0) + return -1; + if (fp->_flags & __SOPT) + { + fp->_offset =3D pos; + fp->_flags |=3D __SOFF; + } + } + + if (fp->_flags & __SRD) + { + /* Now consider buffered and pushed-back bytes from ungetc. */ + if (fp->_ub._base !=3D NULL) + /* Considering the buffered bytes, we are at position + pos - fp->_ur. + Considering also the pushed-back bytes, we are at position + pos - fp->_ur - fp->_r. */ + pos =3D pos - fp->_ur - fp->_r; + else + /* Considering the buffered bytes, we are at position + pos - fp->_r. */ + pos =3D pos - fp->_r; + if (pos < 0) + { + errno =3D EIO; + return -1; + } + } + else if ((fp->_flags & __SWR) && fp->_p !=3D NULL) + { + /* Consider the buffered bytes. */ + off_t buffered =3D fp->_p - fp->_bf._base; + + /* Compute pos + buffered, with overflow check. */ + off_t sum; + if (! INT_ADD_OK (pos, buffered, &sum)) + { + errno =3D EOVERFLOW; + return -1; + } + pos =3D sum; + } + + return pos; + +#else + +# if LSEEK_PIPE_BROKEN /* mingw gives bogus answers rather than failure on non-seekable files. = */ if (lseek (fileno (fp), 0, SEEK_CUR) =3D=3D -1) return -1; =2D#endif +# endif =20 =2D#if FTELLO_BROKEN_AFTER_SWITCHING_FROM_READ_TO_WRITE /* Solaris */ +# if FTELLO_BROKEN_AFTER_SWITCHING_FROM_READ_TO_WRITE /* Solaris */ /* The Solaris stdio leaves the _IOREAD flag set after reading from a fi= le reaches EOF and the program then starts writing to the file. ftello gets confused by this. */ @@ -66,9 +135,9 @@ ftello (FILE *fp) } return pos; } =2D#endif +# endif =20 =2D#if defined __SL64 && defined __SCLE /* Cygwin */ +# if defined __SL64 && defined __SCLE /* Cygwin */ if ((fp->_flags & __SL64) =3D=3D 0) { /* Cygwin 1.5.0 through 1.5.24 failed to open stdin in 64-bit @@ -80,6 +149,9 @@ ftello (FILE *fp) fp->_seek64 =3D tmp->_seek64; fclose (tmp); } =2D#endif +# endif + return ftello (fp); + +#endif } diff --git a/m4/ftello.m4 b/m4/ftello.m4 index 1a0f7bc..65bec39 100644 =2D-- a/m4/ftello.m4 +++ b/m4/ftello.m4 @@ -1,4 +1,4 @@ =2D# ftello.m4 serial 13 +# ftello.m4 serial 14 dnl Copyright (C) 2007-2021 Free Software Foundation, Inc. dnl This file is free software; the Free Software Foundation dnl gives unlimited permission to copy and/or distribute it, @@ -130,6 +130,15 @@ main (void) ;; esac fi + if test $REPLACE_FTELLO =3D 0; then + dnl Detect bug on macOS >=3D 10.15. + gl_FUNC_UNGETC_WORKS + if test $gl_ftello_broken_after_ungetc =3D yes; then + REPLACE_FTELLO=3D1 + AC_DEFINE([FTELLO_BROKEN_AFTER_UNGETC], [1], + [Define to 1 if the system's ftello function has the macOS bug.]) + fi + fi fi ]) =20 diff --git a/m4/ungetc.m4 b/m4/ungetc.m4 index b648fea..dd5d1dd 100644 =2D-- a/m4/ungetc.m4 +++ b/m4/ungetc.m4 @@ -1,4 +1,4 @@ =2D# ungetc.m4 serial 9 +# ungetc.m4 serial 10 dnl Copyright (C) 2009-2021 Free Software Foundation, Inc. dnl This file is free software; the Free Software Foundation dnl gives unlimited permission to copy and/or distribute it, @@ -55,11 +55,19 @@ AC_DEFUN_ONCE([gl_FUNC_UNGETC_WORKS], esac ]) ]) + gl_ftello_broken_after_ungetc=3Dno case "$gl_cv_func_ungetc_works" in *yes) ;; *) =2D AC_DEFINE([FUNC_UNGETC_BROKEN], [1], =2D [Define to 1 if ungetc is broken when used on arbitrary bytes.]) + dnl On macOS >=3D 10.15, where the above program fails with exit cod= e 6, + dnl we fix it through an ftello override. + case "$host_os" in + darwin*) gl_ftello_broken_after_ungetc=3Dyes ;; + *) + AC_DEFINE([FUNC_UNGETC_BROKEN], [1], + [Define to 1 if ungetc is broken when used on arbitrary bytes.= ]) + ;; + esac ;; esac ]) diff --git a/modules/ftello b/modules/ftello index 6de0859..fc6bafb 100644 =2D-- a/modules/ftello +++ b/modules/ftello @@ -6,6 +6,7 @@ lib/ftello.c lib/stdio-impl.h m4/fseeko.m4 m4/ftello.m4 +m4/ungetc.m4 =20 Depends-on: stdio @@ -13,6 +14,7 @@ extensions largefile sys_types lseek [test $HAVE_FTELLO =3D 0 || test $REPLACE_FTELLO =3D 1] +intprops [test $HAVE_FTELLO =3D 0 || test $REPLACE_FTELLO =3D 1] # Just to guarantee consistency between ftell() and ftello(). ftell =20