Re: fseeko issues in mys2/mingw64

[John Donoghue <john.david.donoghue@gmail.com>]

On 1/25/20 11:23 AM, John Donoghue wrote:
> This is a question/perhaps bug on bigfile support with fseeko in 
> gnulib when being used in MSYS2/mingw64.
>
> As a quick overview:
>
> I have a demo program that opens a 5 Gig file add just does a seeko to 
> the end of the file and then does a ftellp to get the end position.
>
> I am importing ftello and fseeko and fflush and it replaces the fseeko 
> and lseek functions which then in the depths in the gnulib calls 
> _lseek rather than _lseeki64.
>
>
> As an additional note, it doesnt seem to replace the lseek if we arent 
> importing fflush
>
> --------------------------------------------------
> main.cpp:
>
> #include <config.h>
>
> #include <stdio.h>
> #include <stdint.h>
>
> int main()
> {
>   fprintf(stderr, "test big file\n");
>
>   /* create 5G file if not there yet */
>   FILE *f1 = fopen("big.txt", "r");
>   if(!f1)
>   {
>    fprintf(stderr, "creating big file ...");
>    fflush(stderr);
>    f1 = fopen("big.txt", "w");
>    if(f1)
>    {
>      char buf[1024];
>
>      int64_t sz = 5LL*1024*1024*1024;
>
>      while(sz > 0)
>      {
>        fwrite(buf, sizeof(buf), 1, f1);
>        sz -= sizeof(buf);
>      }
>
>    }
>    fprintf(stderr, "done\n");
>   }
>   fclose(f1);
>
>   /* open file and test */
>   FILE *f = fopen("big.txt", "r");
>
>   fprintf(stderr, "sizeof(off_t)=%d\n",(int)sizeof(off_t));
>
>   if(f)
>   {
>     int status = fseeko(f, 0, SEEK_END);
>     fprintf(stderr, "seek status=%d\n", status);
>
>     off_t p = ftello(f);
>     fprintf(stderr, "tell pos=%lld\n", (long long)p);
>     fclose(f);
>   }
>
>   return 0;
> }
>
> configure.ac:
>
> AC_INIT([bigfile], [1.0.0])
> AM_INIT_AUTOMAKE([-Wall -Werror foreign])
>
> AC_CONFIG_HEADERS([config.h])
> AC_CONFIG_MACRO_DIRS([m4])
>
> # Checks for programs.
> AC_PROG_CXX
> AC_PROG_CC
> AC_PROG_CC_STDC
> gl_EARLY
>
> # For gnulib.
> gl_INIT
>
> AC_CONFIG_FILES([lib/Makefile Makefile])
> AC_OUTPUT
>
> -------------------------------------------------
> Makefile.am:
>
> ACLOCAL_AMFLAGS = -I m4
> SUBDIRS = lib
>
> AM_CPPFLAGS = -I$(top_builddir)/lib -I$(top_srcdir)/lib
> EXTRA_DIST = m4/gnulib-cache.m4
>
> bin_PROGRAMS = bigtest
> bigtest_SOURCES = main.cpp
> bigtest_LDADD = lib/libgnu.a
>
> --------------------------------------------------
> bootstrap.sh:
>
> if ! [ -d gnulib ]; then \
>   git clone git://git.sv.gnu.org/gnulib.git gnulib
> fi
>
> ./gnulib/gnulib-tool  --import fflush fseeko ftello largefile stdio
>
> touch NEWS README AUTHORS ChangeLog
>
> autoreconf --install
>
> --------------------------------------------------
> Running bootsrap shows the actual modules imported as:
>
> Module list with included dependencies (indented):
>     absolute-header
>     errno
>     extensions
>     extern-inline
>   fflush
>     fpurge
>     freading
>     fseek
>   fseeko
>     fstat
>     ftell
>   ftello
>     include_next
>     intprops
>   largefile
>     lseek
>     msvc-inval
>     msvc-nothrow
>     pathmax
>     snippet/arg-nonnull
>     snippet/c++defs
>     snippet/unused-parameter
>     snippet/warn-on-use
>     ssize_t
>     stat-time
>     stdbool
>     stddef
>   stdio
>     sys_stat
>     sys_types
>     time
>     unistd
>     verify
>
> --------------------------------------------------
>
> Running native configure and make in linux builds ok.
>
> The following modules are added to libnu.a:
>   freading.o stat-time.o unistd.o fflush.o fpurge.o fseek.o fseeko.o
>
> Running the app:
>
> ./bigtest
> test big file
> sizeof(off_t)=8
> seek status=0
> tell pos=5368709120
>
> Running in debug, it has replaced fseek with the gnulib fseeko and 
> calls lseek at line 117 of fseeko.c
>
> ------------------------------------------------
> Running in a native configure in msys2/mingw64 and make builds ok;
>
>
> The following modules are added to libnu.a:
> freading.o stat-time.o unistd.o fflush.o fpurge.o fseek.o fseeko.o 
> fstat.o lseek.o msvc-inval.o msvc-nothrow.o stat-w32.o
>
> Running the app:
>
> ./bigtest.exe
> test big file
> creating big file ...done
> sizeof(off_t)=8
> seek status=-1
> tell pos=0
>
> Running in debug, and doing a break on _lseek
>
> (gdb) bt
> #0  0x00007ffd2f7ad1cb in msvcrt!_lseek () from 
> C:\WINDOWS\System32\msvcrt.dll
> #1  0x00000000004016e7 in rpl_lseek (fd=3, offset=offset@entry=0,
>     whence=whence@entry=1) at lseek.c:69
> #2  0x0000000000401636 in rpl_fseeko (
>     fp=fp@entry=0x7ffd2f81fa90 <msvcrt!_iob+144>, offset=offset@entry=0,
>     whence=whence@entry=2) at fseeko.c:45
> #3  0x000000000040838c in main () at main.cpp:41
>
> And for the lseek call that fails:
>
> (gdb) bt
> #0  0x00007ffd2f7ad1cb in msvcrt!_lseek () from 
> C:\WINDOWS\System32\msvcrt.dll
> #1  0x00000000004016e7 in rpl_lseek (fd=3, offset=offset@entry=0,
>     whence=whence@entry=2) at lseek.c:69
> #2  0x0000000000401683 in rpl_fseeko (
>     fp=fp@entry=0x7ffd2f81fa90 <msvcrt!_iob+144>, offset=offset@entry=0,
>     whence=whence@entry=2) at fseeko.c:117
> #3  0x000000000040838c in main () at main.cpp:41
>
> ------------------------------------------------
>
> configure output:
> .
> checking build system type... x86_64-w64-mingw32
> checking host system type... x86_64-w64-mingw32
> checking for _LARGEFILE_SOURCE value needed for large files... no
> checking for special C compiler options needed for large files... no
> checking for _FILE_OFFSET_BITS value needed for large files... 64
> checking whether the preprocessor supports include_next... yes
> .
> checking for 64-bit off_t... yes
> checking for 64-bit st_size... no
> checking whether fseeko is declared... yes
> checking for fseeko... yes
> checking whether fflush works on input streams... no
> checking whether stat file-mode macros are broken... no
> checking for nlink_t... no
> checking whether ftello is declared... yes
> checking for ftello... yes
> checking whether ftello works... yes
> .
> checking whether fflush works on input streams... (cached) no
> checking whether fpurge is declared... no
> checking for fseeko... (cached) yes
> checking whether fflush works on input streams... (cached) no
> checking for _fseeki64... yes
> checking whether _fseeki64 is declared... yes
> checking for ftello... (cached) yes
> checking whether ftello works... (cached) yes
> checking whether lseek detects pipes... no
> .
>
> ------------------------------------------------
>
> Looking at the sources for gnulib lseek.c:
>
> It is entering the lseek call which is really _lseek (parameters that 
> are long) rather than 64bit instead of using the _lseeki64 call.
>
> #if _GL_WINDOWS_64_BIT_OFF_T
>   return _lseeki64 (fd, offset, whence);
> #else
>   return lseek (fd, offset, whence);
> #endif
>
>
> Looking for _GL_WINDOWS_64_BIT_OFF_T, it only gets set in 
> lib/sys_types.in.h if WINDOWS_64_BIT_OFF_T is set.
>
> Should lseek.c  call _lseeki64 regardless of _GL_WINDOWS_64_BIT_OFF_T 
> value if _lseeki64 is available ? Or perhaps if (_FILE_OFFSET_BITS == 
> 64 && HAVE_DECL__FSEEKI64 == 1)
>
>
>
>
>
>
Some additional info,


the gnulib fseeko is being used in this case as m4/fseeko.m4 replaces it 
even if it exists, if fflush is one of the checked functions and fflush 
stdin is not yes (in this case configure whether fflush works on input 
streams ... no )