Re: [PATCH 1/3] t5004: test ZIP archives with many entries

["\"René Scharfe\"" <l.s.r@web.de>]

Am 23.08.2015 um 07:54 schrieb Eric Sunshine:
> On Sat, Aug 22, 2015 at 3:06 PM, René Scharfe <l.s.r@web.de> wrote:
>> diff --git a/t/t5004-archive-corner-cases.sh b/t/t5004-archive-corner-cases.sh
>> index 654adda..c6bd729 100755
>> --- a/t/t5004-archive-corner-cases.sh
>> +++ b/t/t5004-archive-corner-cases.sh
>> @@ -115,4 +115,44 @@ test_expect_success 'archive empty subtree by direct pathspec' '
>>          check_dir extract sub
>>   '
>>
>> +ZIPINFO=zipinfo
>> +
>> +test_lazy_prereq ZIPINFO '
>> +       n=$("$ZIPINFO" "$TEST_DIRECTORY"/t5004/empty.zip | sed -n "2s/.* //p")
>> +       test "x$n" = "x0"
>> +'
> 
> Unfortunately, this sed expression isn't portable due to dissimilar
> output of various zipinfo implementations. On Linux, the output of
> zipinfo is:
> 
>      $ zipinfo t/t5004/empty.zip
>      Archive:  t/t5004/empty.zip
>      Zip file size: 62 bytes, number of entries: 0
>      Empty zipfile.
>      $
> 
> however, on Mac OS X:
> 
>      $ zipinfo t/t5004/empty.zip
>      Archive:  t/t5004/empty.zip   62 bytes   0 files
>      Empty zipfile.
>      $
> 
> and on FreeBSD, the zipinfo command seems to have been removed
> altogether in favor of "unzip -Z" (emulate zipinfo).

Thanks for your thorough checks!

I suspected that zipinfo's output might be formatted differently on
different platforms and tried to guard against it by checking for the
number zero there. Git's ZIP file creation is platform independent
(modulo bugs), so having a test run at least somewhere should
suffice. In theory.

We could add support for the one-line-summary variant on OS X easily,
though.

> One might hope that "unzip -Z" would be a reasonable replacement for
> zipinfo, however, it is apparently only partially implemented on
> FreeBSD, and requires that -1 be passed, as well. Even with "unzip -Z
> -1", there are issues. The output on Linux and Mac OS X is:
> 
>      $ unzip -Z -1 t/t5004/empty.zip
>      Empty zipfile.
>      $
> 
> but FreeBSD differs:
> 
>      $ unzip -Z -1 t/t5004/empty.zip
>      $
> 
> With a non-empty zip file, the output is identical on all platforms:
> 
>      $ unzip -Z -1 twofiles.zip
>      file1
>      file2
>      $
> 
> So, if you combine that with "wc -l" or test_line_count, you may have
> a portable and reliable entry counter.

Counting all entries is slow, and more importantly it's not what we
want. In this test we need the number of entries recorded in the ZIP
directory, not the actual number of entries found by scanning the
archive, or the directory.

On Linux "unzip -Z -1 many.zip | wc -l" reports 65792 even before
adding ZIP64 support; only without -1 we get the interesting numbers
(specifically with "unzip -Z many.zip | sed -n '2p;$p'"):

    Zip file size: 6841366 bytes, number of entries: 256
    65792 files, 0 bytes uncompressed, 0 bytes compressed: 0.0%

> With these three patches applied, Mac OS X has trouble with 'many.zip':
> 
>      $ unzip -Z -1 many.zip
>      warning [many.zip]:  76 extra bytes at beginning or within zipfile
>        (attempting to process anyway)
>      error [many.zip]:  reported length of central directory is
>        -76 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
>        zipfile?).  Compensating...
>      00/
>      00/00
>      ...
>      ff/ff
>      error: expected central file header signature not found (file
>        #65793). (please check that you have transferred or created the
>        zipfile in the appropriate BINARY mode and that you have compiled
>        UnZip properly)
> 
> And FreeBSD doesn't like it either:
> 
>      $ unzip -Z -1 many.zip
>      unzip: Invalid central directory signature
>      $
> 

Looks like they don't support ZIP64. Or I got some of the fields wrong
after all.

https://en.wikipedia.org/wiki/Zip_%28file_format%29#ZIP64 says: "OS X
Yosemite does support the creation of ZIP64 archives, but does not
support unzipping these archives using the shipped unzip command-line
utility or graphical Archive Utility.[citation needed]".

How does unzip react to a ZIP file with more than 65535 entries that
was created natively on these platforms? And what does zipinfo (a real
one, without -1) report at the top for such files?

Thanks,
René