Re: RFC: git-commit based mtime-reproducible tarballs

[Jim Meyering <jim@meyering.net>]

[-- Attachment #1: Type: text/plain, Size: 4216 bytes --]

On Mon, Jan 16, 2023, 12:41 AM Simon Josefsson via Gnulib discussion list <
bug-gnulib@gnu.org> wrote:

> Bruno Haible <bruno@clisp.org> writes:
>
> > Paul Eggert wrote:
> >> some users want to "trust but verify" and a reproducible
> >> tarball is easier to audit than a non-reproducible one, so for these
> >> users it can be a win to omit the irrelevant data from the tarball.
> >
> > Reproducibility can be implemented in different ways:
> >   - by omitting irrelevant data from the tarball,
> >   - by having a customized comparison program 'diff', such that
> >     "diff --ignore-irrelevant-metadata contents1 contents2"
> >     would ignore the irrelevant parts.
>
> The problem with a --ignore-irrelevant-metadata approach is that it will
> be a judgement call what is irrelevant, and two projects may have
> different philosophies that are mutually incompatible.
>
> A devils advocate case: consider a build-system that embeds the
> source-code timestamp information in the binary, and the binary sends of
> a hash of its executable binary to a remote server for verification
> purposes.  In some projects this may be what you want to achieve.  Then
> ignoring this particular metadata will be a critical failure for that
> project.
>
> I think it is a worthy goal to reach a tarball that is deterministically
> and one-way reproducable from git source code [for the same set of tool
> versions].
>
> >> when I do an 'ls
> >> -l' of a source directory that I got from a distribution tarball, it's
> >> useful to see the last time the contents of each source file was
> changed
> >> upstream.
> >
> > OK, now we're discussing different ways to make a tarball reproducible.
> > That's nice, because Simon's proposal was to make all timestamps equal,
> > and that puts me off.
> > In binutils-2.40.tar.bz2 all files are from 2023-01-14.
> > In android-studio-2021.3.1.17-linux.tar.gz all files are from 2010-01-01.
> > It gives me as a user no idea whether this tarball is 13 years old,
> > 2 years old, or from yesterday.
> >
> > I much prefer Paul's approach, since it still conveys meaningful
> > timestamps:
>
> I agree!
>
> I even wonder if the binutils tarball build properly on say HP-UX then?
>
> >> For TZDB, where users have long wanted reproducibility, I use something
> >> like this in a Makefile recipe for each source file $$file:
> >>
> >>            time=`git log -1 --format='tformat:%ct' $$file` &&
> >>            touch -cmd @$$time $$file
> >
> > That's good for the files that are under version control.
> >
> >> 2. What about platform-independent files that are automatically created
> >> from source files from the repository, and that are shipped in the
> >> release tarball?
> >
> > For these, you could unpack the tarball, see in which order the
> timestamps
> > are, and then assign artificial timestamps, in the same order but exactly
> > 2 seconds apart. For example, if the tarball contains
> > under version control:
> >   hello.c         2023-01-14 13:28:14
> >   configure.ac    2023-01-01 14:03:07
> > and not under version control:
> >   configure       2023-01-15 04:09:10
> >   config.h.in     2023-01-15 04:05:19
> > then you would determine the
> >   max_timestamp_under_vc = max { 2023-01-14 13:28:14, 2023-01-01
> 14:03:07 }
> >                          = 2023-01-14 13:28:14
> > and then, since config.h.in is older than configure:
> >   touch -m (max_timestamp_under_vc + 2 seconds) config.h.in
> >   touch -m (max_timestamp_under_vc + 4 seconds) configure
> >
> > You can do this without knowing the Makefile rules or scripts which
> created
> > config.h.in and configure.
> >
> > The increment of 2 seconds is, of course, for VFAT file systems, which
> have
> > only 2 seconds of resolution for file modification times.
>
> Clever!
>
> To implement this we would need a dist-hook to do the 'touch -m ...'
> dance on all files.
>
> I somewhat fear that the solution here will be more of a problem than
> the original problem due to the complexity.
>
> Does anyone see a problem with this approach?  Do you think it is a good
> idea?  I like it and don't see any further problems, except for the
> complexity but I don't see a way to reduce it.
>

I like it, too.

>

[-- Attachment #2: Type: text/html, Size: 5823 bytes --]