bug-gnulib@gnu.org mirror (unofficial)
 help / color / mirror / Atom feed
From: Simon Josefsson via Gnulib discussion list <bug-gnulib@gnu.org>
To: Bruno Haible <bruno@clisp.org>
Cc: Paul Eggert <eggert@cs.ucla.edu>,  bug-gnulib@gnu.org
Subject: Re: RFC: git-commit based mtime-reproducible tarballs
Date: Mon, 16 Jan 2023 09:40:37 +0100	[thread overview]
Message-ID: <875yd6dg8q.fsf@josefsson.org> (raw)
In-Reply-To: <2740098.11c6FMkHaZ@nimes> (Bruno Haible's message of "Sun, 15 Jan 2023 23:25:58 +0100")

[-- Attachment #1: Type: text/plain, Size: 4010 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> Paul Eggert wrote:
>> some users want to "trust but verify" and a reproducible 
>> tarball is easier to audit than a non-reproducible one, so for these 
>> users it can be a win to omit the irrelevant data from the tarball.
> Reproducibility can be implemented in different ways:
>   - by omitting irrelevant data from the tarball,
>   - by having a customized comparison program 'diff', such that
>     "diff --ignore-irrelevant-metadata contents1 contents2"
>     would ignore the irrelevant parts.

The problem with a --ignore-irrelevant-metadata approach is that it will
be a judgement call what is irrelevant, and two projects may have
different philosophies that are mutually incompatible.

A devils advocate case: consider a build-system that embeds the
source-code timestamp information in the binary, and the binary sends of
a hash of its executable binary to a remote server for verification
purposes.  In some projects this may be what you want to achieve.  Then
ignoring this particular metadata will be a critical failure for that

I think it is a worthy goal to reach a tarball that is deterministically
and one-way reproducable from git source code [for the same set of tool

>> when I do an 'ls 
>> -l' of a source directory that I got from a distribution tarball, it's 
>> useful to see the last time the contents of each source file was changed 
>> upstream.
> OK, now we're discussing different ways to make a tarball reproducible.
> That's nice, because Simon's proposal was to make all timestamps equal,
> and that puts me off.
> In binutils-2.40.tar.bz2 all files are from 2023-01-14.
> In android-studio-2021.3.1.17-linux.tar.gz all files are from 2010-01-01.
> It gives me as a user no idea whether this tarball is 13 years old,
> 2 years old, or from yesterday.
> I much prefer Paul's approach, since it still conveys meaningful
> timestamps:

I agree!

I even wonder if the binutils tarball build properly on say HP-UX then?

>> For TZDB, where users have long wanted reproducibility, I use something 
>> like this in a Makefile recipe for each source file $$file:
>> 	      time=`git log -1 --format='tformat:%ct' $$file` &&
>> 	      touch -cmd @$$time $$file
> That's good for the files that are under version control.
>> 2. What about platform-independent files that are automatically created 
>> from source files from the repository, and that are shipped in the 
>> release tarball?
> For these, you could unpack the tarball, see in which order the timestamps
> are, and then assign artificial timestamps, in the same order but exactly
> 2 seconds apart. For example, if the tarball contains
> under version control:
>   hello.c         2023-01-14 13:28:14
>   configure.ac    2023-01-01 14:03:07
> and not under version control:
>   configure       2023-01-15 04:09:10
>   config.h.in     2023-01-15 04:05:19
> then you would determine the
>   max_timestamp_under_vc = max { 2023-01-14 13:28:14, 2023-01-01 14:03:07 }
>                          = 2023-01-14 13:28:14
> and then, since config.h.in is older than configure:
>   touch -m (max_timestamp_under_vc + 2 seconds) config.h.in
>   touch -m (max_timestamp_under_vc + 4 seconds) configure
> You can do this without knowing the Makefile rules or scripts which created
> config.h.in and configure.
> The increment of 2 seconds is, of course, for VFAT file systems, which have
> only 2 seconds of resolution for file modification times.


To implement this we would need a dist-hook to do the 'touch -m ...'
dance on all files.

I somewhat fear that the solution here will be more of a problem than
the original problem due to the complexity.

Does anyone see a problem with this approach?  Do you think it is a good
idea?  I like it and don't see any further problems, except for the
complexity but I don't see a way to reduce it.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

  reply	other threads:[~2023-01-16  8:41 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <87h6wtgmhy.fsf__22556.7857896507$1673713908$gmane$org@redhat.com>
2023-01-15 11:01 ` RFC: git-commit based mtime-reproducible tarballs Simon Josefsson via Gnulib discussion list
2023-01-15 13:21   ` Bruno Haible
2023-01-15 16:03     ` Paul Eggert
2023-01-15 22:25       ` Bruno Haible
2023-01-16  8:40         ` Simon Josefsson via Gnulib discussion list [this message]
2023-01-16  8:51           ` Jim Meyering
2023-01-16  9:45       ` Vivien Kraus
2023-01-16 11:48         ` Bruno Haible
2023-01-16 23:00         ` Simon Josefsson via Gnulib discussion list
2023-01-16  8:28     ` Simon Josefsson via Gnulib discussion list

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information: https://lists.gnu.org/mailman/listinfo/bug-gnulib

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=875yd6dg8q.fsf@josefsson.org \
    --to=bug-gnulib@gnu.org \
    --cc=bruno@clisp.org \
    --cc=eggert@cs.ucla.edu \
    --cc=simon@josefsson.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).