RFC: git-commit based mtime-reproducible tarballs

[Simon Josefsson via Gnulib discussion list <bug-gnulib@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 3772 bytes --]

Hi.  Quoting the recent binutils announcement:

>   As an experiment these tarballs were made with the new "-r <date>"
>   option supported by the src-release.sh script.  This attempts to make
>   reproducible tarballs by sorting the files and passing the
>   "--mtime=<date>" option to tar.  The date used for these tarballs was
>   obtained by running:
>   
>     git log -1 --format=%cd --date=format:%F bfd/version.m4

This got me thinking about git-version-gen and GNUmakefile, and I came
up with the patch below to use the most recent commit as the timestamp
for all files in the tarball.  What do you think?

There are some concerns about this:

1) Having the same mtime on all files in a tarball may cause problems
for some projects that have fragile dependency-systems.  While I think
all dependency checks really should be using >= timestamp tests, I
wouldn't rule out that some use > timestamp tests, which would cause
(sometimes unwanted) rebuilding of some files.  Are there
dependency-constructs where the same mtime for all files in a tarball is
just a bad idea, with no better approach available?

2) The use of TAR_OPTIONS in GNUmakefile is complex and somewhat hard to
debug.  I can't find any cleaner way to provide options to tar for 'make
dist' though.  Automake defines $(AMTAR) but looks like an internal
symbol which also isn't used (bug?), instead $(am__tar) is used and
defined as am__tar = $${TAR-tar} chof - "$$tardir".  So we can override
TAR in Makefile.am but it looks like a user-variable that we shouldn't
override.  So pending support for a AMTAR (or AM_TAR?) variable in
Makefile.am that actually works, I guess we are stuck with the
TAR_OPTIONS approach.  We could do 'TAR = env TAR_OPTIONS_=... tar' in
Makefile.am but it looks like the wrong approach.

3) The Makefile.am snippet in git-version-gen is difficult to maintain,
can't we put such snippets in a gnulib-owned file and suggest use of
'include gl/top-gl-Makefile.am-include.mk' instead?  The same applies to
gen-ChangeLog rule.  The logic would have to be a bit more complex to
support per-project modifications to these rules though.

Two small bugs that are possible to fix but not important before we know
if mtime-reproducible tarballs is useful or not:

4) If there is no .version file when you type 'make dist' my patch below
would fail to provide --mtime=... to tar.  So it fails if you didn't do
'make' before 'make dist' after ./bootstrap + ./configure in a clean
checkout.

5) It is also a bit fragile that it assume 'git log -1' works without
checking for errors before invoking touch.

/Simon

diff --git a/build-aux/git-version-gen b/build-aux/git-version-gen
index a72057bf2c..0a98cb12dd 100755
--- a/build-aux/git-version-gen
+++ b/build-aux/git-version-gen
@@ -66,6 +66,7 @@ scriptversion=2022-07-09.08; # UTC
 # BUILT_SOURCES = $(top_srcdir)/.version
 # $(top_srcdir)/.version:
 #      echo '$(VERSION)' > $@-t
+#      touch -m -d @$(shell git log -1 --format=%cd --date=unix) $@-t
 #      mv $@-t $@
 # dist-hook:
 #      echo '$(VERSION)' > $(distdir)/.tarball-version
diff --git a/top/GNUmakefile b/top/GNUmakefile
index 07b331fe53..f0dd41b5b4 100644
--- a/top/GNUmakefile
+++ b/top/GNUmakefile
@@ -25,8 +25,14 @@
 _gl-Makefile := $(wildcard [M]akefile)
 ifneq ($(_gl-Makefile),)
 
+_gl-.version := $(wildcard .version)
+ifneq ($(_gl-.version),)
+_tar_mtime := --mtime=.version
+endif
+
 # Make tar archive easier to reproduce.
-export TAR_OPTIONS = --owner=0 --group=0 --numeric-owner --sort=name
+export TAR_OPTIONS = --owner=0 --group=0 --numeric-owner --sort=name \
+       $(_tar_mtime)
 
 # Allow the user to add to this in the Makefile.
 ALL_RECURSIVE_TARGETS =

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]