Fetch from existing gnulib Git repository if needed

Fetch from existing gnulib Git repository if needed

[Markus Mützel]

[-- Attachment #1: Type: text/plain, Size: 1391 bytes --]

Dear gnulib developers,

GNU Octave uses Mercurial as the VCS of its main repository.
Developers are using the bootstrap script of gnulib to automatically clone its Git repository in a subdirectory of Octave's source tree. The revision that we'd like to use is set in the bootstrap.conf script. Currently, that is

: ${GNULIB_REVISION=d4ec02b3cc70cddaaa5183cc5a45814e0afb2292}


This is working perfectly for a fresh clone of Octave's source tree. However, when we update GNULIB_REVISION to a newer revision and a user/developer ran the bootstrap script before, running the bootstrap script again fails with an error like the following:

./bootstrap: Bootstrapping from checked-out octave sources...
fatal: reference is not a tree: d4ec02b3cc70cddaaa5183cc5a45814e0afb2292
program finished with exit code 128


To work around that, a user/developer could manually fetch from the remote repository. That is a bit more tedious when it comes to CI installations that usually need no manual interaction.

As a workaround we are applying the attached patch to the bootstrap-funclib.sh script to automatically fetch from the remote gnulib repository if the GNULIB_REVISION isn't found in the local gnulib Git repository.

Would it be possible to make a similar change in gnulib so that updating to a newer gnulib revision becomes a bit easier for that configuration?

Markus

[-- Attachment #2: gnulib-bootstrap-git-fetch.diff --]
[-- Type: text/plain, Size: 774 bytes --]

Update bootstrap script from upstream gnulib to automatically fetch from repository if needed

diff -r c51b07a71421 bootstrap-funclib.sh
--- a/bootstrap-funclib.sh	Fri Apr 26 13:33:37 2024 -0400
+++ b/bootstrap-funclib.sh	Fri Apr 26 20:00:21 2024 +0200
@@ -532,6 +532,10 @@
         # The subdirectory 'gnulib' already exists.
         if test -n "$GNULIB_REVISION"; then
           if test -d "$gnulib_path/.git"; then
+            if ! git --git-dir="$gnulib_path"/.git cat-file \
+                 commit "$GNULIB_REVISION"; then
+              git --git-dir="$gnulib_path"/.git fetch
+            fi
             (cd "$gnulib_path" && git checkout "$GNULIB_REVISION") || exit 1
           else
             die "Error: GNULIB_REVISION is specified in bootstrap.conf," \

Re: Fetch from existing gnulib Git repository if needed

[Bruno Haible]

Hi Markus,

> GNU Octave uses Mercurial as the VCS of its main repository.
> Developers are using the bootstrap script of gnulib to automatically clone its Git repository in a subdirectory of Octave's source tree. The revision that we'd like to use is set in the bootstrap.conf script. Currently, that is
> 
> : ${GNULIB_REVISION=d4ec02b3cc70cddaaa5183cc5a45814e0afb2292}
> 
> 
> This is working perfectly for a fresh clone of Octave's source tree. However, when we update GNULIB_REVISION to a newer revision and a user/developer ran the bootstrap script before, running the bootstrap script again fails with an error like the following:
> 
> ./bootstrap: Bootstrapping from checked-out octave sources...
> fatal: reference is not a tree: d4ec02b3cc70cddaaa5183cc5a45814e0afb2292
> program finished with exit code 128

I see. Indeed there is no 'git fetch' in the code branch that you are
highlighting.

> As a workaround we are applying the attached patch to the bootstrap-funclib.sh script to automatically fetch from the remote gnulib repository if the GNULIB_REVISION isn't found in the local gnulib Git repository.

Thanks for the patch. But note that GNULIB_REVISION can hold either a commit
hash or the name of a branch (such as 'stable-202401'). So, we have 4 cases:
  (a) a commit hash, already present
  (b) a commit hash, not known
  (c) a branch, already present
  (d) a branch, not known

The command 'git cat-file commit $GNULIB_REVISION' returns true in the cases
(a) and (c). So, your patch would trigger a 'git fetch' in the cases (b) and
(d). But in case (d), the 'git fetch' is useless:
'git cat-file commit $GNULIB_REVISION' would still fail afterwards.

One can distinguish the four cases in more detail using the commands
  git rev-list --quiet $GNULIB_REVISION --
which tests for case (a) and
  git show-ref --verify --quiet refs/heads/$GNULIB_REVISION
which tests for case (c). This would allow us to do 'git fetch' only in case
(b).

However, I believe the patch below is simpler and achieves the same goal.


2024-04-27  Bruno Haible  <bruno@clisp.org>

	bootstrap: Support checking out a recent GNULIB_REVISION.
	Reported by Markus Mützel <markus.muetzel@gmx.de> in
	<https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00462.html>.
	* top/bootstrap-funclib.sh (prepare_GNULIB_SRCDIR): If the
	'git checkout' command fails, fetch the newer commits and then retry it.
	* build-aux/bootstrap: Regenerated.

diff --git a/top/bootstrap-funclib.sh b/top/bootstrap-funclib.sh
index 7511d62d9b..620006d320 100644
--- a/top/bootstrap-funclib.sh
+++ b/top/bootstrap-funclib.sh
@@ -1,6 +1,6 @@
 # A library of shell functions for autopull.sh, autogen.sh, and bootstrap.
 
-scriptlibversion=2024-04-13.15; # UTC
+scriptlibversion=2024-04-27.17; # UTC
 
 # Copyright (C) 2003-2024 Free Software Foundation, Inc.
 #
@@ -462,7 +462,17 @@ prepare_GNULIB_SRCDIR ()
       || die "Error: --gnulib-srcdir or \$GNULIB_SRCDIR is specified," \
              "but does not contain gnulib-tool"
     if test -n "$GNULIB_REVISION" && $use_git; then
-      (cd "$GNULIB_SRCDIR" && git checkout "$GNULIB_REVISION") || exit $?
+      # The 'git checkout "$GNULIB_REVISION"' command succeeds if the
+      # GNULIB_REVISION is a commit hash that exists locally, or if it is
+      # branch name that can be fetched from origin. It fails, however,
+      # if the GNULIB_REVISION is a commit hash that only exists in origin.
+      # In this case, we need a 'git fetch' and then retry
+      # 'git checkout "$GNULIB_REVISION"'.
+      (cd "$GNULIB_SRCDIR" \
+       && { git checkout "$GNULIB_REVISION" 2>/dev/null \
+            || { git fetch origin && git checkout "$GNULIB_REVISION"; }
+          }
+      ) || exit $?
     fi
   else
     if ! $use_git; then

Re: Fetch from existing gnulib Git repository if needed

[Markus Mützel]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

Hi Bruno,

> > As a workaround we are applying the attached patch to the bootstrap-funclib.sh script to automatically fetch from the remote gnulib repository if the GNULIB_REVISION isn't found in the local gnulib Git repository.
>
> Thanks for the patch. But note that GNULIB_REVISION can hold either a commit
> hash or the name of a branch (such as 'stable-202401'). So, we have 4 cases:
>   (a) a commit hash, already present
>   (b) a commit hash, not known
>   (c) a branch, already present
>   (d) a branch, not known
>
> The command 'git cat-file commit $GNULIB_REVISION' returns true in the cases
> (a) and (c). So, your patch would trigger a 'git fetch' in the cases (b) and
> (d). But in case (d), the 'git fetch' is useless:
> 'git cat-file commit $GNULIB_REVISION' would still fail afterwards.
>
> One can distinguish the four cases in more detail using the commands
>   git rev-list --quiet $GNULIB_REVISION --
> which tests for case (a) and
>   git show-ref --verify --quiet refs/heads/$GNULIB_REVISION
> which tests for case (c). This would allow us to do 'git fetch' only in case
> (b).
>
> However, I believe the patch below is simpler and achieves the same goal.

Thank you for looking into this. However, it looks like $GNULIB_SRCDIR is empty for us. So, the change doesn't seem to make a difference. Executing bootstrap after a revision bump still fails if the bootstrap script was already run before.
I like your idea of fetching if the checkout failed though.

The attached diff seems to be working for our use case. Would it be possible to apply something like that in gnulib?

Markus

[-- Attachment #2: gnulib-bootstrap-git-fetch-v2.diff --]
[-- Type: text/plain, Size: 2117 bytes --]

diff -r 0fbf06e4b460 bootstrap-funclib.sh
--- a/bootstrap-funclib.sh	Sat Apr 27 17:33:00 2024 +0200
+++ b/bootstrap-funclib.sh	Sun Apr 28 11:04:52 2024 +0200
@@ -462,7 +462,17 @@
       || die "Error: --gnulib-srcdir or \$GNULIB_SRCDIR is specified," \
              "but does not contain gnulib-tool"
     if test -n "$GNULIB_REVISION" && $use_git; then
-      (cd "$GNULIB_SRCDIR" && git checkout "$GNULIB_REVISION") || exit $?
+      # The 'git checkout "$GNULIB_REVISION"' command succeeds if the
+      # GNULIB_REVISION is a commit hash that exists locally, or if it is
+      # branch name that can be fetched from origin. It fails, however,
+      # if the GNULIB_REVISION is a commit hash that only exists in origin.
+      # In this case, we need a 'git fetch' and then retry
+      # 'git checkout "$GNULIB_REVISION"'.
+      (cd "$GNULIB_SRCDIR" \
+        && { git checkout "$GNULIB_REVISION" 2>/dev/null \
+          || { git fetch origin && git checkout "$GNULIB_REVISION"; }
+        }
+      ) || exit $?
     fi
   else
     if ! $use_git; then
@@ -532,7 +542,17 @@
         # The subdirectory 'gnulib' already exists.
         if test -n "$GNULIB_REVISION"; then
           if test -d "$gnulib_path/.git"; then
-            (cd "$gnulib_path" && git checkout "$GNULIB_REVISION") || exit 1
+            # The 'git checkout "$GNULIB_REVISION"' command succeeds if the
+            # GNULIB_REVISION is a commit hash that exists locally, or if it is
+            # branch name that can be fetched from origin. It fails, however,
+            # if the GNULIB_REVISION is a commit hash that only exists in origin.
+            # In this case, we need a 'git fetch' and then retry
+            # 'git checkout "$GNULIB_REVISION"'.
+            (cd "$gnulib_path" \
+              && { git checkout "$GNULIB_REVISION" 2>/dev/null \
+               || { git fetch origin && git checkout "$GNULIB_REVISION"; }
+              }
+            ) || exit $?
           else
             die "Error: GNULIB_REVISION is specified in bootstrap.conf," \
                 "but '$gnulib_path' contains no git history"

Re: Fetch from existing gnulib Git repository if needed

[Bruno Haible]

Markus Mützel wrote:
> However, it looks like $GNULIB_SRCDIR is empty for us. So, the change doesn't seem to make a difference. Executing bootstrap after a revision bump still fails if the bootstrap script was already run before.

Oh, there were two 'git checkout' commands and I enhanced only one of them...

Should now be fixed, like this. Thanks for the feedback!


2024-04-28  Bruno Haible  <bruno@clisp.org>

	bootstrap: Support checking out a recent GNULIB_REVISION, part 2.
	Reported by Markus Mützel <markus.muetzel@gmx.de> in
	<https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00507.html>.
	* top/bootstrap-funclib.sh (prepare_GNULIB_SRCDIR): If using a submodule
	and the 'git checkout' command fails, fetch the newer commits and then
	retry it.
	* build-aux/bootstrap: Regenerated.

diff --git a/top/bootstrap-funclib.sh b/top/bootstrap-funclib.sh
index 620006d320..f7905eb208 100644
--- a/top/bootstrap-funclib.sh
+++ b/top/bootstrap-funclib.sh
@@ -1,6 +1,6 @@
 # A library of shell functions for autopull.sh, autogen.sh, and bootstrap.
 
-scriptlibversion=2024-04-27.17; # UTC
+scriptlibversion=2024-04-28.09; # UTC
 
 # Copyright (C) 2003-2024 Free Software Foundation, Inc.
 #
@@ -465,8 +465,8 @@ prepare_GNULIB_SRCDIR ()
       # The 'git checkout "$GNULIB_REVISION"' command succeeds if the
       # GNULIB_REVISION is a commit hash that exists locally, or if it is
       # branch name that can be fetched from origin. It fails, however,
-      # if the GNULIB_REVISION is a commit hash that only exists in origin.
-      # In this case, we need a 'git fetch' and then retry
+      # if the GNULIB_REVISION is a commit hash that only exists in
+      # origin. In this case, we need a 'git fetch' and then retry
       # 'git checkout "$GNULIB_REVISION"'.
       (cd "$GNULIB_SRCDIR" \
        && { git checkout "$GNULIB_REVISION" 2>/dev/null \
@@ -542,7 +542,17 @@ prepare_GNULIB_SRCDIR ()
         # The subdirectory 'gnulib' already exists.
         if test -n "$GNULIB_REVISION"; then
           if test -d "$gnulib_path/.git"; then
-            (cd "$gnulib_path" && git checkout "$GNULIB_REVISION") || exit 1
+            # The 'git checkout "$GNULIB_REVISION"' command succeeds if the
+            # GNULIB_REVISION is a commit hash that exists locally, or if it is
+            # branch name that can be fetched from origin. It fails, however,
+            # if the GNULIB_REVISION is a commit hash that only exists in
+            # origin. In this case, we need a 'git fetch' and then retry
+            # 'git checkout "$GNULIB_REVISION"'.
+            (cd "$gnulib_path" \
+             && { git checkout "$GNULIB_REVISION" 2>/dev/null \
+                  || { git fetch origin && git checkout "$GNULIB_REVISION"; }
+                }
+            ) || exit $?
           else
             die "Error: GNULIB_REVISION is specified in bootstrap.conf," \
                 "but '$gnulib_path' contains no git history"

Re: Fetch from existing gnulib Git repository if needed

[Markus Mützel]

Hi Bruno,

Bruno Haible wrote:
> Markus Mützel wrote:
> > However, it looks like $GNULIB_SRCDIR is empty for us. So, the change doesn't seem to make a difference. Executing bootstrap after a revision bump still fails if the bootstrap script was already run before.
> 
> Oh, there were two 'git checkout' commands and I enhanced only one of them...
> 
> Should now be fixed, like this. Thanks for the feedback!

Thank you for the quick fix. It seems to automatically fetch if needed for us now.

Markus