'relocatable' project built without --enable-relocatable

'relocatable' project built without --enable-relocatable

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 1230 bytes --]

relocatable.h has partial support for building without
--enable-relocatable: it #defines relocate and relocate2 to empty macros.
This is great! Could macros be similarly added for the other relocatable.h
APIs, namely set_relocation_prefix and compute_curr_prefix? This would
allow relocate-using code to be entirely #ifdef free.

This would help me currently because I am using this module from Vala.
Vala's compiler outputs C sources. In order to avoid the user of my package
needing a Vala compiler, I am taking advantage of automake's support for
shipping the generated C sources, and allowing the user to compile those
directly. However, this means that my Vala source code cannot depend on
configure-time settings. Further, Vala does not offer any way to generate
#ifdef-guarded C code.

My current solution for this situation, which I also face for
Windows-specific code, is to have separate Vala files which are built or
not according to automake settings.

For relocatable, I can also simply define my own empty macros for the above
APIs, and insert them in config.h with AH_BOTTOM. But this made my wonder
why some of the APIs, but not all, are already provided with "empty"
versions like this.

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 2195 bytes --]

Re: 'relocatable' project built without --enable-relocatable

[Bruno Haible]

Reuben Thomas wrote:
> Could macros be similarly added for the other relocatable.h
> APIs, namely set_relocation_prefix and compute_curr_prefix? This would
> allow relocate-using code to be entirely #ifdef free.

How does or would the code you are talking about look like, with #ifs?
And would it be code for a library, or for a program?

I'm asking because no package I know of is using set_relocation_prefix
explicitly [1], nor is it documented by Gnulib [2].

Bruno

[1] https://codesearch.debian.net/search?q=+set_relocation_prefix+-path%3Arelocatable.c+-path%3Arelocatable.h+-path%3Aconfigure&literal=1&page=1
[2] https://www.gnu.org/software/gnulib/manual/html_node/Supporting-Relocation.html

Re: 'relocatable' project built without --enable-relocatable

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

On Tue, 23 Apr 2024 at 21:46, Bruno Haible <bruno@clisp.org> wrote:

> How does or would the code you are talking about look like, with #ifs?
> And would it be code for a library, or for a program?
>

For a library. For example, from the libpaper commit referenced below,
using #ifdefs:

/* Set the prefix directory for relocation. */
void papersetprefixdir(const char *new_prefix)
{
#ifdef ENABLE_RELOCATABLE
    set_relocation_prefix (INSTALLPREFIX, new_prefix);
#else
    (void)new_prefix;
#endif
}

I'm asking because no package I know of is using set_relocation_prefix
> explicitly [1]

I've been using it in Enchant since 2017[1] and in libpaper since 2021[2]

[1] https://github.com/AbiWord/enchant/commit/a8e771a

(this is definitely in Debian by now)

[2]
https://github.com/rrthomas/libpaper/commit/13bb68b87fdc0b20e08adf58146576fd120d3f3b

(not yet in Debian, sadly, as they don't like me "vendoring gnulib", as FTP
Master calls it, or "using gnulib as other packages like Enchant do, and as
designed", as I call it).

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 3289 bytes --]

Re: 'relocatable' project built without --enable-relocatable

[Bruno Haible]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

Reuben Thomas wrote:
> For a library. For example, from the libpaper commit referenced below,
> using #ifdefs:
> 
> /* Set the prefix directory for relocation. */
> void papersetprefixdir(const char *new_prefix)
> {
> #ifdef ENABLE_RELOCATABLE
>     set_relocation_prefix (INSTALLPREFIX, new_prefix);
> #else
>     (void)new_prefix;
> #endif
> }
> ...
> I'm asking because no package I know of is using set_relocation_prefix
> > explicitly [1]
> 
> I've been using it in Enchant since 2017[1] and in libpaper since 2021[2]
> 
> [1] https://github.com/AbiWord/enchant/commit/a8e771a

OK, I see. What you are doing is perfectly like it should be done:
  - papersetprefixdir is a function part of the public API of your library.
    It needs to be defined unconditionally.
  - set_relocation_prefix is defined in relocatable.c (module relocatable-lib
    or relocatable-lib-lgpl). This file is only compiled when $RELOCATABLE
    is 'yes' or (equivalently) 'defined ENABLE_RELOCATABLE'. When $RELOCATABLE
    is 'no', you get a link error when linking the library.

Can you please try the attached patch?

If it does not help, I would suggest that you move out the papersetprefixdir
function from a .vala compilation unit to a .c compilation unit, so that the
requirements of the Vala language and processor become irrelevant.

Bruno

[-- Attachment #2: relocatable.diff --]
[-- Type: text/x-patch, Size: 506 bytes --]

diff --git a/lib/relocatable.h b/lib/relocatable.h
index 162f9d82a4..0c10ebe2a1 100644
--- a/lib/relocatable.h
+++ b/lib/relocatable.h
@@ -109,6 +109,8 @@ extern char * compute_curr_prefix (const char *orig_installprefix,
 #else
 
 /* By default, we use the hardwired pathnames.  */
+#define set_relocation_prefix(orig_prefix, curr_prefix) \
+  ((void) (orig_prefix), (void) (curr_prefix))
 #define relocate(pathname) (pathname)
 #define relocate2(pathname,allocatedp) (*(allocatedp) = NULL, (pathname))
 

Re: Gnulib in Debian

[Bruno Haible]

Reuben Thomas wrote:
> (not yet in Debian, sadly, as they don't like me "vendoring gnulib", as FTP
> Master calls it, or "using gnulib as other packages like Enchant do, and as
> designed", as I call it).

I assume you are alluding to the mail thread that starts at
<https://lists.debian.org/debian-devel/2022/11/msg00110.html> ?

I haven't read the thread. But you write:
  "I am the upstream maintainer of libpaper ..., and also a Debian Maintainer
   trying to get a new version of libpaper into Debian."

Is the problem something that affects the package upstream, or only something
that is specific to Debian?

In the latter case, I don't want to interfere with that. Distros package the
software like they want to. Debian, in particular, has hundreds of pages of
policy documents. It's not my business as an upstream maintainer to interfere
with that. (*)

Bruno

(*) Except when there's a copyright infringement, like it happened in the
Oracle Solaris downstream distribution of GNU libsigsegv. I had to write them,
so they stopped doing that.

Re: Gnulib in Debian

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 2685 bytes --]

On Wed, 24 Apr 2024 at 01:51, Bruno Haible <bruno@clisp.org> wrote:

> Reuben Thomas wrote:
> > (not yet in Debian, sadly, as they don't like me "vendoring gnulib", as
> FTP
> > Master calls it, or "using gnulib as other packages like Enchant do, and
> as
> > designed", as I call it).
>
> I assume you are alluding to the mail thread that starts at
> <https://lists.debian.org/debian-devel/2022/11/msg00110.html> ?
>

Yes.


> I haven't read the thread. But you write:
>   "I am the upstream maintainer of libpaper ..., and also a Debian
> Maintainer
>    trying to get a new version of libpaper into Debian."
>

TLDR: FTP Master rejected my libpaper package because it contains gnulib
source files. I pointed out that other Debian packages for which I am
upstream do exactly this and have been accepted, and that it is the
standard way to use gnulib. A few senior Debian Developers said they did
not consider this use of gnulib to be against Debian policy. But FTP
Master's stance appears to be that they will not let any new packages into
the archive that contain gnulib sources (or in general, vendored
sources—they don't have anything against gnulib in particular!). I also
argued that building against Debian's version of gnulib would risk
introducing bugs (I have found that updating gnulib in my projects can make
previously-working code fail).

Is the problem something that affects the package upstream, or only
> something
> that is specific to Debian?
>

It's Debian-specific, though I imagine other distros might also take a
similar stance.

In this case, the solution is for someone else to repackage libpaper
without the offending files (by generating a new source tarball). I have
said I don't want to do this myself; to be honest it's just a depressing
thought to spend hours doing something that makes no sense to me, and that
will potentially cause me bug reports in future.

I do sympathise with Debian's aim here, and the long-mooted "libposix"
project, or rather an extended "libgnu" version—that is, an installable
version of gnulib that one can use like any other library—would solve this
problem for both me and Debian. Maybe I'll summon the energy to tackle some
of the libposix to-do list one day.


> In the latter case, I don't want to interfere with that. Distros package
> the
> software like they want to. Debian, in particular, has hundreds of pages of
> policy documents. It's not my business as an upstream maintainer to
> interfere
> with that.
>

Sure, I'm just complaining, not asking for a solution. I should have been
clearer about that, sorry.

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 5531 bytes --]

Re: Gnulib in Debian

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

Reuben Thomas <rrt@sc3d.org> writes:

> TLDR: FTP Master rejected my libpaper package because it contains gnulib
> source files. I pointed out that other Debian packages for which I am
> upstream do exactly this and have been accepted, and that it is the
> standard way to use gnulib. A few senior Debian Developers said they did
> not consider this use of gnulib to be against Debian policy. But FTP
> Master's stance appears to be that they will not let any new packages into
> the archive that contain gnulib sources (or in general, vendored
> sources—they don't have anything against gnulib in particular!). I also
> argued that building against Debian's version of gnulib would risk
> introducing bugs (I have found that updating gnulib in my projects can make
> previously-working code fail).

The last aspect should be solved: the latest gnulib in Debian contains a
git bundle of gnulib, so you can Build-Depends on gnulib and via
GNULIB_REVISION pick out exactly the gnulib git revision that libpaper
needs.  This avoids including gnulib files in the tarball that is
uploaded to Debian, and there is no risk that you will get gnulib code
from a different git commit.  It requires an added 'Build-Depends: git'
in libpaper, though, which is unfortunate but I don't see how to avoid
it.  I should write a post to debian-devel describing this pattern on
how to use gnulib in Debian packages, but you can infer everything from
the links given in my blog post [1] and the latest upload of libntlm
into Debian.

/Simon

[1] https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/
[2] https://salsa.debian.org/auth-team/libntlm/-/tree/master/debian

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: Gnulib in Debian

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 778 bytes --]

On Wed, 24 Apr 2024 at 15:56, Simon Josefsson <simon@josefsson.org> wrote:

>
> The last aspect should be solved: the latest gnulib in Debian contains a
> git bundle of gnulib,


That's fantastic, I wish I had thought of that. I still don't fancy doing
the necessary packaging work, but I'll let those who have been helping me
know that there's a viable alternative.

 I should write a post to debian-devel describing this pattern on
> how to use gnulib in Debian packages, but you can infer everything from
> the links given in my blog post [1] and the latest upload of libntlm
> into Debian.
>

Thanks very much for writing this up. If in a few years libpaper2 has still
not made it into Debian I shall probably find the energy to look at it
myself!

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 1833 bytes --]

Re: 'relocatable' project built without --enable-relocatable

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

On Wed, 24 Apr 2024 at 01:24, Bruno Haible <bruno@clisp.org> wrote:

>
> Can you please try the attached patch?
>

Works beautifully, thanks. (Sorry to cause confusion: I was actually
quoting code from a C-based project, not a Vala-based project, where I
don't need the fix. But if you install this patch then I will certainly use
it in my C-based projects to reduce the use of #ifdef.)

Is there a reason not to make a similar macro for compute_curr_prefix?

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 1281 bytes --]

Re: 'relocatable' project built without --enable-relocatable

[Bruno Haible]

Reuben Thomas wrote:
> > Can you please try the attached patch?
> 
> Works beautifully, thanks.

OK, I'm committing as shown below.

> Is there a reason not to make a similar macro for compute_curr_prefix?

Yes:
  - For compute_curr_prefix, the need has not been demonstrated.
  - Even if ENABLE_RELOCATABLE is 1, there are cases when compute_curr_prefix
    is not defined.


2024-04-25  Bruno Haible  <bruno@clisp.org>

	relocatable-lib-lgpl: Allow unconditional use of set_relocation_prefix.
	Reported by Reuben Thomas <rrt@sc3d.org> in
	<https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00384.html>.
	* lib/relocatable.h (set_relocation_prefix): Define in a dummy way if
	ENABLE_RELOCATABLE is not defined.

diff --git a/lib/relocatable.h b/lib/relocatable.h
index 162f9d82a4..0c10ebe2a1 100644
--- a/lib/relocatable.h
+++ b/lib/relocatable.h
@@ -109,6 +109,8 @@ extern char * compute_curr_prefix (const char *orig_installprefix,
 #else
 
 /* By default, we use the hardwired pathnames.  */
+#define set_relocation_prefix(orig_prefix, curr_prefix) \
+  ((void) (orig_prefix), (void) (curr_prefix))
 #define relocate(pathname) (pathname)
 #define relocate2(pathname,allocatedp) (*(allocatedp) = NULL, (pathname))
 

Re: 'relocatable' project built without --enable-relocatable

[Reuben Thomas]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

On Thu, 25 Apr 2024 at 14:07, Bruno Haible <bruno@clisp.org> wrote:

> OK, I'm committing as shown below.
>

Great, thanks!

> Is there a reason not to make a similar macro for compute_curr_prefix?
>
> Yes:
>   - For compute_curr_prefix, the need has not been demonstrated.
>   - Even if ENABLE_RELOCATABLE is 1, there are cases when
> compute_curr_prefix
>     is not defined.
>

Good reasons.

-- 
https://rrt.sc3d.org

[-- Attachment #2: Type: text/html, Size: 1510 bytes --]

Re: GNULIB_REVISION

[Bruno Haible]

Hi Simon,

> you can ... via
> GNULIB_REVISION pick out exactly the gnulib git revision that libpaper
> needs. ...
> [1] https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/
> [2] https://salsa.debian.org/auth-team/libntlm/-/tree/master/debian

I see GNULIB_REVISION as an obsolete alternative to git submodules, and
would therefore discourage rather than propagate its use.

Currently libntlm has this in its bootstrap.conf:

  GNULIB_REVISION=dfb71172a46ef41f8cf8ab7ca529c1dd3097a41d

and GNU make has this:

  GNULIB_REVISION=stable-202307

Both can be done with git submodules. Git submodules do support branches [1],
and the command 'git submodule update --remote gnulib' updates the
submodule while staying on the indicated branch.

When using submodules, the target revision is stored in a versionable way,
that both gitweb and cgit can show appropriately [2][3]. If a branch is
used, it is stored in the .gitmodules file.

The differences between both approaches are:

  - GNULIB_REVISION works only with the 'bootstrap' program. The submodules
    approach works also without 'bootstrap'.

  - For GNULIB_REVISION, the user is on their own regarding tooling, aside
    from 'bootstrap'. In the submodules approach, the 'git' suite provides
    the tooling, and many developers are familiar with it.

  - .tar.gz files created by the gitweb "snapshot" link, by the cgit "refs >
    Download" section, or the GitHub "Download ZIP" button contain an empty
    directory in place of the submodule, and no information about the revision.
    Whereas they contain the file with the GNULIB_REVISION assignment.

> I should write a post to debian-devel describing this pattern on
> how to use gnulib in Debian packages

It feels wrong to me if, in order to get meta-information about required
dependencies of a package, Debian tools grep a particular file for a specific
string. This approach is simply too limited.

The correct way, IMO, would be that 'git' provides this meta-information,
either embedded in the .tar.gz generated by the web tooling, or in a
separate .tar.gz. AFAICT, 'git' currently does not have this ability.
Therefore we need to approach the 'git' team, in order to find a solution
that scales across the whole set of software package — not specific to
gnulib and not specific to 'bootstrap'.

Bruno

[1] https://stackoverflow.com/questions/1777854/how-can-i-specify-a-branch-tag-when-adding-a-git-submodule
[2] https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=tree
[3] https://git.savannah.gnu.org/cgit/coreutils.git/tree/

Re: GNULIB_REVISION

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 7465 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> Hi Simon,
>
>> you can ... via
>> GNULIB_REVISION pick out exactly the gnulib git revision that libpaper
>> needs. ...
>> [1] https://blog.josefsson.org/2024/04/13/reproducible-and-minimal-source-only-tarballs/
>> [2] https://salsa.debian.org/auth-team/libntlm/-/tree/master/debian
>
> I see GNULIB_REVISION as an obsolete alternative to git submodules, and
> would therefore discourage rather than propagate its use.

I think it will be challenging for gnulib to insists on always being
used as a git submodule, and I would prefer if we continue support
multiple ways of working.  Personally I have been migrating towards
gnulib git submodules because most other projects use gnulib like that,
but I've never really felt comfortable with them.  Some of the concerns
I have:

- git submodules leads to -- in my subjective opinion -- complexity
  which leads to a worse user experience for developers.  I have learned
  to work with git submodules over the years, but it was a hurdle that I
  don't want to force on everyone.

- the gnulib git submodule is huge.  Not rarely I get out of memory
  errors during 'git clone' in CI/CD jobs.  I can restart the jobs
  manually, but this indicate that there is a resource drain here.  For
  a tiny project like libntlm the imbalance if the small project code
  and large gnulib is troubling.

- often CI/CD platforms have different ways of working with git
  submodules which adds complexity which leads to bugs.  Allowing
  maintainers to decide if they want to work with git submodules or not
  seems like a good thing.

- we don't offer any way for people receiving tarballs to learn which
  gnulib git commit was used (you noticed this too below) but with a
  GNULIB_REVISION approach this is part of the tarball, just like any
  other versioned dependency on autoconf, automake etc

- I think gnulib could be regarded as any other external dependency,
  just like autoconf, automake, libtool etc that also generate files in
  my build tree during bootstrapping.  I don't put autoconf as a git
  submodule, why should I put gnulib as one?

Granted, these concerns are a bit vague and subjective.

> Currently libntlm has this in its bootstrap.conf:
>
>   GNULIB_REVISION=dfb71172a46ef41f8cf8ab7ca529c1dd3097a41d
>
> and GNU make has this:
>
>   GNULIB_REVISION=stable-202307

Interesting.  This suggests the GNULIB_REVISION approach isn't the
entire solution either.

I think it is useful to record the gnulib git commit used to prepare a
tarball, and have that git commit id be part of the shipped tarball, and
stored inside the git repository.  The first use above achieve this, but
the second one doesn't (branches/tags are moving targets).

If I download the gzip tarball I can't find anywhere what gnulib commit
was used for bootstrapping.  It is quite cumbersome to verify that the
tarball didn't contain any modified gnulib code.  This is even harder
when projects INTENTIONALLY modify gnulib code compared to what's in
gnulib git, which coreutils and several others projects does through
gnulib *.diff/*.patch files.

Ultimately, I think there is an important use-case to build projects
directly from source code without having tarballs with pre-generated
files that are not reproduced by the user.

> The differences between both approaches are:
>
>   - GNULIB_REVISION works only with the 'bootstrap' program. The submodules
>     approach works also without 'bootstrap'.

What use case are you thinking of?  The gnulib git commit information
consumers that I can think of are gnulib-aware.

>   - For GNULIB_REVISION, the user is on their own regarding tooling, aside
>     from 'bootstrap'. In the submodules approach, the 'git' suite provides
>     the tooling, and many developers are familiar with it.

Yes, but developers also like flexibility, and in some situations I
think the git approach is not the best way of working.

>   - .tar.gz files created by the gitweb "snapshot" link, by the cgit "refs >
>     Download" section, or the GitHub "Download ZIP" button contain an empty
>     directory in place of the submodule, and no information about the revision.
>     Whereas they contain the file with the GNULIB_REVISION assignment.

Indeed, this was the main challenge for me.  That is critical
information for anyone who wants to avoid touching tarballs with
pre-generated content.

>> I should write a post to debian-devel describing this pattern on
>> how to use gnulib in Debian packages
>
> It feels wrong to me if, in order to get meta-information about required
> dependencies of a package, Debian tools grep a particular file for a specific
> string. This approach is simply too limited.

Meta-information about dependencies are normally always hand-curated in
Debian (the Build-Depends: header).  The simplest solution is for the
Debian package maintainer to figure out which gnulib git commit version
was used for a release and pin that manually in the debian/rules
makefile.  If the information is available in bootstrap.conf via
GNULIB_REVISION that saves time.  I believe this is conceptually the
same thing as pinning version information for any other dependency.

> The correct way, IMO, would be that 'git' provides this meta-information,
> either embedded in the .tar.gz generated by the web tooling, or in a
> separate .tar.gz. AFAICT, 'git' currently does not have this ability.
> Therefore we need to approach the 'git' team, in order to find a solution
> that scales across the whole set of software package — not specific to
> gnulib and not specific to 'bootstrap'.

Yes I was quite disappointed when I realized that 'git archive' doesn't
record the git submodule git commit anywhere.

Couldn't the .gitmodules file be extended to allow specifying the git
commit of the submodule?

I think GitLab/GitHub/etc use 'git archive' under the hood, so we could
ask that the .gitmodules file is extended to hold the commit (just like
it holds branch name now).

Alternatively, get 'git archive' to somehow record the submodule commit
in some other way.

I suppose we could recommend a practice for gnulib users that use gnulib
git submodules to put this in .gitmodules:

    # GNULIB_REVISION=dfb71172a46ef41f8cf8ab7ca529c1dd3097a41d

Then this will be part of the 'git archive' output (I think?) and we
would have to also recommend to 'EXTRA_DIST += .gitmodules' so this
information is included in the tarballs.

However.  I want to deploy a solution that works now while we wait for
git to add this feature (or not).

I think we also may find that requiring 'git' for building packages
cause a cyclic dependency for boostrap people.  Requiring 'tar' is okay
because 'tar' has very few other pre-dependencies.  I don't know how to
solve this problem though: some way to ship all gnulib git commits in a
compact way that can be extracted easily without complex tools are
necessary.  Maybe a minimal git like 'bootstrap-git' could be written
that just supports cloning from a git bundle and nothing else.

/Simon


> Bruno
>
> [1] https://stackoverflow.com/questions/1777854/how-can-i-specify-a-branch-tag-when-adding-a-git-submodule
> [2] https://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=tree
> [3] https://git.savannah.gnu.org/cgit/coreutils.git/tree/
>
>
>
>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: GNULIB_REVISION

[Paul Eggert]

You raise several good points. A couple of quick reaction:

On 2024-04-25 09:26, Simon Josefsson via Gnulib discussion list wrote:

> - the gnulib git submodule is huge.  Not rarely I get out of memory
>    errors during 'git clone' in CI/CD jobs.

First I've heard of this problem (other than with Git LFS, which Gnulib 
doesn't use). What part of the clone operation fails? Is this 
server-side RAM, or client-side RAM, or something else? How much memory 
does 'git clone' require now for Gnulib?

Is there some way to cajole 'git clone' into using less memory, with a 
'--depth 1' or similar options? Cloning shallowly would clone Gnulib a 
lot faster, if you're cloning from a remote repository.


> - we don't offer any way for people receiving tarballs to learn which
>    gnulib git commit was used

Isn't the real problem that we don't put (for example) gzip's own commit 
ID into the coreutils tarball? If we did that, Gnulib's commit ID would 
come for free, since it can be derived from gzip's commit ID.

Re: GNULIB_REVISION

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 2267 bytes --]

Paul Eggert <eggert@cs.ucla.edu> writes:

> You raise several good points. A couple of quick reaction:
>
> On 2024-04-25 09:26, Simon Josefsson via Gnulib discussion list wrote:
>
>> - the gnulib git submodule is huge.  Not rarely I get out of memory
>>    errors during 'git clone' in CI/CD jobs.
>
> First I've heard of this problem (other than with Git LFS, which
> Gnulib doesn't use). What part of the clone operation fails? Is this 
> server-side RAM, or client-side RAM, or something else? How much
> memory does 'git clone' require now for Gnulib?
>
> Is there some way to cajole 'git clone' into using less memory, with a
> '--depth 1' or similar options? Cloning shallowly would clone Gnulib a 
> lot faster, if you're cloning from a remote repository.

It only happens once in a while, for example:

https://gitlab.com/gsasl/inetutils/-/jobs/6706396721

This is gitlab's own git submodule checkout system working, and it is
using --depth 150 which shouldn't be too heavy, so it not even getting
to ./bootstrap's git clone.

Btw, using --depth 1 is incompatible with gnulib's git-version-gen: it
will not find a suitable version number for the build.  Even gitlab's
default depth of 50 (or if it was 100, can't remember) is often not
enough if you have >50 commits since the last release.  This cause
problems when building from 'git archive' tarballs.

>> - we don't offer any way for people receiving tarballs to learn which
>>    gnulib git commit was used
>
> Isn't the real problem that we don't put (for example) gzip's own
> commit ID into the coreutils tarball? If we did that, Gnulib's commit
> ID would come for free, since it can be derived from gzip's commit ID.

I suppose you meant s/coreutils/gzip/, otherwise I don't follow?

Yes that is a good idea!  The git commit of the project should be part
of the announce-gen output.  The git tag name could be mentioned too.
Tags are not long-term stable since they can be moved later on, so I
think the full git commit id should be mentioned too.  Current SHA1 git
commits aren't long-term stable either, since SHA1 is broken, but at
least this approach is the best we can do right now and when we move to
SHA256 git things will be better.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: GNULIB_REVISION

[Collin Funk]

On 4/25/24 10:00 AM, Paul Eggert wrote:
> Is there some way to cajole 'git clone' into using less memory, with a
> '--depth 1' or similar options? Cloning shallowly would clone Gnulib a
> lot faster, if you're cloning from a remote repository.

Maybe '--single-branch' would help too. If space is really desperate
the snapshots on gitweb are pretty small.

    https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=commit;h=648aeb575db7af9ddd51cf17d1b56ee91e9ef3c5

7.6 MB for me vs. my full clone

$ du -s --human-readable .git
114M	.git

Collin

Re: GNULIB_REVISION

[Paul Eggert]

On 4/25/24 10:43, Simon Josefsson wrote:

> https://gitlab.com/gsasl/inetutils/-/jobs/6706396721

That's an malloc failure on the server side. The savannah admins should 
be told about the issue soon after it happens. Routine clones of GNU 
projects shouldn't fail like that. (It's never happened to me, but then 
I don't clone from Savannah all that often - maybe Savannah is giving 
less server memory to clients that seem to hog?)


> Btw, using --depth 1 is incompatible with gnulib's git-version-gen

We could fix that by not requiring git-version-gen for the Gnulib 
submodule. git-version-gen is pretty useless for that anyway, as for 
Gnulib it currently outputs a string like "0.1.7513-648ae" which is not 
much more useful than the commit ID 
"648aeb575db7af9ddd51cf17d1b56ee91e9ef3c5".

>> Isn't the real problem that we don't put (for example) gzip's own
>> commit ID into the coreutils tarball? If we did that, Gnulib's commit
>> ID would come for free, since it can be derived from gzip's commit ID.
> 
> I suppose you meant s/coreutils/gzip/

Yes, sorry.

> SHA1 git
> commits aren't long-term stable either, since SHA1 is broken

As you say, they're good enough for now. And anyway I would think SHA1 
is good enough longer term unless an adversary infects your Git 
repository (and in that case you probably have bigger problems...).