beta-tester call draft

beta-tester call draft

[Bruno Haible]

Hi,

It's now time to call for beta-testers of the Python gnulib-tool.
I plan to post the same text to info-gnu and to planet.gnu.org.

Here is a draft. Please comment!

 ---------------------------------------------------------------------
GNU gnulib: calling for beta-testers

If you are developer on a package that uses GNU gnulib as part of its
build system:

gnulib-tool has been known for being slow for many years. We have
listened to your complaints. A rewrite of gnulib-tool in another
programming language (Python) is ready for beta-testing. It is
between 8 times and 100 times faster than the original gnulib-tool.

Both implementations should behave identically, that is, produce
the same generated files and the same output. You can help us ensure
this, through the following steps:

  1. Make sure you have Python (version 3.7 or newer) installed on
     your machine.

  2. Update your gnulib checkout. (For some packages, it comes as a
     git submodule named 'gnulib'.) Like this:
       $ git pull
     Set the environment variable GNULIB_SRCDIR, pointing to this checkout.

  3. Set an environment variable that enables checking that the two
     implementations behave the same:
       $ export GNULIB_TOOL_IMPL=sh+py

  4. Clean the built files of your package:
       $ make -k distclean

  5. Regenerate the fetched and generated files of your package.
     Depending on the packge, this may be a command such as
       $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR
     or
       $ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh
     or, if no such script is available:
       $ $GNULIB_SRCDIR/gnulib-tool --update
     If there is a failure, due to differences between the 'sh' and 'py'
     results, please report it to <bug-gnulib@gnu.org>.

  6. If this invocation was successful, you can trust the rewritten
     gnulib-tool and use it from now on, by setting the environment
     variable
       $ export GNULIB_TOOL_IMPL=py

  7. Continue with
       $ ./configure
       $ make
     as usual.

And enjoy the speed! The rewritten gnulib-tool was implemented by
Dmitry Selyutin, Collin Funk, and me.
 ---------------------------------------------------------------------

Re: beta-tester call draft

[Bruno Haible]

PS:
>   2. Update your gnulib checkout. (For some packages, it comes as a
>      git submodule named 'gnulib'.) Like this:
>        $ git pull

That should be:

         $ git checkout master
         $ git pull

Re: beta-tester call draft

[Collin Funk]

On 4/19/24 5:22 PM, Bruno Haible wrote:
>   5. Regenerate the fetched and generated files of your package.
>      Depending on the packge, this may be a command such as
>        $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR

Forgive me if this is a silly question, I have focused on
pygnulib/*.py and have not looked at the bootstrap script or the
recent thread on it. But would this work on stable branches?

Since the stable branches don't have the updated gnulib-tool.py files
I could see a checkout breaking things. I'm not familiar enough to know
if you worked around this already though.

Other than that the announcement message looks good to me. Thanks!

Collin

Re: beta-tester call draft

[Bruno Haible]

Collin Funk wrote:
> But would this work on stable branches?

It would not work, no. Also it would not work in submodules (which fix
a certain commit of gnulib). That's why in step 2 we need a
  $ git checkout master

Bruno

Re: beta-tester call draft

[Paul Eggert]

On 2024-04-19 17:22, Bruno Haible wrote:
> Hi,

>    2. Update your gnulib checkout. (For some packages, it comes as a
>       git submodule named 'gnulib'.) Like this:
>         $ git pull
>       Set the environment variable GNULIB_SRCDIR, pointing to this checkout.

Might help to spell out what to do with submodules. Something like this, 
perhaps:

   2. Update your gnulib checkout. If you are not using git submodules,
      like this:
        $ git pull
      and then set the environment variable GNULIB_SRCDIR, pointing to
      this checkout. If you are using git submodules, something like
      this:
        $ git -C gnulib pull origin master
        $ git commit -m 'build: update gnulib submodule to latest' gnulib


>    5. Regenerate the fetched and generated files of your package.
>       Depending on the packge, this may be a command such as
>         $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR
>       or
>         $ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh
>       or, if no such script is available:
>         $ $GNULIB_SRCDIR/gnulib-tool --update

Here, perhaps you can add, "If you are using git submodules, just run a 
plain './bootstrap'."

>       If there is a failure, due to differences between the 'sh' and 'py'
>       results, please report it to <bug-gnulib@gnu.org>.

I tried this on my well-worn copy of GNU diffutils, and got the 
following diagnostics. These were all diagnostics about backup files, or 
files I manually deleted by moving them into a '.del' subdirectory (an 
old habit of mine), or symlink loops that I had created to test.

Perhaps the advice should start with, "Start with a fresh checkout from 
Git."

-----
diff: ./.del/gnulib-tests/test-fnmatch-1.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-1.sh: No 
such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-2.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-2.sh: No 
such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-3.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-3.sh: No 
such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-4.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-4.sh: No 
such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-5.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-5.sh: No 
such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32-1.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32-1.sh: 
No such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32-2.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32-2.sh: 
No such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32-3.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32-3.sh: 
No such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32-4.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32-4.sh: 
No such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32-5.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32-5.sh: 
No such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32-6.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32-6.sh: 
No such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32-7.sh: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32-7.sh: 
No such file or directory
diff: ./.del/gnulib-tests/test-fnmatch-w32.c: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-w32.c: No 
such file or directory
diff: ./.del/lib/mbszero.c: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/.del/lib/mbszero.c: No such file 
or directory
diff: ./.del/lib/mbuiterf.c: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/.del/lib/mbuiterf.c: No such file 
or directory
diff: ./.del/lib/mbuiterf.h: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/.del/lib/mbuiterf.h: No such file 
or directory
diff: ./.del/lib/propername-lite.c: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/.del/lib/propername-lite.c: No 
such file or directory
diff: ./gnulib-tests/rand-digit.h~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/rand-digit.h~: No 
such file or directory
diff: ./gnulib-tests/test-btowc1.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-btowc1.sh~: No 
such file or directory
diff: ./gnulib-tests/test-btowc2.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-btowc2.sh~: No 
such file or directory
diff: ./gnulib-tests/test-btowc3.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-btowc3.sh~: No 
such file or directory
diff: ./gnulib-tests/test-c32rtomb-w32-1.sh~: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-c32rtomb-w32-1.sh~: No 
such file or directory
diff: ./gnulib-tests/test-fnmatch-w32-1.sh~: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-fnmatch-w32-1.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbrtoc32-w32-1.sh~: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbrtoc32-w32-1.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbrtowc-w32-1.sh~: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbrtowc-w32-1.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbrtowc1.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbrtowc1.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbrtowc2.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbrtowc2.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbrtowc3.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbrtowc3.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbrtowc4.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbrtowc4.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbrtowc5.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbrtowc5.sh~: No 
such file or directory
diff: ./gnulib-tests/test-mbsrtowcs1.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbsrtowcs1.sh~: 
No such file or directory
diff: ./gnulib-tests/test-mbsrtowcs2.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbsrtowcs2.sh~: 
No such file or directory
diff: ./gnulib-tests/test-mbsrtowcs3.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbsrtowcs3.sh~: 
No such file or directory
diff: ./gnulib-tests/test-mbsrtowcs4.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbsrtowcs4.sh~: 
No such file or directory
diff: ./gnulib-tests/test-mbsrtowcs5.sh~: No such file or directory
diff: /home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-mbsrtowcs5.sh~: 
No such file or directory
diff: ./gnulib-tests/test-wcrtomb-w32-1.sh~: No such file or directory
diff: 
/home/eggert/src/gnu/glpyXeRWfy/gnulib-tests/test-wcrtomb-w32-1.sh~: No 
such file or directory
diff: ./src/loop: Too many levels of symbolic links
diff: /home/eggert/src/gnu/glpyXeRWfy/src/loop: Too many levels of 
symbolic links
diff: ./src/subdir/loop: Too many levels of symbolic links
diff: /home/eggert/src/gnu/glpyXeRWfy/src/subdir/loop: Too many levels 
of symbolic links
gnulib/gnulib-tool: *** gnulib-tool.py produced different files than 
gnulib-tool.sh! Compare /home/eggert/src/gnu/diffutils and 
/home/eggert/src/gnu/glpyXeRWfy.
gnulib/gnulib-tool: *** Stop.
./bootstrap: gnulib-tool failed

Re: beta-tester call draft

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> Hi,
>
> It's now time to call for beta-testers of the Python gnulib-tool.
> I plan to post the same text to info-gnu and to planet.gnu.org.

Confirmed success with oath-toolkit; identical generated files.

Old execution time was ~48 seconds, now it is at 0.7 seconds.

The slow gnulib-tool runtime was the primary reason for putting gnulib
generated files into git for oath-toolkit.  Waiting close to a minute
for a fresh rebuild became unbearable during development cycles, and
this is not on the slowest of machines (i7-1260P, 64GB RAM, Samsung SSD
990 PRO).  Now I can experiment with removing gnulib files from version
control again.

jas@kaka:~/src/oath-toolkit$ time make -f cfg.mk 
...
real	0m48,169s
user	0m49,900s
sys	0m9,658s
jas@kaka:~/src/oath-toolkit$ export GNULIB_TOOL_IMPL=py
jas@kaka:~/src/oath-toolkit$ time make -f cfg.mk 
...
real	0m0,704s
user	0m0,527s
sys	0m0,179s
jas@kaka:~/src/oath-toolkit$

In case you doubt this was due to a caching speedup, here is another
invocation right after the previous python run:

jas@kaka:~/src/oath-toolkit$ export GNULIB_TOOL_IMPL=sh
jas@kaka:~/src/oath-toolkit$ time make -f cfg.mk 
...
real	0m49,414s
user	0m50,742s
sys	0m10,332s
jas@kaka:~/src/oath-toolkit$

Thank you so much,
/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: beta-tester call draft

[Pádraig Brady]

On 20/04/2024 01:22, Bruno Haible wrote:
> Hi,
> 
> It's now time to call for beta-testers of the Python gnulib-tool.
> I plan to post the same text to info-gnu and to planet.gnu.org.
> 
> Here is a draft. Please comment!

coreutils results...

>    4. Clean the built files of your package:
>         $ make -k distclean

I also needed to clean out old temp files / dirs that were otherwise gitignored,
which I did with: git clean -f -d -X
One can audit that command first by using -n instead of -f.

With that, the sh+py checks pass.

As for timings:

$ export GNULIB_TOOL_IMPL=sh
$ ./bootstrap
real	2m17.872s
user	1m24.687s
sys	1m20.877s

$ export GNULIB_TOOL_IMPL=py
$ ./bootstrap
real	0m27.423s
user	0m22.367s
sys	0m3.382s


$ time AUTORECONF=true ./bootstrap --skip-po
real	0m7.302s
user	0m5.172s
sys	0m2.755s

Amazing.

thank you!

Pádraig

Re: beta-tester call draft

[Bernhard Voelker]

On 4/20/24 02:22, Bruno Haible wrote:
> Hi,
>
> It's now time to call for beta-testers of the Python gnulib-tool.
> I plan to post the same text to info-gnu and to planet.gnu.org.
>
> Here is a draft. Please comment!

findutils results ...

>   ---------------------------------------------------------------------
> GNU gnulib: calling for beta-testers
>
> If you are developer on a package that uses GNU gnulib as part of its
> build system:
>
> gnulib-tool has been known for being slow for many years. We have
> listened to your complaints. A rewrite of gnulib-tool in another
> programming language (Python) is ready for beta-testing. It is
> between 8 times and 100 times faster than the original gnulib-tool.
>
> Both implementations should behave identically, that is, produce
> the same generated files and the same output. You can help us ensure
> this, through the following steps:
>
>    1. Make sure you have Python (version 3.7 or newer) installed on
>       your machine.

$ python3 --version
Python 3.11.8

>    2. Update your gnulib checkout. (For some packages, it comes as a
>       git submodule named 'gnulib'.) Like this:
>         $ git pull
>       Set the environment variable GNULIB_SRCDIR, pointing to this checkout.

done: gnulib 237cbf1c

>    3. Set an environment variable that enables checking that the two
>       implementations behave the same:
>         $ export GNULIB_TOOL_IMPL=sh+py

done

>    4. Clean the built files of your package:
>         $ make -k distclean

Used
   $ git clean -xdfq && ./bootstrap && ./configure && && make -k distclean
to get a clean working environment.

>    5. Regenerate the fetched and generated files of your package.
>       Depending on the packge, this may be a command such as
>         $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR
>       or
>         $ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh
>       or, if no such script is available:
>         $ $GNULIB_SRCDIR/gnulib-tool --update
>       If there is a failure, due to differences between the 'sh' and 'py'
>       results, please report it to <bug-gnulib@gnu.org>.

How would such a failure look like? Or how should one check?

   $ ./bootstrap

no error. :-)

Regarding the timings:

$ export GNULIB_TOOL_IMPL=sh
$ time ./bootstrap
...
real	1m29.778s
ser	1m21.919s
sys	0m19.456s

$ export GNULIB_TOOL_IMPL=py
$ time ./bootstrap
...
real	0m26.536s
user	0m22.881s
sys	0m0.884s

$ time AUTORECONF=true ./bootstrap --skip-po
...
real	0m3.369s
user	0m2.979s
sys	0m0.520s

Cool!

>    6. If this invocation was successful, you can trust the rewritten
>       gnulib-tool and use it from now on, by setting the environment
>       variable
>         $ export GNULIB_TOOL_IMPL=py
>
>    7. Continue with
>         $ ./configure
>         $ make
>       as usual.
>
> And enjoy the speed! The rewritten gnulib-tool was implemented by
> Dmitry Selyutin, Collin Funk, and me.
>   ---------------------------------------------------------------------

There will come the question whether there are plans to drop the 'sh' mode,
or if or how long both modes will be maintained in parallel.

Have a nice day,
Berny

gnulib-tool: In sh+py mode, don't fail because of dangling symlinks

[Bruno Haible]

Paul Eggert reported:
> I tried this on my well-worn copy of GNU diffutils, and got the 
> following diagnostics. These were all diagnostics about backup files, or 
> files I manually deleted by moving them into a '.del' subdirectory (an 
> old habit of mine), or symlink loops that I had created to test.
> 
> Perhaps the advice should start with, "Start with a fresh checkout from 
> Git."
> 
> -----
> diff: ./.del/gnulib-tests/test-fnmatch-1.sh: No such file or directory
> diff: /home/eggert/src/gnu/glpyXeRWfy/.del/gnulib-tests/test-fnmatch-1.sh: No such file or directory
> ...

These old files most likely were not backup files, but dangling symlinks.
With dangling symlinks, I can reproduce the error, and this patch fixes it:


2024-04-20  Bruno Haible  <bruno@clisp.org>

	gnulib-tool: In sh+py mode, don't fail because of dangling symlinks.
	Reported by Paul Eggert in
	<https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00318.html>
	and Pádraig Brady in
	<https://lists.gnu.org/archive/html/bug-gnulib/2024-04/msg00321.html>.
	* gnulib-tool: In sh+py mode, use diff option '--no-dereference' if
	available.

diff --git a/gnulib-tool b/gnulib-tool
index d7b6d33912..6d430e56e6 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -192,7 +192,14 @@ case "$GNULIB_TOOL_IMPL" in
           func_fatal_error "gnulib-tool.sh succeeded but gnulib-tool.py failed! Inspect $tmp/ and $tmp-py-err."
         fi
         # Compare the two results on the file system.
-        diff -r -q . "$tmp" >/dev/null ||
+        # GNU diffutils 3.3 or newer support option --no-dereference. This
+        # option avoids errors on dangling links.
+        if LC_ALL=C diff --help 2>/dev/null | grep no-dereference >/dev/null; then
+          diff_options='--no-dereference'
+        else
+          diff_options=
+        fi
+        diff -r $diff_options -q . "$tmp" >/dev/null ||
           func_fatal_error "gnulib-tool.py produced different files than gnulib-tool.sh! Compare `pwd` and $tmp."
         # Compare the two outputs.
         diff -q "$tmp-sh-out" "$tmp-py-out" >/dev/null ||
@@ -235,7 +242,14 @@ case "$GNULIB_TOOL_IMPL" in
           func_fatal_error "gnulib-tool.sh succeeded but gnulib-tool.py failed! Inspect $tmp/ and $tmp-py-err."
         fi
         # Compare the two results on the file system.
-        diff -r -q "$dir" "$tmp" >/dev/null ||
+        # GNU diffutils 3.3 or newer support option --no-dereference. This
+        # option avoids errors on dangling links.
+        if LC_ALL=C diff --help 2>/dev/null | grep no-dereference >/dev/null; then
+          diff_options='--no-dereference'
+        else
+          diff_options=
+        fi
+        diff -r $diff_options -q "$dir" "$tmp" >/dev/null ||
           func_fatal_error "gnulib-tool.py produced different files than gnulib-tool.sh! Compare $dir and $tmp."
         # Compare the two outputs.
         diff -q "$tmp-sh-out" "$tmp-py-out" >/dev/null ||

Re: beta-tester call draft

[Bruno Haible]

Hi Paul,

Thanks for the feedback.

>       If you are using git submodules, something like this:
>         $ git -C gnulib pull origin master
>         $ git commit -m 'build: update gnulib submodule to latest' gnulib

With the newest 'bootstrap', this would not be needed. But many packages
still use older or forked 'bootstrap' variants, for which the option
'--no-git' does not do what one would expect. I'm adding this paragraph:

     If the package is using a git submodule named 'gnulib', it is
     also advisable to do
       $ git commit -m 'build: Update gnulib submodule to latest.' gnulib
     (as a preparation for step 5, because the --no-git option does
     not work as expected in all variants of 'bootstrap').

> >    5. Regenerate the fetched and generated files of your package.
> >       Depending on the packge, this may be a command such as
> >         $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR
> >       or
> >         $ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh
> >       or, if no such script is available:
> >         $ $GNULIB_SRCDIR/gnulib-tool --update
> 
> Here, perhaps you can add, "If you are using git submodules, just run a 
> plain './bootstrap'."

This would be a bit dangerous, because 'bootstrap', by default, switches
back to the stored commit.

> >       If there is a failure, due to differences between the 'sh' and 'py'
> >       results, please report it to <bug-gnulib@gnu.org>.
> 
> I tried this on my well-worn copy of GNU diffutils, and got the 
> following diagnostics. These were all diagnostics about backup files, or 
> files I manually deleted by moving them into a '.del' subdirectory (an 
> old habit of mine), or symlink loops that I had created to test.
> 
> Perhaps the advice should start with, "Start with a fresh checkout from 
> Git."

We want to give people trust in using the Python implementation also
in worn working directories, not only in fresh checkouts. And I want to
hear about bugs in worn working directories.

Now that the bug with dangling symlinks is fixed, I feel confident
*not* to ask for a fresh checkout.

Bruno

Re: gnulib-tool.py speedup

[Bruno Haible]

Simon Josefsson wrote:
> jas@kaka:~/src/oath-toolkit$ time make -f cfg.mk 
> ...
> real	0m48,169s
> user	0m49,900s
> sys	0m9,658s
> jas@kaka:~/src/oath-toolkit$ export GNULIB_TOOL_IMPL=py
> jas@kaka:~/src/oath-toolkit$ time make -f cfg.mk 
> ...
> real	0m0,704s
> user	0m0,527s
> sys	0m0,179s
> jas@kaka:~/src/oath-toolkit$

I see similar speedups, using the test suite:

On Linux:                                   On Cygwin 2.9.0:

in create-tests: time ./test-all.sh         in create-tests: time ./test-all.sh
  sh: 1225 sec                                sh: 27406 sec
  py: 155 sec                                 py: 2400 sec
  => about 8 times faster                   => more than 11 times faster

in import-tests: time ./test-wget2-1.sh     in import-tests: time ./test-wget2-1.sh
  sh: 31.9 sec                                sh: 831 sec
  py: 0.35 sec                                py: 43 sec
  => more than 91 times faster              => more than 19 times faster

Bruno

Re: beta-tester call draft

[Bruno Haible]

Bernhard Voelker wrote:
> There will come the question whether there are plans to drop the 'sh' mode,
> or if or how long both modes will be maintained in parallel.

Sure, the question will come. And the conclusion will be based on two
considerations:
  - No package maintainer is expected to say "I don't want to install Python".
  - It is more expensive to maintain two implementations in parallel, than
    only one implementation.

Bruno

Re: beta-tester call draft

[Paul Eggert]

I hope we can drop the sh implementation at the first opportunity - 
which would be after we've tested the Python implementation enough, and 
when we want to add some features and it's a pain to add them to both sh 
and Python.

Re: gnulib-tool.py speedup

[Collin Funk]

Hi Bruno,

On 4/20/24 3:50 PM, Bruno Haible wrote:
> On Linux:                                   On Cygwin 2.9.0:
> 
> in create-tests: time ./test-all.sh         in create-tests: time ./test-all.sh
>   sh: 1225 sec                                sh: 27406 sec
>   py: 155 sec                                 py: 2400 sec
>   => about 8 times faster                   => more than 11 times faster

What shell did you use for this test? Running ./test-all.sh in the
create-tests took me ~1800 (30 minutes) last time I checked. That was
on GNU/Linux with Bash. Would other shells even make a difference?

Also those Cygwin times don't look very fun. The Python one is much
more bearable at least LOL.

Collin

Re: beta-tester call draft

[Bruno Haible]

Paul Eggert wrote:
> I hope we can drop the sh implementation at the first opportunity - 
> which would be after we've tested the Python implementation enough, and 
> when we want to add some features and it's a pain to add them to both sh 
> and Python.

Of course it's a pain to maintain two implementations in parallel.

But we need to give the developers a bit of slack, so that they have time to
  - install Python on their development machines,
  - update their continuous integration recipes accordingly.

Bruno

Re: gnulib-tool.py speedup

[Bruno Haible]

On Sonntag, 21. April 2024 01:01:01 CEST Collin Funk wrote:
> Hi Bruno,
> 
> On 4/20/24 3:50 PM, Bruno Haible wrote:
> > On Linux:                                   On Cygwin 2.9.0:
> > 
> > in create-tests: time ./test-all.sh         in create-tests: time ./test-all.sh
> >   sh: 1225 sec                                sh: 27406 sec
> >   py: 155 sec                                 py: 2400 sec
> >   => about 8 times faster                   => more than 11 times faster
> 
> What shell did you use for this test?

On Linux: dash.                       On Cygwin: bash

> Would other shells even make a difference?

You just have to replace the first line of gnulib-tool.sh:
  #!/bin/sh -> #!/bin/bash

What I measure (with "GNULIB_TOOL_IMPL=sh time ./test-create-testdir-1.sh") is:
  dash  22 sec
  bash  20 sec

I think that 'dash' is generally somewhat faster than 'bash'. However,
gnulib-tool uses special bash syntax for appending to a list and for the
module caching; this probably makes it faster with 'bash' than with 'dash',

What matters most, in the comparison shell vs. Python, IMO, is the string
processing [1].

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00160.html

Re: gnulib-tool.py speedup

[Collin Funk]

On 4/20/24 4:50 PM, Bruno Haible wrote:
> What I measure (with "GNULIB_TOOL_IMPL=sh time ./test-create-testdir-1.sh") is:
>   dash  22 sec
>   bash  20 sec

dash  31 sec
bash  28 sec

Similar here. Looks like my desktop is just older than I remembered
originally.

> I think that 'dash' is generally somewhat faster than 'bash'. However,
> gnulib-tool uses special bash syntax for appending to a list and for the
> module caching; this probably makes it faster with 'bash' than with 'dash',

Ah, I see. I remember seeing the associative arrays for module
caching. I haven't spent much time using shells outside of bash so I
don't know how common support for those extensions are.

> What matters most, in the comparison shell vs. Python, IMO, is the string
> processing [1].

Yes, I remember reading that email. Thank you for the detailed
explanation. I guess that would also explain the Cygwin speed too?
Since as far as I know Windows doesn't have fork() and deals with
creating processes differently. I assume the Cygwin implementation is
limited by that.

Collin

Re: beta-tester call draft

[Bernhard Voelker]

On 4/21/24 01:14, Bruno Haible wrote:
>    - install Python on their development machines,

I'd guess most hosts have python installed nowadays ... the question is
rather which version of it, and how compatible it is:
now it's <3.7 which is incompatible (according to your mail),
but in future there might come more incompatibilities with newer versions.
The shell seems have to been more safe in that regard.

Have a nice day,
Berny

Re: future Python evolution

[Bruno Haible]

Bernhard Voelker wrote:
> The shell seems have to been more safe in that regard.

But the concepts of the shell are stuck in the 40-years-ago past.
That's why it is not recommendable as a programming language for real
programs [1].

> I'd guess most hosts have python installed nowadays ... the question is
> rather which version of it, and how compatible it is:
> now it's <3.7 which is incompatible (according to your mail),
> but in future there might come more incompatibilities with newer versions.

Good point. Yes, Python occasionally (rarely?) makes incompatible changes.
So, I've now created a continuous integration at [2]. If a Python release
is made that breaks gnulib-tool, this CI will notify me shortly afterwards,
and we will have time to adapt gnulib-tool, even before the new Python
release lands in the various distros.

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnulib/2024-03/msg00160.html
[2] https://gitlab.com/gnulib/gnulib-tool-ci

Re: future Python evolution

[Paul Eggert]

On 2024-04-21 07:50, Bruno Haible wrote:
> But the concepts of the shell are stuck in the 40-years-ago past.

True, but keeping things simple allows for optimizations like PaSH that 
can't reasonably be done to Python.

https://binpa.sh/

Re: beta-tester call draft

[Janneke Nieuwenhuizen]

Bernhard Voelker writes:

> On 4/21/24 01:14, Bruno Haible wrote:
>>    - install Python on their development machines,
>
> I'd guess most hosts have python installed nowadays ... the question is
> rather which version of it, and how compatible it is:

What a host has installed (or could install) is of no consequence.  A
much more interesting question is: Are are we creating a problem for
bootstrapping (or even a dependency cycle) when introducing this new
dependency into a certain package.

Greetings,
Janneke

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

Re: future Python evolution

[Bernhard Voelker]

On 4/21/24 16:50, Bruno Haible wrote:
> Bernhard Voelker wrote:
> But the concepts of the shell are stuck in the 40-years-ago past.

Hehe, Python also had its 33rd birthday already. :-)

> Good point. Yes, Python occasionally (rarely?) makes incompatible changes.
> So, I've now created a continuous integration at [2]. If a Python release
> is made that breaks gnulib-tool, this CI will notify me shortly afterwards,
> and we will have time to adapt gnulib-tool, even before the new Python
> release lands in the various distros.

great, thanks!

Have a nice day,
Berny

Re: full-source bootstrap and Python

[Bruno Haible]

Janneke Nieuwenhuizen wrote:
> Are are we creating a problem for
> bootstrapping (or even a dependency cycle) when introducing this new
> dependency into a certain package.

I think you answered this question with "no", when writing in [1]:

  "Even more recently (2018), the GNU C Library glibc-2.28 adds Python
   as a build requirement"

So, how do you avoid Python when building glibc? Do you use musl libc as
a first stage, and only build glibc once a python built with musl exists?

Also, from the diagrams in [1][2][3] it looks like the full-source bootstrap
uses tarballs frozen in time (make-3.80, gcc-2.95.3, gcc 4.7.3, etc.). So,
even if newer versions of 'make' or 'gcc' will use a Python-based gnulib-tool,
there won't be a problem, because the bootstrap of these old tarballs will
be unaffected.

Bruno

[1] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/
[2] https://guix.gnu.org/manual/en/html_node/Reduced-Binary-Seed-Bootstrap.html
[3] https://guix.gnu.org/manual/devel/en/html_node/Full_002dSource-Bootstrap.html

Re: future Python evolution

[Bruno Haible]

Hi Paul,

> > But the concepts of the shell are stuck in the 40-years-ago past.
> 
> True, but keeping things simple allows for optimizations like PaSH that 
> can't reasonably be done to Python.
> 
> https://binpa.sh/

Well, I did try PaSh on gnulib-tool — this issue [1] is a testimony.

But what can you expect from a parallelization approach? On, say, a
quad-core processor you can expect at most a 4x speedup. Which means,
the parallelized gnulib-tool.sh would still be 2 times to 25 times
slower than the Python rewrite.

Also, has PaSh been applied to configure scripts? I recall that
configure script parallelization had been a topic for Ralf Wildenhues
(before Google swallowed him).

   Bruno

[1] https://github.com/binpash/pash/issues/573

Re: future Python evolution

[Paul Eggert]

On 2024-04-21 15:38, Bruno Haible wrote:
> Hi Paul,
> 
>>> But the concepts of the shell are stuck in the 40-years-ago past.
>>
>> True, but keeping things simple allows for optimizations like PaSH that
>> can't reasonably be done to Python.
>>
>> https://binpa.sh/
> 
> Well, I did try PaSh on gnulib-tool — this issue [1] is a testimony.

I agree that PaSh is not ready to tackle 'configure' scripts yet. 
However, it's promising and I wouldn't expect similar promise from 
Python script acceleration.

A better way to exploit PaSh would be to modify Autoconf to use it 
effectively. This of course would be nontrivial, though it shouldn't be 
*that* hard.


> But what can you expect from a parallelization approach? On, say, a
> quad-core processor you can expect at most a 4x speedup.

Quad-core is not the wave of the future. Even the three-year-old (and 
now discontinued) Xeon W-1350 I'm typing this on (which was trailing 
edge and bottom of the line when it came out - hey, I'm a cheapskate!) 
is 6 cores and 12 threads. And if you've been following recent CPU news 
you're aware of the big core counts coming down the pipeline. We should 
be engineering for these future systems, and not worry too much about 
yesterday's quad-core CPUs.

And if one can't get a decent single node to develop on, there's always 
DiSh on the horizon....

https://github.com/binpash/dish

Re: full-source bootstrap and Python

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 3653 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> Janneke Nieuwenhuizen wrote:
>> Are are we creating a problem for
>> bootstrapping (or even a dependency cycle) when introducing this new
>> dependency into a certain package.
>
> I think you answered this question with "no", when writing in [1]:
>
>   "Even more recently (2018), the GNU C Library glibc-2.28 adds Python
>    as a build requirement"
>
> So, how do you avoid Python when building glibc? Do you use musl libc as
> a first stage, and only build glibc once a python built with musl exists?
>
> Also, from the diagrams in [1][2][3] it looks like the full-source bootstrap
> uses tarballs frozen in time (make-3.80, gcc-2.95.3, gcc 4.7.3, etc.). So,
> even if newer versions of 'make' or 'gcc' will use a Python-based gnulib-tool,
> there won't be a problem, because the bootstrap of these old tarballs will
> be unaffected.

While I agree, I think there is one nuance that could be added here: it
is true that full-source bootstraps usually needs to use earlier
releases of software tarballs to build more recent projects because of
cyclic dependencies.  However this cause extra work for people involved.
It also means we will have to keep maintaining and patching all the
software that is involved in the full-source bootstrap to keep it
working in the future.  So there is a cost involved here.

The takeaway from that situation should NOT be "don't use python" or
"don't use modern tools", that would be absurd.

The takeaway should be that one should carefully evaluate the
implications of using Python and modern tools, and look at the costs.

If we want to minimize the work for full-source bootstrap people we
increase the cost of people maintaining modern software, and vice versa,
I don't see how we can get away from this conflict.  If everyone wrote
everything in machine code, there would be nothing for the bootstrap
people to do.

However what we can do is to reduce the total amount of work involved by
not introducing too many bootstrap dependency cycles.  Consider the
extreme situation where gnulib-tool version A would require coreutils
verison B, and coreutils version B+1 would require gnulib-tool version
A+1, and gnulib-tool version A+2 would require coreutils version B+1 and
so on for really short release version increments.  Then a full-source
bootstrap will need to package and keep maintain all those coreutils and
gnulib-tool versions -- or start to patch things to avoid the
dependencies.  (I'm ignoring the fact that normally gnulib-tool is not
involved at all when building projects.)

I think gnulib is already quite careful in dependency tracking, more so
than most projects I'm familiar with, but it doesn't mean this can be
improved.

Janneke: is there any recommendation from you as a bootstrapping person
on what dependency we should be careful with, and which dependencies
that are fine?  I suppose/hope that if gnulib-tool required python
version 2, you would not have a serious problem?  I'm certain that
python version 2 is possible to build using really old toolchains.  At
what version of python would it lead require added another bootstrapping
step to the graph?  Python 3.0, 3.7, 3.12?  Maybe it is not easy to
answer this without generating the graph.  But I also think you would
have a better feeling of what the answer would be than most of us.

/Simon


> Bruno
>
> [1] https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/
> [2] https://guix.gnu.org/manual/en/html_node/Reduced-Binary-Seed-Bootstrap.html
> [3] https://guix.gnu.org/manual/devel/en/html_node/Full_002dSource-Bootstrap.html
>
>
>
>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: full-source bootstrap and Python

[Janneke Nieuwenhuizen]

Bruno Haible writes:

Hi!

> Janneke Nieuwenhuizen wrote:
>> Are are we creating a problem for
>> bootstrapping (or even a dependency cycle) when introducing this new
>> dependency into a certain package.
>
> I think you answered this question with "no", when writing in [1]:
>
>   "Even more recently (2018), the GNU C Library glibc-2.28 adds Python
>    as a build requirement"

How so?  In 2013, the GCC folks decided that their gcc-4.8 would no
longer be a directly bootstrappable compiler by introducing a (casual?)
dependency on c++.  That's pretty bad for bootstrapping, and it would be
amazing if someone could revet that mistake.  With glibc-2.28, a similar
bootstrappable mistake was made.

Making an essential GNU package such as GCC or Glibc non-bootstrappable
has severe consequences.  As an example, we have been working on the
RISC-V bootstrap for about a year with three people.  One of the
problems here is that RISC-V was only added to a non-bootstrappable
version of GCC: 7.5.0, while the GCC team failed to maintain their last
bootstrappable version: 4.7.4.  In other words, the RISC-V backend
needed to backported and someone else now needs to maintain a
bootstrappable version of GCC.

When something else like this changes in the future, that isn't as
"easily" backported, what do we do?  Terrible!

> So, how do you avoid Python when building glibc? Do you use musl libc as
> a first stage, and only build glibc once a python built with musl exists?

Currently we aren't directly hit by this because we build glibc-2.2.5
and glibc-2.16 first.  We added these earlier Glibc's because we allowed
ourcelves to cut some corners and there earlier versions were more easy
to bootstrap.  On the roadmap is to remove as many ancient versions as
we can, as they are potential time-bombs.  In fact, glibc-2.2.5 is
problematic for the ARM and the RISC-V bootstrap, so we'll get rid of
that.

We won't be able to get rid of advancing beyond glibc-2.27 without
adding a yet another non-GNU package such as musl libc.  Possibly a nice
hack for now, but what to do when we want to port the bootstrap to the
Hurd?  Again, terrible!

> Also, from the diagrams in [1][2][3] it looks like the full-source bootstrap
> uses tarballs frozen in time (make-3.80, gcc-2.95.3, gcc 4.7.3, etc.). So,
> even if newer versions of 'make' or 'gcc' will use a Python-based gnulib-tool,
> there won't be a problem, because the bootstrap of these old tarballs will
> be unaffected.

indeed.  For the current situtation (that's less than great and are
working on to resolve), making essential GNU packages less
bootstrappable is of no consequence.  Cleaning-up the full-source
bootstrap and making it more or less future-proof, might be challenged
by such a new dependency.

Greetings,
Janneke

-- 
Janneke Nieuwenhuizen <janneke@gnu.org>  | GNU LilyPond https://LilyPond.org
Freelance IT https://www.JoyOfSource.com | Avatar® https://AvatarAcademy.com

Re: full-source bootstrap and Python

[Bruno Haible]

Simon Josefsson wrote:
> If we want to minimize the work for full-source bootstrap people we
> increase the cost of people maintaining modern software, and vice versa,

+1

> Consider the
> extreme situation where gnulib-tool version A would require coreutils
> verison B, and coreutils version B+1 would require gnulib-tool version
> A+1, and gnulib-tool version A+2 would require coreutils version B+1 and
> so on for really short release version increments.  Then a full-source
> bootstrap will need to package and keep maintain all those coreutils and
> gnulib-tool versions -- or start to patch things to avoid the
> dependencies.

Right. This way of working was pretty common in the Lisp and Haskell worlds,
many years ago. For gnulib-tool, we're relying on Python 3.7, which is
already 6 years old; so, we are clearly not doing the extreme thing.

Bruno

Re: full-source bootstrap and Python

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1989 bytes --]

Janneke Nieuwenhuizen <janneke@gnu.org> writes:

>> Also, from the diagrams in [1][2][3] it looks like the full-source bootstrap
>> uses tarballs frozen in time (make-3.80, gcc-2.95.3, gcc 4.7.3, etc.). So,
>> even if newer versions of 'make' or 'gcc' will use a Python-based gnulib-tool,
>> there won't be a problem, because the bootstrap of these old tarballs will
>> be unaffected.
>
> indeed.  For the current situtation (that's less than great and are
> working on to resolve), making essential GNU packages less
> bootstrappable is of no consequence.  Cleaning-up the full-source
> bootstrap and making it more or less future-proof, might be challenged
> by such a new dependency.

Rather than finding out what dependencies are problematic through
tedious manual work, is there a recommendation we can articulate that
would help the bootstrappable effort?

For example, in Libtasn1 (which I guess is fairly low in the
bootstrapping graph) I made the CI/CD pipeline [1] build the tarball on
Debian 4 etch (2010, first amd64 release), and using 'pcc' and 'tcc' as
alternative C compilers.  I'm hoping this has some value, but I have no
good way to tell.  What actual testable environments would it make sense
to test a project in, to help the bootstrappable effort?  Right now
these targets build fine, but if at some point 'pcc' stops building, I
may be inclinced to simply drop this target rather than to fix the bugs
since I have no idea if supporting building with 'pcc' helps anyone.

I'm thinking suggestions like 'Build and test project on i386 Debian 3',
or 'Cross-build project from amd64 to mipsel on Debian 4'.  I can't seem
to find docker images for CentOS 3-6, maybe old CentOS is a good
long-term target too.  If there were concrete fact-based suggestions
like that, I would make an effort to CI/CD build libidn, libidn2,
inetutils, and some other projects to make sure they continue to work on
old platforms.

/Simon

[1] https://gitlab.com/gnutls/libtasn1/-/pipelines/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: full-source bootstrap and Python

[Bruno Haible]

Janneke Nieuwenhuizen wrote:
> For the current situtation (that's less than great and are
> working on to resolve), making essential GNU packages less
> bootstrappable is of no consequence.

OK. That's what I conjectured. Thanks for confirming.

> Cleaning-up the full-source
> bootstrap and making it more or less future-proof, might be challenged
> by such a new dependency.

I see. So, I wondered whether Python can be compiled by the last C-only
GCC, gcc-4.7.3. And yes, Python-3.7.17 (the minimum supported version
for gnulib-tool) compiles with gcc-4.7.3. Whereas a newer version,
Python-3.12.2, doesn't.

> As an example, we have been working on the
> RISC-V bootstrap for about a year with three people.  One of the
> problems here is that RISC-V was only added to a non-bootstrappable
> version of GCC: 7.5.0, while the GCC team failed to maintain their last
> bootstrappable version: 4.7.4.  In other words, the RISC-V backend
> needed to backported and someone else now needs to maintain a
> bootstrappable version of GCC.

Oh, you don't allow yourself to use a cross-compiler from, say, x86?
I mean, running a complete x86 bootstrap, followed by the creation of
a cross-compiled RISC-V compiler, in a virtualized environment on RISC-V
could work in theory (although it would be very slow).

Bruno

Re: future Python evolution

[Bernhard Voelker]

On 4/21/24 16:50, Bruno Haible wrote:
> Good point. Yes, Python occasionally (rarely?) makes incompatible changes.
> So, I've now created a continuous integration at [2]. If a Python release
> is made that breaks gnulib-tool, this CI will notify me shortly afterwards,
> and we will have time to adapt gnulib-tool, even before the new Python
> release lands in the various distros.

great, thanks!  (sorry, the mail didn't go out earlier.)

Have a nice day,
Berny