bug-gnulib@gnu.org mirror (unofficial)
 help / color / mirror / Atom feed
* [PATCH] gnulib-tool.py: Fix function call on incorrect object.
@ 2024-02-19  1:24 Collin Funk
  2024-02-19  2:17 ` Bruno Haible
  0 siblings, 1 reply; 12+ messages in thread
From: Collin Funk @ 2024-02-19  1:24 UTC (permalink / raw)
  To: bug-gnulib

[-- Attachment #1: Type: text/plain, Size: 1451 bytes --]

I was looking around the gnulib code and decided to try out the Python
gnulib-tool. It looks like it hasn't been touched in a while so I
wanted to see what state it was in. Coreutils fails because
--automake-subdir is not supported (already documented in
gnulib-tool.py.TODO). GNU m4 gave the following error:

bootstrap: running: gnulib/gnulib-tool.py --no-changelog --no-libtool --symlink --update
Traceback (most recent call last):
  File "/home/collin/.local/src/m4/gnulib/gnulib-tool.py", line 1171, in <module>
    main()
  File "/home/collin/.local/src/m4/gnulib/gnulib-tool.py", line 886, in main
    importer = classes.GLImport(config, mode)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/collin/.local/src/m4/gnulib/pygnulib/GLImport.py", line 243, in __init__
    if self.checkInclTestCategory(TESTS['tests']) and self.config['conddeps']:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'GLImport' object has no attribute 'checkInclTestCategory'

I assume this was just caused by a small coding typo (if that is a
thing). This patch should fix it. The M4 build still fails because of
the !@NMD@ which is documented in TODO along with a few other errors
with the generated Makefile.

What is the status of the Python gnulib tool? I'm not sure how far
behind it is compared to the shell script but it seems like it would
be much faster. I would say more maintainable but I might just be bad
at writing shell scripts. :)

Collin

[-- Attachment #2: 0001-gnulib-tool.py-Fix-function-call-on-incorrect-object.patch --]
[-- Type: text/x-patch, Size: 2766 bytes --]

From 49adb8c6c480d42a2ad90f52fde10df9929506fb Mon Sep 17 00:00:00 2001
From: Collin Funk <collin.funk1@gmail.com>
Date: Sun, 18 Feb 2024 16:52:45 -0800
Subject: [PATCH] gnulib-tool.py: Fix function call on incorrect object.

* pygnulib/GLImport.py: Call checkInclTestCategory on the
GLConfig member instead of the GLImport object itself.
* pygnulib/__init__.py: Update copyright dates.
* pygnulib/constants.py: Update copyright dates.
---
 ChangeLog             | 8 ++++++++
 pygnulib/GLImport.py  | 2 +-
 pygnulib/__init__.py  | 2 +-
 pygnulib/constants.py | 2 +-
 4 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 3a8559bfb3..b8d484e8da 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2024-02-18  Collin Funk  <collin.funk1@gmail.com>
+
+	gnulib-tool.py: Fix function call on incorrect object.
+	* pygnulib/GLImport.py: Call checkInclTestCategory on the
+	GLConfig member instead of the GLImport object itself.
+	* pygnulib/__init__.py: Update copyright dates.
+	* pygnulib/constants.py: Update copyright dates.
+
 2024-02-18  Bruno Haible  <bruno@clisp.org>
 
 	maint.mk: Add more comments.
diff --git a/pygnulib/GLImport.py b/pygnulib/GLImport.py
index b09ba9868c..68b2e7b0bb 100644
--- a/pygnulib/GLImport.py
+++ b/pygnulib/GLImport.py
@@ -240,7 +240,7 @@ class GLImport(object):
                 modules = self.cache.getModules()
 
             # If user tries to apply conddeps and TESTS['tests'] together.
-            if self.checkInclTestCategory(TESTS['tests']) and self.config['conddeps']:
+            if self.config.checkInclTestCategory(TESTS['tests']) and self.config['conddeps']:
                 raise GLError(10, None)
 
             # Update configuration dictionary.
diff --git a/pygnulib/__init__.py b/pygnulib/__init__.py
index 37cd33ef90..b9050a98ac 100644
--- a/pygnulib/__init__.py
+++ b/pygnulib/__init__.py
@@ -30,6 +30,6 @@ coding standards, the GNU maintainer information, the GPL and other licenses (in
 Texinfo), assorted configuration scripts, and more. The goal is to provide all
 the common infrastructure needed by GNU packages.'''
 
-__copyright__ = '2012-2022 Free Software Foundation, Inc.'
+__copyright__ = '2012-2024 Free Software Foundation, Inc.'
 __author__ = 'Dmitry Selyutin'
 __license__ = 'GNU GPLv3+'
diff --git a/pygnulib/constants.py b/pygnulib/constants.py
index a7e3d99ceb..a5554956c1 100644
--- a/pygnulib/constants.py
+++ b/pygnulib/constants.py
@@ -42,7 +42,7 @@ __author__ = \
         'Dmitry Selyutin',
     ]
 __license__ = 'GNU GPLv3+'
-__copyright__ = '2002-2022 Free Software Foundation, Inc.'
+__copyright__ = '2002-2024 Free Software Foundation, Inc.'
 
 
 #===============================================================================
-- 
2.39.2


^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH] gnulib-tool.py: Fix function call on incorrect object.
  2024-02-19  1:24 [PATCH] gnulib-tool.py: Fix function call on incorrect object Collin Funk
@ 2024-02-19  2:17 ` Bruno Haible
  2024-02-19  8:59   ` Simon Josefsson via Gnulib discussion list
  2024-02-19 18:37   ` [PATCH] gnulib-tool.py: Fix function call on incorrect object Collin Funk
  0 siblings, 2 replies; 12+ messages in thread
From: Bruno Haible @ 2024-02-19  2:17 UTC (permalink / raw)
  To: bug-gnulib; +Cc: Collin Funk

Hi Collin,

> GNU m4 gave the following error:
> 
> bootstrap: running: gnulib/gnulib-tool.py --no-changelog --no-libtool --symlink --update
> Traceback (most recent call last):
>   File "/home/collin/.local/src/m4/gnulib/gnulib-tool.py", line 1171, in <module>
>     main()
>   File "/home/collin/.local/src/m4/gnulib/gnulib-tool.py", line 886, in main
>     importer = classes.GLImport(config, mode)
>                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   File "/home/collin/.local/src/m4/gnulib/pygnulib/GLImport.py", line 243, in __init__
>     if self.checkInclTestCategory(TESTS['tests']) and self.config['conddeps']:
>        ^^^^^^^^^^^^^^^^^^^^^^^^^^
> AttributeError: 'GLImport' object has no attribute 'checkInclTestCategory'
> 
> I assume this was just caused by a small coding typo (if that is a
> thing). This patch should fix it.

Thanks! Applied.

> What is the status of the Python gnulib tool? I'm not sure how far
> behind it is compared to the shell script but it seems like it would
> be much faster. I would say more maintainable but I might just be bad
> at writing shell scripts. :)

Yes, it's the hope that it will be faster that is the main motivation
behind the Python rewrite.

The status: It's about 2/3 complete. 4 months of work have gone into it,
and 2 months of work still remaining. (That's my estimation, based on
two weeks of work that I put in, in 2022.)
65% of the .py code has been verified to be in sync with the bash code;
35% still to go. And then, the changes from the gnulib-tool.py.TODO list
need to me implemented on the Python side.
Then would come a phase of public testing...

Bruno





^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] gnulib-tool.py: Fix function call on incorrect object.
  2024-02-19  2:17 ` Bruno Haible
@ 2024-02-19  8:59   ` Simon Josefsson via Gnulib discussion list
  2024-02-19 10:38     ` gnulib-tool caching Bruno Haible
  2024-02-19 18:37   ` [PATCH] gnulib-tool.py: Fix function call on incorrect object Collin Funk
  1 sibling, 1 reply; 12+ messages in thread
From: Simon Josefsson via Gnulib discussion list @ 2024-02-19  8:59 UTC (permalink / raw)
  To: Bruno Haible; +Cc: bug-gnulib, Collin Funk

[-- Attachment #1: Type: text/plain, Size: 1397 bytes --]

Bruno Haible <bruno@clisp.org> writes:

>> What is the status of the Python gnulib tool? I'm not sure how far
>> behind it is compared to the shell script but it seems like it would
>> be much faster. I would say more maintainable but I might just be bad
>> at writing shell scripts. :)
>
> Yes, it's the hope that it will be faster that is the main motivation
> behind the Python rewrite.

Orthogonal to a rewrite in python: is it possible to design a reliable
caching mechanism?  Something similar to CONFIG_SITE for autoconf?

I find that ./gnulib-tool takes a long time and 95% of the time I use
it, it ended up doing exactly the same thing as it did last time I ran
it: copying a set of possibly patched files out of the gnulib directory.

How about logic like this:

. $GNULIB_SITE
if test -d $gnulib_cache_dir; then
  rsync -av $gnulib_cache_dir .
else if test -n "$gnulib_cache_dir"; then
  mkdir $savedir
  rsync -av . $savedir

  # do whatever gnulib normally is doing

  # compare . with $savedir, saving a copy of each modified
  # file into $gnulib_cache_dir
fi

then I could put something like this into a $GNULIB_SITE script:

if test -z "$gnulib_cache_dir"; then
    hash=`echo $PWD|md5sum|cut -d' ' -f1`
    my_cache_dir=$HOME/.cache/gnulib.site
    gnulib_cache_dir=$my_cache_dir/cache.`basename $PWD`.$hash
    test -d $gnulib_cache_dir || mkdir -p $gnulib_cache_dir
fi

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: gnulib-tool caching
  2024-02-19  8:59   ` Simon Josefsson via Gnulib discussion list
@ 2024-02-19 10:38     ` Bruno Haible
  2024-02-19 12:55       ` Simon Josefsson via Gnulib discussion list
  0 siblings, 1 reply; 12+ messages in thread
From: Bruno Haible @ 2024-02-19 10:38 UTC (permalink / raw)
  To: Simon Josefsson; +Cc: bug-gnulib, Collin Funk

Simon Josefsson wrote:
> is it possible to design a reliable
> caching mechanism?  Something similar to CONFIG_SITE for autoconf?

CONFIG_SITE is not reliable; that's the problem with it...

> I find that ./gnulib-tool takes a long time and 95% of the time I use
> it, it ended up doing exactly the same thing as it did last time I ran
> it: copying a set of possibly patched files out of the gnulib directory.

I use gnulib-tool with option --symlink in such cases. As long as the
module descriptions don't change, you don't need to re-run gnulib-tool
then.

Bruno





^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: gnulib-tool caching
  2024-02-19 10:38     ` gnulib-tool caching Bruno Haible
@ 2024-02-19 12:55       ` Simon Josefsson via Gnulib discussion list
  2024-02-19 21:50         ` Bruno Haible
  0 siblings, 1 reply; 12+ messages in thread
From: Simon Josefsson via Gnulib discussion list @ 2024-02-19 12:55 UTC (permalink / raw)
  To: Bruno Haible; +Cc: bug-gnulib, Collin Funk

[-- Attachment #1: Type: text/plain, Size: 2259 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> Simon Josefsson wrote:
>> is it possible to design a reliable
>> caching mechanism?  Something similar to CONFIG_SITE for autoconf?
>
> CONFIG_SITE is not reliable; that's the problem with it...
>
>> I find that ./gnulib-tool takes a long time and 95% of the time I use
>> it, it ended up doing exactly the same thing as it did last time I ran
>> it: copying a set of possibly patched files out of the gnulib directory.
>
> I use gnulib-tool with option --symlink in such cases. As long as the
> module descriptions don't change, you don't need to re-run gnulib-tool
> then.

My usage pattern is to frequently re-bootstrap from a clean git checkout
to confirm that my changes still work properly for fresh rebuilds.  The
reason is that I don't trust 'make clean', 'make distclean', etc.  That
may not be a common work flow, but I'm pretty committed to it.  I now
remember that something like this was discussed before:

https://git.savannah.gnu.org/cgit/libidn.git/commit/?id=9ae53e866a6fafa56db26d184ccae9c39dae7446
https://lists.gnu.org/archive/html/bug-gnulib/2021-05/msg00077.html

I don't recall if I actually had any real problems with that approach...
however I got a faster laptop and stoped using it to mimize any
deviation from standard gnulib workflows.

Maybe a hook below would allow further experiments?  I am not proposing
to commit this now since it is relatively easy for me to experiment with
improvements by adding this to the bootstrap.conf script on projects
that I work on.  The GNULIB_BOOTSTRAP_SITE also allows me to change my
setup without altering bootstrap.conf for every project I work on.

/Simon

diff --git a/top/bootstrap-funclib.sh b/top/bootstrap-funclib.sh
index 9e40f4a3e4..9b8972f56c 100644
--- a/top/bootstrap-funclib.sh
+++ b/top/bootstrap-funclib.sh
@@ -194,6 +194,7 @@ bootstrap_sync=false
 # Make sure that bootstrap.conf is sourced from the current directory
 # if we were invoked as "sh bootstrap".
 conffile=`dirname "$me"`/bootstrap.conf
+test -r "$GNULIB_BOOTSTRAP_SITE" && . "$GNULIB_BOOTSTRAP_SITE"
 test -r "$conffile" && . "$conffile"
 
 # ------------------------- Build-time prerequisites -------------------------

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: [PATCH] gnulib-tool.py: Fix function call on incorrect object.
  2024-02-19  2:17 ` Bruno Haible
  2024-02-19  8:59   ` Simon Josefsson via Gnulib discussion list
@ 2024-02-19 18:37   ` Collin Funk
  2024-02-19 21:36     ` Bruno Haible
  1 sibling, 1 reply; 12+ messages in thread
From: Collin Funk @ 2024-02-19 18:37 UTC (permalink / raw)
  To: Bruno Haible; +Cc: bug-gnulib

Bruno Haible <bruno@clisp.org> writes:
> The status: It's about 2/3 complete. 4 months of work have gone into it,
> and 2 months of work still remaining. (That's my estimation, based on
> two weeks of work that I put in, in 2022.)
> 65% of the .py code has been verified to be in sync with the bash code;
> 35% still to go. And then, the changes from the gnulib-tool.py.TODO list

I see. I had it open and compared some of it to the shell script and it
wasn't too hard to follow. Maybe I could help hack away at some of in my
free time.

I assume the best way to test it would be to create a small test
directory with a few modules? I think all of the header file
replacements would break the build so whichever ones avoid using those.


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] gnulib-tool.py: Fix function call on incorrect object.
  2024-02-19 18:37   ` [PATCH] gnulib-tool.py: Fix function call on incorrect object Collin Funk
@ 2024-02-19 21:36     ` Bruno Haible
  2024-02-19 22:42       ` Collin Funk
  0 siblings, 1 reply; 12+ messages in thread
From: Bruno Haible @ 2024-02-19 21:36 UTC (permalink / raw)
  To: Collin Funk; +Cc: bug-gnulib

Collin Funk wrote:
> > 65% of the .py code has been verified to be in sync with the bash code;
> > 35% still to go. And then, the changes from the gnulib-tool.py.TODO list
> 
> I see. I had it open and compared some of it to the shell script and it
> wasn't too hard to follow. Maybe I could help hack away at some of in my
> free time.

If you want to do help with gnulib-tool.py, I would suggest not to continue
directly with the comparison / review (because it's hard to keep track of
what has been reviewed / tested and what has not), but rather work across
the gnulib-tool.py.TODO file, from the bottom to the top.

> I assume the best way to test it would be to create a small test
> directory with a few modules?

The best way to test it is to take a specific source package that uses a
number of modules (I used wget2, but you can use coreutils or any other
package), run gnulib-tool and gnulib-tool.py with identical command-line
options, and compare/verify the results on either side.

> I think all of the header file
> replacements would break the build so whichever ones avoid using those.

You don't need to go to the './configure' and 'make' stages at this point.
Just ensuring that the outputs of gnulib-tool and gnulib-tool.py are the
same will sufficiently guide you.

Bruno





^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: gnulib-tool caching
  2024-02-19 12:55       ` Simon Josefsson via Gnulib discussion list
@ 2024-02-19 21:50         ` Bruno Haible
  2024-02-19 22:24           ` Sam James
  0 siblings, 1 reply; 12+ messages in thread
From: Bruno Haible @ 2024-02-19 21:50 UTC (permalink / raw)
  To: Simon Josefsson; +Cc: bug-gnulib

Simon Josefsson wrote:
> I now
> remember that something like this was discussed before:
> 
> https://git.savannah.gnu.org/cgit/libidn.git/commit/?id=9ae53e866a6fafa56db26d184ccae9c39dae7446
> https://lists.gnu.org/archive/html/bug-gnulib/2021-05/msg00077.html

I see... you are building a cache that will become invalid when either
  - the bootstrap.conf changes, or
  - there is a change in gnulib in one of the request modules (in the
    module description or in code).

> My usage pattern is to frequently re-bootstrap from a clean git checkout
> to confirm that my changes still work properly for fresh rebuilds.  The
> reason is that I don't trust 'make clean', 'make distclean', etc.

With your cache, you'll now have to trust your decision whether to use
or erase the cache. But at least, you can automate this decision:
erase the cache when bootstrap.conf or the referenced gnulib submodule
was changed.

In other words, replace the cache file name
/home/jas/.local/gnulib-bootstrap-cache/bootstrap-files-for-libidn.tar.gz
with
/home/jas/.local/gnulib-bootstrap-cache/bootstrap-files-for-libidn-<gnulib_submodule_commit>-<hash_code_of_bootstrap.conf>.tar.gz

Bruno





^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: gnulib-tool caching
  2024-02-19 21:50         ` Bruno Haible
@ 2024-02-19 22:24           ` Sam James
  2024-02-19 23:11             ` Bruno Haible
  0 siblings, 1 reply; 12+ messages in thread
From: Sam James @ 2024-02-19 22:24 UTC (permalink / raw)
  To: Bruno Haible; +Cc: Simon Josefsson, bug-gnulib


Bruno Haible <bruno@clisp.org> writes:

> Simon Josefsson wrote:
>> I now
>> remember that something like this was discussed before:
>> 
>> https://git.savannah.gnu.org/cgit/libidn.git/commit/?id=9ae53e866a6fafa56db26d184ccae9c39dae7446
>> https://lists.gnu.org/archive/html/bug-gnulib/2021-05/msg00077.html
>
> I see... you are building a cache that will become invalid when either
>   - the bootstrap.conf changes, or
>   - there is a change in gnulib in one of the request modules (in the
>     module description or in code).

We could also probably cache based on (g)libc version, kernel version,
compiler & linker version, and any dependencies which gnulib modules may
use (e.g. OpenSSL).

While that might sound a bit much, it should be pretty quick to
calculate compared to no cache at all.

With that, you can do pretty well, I think. Whether or not it's worth it
is another question. We'd still need people to be responsible about it
(not use it recklessly, just when they're developing and know when their
system changes).

I wasn't aware of the Python tool work and am intrigued.



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] gnulib-tool.py: Fix function call on incorrect object.
  2024-02-19 21:36     ` Bruno Haible
@ 2024-02-19 22:42       ` Collin Funk
  0 siblings, 0 replies; 12+ messages in thread
From: Collin Funk @ 2024-02-19 22:42 UTC (permalink / raw)
  To: Bruno Haible; +Cc: bug-gnulib

On 2/19/24 1:36 PM, Bruno Haible wrote:
> You don't need to go to the './configure' and 'make' stages at this point.
> Just ensuring that the outputs of gnulib-tool and gnulib-tool.py are the
> same will sufficiently guide you.

Ah, I see. Thanks for this and the rest of the advice.

Collin



^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: gnulib-tool caching
  2024-02-19 22:24           ` Sam James
@ 2024-02-19 23:11             ` Bruno Haible
  2024-02-19 23:17               ` Sam James
  0 siblings, 1 reply; 12+ messages in thread
From: Bruno Haible @ 2024-02-19 23:11 UTC (permalink / raw)
  To: Sam James; +Cc: Simon Josefsson, bug-gnulib

Sam James wrote:
> > I see... you are building a cache that will become invalid when either
> >   - the bootstrap.conf changes, or
> >   - there is a change in gnulib in one of the request modules (in the
> >     module description or in code).
> 
> We could also probably cache based on (g)libc version, kernel version,
> compiler & linker version, and any dependencies which gnulib modules may
> use (e.g. OpenSSL).

We are talking about different things.
1) First comes the './autogen.sh' execution (or 'bootstrap'), which
   generates files, by calling gnulib-tool and the Autotools.
2) Then comes the './configure' step.

We are talking about caching the result of 1).
You seem to be thinking about 2).

Bruno





^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: gnulib-tool caching
  2024-02-19 23:11             ` Bruno Haible
@ 2024-02-19 23:17               ` Sam James
  0 siblings, 0 replies; 12+ messages in thread
From: Sam James @ 2024-02-19 23:17 UTC (permalink / raw)
  To: Bruno Haible; +Cc: Sam James, Simon Josefsson, bug-gnulib


Bruno Haible <bruno@clisp.org> writes:

> Sam James wrote:
>> > I see... you are building a cache that will become invalid when either
>> >   - the bootstrap.conf changes, or
>> >   - there is a change in gnulib in one of the request modules (in the
>> >     module description or in code).
>> 
>> We could also probably cache based on (g)libc version, kernel version,
>> compiler & linker version, and any dependencies which gnulib modules may
>> use (e.g. OpenSSL).
>
> We are talking about different things.
> 1) First comes the './autogen.sh' execution (or 'bootstrap'), which
>    generates files, by calling gnulib-tool and the Autotools.
> 2) Then comes the './configure' step.
>
> We are talking about caching the result of 1).
> You seem to be thinking about 2).

Oh, I see. Thanks!

>
> Bruno



^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2024-02-19 23:18 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-19  1:24 [PATCH] gnulib-tool.py: Fix function call on incorrect object Collin Funk
2024-02-19  2:17 ` Bruno Haible
2024-02-19  8:59   ` Simon Josefsson via Gnulib discussion list
2024-02-19 10:38     ` gnulib-tool caching Bruno Haible
2024-02-19 12:55       ` Simon Josefsson via Gnulib discussion list
2024-02-19 21:50         ` Bruno Haible
2024-02-19 22:24           ` Sam James
2024-02-19 23:11             ` Bruno Haible
2024-02-19 23:17               ` Sam James
2024-02-19 18:37   ` [PATCH] gnulib-tool.py: Fix function call on incorrect object Collin Funk
2024-02-19 21:36     ` Bruno Haible
2024-02-19 22:42       ` Collin Funk

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).