backupfile and backup-replace are introducing the same object to make

backupfile and backup-replace are introducing the same object to make

[(GalaxyMaster)]

Hello,

[I am not a subscriber, so if you need anything from me, please CC me]

I just stumbled upon the following: when I include both backupfile and
backup-replace modules I am getting a broken build due to multiple definitions
of "simple_backup_suffix'.

When I compared modules, sure enough I found the issue:
===
--- backupfile  2023-01-23 03:29:03.000000000 +1100
+++ backup-rename       2023-01-23 03:29:03.000000000 +1100
[...]
 Makefile.am:
-lib_SOURCES += backupfile.c backup-find.c
+lib_SOURCES += backupfile.c backup-rename.c
[...]
===

I could not find any information on whether these modules are mutually
exclusive or inter-dependent, but the fact is that you cannot have them both
in your project unless one of them drops "backupfile.c" from the "lib_SOURCES"
list.

-- 
(GM)

Re: backupfile and backup-rename are introducing the same object to make

[Bruno Haible]

Hi,

(GalaxyMaster) wrote:
> backup-replace

You mean 'backup-rename', right?

> I could not find any information on whether these modules are mutually
> exclusive or inter-dependent

Given the description of these two modules, it should be possible for a
program to use one, or the other, or both.

> I just stumbled upon the following: when I include both backupfile and
> backup-replace modules I am getting a broken build due to multiple definitions
> of "simple_backup_suffix'.

This is not expected, because usually the .o files are collected in a .a file,
from which one of the two backupfile.o will be picked, and it does not matter
which one since they are identical.

Please describe your situation:
  - operating system?
  - building what kind of library or binary in which way?
  - which compiler: $CC = ? $CFLAGS = ? $CPPFLAGS = ? $LDFLAGS = ?
  - which linker is being used by the compiler?,
and show the precise error message log.

Bruno

Re: backupfile and backup-rename are introducing the same object to make

[Bruno Haible]

Btw, if we have a problem here, we also have it in other places.
The .c files that are listed in lib_SOURCES of more than one module are:

      2 backupfile.c
      2 pipe-filter-aux.c

Bruno

Re: backupfile and backup-rename are introducing the same object to make

[(GalaxyMaster)]

Bruno,

On Wed, Jan 25, 2023 at 06:53:57PM +0100, Bruno Haible wrote:
> Hi,
> 
> (GalaxyMaster) wrote:
> > backup-replace
> You mean 'backup-rename', right?

Yeah, it was a typo, it was in the middle of the night and my brain was not
working efficiently.

> > I just stumbled upon the following: when I include both backupfile and
> > backup-replace modules I am getting a broken build due to multiple definitions
> > of "simple_backup_suffix'.
> 
> This is not expected, because usually the .o files are collected in a .a file,
> from which one of the two backupfile.o will be picked, and it does not matter
> which one since they are identical.

This makes sense.  My use case in not standard since I am actually building
these into a shared library using libtool, so libtool sees the same object file
listed twice and tries to resolve it by renaming one of them and then includes
both in the resulting link command to produce a shared object.  The linker
rightfully complains about duplicates.  Anyway, I think my use case is out of
scope for this project, but the reason why I reported it was that it just does
not look tidy to have two modules claiming ownership of the same files and
including the same configuration to configure.ac/Makefile.am files.

For my project, I am just patching backup-rename module's definition to only
include parts that are different from backupfile and I also made a dependency
on backupfile.

> Please describe your situation:
>   - operating system?

I am building a Linux distribution from ground up.

>   - building what kind of library or binary in which way?

I am basically building what is directly against GNU Lib's philosophy (section
2.2 of the manual), a shared library containing every single compilable module
with all functions defined as weak symbols.  The resulting shared library then
also stripped of all DT_NEEDED sections except for the system libc one, so that
it does not have lots of dependencies (and with weak symbols it is usable in
parts even if the dependencies are not met as long as the corresponding
functions are not called).

I clearly understand that my use case is not something supported or endorsed by
the GNU Lib project and am not trying to change that, but it is another avenue
to test stuff, which may be valueable to the project.

>   - which compiler: $CC = ? $CFLAGS = ? $CPPFLAGS = ? $LDFLAGS = ?
>   - which linker is being used by the compiler?,

I am not sure this would be helpful, since as I described my use case is
definitely not something GNU Lib supports, but for the context purposes: the
system is musl based with GCC as the primary compiler.

So, as you wrote in your message, I think there is no bug and the way how these
two modules are currently set up is fine (unless one does something
non-standard like myself).  However, I would argue that it would still be nicer
not to polute project's build files with duplicate entries if possible, but
this is up to you of course.

-- 
(GM)

Re: backupfile and backup-rename are introducing the same object to make

[(GalaxyMaster)]

Bruno,

On Wed, Jan 25, 2023 at 08:44:34PM +0100, Bruno Haible wrote:
> Btw, if we have a problem here, we also have it in other places.
> The .c files that are listed in lib_SOURCES of more than one module are:
> 
>       2 backupfile.c
>       2 pipe-filter-aux.c

With pipe-filter-aux.c I experienced a different issue.  In comparison to all
other modules that work with nonintr_close(), nonintr_open(), nonintr_read(),
and nonintr_write(), this module defines the function as just "inlined", while
execute and spawn-pipe modules declares them as static.  So, to build
pipe-filter-aux as a shareable object I redefined "PIPE_FILTER_AUX_INLINE" to
be "static _GL_INLINE" instead of just "_GL_INLINE" and this basically resolved
the issue.  I think this change may be considered for the inclusion into the
library since I see no reason why every other module is declaring these
functions as static and pipe-filter-aux.c is not.  But again, my use case is
non-standard and is likely not supported :).

-- 
(GM)

Re: backupfile and backup-rename are introducing the same object to make

[Bruno Haible]

(GalaxyMaster) wrote:
> My use case in not standard since I am actually building
> these into a shared library using libtool, so libtool sees the same object file
> listed twice and tries to resolve it by renaming one of them and then includes
> both in the resulting link command to produce a shared object.

Indeed, this explains why you get a link error then.

> I am building a Linux distribution from ground up.
> 
> >   - building what kind of library or binary in which way?
> 
> I am basically building what is directly against GNU Lib's philosophy (section
> 2.2 of the manual), a shared library containing every single compilable module
> with all functions defined as weak symbols.  ...
> 
> I clearly understand that my use case is not something supported or endorsed by
> the GNU Lib project and am not trying to change that, but it is another avenue
> to test stuff, which may be valueable to the project.

I have nothing against experiments. But you have to take into account that
GNU gnulib (that's the actual name, not "GNU Lib") does not attempt to have
backwards compatibility constraints as usual for shared libraries. This is
written in the documentation
  https://www.gnu.org/software/gnulib/manual/html_node/Steady-Development.html
and can be highlighted by a few entries of the NEWS file:

            getprogname     The include file is changed from "getprogname.h"
                            to <stdlib.h>.

This means that source code that uses gnulib needs adaptation.

            scratch_buffer  The function 'gl_scratch_buffer_dupfree' is removed.

            careadlinkat    This module no longer provides the careadlinkatcwd
                            function.

A function was removed. Binaries that used this function would no longer work.

            diacrit         This deprecated module is removed.

A module with all its functions was removed. Binaries that used it would no
longer work.

            read-file       The functions provided by this module now take an
                            'int flags' argument to modify the file reading
                            behavior.  The read_binary_file function has been
                            removed as it is no longer necessary.

These are backward-incompatible changes on existing function names.

            fatal-signal    The function that you pass to at_fatal_signal now
                            takes the signal as argument.

Likewise.

You need to think about how you will handle these changes.

Because if you don't handle these changes, you are stuck with a gnulib
snapshot from 2023 for all times, and in the year 2030 it will be useless
to ship seven years old code.

Bruno

Re: backupfile and backup-rename are introducing the same object to make

[GalaxyMaster]

Bruno,

On Thu, Jan 26, 2023 at 11:19:39AM +0100, Bruno Haible wrote:
> > I clearly understand that my use case is not something supported or endorsed by
> > the GNU Lib project and am not trying to change that, but it is another avenue
> > to test stuff, which may be valueable to the project.
> 
> I have nothing against experiments. But you have to take into account that
> GNU gnulib (that's the actual name, not "GNU Lib")

Thanks, noted.

> does not attempt to have backwards compatibility constraints as usual for
> shared libraries. This is written in the documentation

Yes, I am aware of that and it is not a big issue for my project.  I am
currently torn between two approaches:

 1. closely monitor changes between sync points and follow the libtool
    recommended approach on library versioning (when revision is incrementally
    changed on any change, age is incremented on additions, and current is
    incremented on interface changes).  Despite gnulib's changelog is good in
    this regard, I am still thinking of finding an automated way to monitor this;

 2. do a sync quarterly aligned with the release of a new version, which
    includes rebuilding everything.  This is more aligned with gnulib's
    philosophy in terms that the whole distribution I am creating can be
    considered as a package that uses GNU gnulib and everything beneath like
    coreutils or bison, for example, are just subcomponents.

My preference is automated version of the first approach.  And, if I am
successful and can prove it to be viable, I will contribute back in a form of a
separate project that is focusing on making GNU gnulib buildable as a
standalone library.  Looking around and seeing Gentoo packaging a system-wide
gnulib (https://packages.gentoo.org/packages/dev-libs/gnulib) and using it in
their package builds, NixOS doing something similar
(https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/tools/gnulib/default.nix),
and many others -- I think it may be valuable, if not it would be still
valuable to me :).

One thing that I really appreciate (and am not trying to conquer) is that GNU
gnulib as a project cares about all the different platforms and my effort is
focused purely around Linux-based ecosystem, hence I do not see a possibility
to merge my work back into gnulib (testing across all architectures supported
by GNU gnulib is a huge task).  This does not preclude me from contributing
in other ways, though :)

> You need to think about how you will handle these changes.

Thanks for kind words, I do consider these and thought quite a bit on the pros
and cons.  In the end, I found that it would be more efficient to spend time on
one of the two approaches described above rather then try to hunt down each
individual package that introduces local changes to GNU gnulib they are
bundling.  E.g. make (https://git.savannah.gnu.org/cgit/make.git/tree/gl) and
coreutils (https://github.com/coreutils/coreutils/tree/master/gl/modules) are
extending the GNU gnulib, some packages (I cannot easily recall the names) are
introducing non-GNU modules or patched versions of gnulib's modules, etc.

From the system engineering point of view, it is more viable to perform a
"surgery" on a package once, when it is entering the build queue and strip it
off its ties to the bundled, perhaps hacked version of gnulib and convince
package's build system to use a centralised, verified copy.  At least, this is
my theory, I am at the begining of the path with less than 20 packages using the
shared version of GNU gnulib, but among these are systemd and coreutils, so if
I do not tread carefully it will result in a borked system :).

-- 
(GM)

Re: backupfile and backup-rename are introducing the same object to make

[Bruno Haible]

GalaxyMaster wrote:
>     follow the libtool
>     recommended approach on library versioning (when revision is incrementally
>     changed on any change, age is incremented on additions, and current is
>     incremented on interface changes).  Despite gnulib's changelog is good in
>     this regard, I am still thinking of finding an automated way to monitor this;

Unfortunately this typically involves developer knowledge about the package.

There is some helper script gnulib/build-aux/libtool-next-version that
helps avoiding mistakes when assigning the LTV triple of a shared library.
It looks at the "nm" output of the library in the current and in the next
version. But it still requires judgement: When a function's implementation
changed in such a way that its results will change, are these modified
results a "major" or a "minor" change?

And when you say "ok I'll try to err on the safe side, if in doubt I'll
bump the major version", then it's an effort for the distro because they
have to rebuild some more packages. Cf.
  https://lists.gnu.org/archive/html/bug-libunistring/2022-12/msg00001.html

>  2. do a sync quarterly aligned with the release of a new version, which
>     includes rebuilding everything.

Note that it's quite frequent that changes to gnulib are being made in the
prerelease phase of some other package. For example, there were some gnulib
changes yesterday for the GNU poke 3.0 release that happened today. If you
have a quarterly schedule regarding gnulib, it means that you will have to
wait until 2023-04-01 until you can package poke 3.0. (You'll decide whether
that is acceptable for your distro. I'm just giving you a heads-up.)

> > You need to think about how you will handle these changes.
> 
> Thanks for kind words, I do consider these and thought quite a bit on the pros
> and cons.

Maybe we can get some advancement if you describe why, in the first place,
you want to do this package "surgery" in the first place. I don't want to
judge what you are doing; I wish to listen and understand if on the Gnulib
side we can do something a little bit differently, so that it would help
regarding your points.

> In the end, I found that it would be more efficient to spend time on
> one of the two approaches described above rather then try to hunt down each
> individual package that introduces local changes to GNU gnulib they are
> bundling.  E.g. make (https://git.savannah.gnu.org/cgit/make.git/tree/gl) and
> coreutils (https://github.com/coreutils/coreutils/tree/master/gl/modules) are
> extending the GNU gnulib, some packages (I cannot easily recall the names) are
> introducing non-GNU modules or patched versions of gnulib's modules, etc.
> 
> From the system engineering point of view, it is more viable to perform a
> "surgery" on a package once, when it is entering the build queue and strip it
> off its ties to the bundled, perhaps hacked version of gnulib and convince
> package's build system to use a centralised, verified copy.

You say "centralised". Why is it a problem if, e.g. GNU make uses some set
of (possibly modified) gnulib modules, and GNU coreutils another set, and
they use different versions? Binary code size / efficiency of using the RAM?

You say "verified". Why is it a problem if some GNU package has decided to
modify a gnulib module? This is supported, see
https://www.gnu.org/software/gnulib/manual/html_node/Extending-Gnulib.html
Are there reasons not to trust the modifications done e.g. by the GNU make
maintainer, when you do trust him for the other parts of the GNU make source
code?

Bruno

Re: backupfile and backup-rename are introducing the same object to make

[(GalaxyMaster)]

Bruno,

On Thu, Jan 26, 2023 at 03:43:01PM +0100, Bruno Haible wrote:
> GalaxyMaster wrote:
> >     incremented on interface changes).  Despite gnulib's changelog is good in
> >     this regard, I am still thinking of finding an automated way to monitor this;
> 
> Unfortunately this typically involves developer knowledge about the package.

You've raised a lot of really good questions and I appreciate your insights.  I
feel this discussion has already deviated a lot from the original topic and I
am thinking of responding in a different thread.  Please be assured that everything
you wrote in the previous message was read and been thought over, however,
responding would produce an even bigger e-mail message and I am not really
ready to type it out right now.

I will try my best to find a way on how to address your questions and still
maintain a message valuable for the rest of the community.

Thanks again for your patience and time!

-- 
(GM)