* backupfile and backup-replace are introducing the same object to make @ 2023-01-25 12:04 (GalaxyMaster) 2023-01-25 17:53 ` backupfile and backup-rename " Bruno Haible 0 siblings, 1 reply; 9+ messages in thread From: (GalaxyMaster) @ 2023-01-25 12:04 UTC (permalink / raw) To: bug-gnulib Hello, [I am not a subscriber, so if you need anything from me, please CC me] I just stumbled upon the following: when I include both backupfile and backup-replace modules I am getting a broken build due to multiple definitions of "simple_backup_suffix'. When I compared modules, sure enough I found the issue: === --- backupfile 2023-01-23 03:29:03.000000000 +1100 +++ backup-rename 2023-01-23 03:29:03.000000000 +1100 [...] Makefile.am: -lib_SOURCES += backupfile.c backup-find.c +lib_SOURCES += backupfile.c backup-rename.c [...] === I could not find any information on whether these modules are mutually exclusive or inter-dependent, but the fact is that you cannot have them both in your project unless one of them drops "backupfile.c" from the "lib_SOURCES" list. -- (GM) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-25 12:04 backupfile and backup-replace are introducing the same object to make (GalaxyMaster) @ 2023-01-25 17:53 ` Bruno Haible 2023-01-25 19:44 ` Bruno Haible 2023-01-26 7:19 ` (GalaxyMaster) 0 siblings, 2 replies; 9+ messages in thread From: Bruno Haible @ 2023-01-25 17:53 UTC (permalink / raw) To: galaxy; +Cc: bug-gnulib Hi, (GalaxyMaster) wrote: > backup-replace You mean 'backup-rename', right? > I could not find any information on whether these modules are mutually > exclusive or inter-dependent Given the description of these two modules, it should be possible for a program to use one, or the other, or both. > I just stumbled upon the following: when I include both backupfile and > backup-replace modules I am getting a broken build due to multiple definitions > of "simple_backup_suffix'. This is not expected, because usually the .o files are collected in a .a file, from which one of the two backupfile.o will be picked, and it does not matter which one since they are identical. Please describe your situation: - operating system? - building what kind of library or binary in which way? - which compiler: $CC = ? $CFLAGS = ? $CPPFLAGS = ? $LDFLAGS = ? - which linker is being used by the compiler?, and show the precise error message log. Bruno ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-25 17:53 ` backupfile and backup-rename " Bruno Haible @ 2023-01-25 19:44 ` Bruno Haible 2023-01-26 7:28 ` (GalaxyMaster) 2023-01-26 7:19 ` (GalaxyMaster) 1 sibling, 1 reply; 9+ messages in thread From: Bruno Haible @ 2023-01-25 19:44 UTC (permalink / raw) To: bug-gnulib; +Cc: galaxy Btw, if we have a problem here, we also have it in other places. The .c files that are listed in lib_SOURCES of more than one module are: 2 backupfile.c 2 pipe-filter-aux.c Bruno ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-25 19:44 ` Bruno Haible @ 2023-01-26 7:28 ` (GalaxyMaster) 0 siblings, 0 replies; 9+ messages in thread From: (GalaxyMaster) @ 2023-01-26 7:28 UTC (permalink / raw) To: Bruno Haible; +Cc: bug-gnulib Bruno, On Wed, Jan 25, 2023 at 08:44:34PM +0100, Bruno Haible wrote: > Btw, if we have a problem here, we also have it in other places. > The .c files that are listed in lib_SOURCES of more than one module are: > > 2 backupfile.c > 2 pipe-filter-aux.c With pipe-filter-aux.c I experienced a different issue. In comparison to all other modules that work with nonintr_close(), nonintr_open(), nonintr_read(), and nonintr_write(), this module defines the function as just "inlined", while execute and spawn-pipe modules declares them as static. So, to build pipe-filter-aux as a shareable object I redefined "PIPE_FILTER_AUX_INLINE" to be "static _GL_INLINE" instead of just "_GL_INLINE" and this basically resolved the issue. I think this change may be considered for the inclusion into the library since I see no reason why every other module is declaring these functions as static and pipe-filter-aux.c is not. But again, my use case is non-standard and is likely not supported :). -- (GM) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-25 17:53 ` backupfile and backup-rename " Bruno Haible 2023-01-25 19:44 ` Bruno Haible @ 2023-01-26 7:19 ` (GalaxyMaster) 2023-01-26 10:19 ` Bruno Haible 1 sibling, 1 reply; 9+ messages in thread From: (GalaxyMaster) @ 2023-01-26 7:19 UTC (permalink / raw) To: Bruno Haible; +Cc: bug-gnulib Bruno, On Wed, Jan 25, 2023 at 06:53:57PM +0100, Bruno Haible wrote: > Hi, > > (GalaxyMaster) wrote: > > backup-replace > You mean 'backup-rename', right? Yeah, it was a typo, it was in the middle of the night and my brain was not working efficiently. > > I just stumbled upon the following: when I include both backupfile and > > backup-replace modules I am getting a broken build due to multiple definitions > > of "simple_backup_suffix'. > > This is not expected, because usually the .o files are collected in a .a file, > from which one of the two backupfile.o will be picked, and it does not matter > which one since they are identical. This makes sense. My use case in not standard since I am actually building these into a shared library using libtool, so libtool sees the same object file listed twice and tries to resolve it by renaming one of them and then includes both in the resulting link command to produce a shared object. The linker rightfully complains about duplicates. Anyway, I think my use case is out of scope for this project, but the reason why I reported it was that it just does not look tidy to have two modules claiming ownership of the same files and including the same configuration to configure.ac/Makefile.am files. For my project, I am just patching backup-rename module's definition to only include parts that are different from backupfile and I also made a dependency on backupfile. > Please describe your situation: > - operating system? I am building a Linux distribution from ground up. > - building what kind of library or binary in which way? I am basically building what is directly against GNU Lib's philosophy (section 2.2 of the manual), a shared library containing every single compilable module with all functions defined as weak symbols. The resulting shared library then also stripped of all DT_NEEDED sections except for the system libc one, so that it does not have lots of dependencies (and with weak symbols it is usable in parts even if the dependencies are not met as long as the corresponding functions are not called). I clearly understand that my use case is not something supported or endorsed by the GNU Lib project and am not trying to change that, but it is another avenue to test stuff, which may be valueable to the project. > - which compiler: $CC = ? $CFLAGS = ? $CPPFLAGS = ? $LDFLAGS = ? > - which linker is being used by the compiler?, I am not sure this would be helpful, since as I described my use case is definitely not something GNU Lib supports, but for the context purposes: the system is musl based with GCC as the primary compiler. So, as you wrote in your message, I think there is no bug and the way how these two modules are currently set up is fine (unless one does something non-standard like myself). However, I would argue that it would still be nicer not to polute project's build files with duplicate entries if possible, but this is up to you of course. -- (GM) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-26 7:19 ` (GalaxyMaster) @ 2023-01-26 10:19 ` Bruno Haible 2023-01-26 11:20 ` GalaxyMaster 0 siblings, 1 reply; 9+ messages in thread From: Bruno Haible @ 2023-01-26 10:19 UTC (permalink / raw) To: (GalaxyMaster); +Cc: bug-gnulib (GalaxyMaster) wrote: > My use case in not standard since I am actually building > these into a shared library using libtool, so libtool sees the same object file > listed twice and tries to resolve it by renaming one of them and then includes > both in the resulting link command to produce a shared object. Indeed, this explains why you get a link error then. > I am building a Linux distribution from ground up. > > > - building what kind of library or binary in which way? > > I am basically building what is directly against GNU Lib's philosophy (section > 2.2 of the manual), a shared library containing every single compilable module > with all functions defined as weak symbols. ... > > I clearly understand that my use case is not something supported or endorsed by > the GNU Lib project and am not trying to change that, but it is another avenue > to test stuff, which may be valueable to the project. I have nothing against experiments. But you have to take into account that GNU gnulib (that's the actual name, not "GNU Lib") does not attempt to have backwards compatibility constraints as usual for shared libraries. This is written in the documentation https://www.gnu.org/software/gnulib/manual/html_node/Steady-Development.html and can be highlighted by a few entries of the NEWS file: getprogname The include file is changed from "getprogname.h" to <stdlib.h>. This means that source code that uses gnulib needs adaptation. scratch_buffer The function 'gl_scratch_buffer_dupfree' is removed. careadlinkat This module no longer provides the careadlinkatcwd function. A function was removed. Binaries that used this function would no longer work. diacrit This deprecated module is removed. A module with all its functions was removed. Binaries that used it would no longer work. read-file The functions provided by this module now take an 'int flags' argument to modify the file reading behavior. The read_binary_file function has been removed as it is no longer necessary. These are backward-incompatible changes on existing function names. fatal-signal The function that you pass to at_fatal_signal now takes the signal as argument. Likewise. You need to think about how you will handle these changes. Because if you don't handle these changes, you are stuck with a gnulib snapshot from 2023 for all times, and in the year 2030 it will be useless to ship seven years old code. Bruno ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-26 10:19 ` Bruno Haible @ 2023-01-26 11:20 ` GalaxyMaster 2023-01-26 14:43 ` Bruno Haible 0 siblings, 1 reply; 9+ messages in thread From: GalaxyMaster @ 2023-01-26 11:20 UTC (permalink / raw) To: Bruno Haible; +Cc: bug-gnulib Bruno, On Thu, Jan 26, 2023 at 11:19:39AM +0100, Bruno Haible wrote: > > I clearly understand that my use case is not something supported or endorsed by > > the GNU Lib project and am not trying to change that, but it is another avenue > > to test stuff, which may be valueable to the project. > > I have nothing against experiments. But you have to take into account that > GNU gnulib (that's the actual name, not "GNU Lib") Thanks, noted. > does not attempt to have backwards compatibility constraints as usual for > shared libraries. This is written in the documentation Yes, I am aware of that and it is not a big issue for my project. I am currently torn between two approaches: 1. closely monitor changes between sync points and follow the libtool recommended approach on library versioning (when revision is incrementally changed on any change, age is incremented on additions, and current is incremented on interface changes). Despite gnulib's changelog is good in this regard, I am still thinking of finding an automated way to monitor this; 2. do a sync quarterly aligned with the release of a new version, which includes rebuilding everything. This is more aligned with gnulib's philosophy in terms that the whole distribution I am creating can be considered as a package that uses GNU gnulib and everything beneath like coreutils or bison, for example, are just subcomponents. My preference is automated version of the first approach. And, if I am successful and can prove it to be viable, I will contribute back in a form of a separate project that is focusing on making GNU gnulib buildable as a standalone library. Looking around and seeing Gentoo packaging a system-wide gnulib (https://packages.gentoo.org/packages/dev-libs/gnulib) and using it in their package builds, NixOS doing something similar (https://github.com/NixOS/nixpkgs/blob/nixos-unstable/pkgs/development/tools/gnulib/default.nix), and many others -- I think it may be valuable, if not it would be still valuable to me :). One thing that I really appreciate (and am not trying to conquer) is that GNU gnulib as a project cares about all the different platforms and my effort is focused purely around Linux-based ecosystem, hence I do not see a possibility to merge my work back into gnulib (testing across all architectures supported by GNU gnulib is a huge task). This does not preclude me from contributing in other ways, though :) > You need to think about how you will handle these changes. Thanks for kind words, I do consider these and thought quite a bit on the pros and cons. In the end, I found that it would be more efficient to spend time on one of the two approaches described above rather then try to hunt down each individual package that introduces local changes to GNU gnulib they are bundling. E.g. make (https://git.savannah.gnu.org/cgit/make.git/tree/gl) and coreutils (https://github.com/coreutils/coreutils/tree/master/gl/modules) are extending the GNU gnulib, some packages (I cannot easily recall the names) are introducing non-GNU modules or patched versions of gnulib's modules, etc. From the system engineering point of view, it is more viable to perform a "surgery" on a package once, when it is entering the build queue and strip it off its ties to the bundled, perhaps hacked version of gnulib and convince package's build system to use a centralised, verified copy. At least, this is my theory, I am at the begining of the path with less than 20 packages using the shared version of GNU gnulib, but among these are systemd and coreutils, so if I do not tread carefully it will result in a borked system :). -- (GM) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-26 11:20 ` GalaxyMaster @ 2023-01-26 14:43 ` Bruno Haible 2023-01-28 14:41 ` (GalaxyMaster) 0 siblings, 1 reply; 9+ messages in thread From: Bruno Haible @ 2023-01-26 14:43 UTC (permalink / raw) To: GalaxyMaster; +Cc: bug-gnulib GalaxyMaster wrote: > follow the libtool > recommended approach on library versioning (when revision is incrementally > changed on any change, age is incremented on additions, and current is > incremented on interface changes). Despite gnulib's changelog is good in > this regard, I am still thinking of finding an automated way to monitor this; Unfortunately this typically involves developer knowledge about the package. There is some helper script gnulib/build-aux/libtool-next-version that helps avoiding mistakes when assigning the LTV triple of a shared library. It looks at the "nm" output of the library in the current and in the next version. But it still requires judgement: When a function's implementation changed in such a way that its results will change, are these modified results a "major" or a "minor" change? And when you say "ok I'll try to err on the safe side, if in doubt I'll bump the major version", then it's an effort for the distro because they have to rebuild some more packages. Cf. https://lists.gnu.org/archive/html/bug-libunistring/2022-12/msg00001.html > 2. do a sync quarterly aligned with the release of a new version, which > includes rebuilding everything. Note that it's quite frequent that changes to gnulib are being made in the prerelease phase of some other package. For example, there were some gnulib changes yesterday for the GNU poke 3.0 release that happened today. If you have a quarterly schedule regarding gnulib, it means that you will have to wait until 2023-04-01 until you can package poke 3.0. (You'll decide whether that is acceptable for your distro. I'm just giving you a heads-up.) > > You need to think about how you will handle these changes. > > Thanks for kind words, I do consider these and thought quite a bit on the pros > and cons. Maybe we can get some advancement if you describe why, in the first place, you want to do this package "surgery" in the first place. I don't want to judge what you are doing; I wish to listen and understand if on the Gnulib side we can do something a little bit differently, so that it would help regarding your points. > In the end, I found that it would be more efficient to spend time on > one of the two approaches described above rather then try to hunt down each > individual package that introduces local changes to GNU gnulib they are > bundling. E.g. make (https://git.savannah.gnu.org/cgit/make.git/tree/gl) and > coreutils (https://github.com/coreutils/coreutils/tree/master/gl/modules) are > extending the GNU gnulib, some packages (I cannot easily recall the names) are > introducing non-GNU modules or patched versions of gnulib's modules, etc. > > From the system engineering point of view, it is more viable to perform a > "surgery" on a package once, when it is entering the build queue and strip it > off its ties to the bundled, perhaps hacked version of gnulib and convince > package's build system to use a centralised, verified copy. You say "centralised". Why is it a problem if, e.g. GNU make uses some set of (possibly modified) gnulib modules, and GNU coreutils another set, and they use different versions? Binary code size / efficiency of using the RAM? You say "verified". Why is it a problem if some GNU package has decided to modify a gnulib module? This is supported, see https://www.gnu.org/software/gnulib/manual/html_node/Extending-Gnulib.html Are there reasons not to trust the modifications done e.g. by the GNU make maintainer, when you do trust him for the other parts of the GNU make source code? Bruno ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: backupfile and backup-rename are introducing the same object to make 2023-01-26 14:43 ` Bruno Haible @ 2023-01-28 14:41 ` (GalaxyMaster) 0 siblings, 0 replies; 9+ messages in thread From: (GalaxyMaster) @ 2023-01-28 14:41 UTC (permalink / raw) To: Bruno Haible; +Cc: bug-gnulib Bruno, On Thu, Jan 26, 2023 at 03:43:01PM +0100, Bruno Haible wrote: > GalaxyMaster wrote: > > incremented on interface changes). Despite gnulib's changelog is good in > > this regard, I am still thinking of finding an automated way to monitor this; > > Unfortunately this typically involves developer knowledge about the package. You've raised a lot of really good questions and I appreciate your insights. I feel this discussion has already deviated a lot from the original topic and I am thinking of responding in a different thread. Please be assured that everything you wrote in the previous message was read and been thought over, however, responding would produce an even bigger e-mail message and I am not really ready to type it out right now. I will try my best to find a way on how to address your questions and still maintain a message valuable for the rest of the community. Thanks again for your patience and time! -- (GM) ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-01-28 14:41 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-25 12:04 backupfile and backup-replace are introducing the same object to make (GalaxyMaster) 2023-01-25 17:53 ` backupfile and backup-rename " Bruno Haible 2023-01-25 19:44 ` Bruno Haible 2023-01-26 7:28 ` (GalaxyMaster) 2023-01-26 7:19 ` (GalaxyMaster) 2023-01-26 10:19 ` Bruno Haible 2023-01-26 11:20 ` GalaxyMaster 2023-01-26 14:43 ` Bruno Haible 2023-01-28 14:41 ` (GalaxyMaster)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).