replacement for 'join'?

replacement for 'join'?

[Bruno Haible]

Alpine Linux does not have the 'join' program.
The GCS [1] don't list it among the essential utilities.
So, what kind of replacement for it would you recommend?
My use-case is

  join -v 1 FILE1 FILE2

Bruno

[1] https://www.gnu.org/prep/standards/html_node/Utilities-in-Makefiles.html

Re: replacement for 'join'?

[Bernhard Voelker]

On 4/5/21 1:06 AM, Bruno Haible wrote:
> Alpine Linux does not have the 'join' program.
> The GCS [1] don't list it among the essential utilities.

FWIW: join(1) is required by POSIX [1]:

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/join.html

> So, what kind of replacement for it would you recommend?
> My use-case is
> 
>   join -v 1 FILE1 FILE2

Maybe awk?

  $ awk -v keyfile=file2 '
      BEGIN { while ((getline < keyfile) > 0) k[$1]=1 }
      !k[$1]
      ' file1
or

  $ awk '
      keys { k[$1]=1; next }
      !k[$1]
      ' keys=1 file2 keys=0 file1

To be honest, I'm not sure whether the variable assignment on the command line
or getline in a BEGIN block is more portable.

Have a nice day,
Berny

Re: replacement for 'join'?

[Bruno Haible]

Hi Berny,

> >   join -v 1 FILE1 FILE2
> 
> Maybe awk?
> 
>   $ awk -v keyfile=file2 '
>       BEGIN { while ((getline < keyfile) > 0) k[$1]=1 }
>       !k[$1]
>       ' file1
> or
> 
>   $ awk '
>       keys { k[$1]=1; next }
>       !k[$1]
>       ' keys=1 file2 keys=0 file1

Wonderful! Thank you. Both solutions work fine with the 'awk' on Alpine Linux
(BusyBox v1.32.1).

> To be honest, I'm not sure whether the variable assignment on the command line
> or getline in a BEGIN block is more portable.

As Alpine Linux is the only platform that lacks 'join', awk portability is
not an issue (so far).

Bruno

Re: replacement for 'join'?

[Paul Eggert]

On 4/5/21 6:01 PM, Bruno Haible wrote:
> Wonderful! Thank you. Both solutions work fine with the 'awk' on Alpine Linux
> (BusyBox v1.32.1).

It looks like you're talking about a simple one-field-per-line join. If 
so, the following should also work and is simpler:

grep -Fvf file2 file1

though it might not scale as well as 'awk'.

Re: replacement for 'join'?

[Bernhard Voelker]

On 4/6/21 4:24 AM, Paul Eggert wrote:
> grep -Fvf file2 file1

I started with that, too, but it is problematic because:
a) it doesn't do a whole-word search ... and 'grep -w' seems not to be portable,
b) it doesn't limit the matching on the key field.
And messing with regular expressions seems to be fragile as well, because the
key field may contain problematic characters.

Have a nice day,
Berny

Re: replacement for 'join'?

[Paul Eggert]

On 4/6/21 2:40 AM, Bernhard Voelker wrote:
>> grep -Fvf file2 file1
> I started with that, too, but it is problematic because:
> a) it doesn't do a whole-word search ... and 'grep -w' seems not to be portable,
> b) it doesn't limit the matching on the key field.
> And messing with regular expressions seems to be fragile as well, because the
> key field may contain problematic characters.

Yes, 'awk' is the way to go if you want more 'join' capability (not just 
treating the entire line as the key). But if the data are not large and 
each key takes up the whole line, 'grep' should work fine.

Regular expressions are not a problem, as the -F option disables them.

Re: replacement for 'join'?

[Bruno Haible]

Paul Eggert wrote:
> Yes, 'awk' is the way to go if you want more 'join' capability (not just 
> treating the entire line as the key). But if the data are not large and 
> each key takes up the whole line, 'grep' should work fine.

The data is large, unfortunately. The two input files contain lists
of symbols, when constructing a shared library. GNU libunistring has
ca. 250 symbols.

For small data, I would have transformed the first file to a 'sed'
script, that I would then apply to the second file. But HP-UX 'sed'
has a limit of 100 -e expressions per invocation.

So, the solution with 'awk' is the best I can see. Thanks again, Berny!

Bruno

Re: replacement for 'join'?

[Bernhard Voelker]

On 4/6/21 10:45 PM, Bruno Haible wrote:
> For small data, I would have transformed the first file to a 'sed'
> script, that I would then apply to the second file. But HP-UX 'sed'
> has a limit of 100 -e expressions per invocation.

I don't have access to HP-UX (and there's none in the GCC compile farm),
but doesn't HP-UX' sed support the POSIX -f option there?

Have a nice day,
Berny

Re: replacement for 'join'?

[Bruno Haible]

Bernhard Voelker wrote:
> but doesn't HP-UX' sed support the POSIX -f option there?

Probably it does. But then the replacement script would have to create
a temporary file. Which is about 30 lines of shell code.

Bruno

Re: replacement for 'join'?

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> Alpine Linux does not have the 'join' program.
> The GCS [1] don't list it among the essential utilities.

I ran into that problem during ./bootstrap -- it seems gnulib-tool
relies on the 'join' tool as well:

./bootstrap: gnulib/gnulib-tool    --no-changelog   --aux-dir=build-aux   --doc-base=doc   --lib=libgnu   --m4-base=m4/   --source-base=gl/   --tests-base=gl/tests   --local-dir=gl      --libtool --import ...
gnulib/gnulib-tool: line 1: join: not found

The ./bootstrap script completes without error exit, but failed to setup
the build properly.

Do we want to support ./boostrap on systems without 'join'?

If not, I suggest the patch below.

/Simon

diff --git a/build-aux/bootstrap.conf b/build-aux/bootstrap.conf
index 9e21d9f45..0e8cbbe4a 100644
--- a/build-aux/bootstrap.conf
+++ b/build-aux/bootstrap.conf
@@ -64,4 +64,5 @@ autoconf   2.59
 automake   1.9.6
 git        1.5.5
 tar        -
+join       -
 "

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: replacement for 'join'?

[Bruno Haible]

Hi Simon,

> > Alpine Linux does not have the 'join' program.
> > The GCS [1] don't list it among the essential utilities.
> 
> I ran into that problem during ./bootstrap -- it seems gnulib-tool
> relies on the 'join' tool as well:
> 
> ./bootstrap: gnulib/gnulib-tool    --no-changelog   --aux-dir=build-aux   --doc-base=doc   --lib=libgnu   --m4-base=m4/   --source-base=gl/   --tests-base=gl/tests   --local-dir=gl      --libtool --import ...
> gnulib/gnulib-tool: line 1: join: not found
> 
> The ./bootstrap script completes without error exit, but failed to setup
> the build properly.

That is not proper behaviour. Fixing it through the patch below.

> Do we want to support ./boostrap on systems without 'join'?

No. gnulib-tool needs more advanced use of 'join' (on tables with 2 columns,
not just on lists of symbols). Unless someone writes a full emulation of
'join' that supports the -t, -v, -a options, it's best to require 'join'.

> If not, I suggest the patch below.
> 
> /Simon
> 
> diff --git a/build-aux/bootstrap.conf b/build-aux/bootstrap.conf
> index 9e21d9f45..0e8cbbe4a 100644
> --- a/build-aux/bootstrap.conf
> +++ b/build-aux/bootstrap.conf
> @@ -64,4 +64,5 @@ autoconf   2.59
>  automake   1.9.6
>  git        1.5.5
>  tar        -
> +join       -
>  "

That is good as well.


2021-05-12  Bruno Haible  <bruno@clisp.org>

	gnulib-tool: Fail properly if the 'join' program is not found.
	Reported by Simon Josefsson <simon@josefsson.org> in
	<https://lists.gnu.org/archive/html/bug-gnulib/2021-05/msg00029.html>.
	* gnulib-tool: Bail out early of the 'join' program is not found.

diff --git a/gnulib-tool b/gnulib-tool
index 182b9b4..9b5c815 100755
--- a/gnulib-tool
+++ b/gnulib-tool
@@ -888,6 +888,15 @@ func_hardlink ()
   }
 }
 
+# The 'join' program does not exist on all platforms.  Where it exists,
+# we can use it.  Where not, bail out.
+if (type join) >/dev/null 2>&1; then
+  :
+else
+  echo "$progname: 'join' program not found. Consider installing GNU coreutils." >&2
+  func_exit 1
+fi
+
 # Ensure an 'echo' command that
 #   1. does not interpret backslashes and
 #   2. does not print an error message "broken pipe" when writing into a pipe

Re: replacement for 'join'?

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 400 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> That is not proper behaviour. Fixing it through the patch below.

Thank you!

It would be nice to write a 'join' replacement for gnulib-tool, as that
is the only thing that needs coreutils for bootstrapping libidn2 on
alpine.  But it is not important, and with your patch things work as
good as they can do both with and without coreutils installed.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: replacement for 'join'?

[Bernhard Voelker]

On 5/12/21 7:58 PM, Simon Josefsson via Gnulib discussion list wrote:
> It would be nice to write a 'join' replacement for gnulib-tool, as that
> is the only thing that needs coreutils for bootstrapping libidn2 on
> alpine.  But it is not important, and with your patch things work as
> good as they can do both with and without coreutils installed.

I'm wondering if we have a list of supported platforms for maintainers vs. that
of end users.  I mean I wouldn't bother if 'bootstrap' fails on some platform
- actually often such platforms lack other prerequisites as well -, and instead
I usually use distribution tarballs for pre-release testing ... like the end
users do.

Have a nice day,
Berny

Re: replacement for 'join'?

[Bruno Haible]

Bernhard Voelker wrote:
> I'm wondering if we have a list of supported platforms for maintainers

Not officially. We just assume that no one will be silly enough to use e.g.
Solaris 10 — without added GNU tools — as their development platform. Every
platform, even Solaris 10 and embedded platforms e.g. µClinux or Alpine
Linux, can be turned into reasonable development platforms by installing
the GNU tools.

Bruno

Re: replacement for 'join'?

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1485 bytes --]

Bernhard Voelker <mail@bernhard-voelker.de> writes:

> On 5/12/21 7:58 PM, Simon Josefsson via Gnulib discussion list wrote:
>> It would be nice to write a 'join' replacement for gnulib-tool, as that
>> is the only thing that needs coreutils for bootstrapping libidn2 on
>> alpine.  But it is not important, and with your patch things work as
>> good as they can do both with and without coreutils installed.
>
> I'm wondering if we have a list of supported platforms for maintainers
> vs. that of end users.  I mean I wouldn't bother if 'bootstrap' fails
> on some platform - actually often such platforms lack other
> prerequisites as well -, and instead I usually use distribution
> tarballs for pre-release testing ... like the end users do.

The manual contains some notes:

https://www.gnu.org/software/gnulib/manual/html_node/Target-Platforms.html
https://www.gnu.org/software/gnulib/manual/html_node/Supported-Platforms.html

However I think it refers to platforms on which you would build the
tarball, not the platform you would do development on.  The best we have
for the latter is this one:

https://git.savannah.gnu.org/cgit/gnulib.git/tree/DEPENDENCIES

Ironically, it does not mention 'join' but mention a lot of other tools.

However I think these lists often become outdated.  In my view, to claim
that a platform is supported by a software project, you should have
continous building for the platform.  Otherwise support is reactive and
tends to be spurious.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: replacement for 'join'?

[Bruno Haible]

Simon Josefsson wrote:
> https://git.savannah.gnu.org/cgit/gnulib.git/tree/DEPENDENCIES
> 
> Ironically, it does not mention 'join' but mention a lot of other tools.

Good point. Fixed through the patch below.

> However I think these lists often become outdated.  In my view, to claim
> that a platform is supported by a software project, you should have
> continous building for the platform.  Otherwise support is reactive and
> tends to be spurious.

Gnulib is a hobbyist, volunteer project. We can not provide the same
level of support as, for example, you do with your company. Therefore —
unless someone comes up and invests the necessary time and money for the
multi-platform continuous integration — our support here will remain
"reactive".


2021-05-14  Bruno Haible  <bruno@clisp.org>

	DEPENDENCIES: Mention the requirement for 'join'.
	Reported by Simon Josefsson <simon@josefsson.org> in
	<https://lists.gnu.org/archive/html/bug-gnulib/2021-05/msg00047.html>.
	* DEPENDENCIES: Mention 'join' among the core POSIX utilities.

diff --git a/DEPENDENCIES b/DEPENDENCIES
index fe70558..5e82be4 100644
--- a/DEPENDENCIES
+++ b/DEPENDENCIES
@@ -58,7 +58,7 @@ at any time.
 
 * Core POSIX utilities, including:
     [ basename cat chgrp chmod chown cp dd echo expand expr
-    false hostname install kill ln ls md5sum mkdir mkfifo
+    false hostname install join kill ln ls md5sum mkdir mkfifo
     mknod mv printenv pwd rm rmdir sleep sort tee test touch
     true uname
   + Mandatory. Using the platform's native utilities gives good portability

Re: replacement for 'join'?

[Bernhard Voelker]

On 5/14/21 6:19 PM, Bruno Haible wrote:
> Gnulib is a hobbyist, volunteer project. We can not provide the same
> level of support as, for example, you do with your company. Therefore —
> unless someone comes up and invests the necessary time and money for the
> multi-platform continuous integration — our support here will remain
> "reactive".

That's fine.  It's anyway better to test for features than for platforms/versions.

Thanks & have a nice day,
Berny

Re: replacement for 'join'?

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]

Bruno Haible <bruno@clisp.org> writes:

>> However I think these lists often become outdated.  In my view, to claim
>> that a platform is supported by a software project, you should have
>> continous building for the platform.  Otherwise support is reactive and
>> tends to be spurious.
>
> Gnulib is a hobbyist, volunteer project. We can not provide the same
> level of support as, for example, you do with your company. Therefore —
> unless someone comes up and invests the necessary time and money for the
> multi-platform continuous integration — our support here will remain
> "reactive".

Certainly!  Was anyone working on setting up CI/CD for gnulib on GitLab?
I recall there was a private project for it, but I don't have access.
Any reason for this?  I have become rather aquinted with GitLab CI/CD
lately so it would be easy for me to setup something from scratch.  Of
course, given all the freedom concerns people may have with using a SaaS
like GitLab, I think it is important that we don't let what features
exists there influence any decisions made in gnulib.  Let's use it as a
way to improve our software, to promote more users of GNU software, just
the way we used proprietary platforms like Solaris or IRIX in the old
days.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: replacement for 'join'?

[Bernhard Voelker]

On 5/14/21 7:42 PM, Simon Josefsson via Gnulib discussion list wrote:
> Certainly!  Was anyone working on setting up CI/CD for gnulib on GitLab?
> I recall there was a private project for it, but I don't have access.
> Any reason for this?  I have become rather aquinted with GitLab CI/CD
> lately so it would be easy for me to setup something from scratch.  Of
> course, given all the freedom concerns people may have with using a SaaS
> like GitLab, I think it is important that we don't let what features
> exists there influence any decisions made in gnulib.  Let's use it as a
> way to improve our software, to promote more users of GNU software, just
> the way we used proprietary platforms like Solaris or IRIX in the old
> days.

IMO that's something for the gnu-prod-discuss mailing list.
We shouldn't blindly use CI/CD platforms which are potentially problematic
or even contradicting or violating the GNU freedoms, just merely because
many people use it.
(I'm no lawyer, so I can't 100% tell for Gitlab, though.)

Isn't there already GNU way for CI/CD?
I know there is hydra.nixos.org, but that looks a bit fallen asleep.
Anything else?

Have a nice day,
Berny

Re: gnulib-ci project

[Bruno Haible]

Simon Josefsson wrote:
> Was anyone working on setting up CI/CD for gnulib on GitLab?
> I recall there was a private project for it, but I don't have access.
> Any reason for this?  I have become rather aquinted with GitLab CI/CD
> lately so it would be easy for me to setup something from scratch.

There's no point in duplicating the effort already done, and creating
confusion by having two different gnulib CIs on the same platform.

https://gitlab.com/gnulib/gnulib-ci/-/pipelines

Tim,

I have asked you twice [1][2] whether it's OK to make the project public.
I didn't receive an answer. When you introduced me to the project (on
the phone), you didn't mention a reason for keeping it private, as far
as I can remember. So, to fulfil the demands of Jim and Simon, I have
now made the project public. I hope that you will agree a posteriori.

Bruno

[1] https://lists.gnu.org/archive/html/bug-gnulib/2020-09/msg00107.html
[2] https://lists.gnu.org/archive/html/bug-gnulib/2020-11/msg00042.html

Re: gnulib-ci project

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

Bruno Haible <bruno@clisp.org> writes:

> Simon Josefsson wrote:
>> Was anyone working on setting up CI/CD for gnulib on GitLab?
>> I recall there was a private project for it, but I don't have access.
>> Any reason for this?  I have become rather aquinted with GitLab CI/CD
>> lately so it would be easy for me to setup something from scratch.
>
> There's no point in duplicating the effort already done, and creating
> confusion by having two different gnulib CIs on the same platform.
>
> https://gitlab.com/gnulib/gnulib-ci/-/pipelines

Great -- seems straightforward, although it would be nice to have it
passing a build before adding more platforms.  It seems there are only
two tests that unexpectedly pass, the test-asyncsafe-linked_list-weak.sh
and test-asyncsafe-linked_list-strong.sh.  Any ideas on these?  I'm not
sure if they have already been discussed.

https://gitlab.com/gnulib/gnulib-ci/-/jobs/1246762022/artifacts/file/gnulib/testdir/gltests/test-asyncsafe-linked_list-strong.sh.log

> Tim,
>
> I have asked you twice [1][2] whether it's OK to make the project public.

I know Tim is busy on a new job so let's make him proud of what he
started by improving it :-)

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 227 bytes --]

Re: gnulib-ci project

[Bruno Haible]

Simon Josefsson wrote:
> It seems there are only
> two tests that unexpectedly pass, the test-asyncsafe-linked_list-weak.sh
> and test-asyncsafe-linked_list-strong.sh.  Any ideas on these?

I was meaning to clean this up soon, but haven't gotten around to it. So,
let me move them into a tests module that does not get included by default.


2021-05-14  Bruno Haible  <bruno@clisp.org>

	linked-list-unportable-test: New module.
	* modules/linked-list-unportable-tests: New file, based on
	modules/linked-list-tests.
	* modules/linked-list-tests: Remove the unportable tests from here.
	Depend on linked-list-unportable-tests.

diff --git a/modules/linked-list-tests b/modules/linked-list-tests
index c45f02b..53472ed 100644
--- a/modules/linked-list-tests
+++ b/modules/linked-list-tests
@@ -1,32 +1,13 @@
 Files:
 tests/test-linked_list.c
-tests/test-asyncsafe-linked_list-weak.sh
-tests/test-asyncsafe-linked_list-weak.c
-tests/test-asyncsafe-linked_list-strong.sh
-tests/test-asyncsafe-linked_list-strong.c
 tests/macros.h
 
 Depends-on:
 array-list
-thread
-yield
-nanosleep
-sigaction
-sigprocmask
+linked-list-unportable-tests
 
 configure.ac:
 
 Makefile.am:
-TESTS += \
-  test-linked_list \
-  test-asyncsafe-linked_list-weak.sh \
-  test-asyncsafe-linked_list-strong.sh
-XFAIL_TESTS += \
-  test-asyncsafe-linked_list-weak.sh \
-  test-asyncsafe-linked_list-strong.sh
-check_PROGRAMS += \
-  test-linked_list \
-  test-asyncsafe-linked_list-weak \
-  test-asyncsafe-linked_list-strong
-test_asyncsafe_linked_list_weak_LDADD = $(LDADD) @LIBMULTITHREAD@ @YIELD_LIB@
-test_asyncsafe_linked_list_strong_LDADD = $(LDADD) @LIBMULTITHREAD@ @YIELD_LIB@
+TESTS += test-linked_list
+check_PROGRAMS += test-linked_list
diff --git a/modules/linked-list-unportable-tests b/modules/linked-list-unportable-tests
new file mode 100644
index 0000000..0c91c72
--- /dev/null
+++ b/modules/linked-list-unportable-tests
@@ -0,0 +1,32 @@
+Files:
+tests/test-asyncsafe-linked_list-weak.sh
+tests/test-asyncsafe-linked_list-weak.c
+tests/test-asyncsafe-linked_list-strong.sh
+tests/test-asyncsafe-linked_list-strong.c
+tests/macros.h
+
+Status:
+unportable-test
+
+Depends-on:
+linked-list
+thread
+yield
+nanosleep
+sigaction
+sigprocmask
+
+configure.ac:
+
+Makefile.am:
+TESTS += \
+  test-asyncsafe-linked_list-weak.sh \
+  test-asyncsafe-linked_list-strong.sh
+XFAIL_TESTS += \
+  test-asyncsafe-linked_list-weak.sh \
+  test-asyncsafe-linked_list-strong.sh
+check_PROGRAMS += \
+  test-asyncsafe-linked_list-weak \
+  test-asyncsafe-linked_list-strong
+test_asyncsafe_linked_list_weak_LDADD = $(LDADD) @LIBMULTITHREAD@ @YIELD_LIB@
+test_asyncsafe_linked_list_strong_LDADD = $(LDADD) @LIBMULTITHREAD@ @YIELD_LIB@