Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

On 10/12/21 9:02 PM, Alexey Dokuchaev wrote:

> Ports framework does several things which affect GNU configure
> scripts, particularly, it replaces build-aux/config.guess file
> with our own, where host/build tuples are derived from.
> 
> x86_64 is spelled as amd64 in FreeBSD

Ouch. When porting, does this mean you need to look for places where GNU 
source code says "x86_64" and replace many of these places with "amd64" 
before building? That sounds error-prone.


>    conftest.c:13:1: error: second parameter of 'main' (argument array)
>      must be of type 'char **'
>      main (int argc, char *argv)
>      ^
>      1 error generated.
>    configure:35726: $? = 1
>    configure:35741: result: unknown

Thanks for reporting that. That's due to a quoting typo in Gnulib's 
m4/stack-direction.m4. I fixed it by installing the attached patch into 
Gnulib. Grep will get this fix the next time it merges from Gnulib. In 
the meantime you should be able work around the problem by changing that 
'char *argv' to 'char *argv[]' in 'configure'.


>> and config.h says:
>>
>> #define STACK_DIRECTION -1

This is after you modified 'configure', right? Otherwise "result: 
unknown" should result in STACK_DIRECTION being 0, which 
gnulib/lib/stackvma.h does not support (perhaps it should?).

[-- Attachment #2: 0001-sigsegv-fix-quoting-problem.patch --]
[-- Type: text/x-patch, Size: 1626 bytes --]

From ff5ef2582300c2d982820ffeaede2ea08aaba3d8 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Wed, 13 Oct 2021 10:24:40 -0700
Subject: [PATCH] sigsegv: fix quoting problem

Problem reported for FreeBSD ports by Alexey Dokuchaev in:
https://bugs.gnu.org/51144
* m4/stack-direction.m4 (SV_STACK_DIRECTION): Quote brackets.
---
 ChangeLog             | 7 +++++++
 m4/stack-direction.m4 | 4 ++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index da5b570ee..ea71de0b2 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2021-10-13  Paul Eggert  <eggert@cs.ucla.edu>
+
+	sigsegv: fix quoting problem
+	Problem reported for FreeBSD ports by Alexey Dokuchaev in:
+	https://bugs.gnu.org/51144
+	* m4/stack-direction.m4 (SV_STACK_DIRECTION): Quote brackets.
+
 2021-10-10  Paul Eggert  <eggert@cs.ucla.edu>
 
 	nproc: port better to OpenBSD
diff --git a/m4/stack-direction.m4 b/m4/stack-direction.m4
index e682be97d..c90d69733 100644
--- a/m4/stack-direction.m4
+++ b/m4/stack-direction.m4
@@ -1,4 +1,4 @@
-# stack-direction.m4 serial 8
+# stack-direction.m4 serial 9
 dnl Copyright (C) 2002-2021 Free Software Foundation, Inc.
 dnl This file is free software, distributed under the terms of the GNU
 dnl General Public License.  As a special exception to the GNU General
@@ -79,7 +79,7 @@ find_stack_direction (int *addr, int depth)
   return dir + dummy;
 }
 int
-main (int argc, char *argv[])
+main (int argc, char *argv[[]])
 {
   printf ("%d\n", find_stack_direction (NULL, argc + 20));
   return 0;
-- 
2.30.2

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Alexey Dokuchaev via Gnulib discussion list]

On Wed, Oct 13, 2021 at 11:47:34AM -0700, Paul Eggert wrote:
> On 10/12/21 9:02 PM, Alexey Dokuchaev wrote:
> 
> >Ports framework does several things which affect GNU configure
> >scripts, particularly, it replaces build-aux/config.guess file
> >with our own, where host/build tuples are derived from.
> >
> >x86_64 is spelled as amd64 in FreeBSD
> 
> Ouch. When porting, does this mean you need to look for places where GNU
> source code says "x86_64" and replace many of these places with "amd64"
> before building? That sounds error-prone.

No, of course not, typically we don't replace anything.  All we do is
use our pre-built templates for config.{guess,site,sub} and pass the
--build=amd64-portbld-freebsd$(version) argument to configure scripts
if they are generated by GNU autotools.

In rare cases, like this one we're dealing with, this difference could
become a problem.  I can't remember when was the last time I've been
bitten by this. :-)

> >   conftest.c:13:1: error: second parameter of 'main' (argument array)
> >     must be of type 'char **'
> >     main (int argc, char *argv)
> >     ^
> >     1 error generated.
> >   configure:35726: $? = 1
> >   configure:35741: result: unknown
> 
> Thanks for reporting that. That's due to a quoting typo in Gnulib's
> m4/stack-direction.m4. I fixed it by installing the attached patch into
> Gnulib. Grep will get this fix the next time it merges from Gnulib. In
> the meantime you should be able work around the problem by changing that
> 'char *argv' to 'char *argv[]' in 'configure'.

Right, that's what I did as I've mentioned above.

> >>and config.h says:
> >>
> >>#define STACK_DIRECTION -1
> 
> This is after you modified 'configure', right?

Right.  So, in retrospect, there were two problems: "amd64" was not in
the list of known architectures, so it could not use known value for
stack direction, and had to fallback to run the program which did not
build because of unescaped brackets.  Fixing either of them allows it
to deduce correct #define STACK_DIRECTION -1.

Now that you've fixed the second problem, would you also consider
adding "amd64" as a synonym to "x86_64" in that switch/case check?

> Otherwise "result: unknown" should result in STACK_DIRECTION being 0,
> which gnulib/lib/stackvma.h does not support (perhaps it should?).

Are you sure "result: unknown" should result in 0, not empty value as
in my first email?

./danfe

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Paul Eggert]

On 10/14/21 7:02 AM, Alexey Dokuchaev wrote:

> All we do is
> use our pre-built templates for config.{guess,site,sub} and pass the
> --build=amd64-portbld-freebsd$(version) argument to configure scripts
> if they are generated by GNU autotools.

Surely this doesn't work if there's a configure.ac or m4/*.m4 file that 
does something like this:

   if "$host_os" = x86_64; then
     ...
   fi

and I see examples of this in gnulib/m4's float_h.m4, host-cpu-c-abi.m4, 
lib-prefix.m4, multiarch.m4, printf.m4, stack-direction.m4. In general 
don't you need to also edit these files and re-run Autoconf?


> I can't remember when was the last time I've been
> bitten by this. :-)

Often Gnulib code will guess the right value anyway. Not always, though. 
You ran into a case where Gnulib didn't guess and aborted. Some other 
times, the guess may differ from what it would be if you did a plain 
'configure; make' from a tarball. You might want to test for that, given 
the above list of potential problem files.

There may well be similar problems in configure.ac or *.m4 files 
maintained in GNU projects other than Gnulib. Autoconf itself has 
"x86_64" hardwired into AC_C_BIGENDIAN; I don't know whether this 
affects FreeBSD compilers.


> would you also consider
> adding "amd64" as a synonym to "x86_64" in that switch/case check?

Yes I suppose we could do that. Bruno, what do you think? You wrote most 
of those "x86_64"s.


>> Otherwise "result: unknown" should result in STACK_DIRECTION being 0,
>> which gnulib/lib/stackvma.h does not support (perhaps it should?).
> 
> Are you sure "result: unknown" should result in 0, not empty value as
> in my first email?

Not at this point, no.

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Alexey Dokuchaev via Gnulib discussion list]

On Thu, Oct 14, 2021 at 10:34:55AM -0700, Paul Eggert wrote:
> On 10/14/21 7:02 AM, Alexey Dokuchaev wrote:
> > On Wed, Oct 13, 2021 at 11:47:34AM -0700, Paul Eggert wrote:
> > > On 10/12/21 9:02 PM, Alexey Dokuchaev wrote:
> > > > Ports framework does several things which affect GNU configure
> > > > scripts, particularly, it replaces build-aux/config.guess file
> > > > with our own, where host/build tuples are derived from.
> > > > 
> > > > x86_64 is spelled as amd64 in FreeBSD
> > > 
> > > Ouch. When porting, does this mean you need to look for places where
> > > GNU source code says "x86_64" and replace many of these places with
> > > "amd64" before building? That sounds error-prone.
> > 
> > No, of course not, typically we don't replace anything.  All we do is
> > use our pre-built templates for config.{guess,site,sub} and pass the
> > --build=amd64-portbld-freebsd$(version) argument to configure scripts
> > if they are generated by GNU autotools.
> 
> Surely this doesn't work if there's a configure.ac or m4/*.m4 file that
> does something like this:
> 
>   if "$host_os" = x86_64; then
>     ...
>   fi
> 
> and I see examples of this in gnulib/m4's float_h.m4, host-cpu-c-abi.m4,
> lib-prefix.m4, multiarch.m4, printf.m4, stack-direction.m4. In general
> don't you need to also edit these files and re-run Autoconf?

In general, we don't edit those files, and typically we only run autoconf
when there's no pre-generated configure script, or there's something so
very wrong about it, or generated Makefiles, that patching isn't feasible
and we have to rebootstrap the whole thing.

About why this x86_64 vs amd64 thing does not cause massive fallout for
us, I better ask our Autotools expert.  Tijl, can you shed some light
here?  I've added back omitted context to the quoted parts of this email,
I hope Paul wouldn't mind.

./danfe

> > I can't remember when was the last time I've been bitten by this. :-)
> 
> Often Gnulib code will guess the right value anyway. Not always, though.
> You ran into a case where Gnulib didn't guess and aborted. Some other
> times, the guess may differ from what it would be if you did a plain
> 'configure; make' from a tarball. You might want to test for that, given
> the above list of potential problem files.
> 
> There may well be similar problems in configure.ac or *.m4 files
> maintained in GNU projects other than Gnulib. Autoconf itself has
> "x86_64" hardwired into AC_C_BIGENDIAN; I don't know whether this
> affects FreeBSD compilers.

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Tijl Coosemans]

On Fri, 15 Oct 2021 14:32:54 +0700 Alexey Dokuchaev <danfe@nsu.ru> wrote:
> On Thu, Oct 14, 2021 at 10:34:55AM -0700, Paul Eggert wrote:
> > On 10/14/21 7:02 AM, Alexey Dokuchaev wrote:  
> > > On Wed, Oct 13, 2021 at 11:47:34AM -0700, Paul Eggert wrote:  
> > > > On 10/12/21 9:02 PM, Alexey Dokuchaev wrote:  
> > > > > Ports framework does several things which affect GNU configure
> > > > > scripts, particularly, it replaces build-aux/config.guess file
> > > > > with our own, where host/build tuples are derived from.
> > > > > 
> > > > > x86_64 is spelled as amd64 in FreeBSD  
> > > > 
> > > > Ouch. When porting, does this mean you need to look for places where
> > > > GNU source code says "x86_64" and replace many of these places with
> > > > "amd64" before building? That sounds error-prone.  
> > > 
> > > No, of course not, typically we don't replace anything.  All we do is
> > > use our pre-built templates for config.{guess,site,sub} and pass the
> > > --build=amd64-portbld-freebsd$(version) argument to configure scripts
> > > if they are generated by GNU autotools.  
> > 
> > Surely this doesn't work if there's a configure.ac or m4/*.m4 file that
> > does something like this:
> > 
> >   if "$host_os" = x86_64; then
> >     ...
> >   fi
> > 
> > and I see examples of this in gnulib/m4's float_h.m4, host-cpu-c-abi.m4,
> > lib-prefix.m4, multiarch.m4, printf.m4, stack-direction.m4. In general
> > don't you need to also edit these files and re-run Autoconf?  
> 
> In general, we don't edit those files, and typically we only run autoconf
> when there's no pre-generated configure script, or there's something so
> very wrong about it, or generated Makefiles, that patching isn't feasible
> and we have to rebootstrap the whole thing.
> 
> About why this x86_64 vs amd64 thing does not cause massive fallout for
> us, I better ask our Autotools expert.  Tijl, can you shed some light
> here?  I've added back omitted context to the quoted parts of this email,
> I hope Paul wouldn't mind.

It is error prone, but it's rare for configure scripts to test the
architecture so no one has bothered to fix this properly.  Where
necessary we simply pass --build=x64_64-portbld-freebsd$(version).  An
example is gcc.

> > > I can't remember when was the last time I've been bitten by this. :-)  
> > 
> > Often Gnulib code will guess the right value anyway. Not always, though.
> > You ran into a case where Gnulib didn't guess and aborted. Some other
> > times, the guess may differ from what it would be if you did a plain
> > 'configure; make' from a tarball. You might want to test for that, given
> > the above list of potential problem files.
> > 
> > There may well be similar problems in configure.ac or *.m4 files
> > maintained in GNU projects other than Gnulib. Autoconf itself has
> > "x86_64" hardwired into AC_C_BIGENDIAN; I don't know whether this
> > affects FreeBSD compilers.  

This one seems to be used when testing Apple C compilers, so it doesn't
affect us.

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Paul Eggert]

On 10/16/21 9:40 AM, Tijl Coosemans wrote:
> Where
> necessary we simply pass --build=x64_64-portbld-freebsd$(version).  An
> example is gcc.

Could you pass --build=x64_64-portbld-freebsd$(version) to every 
'configure' invocation, not just gcc's and a few others?

If I understand you correctly, doing that would have avoided this 
particular problem, and it would avoid other potential problems with 
gnulib/m4's float_h.m4, host-cpu-c-abi.m4, lib-prefix.m4, multiarch.m4, 
printf.m4, as well as in GNU projects other than Gnulib (I haven't tried 
to catalog them).

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Jeffrey Walton]

On Sat, Oct 16, 2021 at 12:53 PM Paul Eggert <eggert@cs.ucla.edu> wrote:
>
> On 10/16/21 9:40 AM, Tijl Coosemans wrote:
> > Where
> > necessary we simply pass --build=x64_64-portbld-freebsd$(version).  An
> > example is gcc.
>
> Could you pass --build=x64_64-portbld-freebsd$(version) to every
> 'configure' invocation, not just gcc's and a few others?
>
> If I understand you correctly, doing that would have avoided this
> particular problem, and it would avoid other potential problems with
> gnulib/m4's float_h.m4, host-cpu-c-abi.m4, lib-prefix.m4, multiarch.m4,
> printf.m4, as well as in GNU projects other than Gnulib (I haven't tried
> to catalog them).

Related, that may affect Solaris, too. I believe Sun prefers the amd64 prefix.

And there's no telling what some of the other lesser known distros are doing.

Jeff

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Paul Eggert]

On 10/16/21 2:06 PM, Jeffrey Walton wrote:
> Related, that may affect Solaris, too. I believe Sun prefers the amd64 prefix.

We should be OK there. I just did a "pkg install editor/gnu-emacs" on 
Solaris 11.4, and its (emacs-version) returns "GNU Emacs 25.2.1 
(x86_64-pc-solaris2.11, X toolkit, Xaw scroll bars)" which means it's 
using x86_64. (Though, eeeeuuuw, Xaw? Really?)


> And there's no telling what some of the other lesser known distros are doing.

Sure, but distros can put any string they want there; we can't possibly 
support them all, and it's good to promote a common standard.

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Bruno Haible]

Alexey Dokuchaev wrote in
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=51144> and
<https://lists.gnu.org/archive/html/bug-gnulib/2021-10/msg00018.html>:

> > >Ports framework does several things which affect GNU configure
> > >scripts, particularly, it replaces build-aux/config.guess file
> > >with our own, where host/build tuples are derived from.
> > >
> > >x86_64 is spelled as amd64 in FreeBSD
...
> All we do is
> use our pre-built templates for config.{guess,site,sub} and pass the
> --build=amd64-portbld-freebsd$(version) argument to configure scripts
> if they are generated by GNU autotools.

This is a recipe for major hassle. The output of config.{guess,sub}
is a *canonicalized* triple. See this comment in config.sub:

  # The goal of this file is to map all the various variations of a given
  # machine specification into a single specification in the form:
  #       CPU_TYPE-MANUFACTURER-OPERATING_SYSTEM
  # or in some cases, the newer four-part form:
  #       CPU_TYPE-MANUFACTURER-KERNEL-OPERATING_SYSTEM

and later:

        # Here we normalize CPU types irrespective of the vendor
        amd64-*)
                cpu=x86_64
                ;;

The rationale for this canonicalization is that on the C preprocessor
level, significant synonyms exist (see [1] for the list), and this has
caused many portability issues over time. So, at the autoconf level,
the GNU project has decided to canonicalize the elements of $host.
Linux prefers x86_64, FreeBSD prefers amd64, Windows prefers x64,
and so on. The canonicalization
1) allows the GNU packages to recognize just one (x86_64) instead of
   multiple ones,
2) allows GNU packages that include arch-dependent files (e.g.
   GNU gmp, GNU lightning, GNU libffcall) to name these files consistently.
   Not asm-x86_64.c in one package and asm-amd64.c in another package.

You can architecture the FreeBSD ports collection and its build system
in the way you like. But you cannot expect dozens of GNU packages to
support a different name for a CPU than the canonical name that GNU
picked 18 years ago:

    2003-05-09  Andreas Jaeger  <aj@suse.de>

    	* config.sub (maybe_os): Add alias amd64 for x86_64.

You can replace the config.guess script before building a package
in the FreeBSD ports collection. But when it's a GNU package you should
better make sure that this replacement script produces the same results
as the GNU config.guess does.

The same holds for the non-GNU non-FreeBSD packages that use Autoconf:
If you force non-GNU-canonical names on them, they may curse the GNU
build system, but in fact the culprit (= origin of the issue) would
still be the choices made in the FreeBSD ports build system.

Paul Eggert asked:
> > would you also consider
> > adding "amd64" as a synonym to "x86_64" in that switch/case check?
>
> Yes I suppose we could do that. Bruno, what do you think? You wrote most 
> of those "x86_64"s.

A firm "no!" from my part.

Btw, a similar problem exists for the aarch64 / arm64 CPU type.
'aarch64' is the name chosen by the GCC people, whereas 'arm64' is the
name chosen by Linux [2] and by Debian [3]. You need to obey the
canonicalization in effect at the level at which you are working;
at the $host_cpu level you should expect to see 'aarch64', never 'arm64' —
because passing --host=arm64-... is wrong.

Bruno

[1] https://sourceforge.net/p/predef/wiki/Architectures/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch?h=v5.0
[3] http://ftp.debian.org/debian/dists/Debian10.11/main/

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Alexey Dokuchaev via Gnulib discussion list]

On Sun, Oct 17, 2021 at 11:20:12PM +0200, Bruno Haible wrote:
> Alexey Dokuchaev wrote in
> ...
> > All we do is
> > use our pre-built templates for config.{guess,site,sub} and pass the
> > --build=amd64-portbld-freebsd$(version) argument to configure scripts
> > if they are generated by GNU autotools.
> 
> This is a recipe for major hassle. The output of config.{guess,sub}
> is a *canonicalized* triple. See this comment in config.sub:
> 
>   # The goal of this file is to map all the various variations of a given
>   # machine specification into a single specification in the form:
>   #       CPU_TYPE-MANUFACTURER-OPERATING_SYSTEM
>   # or in some cases, the newer four-part form:
>   #       CPU_TYPE-MANUFACTURER-KERNEL-OPERATING_SYSTEM
> 
> and later:
> 
>         # Here we normalize CPU types irrespective of the vendor
>         amd64-*)
>                 cpu=x86_64
>                 ;;

Hmm, there's no such normalization code in our /usr/ports/Templates/config.sub
with timestamp='2018-05-24'.

> You can architecture the FreeBSD ports collection and its build system
> in the way you like. But you cannot expect dozens of GNU packages to
> support a different name for a CPU than the canonical name that GNU
> picked 18 years ago:
> 
>     2003-05-09  Andreas Jaeger  <aj@suse.de>
> 
>     	* config.sub (maybe_os): Add alias amd64 for x86_64.

I wonder why it's not in our template if it's from 2003.

> Paul Eggert asked:
> > > would you also consider adding "amd64" as a synonym to "x86_64" in
> > > that switch/case check?
> >
> > Yes I suppose we could do that. Bruno, what do you think? You wrote most
> > of those "x86_64"s.
> 
> A firm "no!" from my part.

Fair enough; I guess we can live with local patches to configure for our
diffutils and grep ports (for now).

./danfe

Re: bug#51144: GNU grep 3.7 fails to build on FreeBSD

[Bruno Haible]

Alexey Dokuchaev wrote:
> I wonder why it's not in our template if it's from 2003.

Just guessing: Maybe because some kernel-related FreeBSD packages want
'amd64'? In other words, don't you need to distinguish original FreeBSD
packages from other packages?

Bruno