test-rwlock1 failing on latest Fedora Rawhide

test-rwlock1 failing on latest Fedora Rawhide

[Richard W.M. Jones]

I cannot reproduce this locally, hence my bug report is rather devoid
of details.  However, it's 100% reproducible in Koji (the Fedora
Rawhide build system) on *all* architectures except armv7:

FAIL: test-rwlock1
==================
Unexpected outcome 3
FAIL: test-thread_create

The full log is here:

https://kojipkgs.fedoraproject.org//work/tasks/4382/32214382/build.log

and the full logs on all other architectures are linked from:

https://koji.fedoraproject.org/koji/taskinfo?taskID=32214341

Oddly the exact same thing was reported last year, but *only* on
armv7.  Unfortunately there was no resolution:

https://lists.gnu.org/archive/html/coreutils/2018-02/msg00023.html

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html

Re: test-rwlock1 failing on latest Fedora Rawhide

[Bruno Haible]

Hi Rich,

> I cannot reproduce this locally, hence my bug report is rather devoid
> of details.  However, it's 100% reproducible in Koji (the Fedora
> Rawhide build system) on *all* architectures except armv7:
> 
> FAIL: test-rwlock1
> ==================
> Unexpected outcome 3
> FAIL: test-thread_create

As written in [1]:
"Unexpected outcome 3" means that the test program could not create a second
thread (other than the main thread).

On glibc systems, this typically means that the 'test-rwlock1' program has not
been linked with '-lpthread'.

> The full log is here:
> 
> https://kojipkgs.fedoraproject.org//work/tasks/4382/32214382/build.log

The configure command-line looks right.

I see
  LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'

A global "-Wl,--as-needed" has the effect of removing libraries like
-lpthread from the link. Which would explain the symptom.

What do the other elements of LDFLAGS do?

Bruno

[1] http://lists.gnu.org/archive/html/coreutils/2018-02/msg00020.html

Re: test-rwlock1 failing on latest Fedora Rawhide

[Richard W.M. Jones]

On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
> Hi Rich,
> 
> > I cannot reproduce this locally, hence my bug report is rather devoid
> > of details.  However, it's 100% reproducible in Koji (the Fedora
> > Rawhide build system) on *all* architectures except armv7:
> > 
> > FAIL: test-rwlock1
> > ==================
> > Unexpected outcome 3
> > FAIL: test-thread_create
> 
> As written in [1]:
> "Unexpected outcome 3" means that the test program could not create a second
> thread (other than the main thread).
> 
> On glibc systems, this typically means that the 'test-rwlock1' program has not
> been linked with '-lpthread'.
> 
> > The full log is here:
> > 
> > https://kojipkgs.fedoraproject.org//work/tasks/4382/32214382/build.log
> 
> The configure command-line looks right.
> 
> I see
>   LDFLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld'
> 
> A global "-Wl,--as-needed" has the effect of removing libraries like
> -lpthread from the link. Which would explain the symptom.

Ahhhhhhh, that would also explain why I couldn't reproduce it.  I was
using the default flags when I was testing locally.  Using the Fedora
flags, I *can* reproduce it locally, thanks!

> What do the other elements of LDFLAGS do?

All the flags AFAIK come from Fedora and are needed for hardening or
other Fedora policy purposes.

I checked the history of the Fedora package which adds these flags, and
it seems like --as-needed was added for:

https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking

Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
gnulib wrong?

Rich.

> Bruno
> 
> [1] http://lists.gnu.org/archive/html/coreutils/2018-02/msg00020.html

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: test-rwlock1 failing on latest Fedora Rawhide

[Dmitry V. Levin]

[-- Attachment #1: Type: text/plain, Size: 714 bytes --]

Hi,

On Wed, Jan 23, 2019 at 08:45:06PM +0000, Richard W.M. Jones wrote:
> On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
[...]
> I checked the history of the Fedora package which adds these flags, and
> it seems like --as-needed was added for:
> 
> https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
> 
> Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
> about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
> gnulib wrong?

We also build everything with --as-needed by default, but
gnulib does the right thing nowadays by using
-Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
instead of plain -lpthread.


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: test-rwlock1 failing on latest Fedora Rawhide

[Richard W.M. Jones]

On Thu, Jan 24, 2019 at 12:03:21AM +0300, Dmitry V. Levin wrote:
> Hi,
> 
> On Wed, Jan 23, 2019 at 08:45:06PM +0000, Richard W.M. Jones wrote:
> > On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
> [...]
> > I checked the history of the Fedora package which adds these flags, and
> > it seems like --as-needed was added for:
> > 
> > https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
> > 
> > Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
> > about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
> > gnulib wrong?
> 
> We also build everything with --as-needed by default, but
> gnulib does the right thing nowadays by using
> -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> instead of plain -lpthread.

hivex is using a very recent gnulib (34290cb926).

The test is linked with:

/bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2
-g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
-Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
-grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
-specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
-fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
-Wl,-z,relro -Wl,--as-needed -Wl,-z,now
-specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-thread_create
test-thread_create.o libtests.a ../../gnulib/lib/libgnu.la libtests.a
-pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state

As you can see from the last bit, the --push-state ... --pop-state
seems correct.  Yet the test still fails.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW

Re: test-rwlock1 failing on latest Fedora Rawhide

[Michael Hudson-Doyle]

[-- Attachment #1: Type: text/plain, Size: 881 bytes --]

On Thu, 24 Jan 2019 at 10:03, Dmitry V. Levin <ldv@altlinux.org> wrote:

> Hi,
>
> On Wed, Jan 23, 2019 at 08:45:06PM +0000, Richard W.M. Jones wrote:
> > On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
> [...]
> > I checked the history of the Fedora package which adds these flags, and
> > it seems like --as-needed was added for:
> >
> > https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
> >
> > Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
> > about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
> > gnulib wrong?
>
> We also build everything with --as-needed by default, but
> gnulib does the right thing nowadays by using
> -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> instead of plain -lpthread.
>

libtool can "helpfully" reorder those flags so they don't do anything
though.

Cheers,
mwh

[-- Attachment #2: Type: text/html, Size: 1456 bytes --]

Re: test-rwlock1 failing on latest Fedora Rawhide

[Dmitry V. Levin]

[-- Attachment #1: Type: text/plain, Size: 2110 bytes --]

On Wed, Jan 23, 2019 at 09:14:30PM +0000, Richard W.M. Jones wrote:
> On Thu, Jan 24, 2019 at 12:03:21AM +0300, Dmitry V. Levin wrote:
> > Hi,
> > 
> > On Wed, Jan 23, 2019 at 08:45:06PM +0000, Richard W.M. Jones wrote:
> > > On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
> > [...]
> > > I checked the history of the Fedora package which adds these flags, and
> > > it seems like --as-needed was added for:
> > > 
> > > https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
> > > 
> > > Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
> > > about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
> > > gnulib wrong?
> > 
> > We also build everything with --as-needed by default, but
> > gnulib does the right thing nowadays by using
> > -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> > instead of plain -lpthread.
> 
> hivex is using a very recent gnulib (34290cb926).
> 
> The test is linked with:
> 
> /bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2
> -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
> -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
> -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
> -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
> -Wl,-z,relro -Wl,--as-needed -Wl,-z,now
> -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-thread_create
> test-thread_create.o libtests.a ../../gnulib/lib/libgnu.la libtests.a
> -pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> 
> As you can see from the last bit, the --push-state ... --pop-state
> seems correct.  Yet the test still fails.

I see libtool is involved, and libtool is known to reorder arguments
in a very unhelpful way.  Could you show the actual link command invoked
by libtool, please?  I expect to see something like
-Wl,--push-state -Wl,--no-as-needed -Wl,--pop-state ... -lpthread
which doesn't make sense.


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: test-rwlock1 failing on latest Fedora Rawhide

[Richard W.M. Jones]

On Thu, Jan 24, 2019 at 01:09:52AM +0300, Dmitry V. Levin wrote:
> On Wed, Jan 23, 2019 at 09:14:30PM +0000, Richard W.M. Jones wrote:
> > On Thu, Jan 24, 2019 at 12:03:21AM +0300, Dmitry V. Levin wrote:
> > > Hi,
> > > 
> > > On Wed, Jan 23, 2019 at 08:45:06PM +0000, Richard W.M. Jones wrote:
> > > > On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
> > > [...]
> > > > I checked the history of the Fedora package which adds these flags, and
> > > > it seems like --as-needed was added for:
> > > > 
> > > > https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
> > > > 
> > > > Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
> > > > about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
> > > > gnulib wrong?
> > > 
> > > We also build everything with --as-needed by default, but
> > > gnulib does the right thing nowadays by using
> > > -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> > > instead of plain -lpthread.
> > 
> > hivex is using a very recent gnulib (34290cb926).
> > 
> > The test is linked with:
> > 
> > /bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2
> > -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> > -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
> > -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
> > -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
> > -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
> > -Wl,-z,relro -Wl,--as-needed -Wl,-z,now
> > -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-thread_create
> > test-thread_create.o libtests.a ../../gnulib/lib/libgnu.la libtests.a
> > -pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> > 
> > As you can see from the last bit, the --push-state ... --pop-state
> > seems correct.  Yet the test still fails.
> 
> I see libtool is involved, and libtool is known to reorder arguments
> in a very unhelpful way.  Could you show the actual link command invoked
> by libtool, please?  I expect to see something like
> -Wl,--push-state -Wl,--no-as-needed -Wl,--pop-state ... -lpthread
> which doesn't make sense.

I've posted the entire build/check with V=1 here:

http://oirase.annexia.org/tmp/2019-01-gnulib-hivex.txt

If you grep through that for "test-thread_create" you should find the
appropriate lines.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: test-rwlock1 failing on latest Fedora Rawhide

[Dmitry V. Levin]

[-- Attachment #1: Type: text/plain, Size: 4069 bytes --]

On Wed, Jan 23, 2019 at 10:24:26PM +0000, Richard W.M. Jones wrote:
> On Thu, Jan 24, 2019 at 01:09:52AM +0300, Dmitry V. Levin wrote:
> > On Wed, Jan 23, 2019 at 09:14:30PM +0000, Richard W.M. Jones wrote:
> > > On Thu, Jan 24, 2019 at 12:03:21AM +0300, Dmitry V. Levin wrote:
> > > > Hi,
> > > > 
> > > > On Wed, Jan 23, 2019 at 08:45:06PM +0000, Richard W.M. Jones wrote:
> > > > > On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
> > > > [...]
> > > > > I checked the history of the Fedora package which adds these flags, and
> > > > > it seems like --as-needed was added for:
> > > > > 
> > > > > https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
> > > > > 
> > > > > Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
> > > > > about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
> > > > > gnulib wrong?
> > > > 
> > > > We also build everything with --as-needed by default, but
> > > > gnulib does the right thing nowadays by using
> > > > -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> > > > instead of plain -lpthread.
> > > 
> > > hivex is using a very recent gnulib (34290cb926).
> > > 
> > > The test is linked with:
> > > 
> > > /bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2
> > > -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> > > -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
> > > -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
> > > -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
> > > -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
> > > -Wl,-z,relro -Wl,--as-needed -Wl,-z,now
> > > -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-thread_create
> > > test-thread_create.o libtests.a ../../gnulib/lib/libgnu.la libtests.a
> > > -pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> > > 
> > > As you can see from the last bit, the --push-state ... --pop-state
> > > seems correct.  Yet the test still fails.
> > 
> > I see libtool is involved, and libtool is known to reorder arguments
> > in a very unhelpful way.  Could you show the actual link command invoked
> > by libtool, please?  I expect to see something like
> > -Wl,--push-state -Wl,--no-as-needed -Wl,--pop-state ... -lpthread
> > which doesn't make sense.
> 
> I've posted the entire build/check with V=1 here:
> 
> http://oirase.annexia.org/tmp/2019-01-gnulib-hivex.txt
> 
> If you grep through that for "test-thread_create" you should find the
> appropriate lines.

According to the log, it was libtool that broke a valid command

/bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-rwlock1 test-rwlock1.o libtests.a ../../gnulib/lib/libgnu.la libtests.a -pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state -lrt 

into an invalid command

libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z -Wl,relro -Wl,--as-needed -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-rwlock1 test-rwlock1.o -pthread -Wl,--push-state -Wl,--no-as-needed -Wl,--pop-state libtests.a ../../gnulib/lib/.libs/libgnu.a libtests.a -lpthread -lrt -pthread 

What's the version of libtool that's produced such a mess?


-- 
ldv

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: test-rwlock1 failing on latest Fedora Rawhide

[Bruno Haible]

Dmitry V. Levin wrote:
> According to the log, it was libtool that broke a valid command
> 
> /bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-rwlock1 test-rwlock1.o libtests.a ../../gnulib/lib/libgnu.la libtests.a -pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state -lrt 
> 
> into an invalid command
> 
> libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z -Wl,relro -Wl,--as-needed -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-rwlock1 test-rwlock1.o -pthread -Wl,--push-state -Wl,--no-as-needed -Wl,--pop-state libtests.a ../../gnulib/lib/.libs/libgnu.a libtests.a -lpthread -lrt -pthread 
> 
> What's the version of libtool that's produced such a mess?

Even the latest released libtool versions do this.

This patch should fix the issue. Pushed.


2019-01-23  Bruno Haible  <bruno@clisp.org>

	thread: Force linking with -lpthread, even when --as-needed is in use.
	Reported by Richard W.M. Jones <rjones@redhat.com> in
	<https://lists.gnu.org/archive/html/bug-gnulib/2019-01/msg00123.html>.
	* lib/glthread/thread.h (pthread_create): Don't declare weak.

diff --git a/lib/glthread/thread.h b/lib/glthread/thread.h
index ca40508..1d2a544 100644
--- a/lib/glthread/thread.h
+++ b/lib/glthread/thread.h
@@ -133,8 +133,6 @@ extern int glthread_in_use (void);
    call to foo(...) in the same function.  To avoid this, we test the
    address of a function in libpthread that we don't use.  */
 
-#  pragma weak pthread_create
-
 #  ifndef pthread_sigmask /* Do not declare rpl_pthread_sigmask weak.  */
 #   pragma weak pthread_sigmask
 #  endif

Re: test-rwlock1 failing on latest Fedora Rawhide

[Bruno Haible]

Let me summarize the issue.

(1) The distro sets LDFLAGS="-Wl,--as-needed". So, libraries that not are not
    required through at least 1 symbol are not linked in.
(2) threadlib.m4 defines LIBMULTITHREAD in a way that is meant to defeat
    -Wl,--as-needed. But it does not work, due to the reordering of
    options done by libtool.
    So, --as-needed is still in effect.
(3) threadlib.m4 enables weak symbols, and thread.h and lock.h define
    specific symbols as weak. Weak symbols are ignored in (1).

So, the fix is to define fewer symbols as weak.

pthread_create was defined as weak, so that users of gnulib could use
glthread_create without linking with -lpthread; it would then return
ENOSYS. Now, the use of glthread_create will require linking with -lpthread,
on platforms like glibc. It may be argued that this is actually a good thing.

Bruno

Re: test-rwlock1 failing on latest Fedora Rawhide

[Bruno Haible]

> (2) threadlib.m4 defines LIBMULTITHREAD in a way that is meant to defeat
>     -Wl,--as-needed. But it does not work, due to the reordering of
>     options done by libtool.

Let me take out the code that does not work.


2019-01-23  Bruno Haible  <bruno@clisp.org>

	threadlib: Revert commit from 2018-06-25. We now have a better fix.
	* m4/threadlib.m4 (gl_THREADLIB_BODY): Don't attempt to defeat a
	preceding -Wl,--as-needed option. Don't check whether the linker
	supports --as-needed/--no-as-needed and --push-state/--pop-state.

diff --git a/m4/threadlib.m4 b/m4/threadlib.m4
index f7841b6..bfc3bac 100644
--- a/m4/threadlib.m4
+++ b/m4/threadlib.m4
@@ -1,4 +1,4 @@
-# threadlib.m4 serial 15
+# threadlib.m4 serial 16
 dnl Copyright (C) 2005-2019 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -151,36 +151,6 @@ int main ()
          *" -static "*) gl_cv_have_weak=no ;;
        esac
       ])
-    dnl Check whether the linker supports the --as-needed/--no-as-needed options.
-    dnl Assume GCC, so that we can use the -Wl option.
-    AC_CACHE_CHECK([whether the linker supports --as-needed],
-      [gl_cv_linker_have_as_needed],
-      [if test -n "$GCC"; then
-         gl_saved_ldflags="$LDFLAGS"
-         LDFLAGS="$gl_saved_ldflags -Wl,--as-needed -Wl,--no-as-needed"
-         AC_LINK_IFELSE([AC_LANG_PROGRAM()],
-           [gl_cv_linker_have_as_needed=yes],
-           [gl_cv_linker_have_as_needed=no])
-         LDFLAGS="$gl_saved_ldflags"
-       else
-         gl_cv_linker_have_as_needed=no
-       fi
-      ])
-    dnl Check whether the linker supports the --push-state/--pop-state options.
-    dnl Assume GCC, so that we can use the -Wl option.
-    AC_CACHE_CHECK([whether the linker supports --push-state],
-      [gl_cv_linker_have_push_state],
-      [if test -n "$GCC"; then
-         gl_saved_ldflags="$LDFLAGS"
-         LDFLAGS="$gl_saved_ldflags -Wl,--push-state -Wl,--pop-state"
-         AC_LINK_IFELSE([AC_LANG_PROGRAM()],
-           [gl_cv_linker_have_push_state=yes],
-           [gl_cv_linker_have_push_state=no])
-         LDFLAGS="$gl_saved_ldflags"
-       else
-         gl_cv_linker_have_push_state=no
-       fi
-      ])
     if test "$gl_use_threads" = yes || test "$gl_use_threads" = posix; then
       # On OSF/1, the compiler needs the flag -pthread or -D_REENTRANT so that
       # it groks <pthread.h>. It's added above, in gl_THREADLIB_EARLY_BODY.
@@ -259,32 +229,6 @@ int main ()
                 [Define if references to the POSIX multithreading library should be made weak.])
               LIBTHREAD=
               LTLIBTHREAD=
-              dnl On platforms where GCC enables --as-needed by default, attempt
-              dnl to make sure that LIBMULTITHREAD really links with -lpthread.
-              dnl Otherwise linking with LIBMULTITHREAD has no effect; then
-              dnl the weak symbols are not defined and thus evaluate to NULL.
-              case "$LIBMULTITHREAD" in
-                "") ;;
-                -pthread)
-                  if test $gl_cv_linker_have_as_needed = yes; then
-                    if test $gl_cv_linker_have_push_state = yes; then
-                      LIBMULTITHREAD="$LIBMULTITHREAD -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state"
-                    else
-                      LIBMULTITHREAD="$LIBMULTITHREAD -Wl,--no-as-needed -lpthread"
-                    fi
-                  fi
-                  ;;
-                *)
-                  if test $gl_cv_linker_have_as_needed = yes; then
-                    if test $gl_cv_linker_have_push_state = yes; then
-                      LIBMULTITHREAD="-Wl,--push-state -Wl,--no-as-needed $LIBMULTITHREAD -Wl,--pop-state"
-                    else
-                      LIBMULTITHREAD="-Wl,--no-as-needed $LIBMULTITHREAD"
-                    fi
-                  fi
-                  ;;
-              esac
-              # TODO: May need to modify LTLIBMULTITHREAD similarly.
             fi
           fi
         fi
@@ -317,18 +261,6 @@ int main ()
               [Define if references to the old Solaris multithreading library should be made weak.])
             LIBTHREAD=
             LTLIBTHREAD=
-            dnl On platforms where GCC enables --as-needed by default, attempt
-            dnl to make sure that LIBMULTITHREAD really links with -lthread.
-            dnl Otherwise linking with LIBMULTITHREAD has no effect; then
-            dnl the weak symbols are not defined and thus evaluate to NULL.
-            if test $gl_cv_linker_have_as_needed = yes; then
-              if test $gl_cv_linker_have_push_state = yes; then
-                LIBMULTITHREAD="-Wl,--push-state -Wl,--no-as-needed $LIBMULTITHREAD -Wl,--pop-state"
-              else
-                LIBMULTITHREAD="-Wl,--no-as-needed $LIBMULTITHREAD"
-              fi
-            fi
-            # TODO: May need to modify LTLIBMULTITHREAD similarly.
           fi
         fi
       fi
@@ -357,18 +289,6 @@ int main ()
               [Define if references to the GNU Pth multithreading library should be made weak.])
             LIBTHREAD=
             LTLIBTHREAD=
-            dnl On platforms where GCC enables --as-needed by default, attempt
-            dnl to make sure that LIBMULTITHREAD really links with -lpth.
-            dnl Otherwise linking with LIBMULTITHREAD has no effect; then
-            dnl the weak symbols are not defined and thus evaluate to NULL.
-            if test $gl_cv_linker_have_as_needed = yes; then
-              if test $gl_cv_linker_have_push_state = yes; then
-                LIBMULTITHREAD="-Wl,--push-state -Wl,--no-as-needed $LIBMULTITHREAD -Wl,--pop-state"
-              else
-                LIBMULTITHREAD="-Wl,--no-as-needed $LIBMULTITHREAD"
-              fi
-            fi
-            # TODO: May need to modify LTLIBMULTITHREAD similarly.
           fi
         fi
       else

Re: test-rwlock1 failing on latest Fedora Rawhide

[Richard W.M. Jones]

On Thu, Jan 24, 2019 at 01:38:28AM +0300, Dmitry V. Levin wrote:
> On Wed, Jan 23, 2019 at 10:24:26PM +0000, Richard W.M. Jones wrote:
> > On Thu, Jan 24, 2019 at 01:09:52AM +0300, Dmitry V. Levin wrote:
> > > On Wed, Jan 23, 2019 at 09:14:30PM +0000, Richard W.M. Jones wrote:
> > > > On Thu, Jan 24, 2019 at 12:03:21AM +0300, Dmitry V. Levin wrote:
> > > > > Hi,
> > > > > 
> > > > > On Wed, Jan 23, 2019 at 08:45:06PM +0000, Richard W.M. Jones wrote:
> > > > > > On Wed, Jan 23, 2019 at 09:01:19PM +0100, Bruno Haible wrote:
> > > > > [...]
> > > > > > I checked the history of the Fedora package which adds these flags, and
> > > > > > it seems like --as-needed was added for:
> > > > > > 
> > > > > > https://fedoraproject.org/wiki/Changes/RemoveExcessiveLinking
> > > > > > 
> > > > > > Removing -Wl,--as-needed fixes the problem.  However I'm still unclear
> > > > > > about this.  Is Fedora wrong?  Is hivex using gnulib wrongly?  Is
> > > > > > gnulib wrong?
> > > > > 
> > > > > We also build everything with --as-needed by default, but
> > > > > gnulib does the right thing nowadays by using
> > > > > -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> > > > > instead of plain -lpthread.
> > > > 
> > > > hivex is using a very recent gnulib (34290cb926).
> > > > 
> > > > The test is linked with:
> > > > 
> > > > /bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2
> > > > -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
> > > > -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong
> > > > -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
> > > > -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic
> > > > -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
> > > > -Wl,-z,relro -Wl,--as-needed -Wl,-z,now
> > > > -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-thread_create
> > > > test-thread_create.o libtests.a ../../gnulib/lib/libgnu.la libtests.a
> > > > -pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state
> > > > 
> > > > As you can see from the last bit, the --push-state ... --pop-state
> > > > seems correct.  Yet the test still fails.
> > > 
> > > I see libtool is involved, and libtool is known to reorder arguments
> > > in a very unhelpful way.  Could you show the actual link command invoked
> > > by libtool, please?  I expect to see something like
> > > -Wl,--push-state -Wl,--no-as-needed -Wl,--pop-state ... -lpthread
> > > which doesn't make sense.
> > 
> > I've posted the entire build/check with V=1 here:
> > 
> > http://oirase.annexia.org/tmp/2019-01-gnulib-hivex.txt
> > 
> > If you grep through that for "test-thread_create" you should find the
> > appropriate lines.
> 
> According to the log, it was libtool that broke a valid command
> 
> /bin/sh ../../libtool --tag=CC --preserve-dup-deps --mode=link gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-rwlock1 test-rwlock1.o libtests.a ../../gnulib/lib/libgnu.la libtests.a -pthread -Wl,--push-state -Wl,--no-as-needed -lpthread -Wl,--pop-state -lrt 
> 
> into an invalid command
> 
> libtool: link: gcc -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wl,-z -Wl,relro -Wl,--as-needed -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o test-rwlock1 test-rwlock1.o -pthread -Wl,--push-state -Wl,--no-as-needed -Wl,--pop-state libtests.a ../../gnulib/lib/.libs/libgnu.a libtests.a -lpthread -lrt -pthread 
> 
> What's the version of libtool that's produced such a mess?

libtool-2.4.6-28.fc30.x86_64

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: test-rwlock1 failing on latest Fedora Rawhide

[Richard W.M. Jones]

I see two patches that went into gnulib overnight.  I have now tried
compiling hivex with gnulib at commit 34881aff4043, and that seems to
have fixed the problem, thanks all.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html