Aarch64 test results and questions

Aarch64 test results and questions

[Steve Ellcey]

Since we are in a freeze for 2.30, I ran the glibc tests on my aarch64
ThunderX2 box and was wondering if my results are consistent with what
other people are getting or not and if these failures need to be
investigated further.

I had to use Florian Weimer's patch to tst-pthread-getattr.c to get
that to build with the latest GCC.

After that I got four failures:

FAIL: elf/check-abi-libc
FAIL: elf/tst-ldconfig-bad-aux-cache
FAIL: elf/tst-pldd
FAIL: nss/tst-nss-files-hosts-long
      4 FAIL


check-abi-libc seems to be PR 14664

tst-nss-files-hosts-long may be PR 24816, I am not running in a docker,
I am on a native system.  The failure is:

	error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
	error: 1 test failures

tst-pldd does not seem to have any known PR associated with it.  The
.out file has:

	error: subprocess failed: pldd
	error:   expected exit status: 0
	error:   actual exit status: 1 [0x100]
	error: subprocess failed: pldd
	error:   unexpected error output from subprocess
	/extra/sellcey/test-glibc/install/bin/pldd: cannot attach to process 3: Operation not permitted

	tst-pldd.c:86: numeric comparison failure
	   left: -1 (0xffffffff); from: fscanf (out, "%u: " STRINPUT (512), &pid, buffer)
	  right: 2 (0x2); from: 2
	tst-pldd.c:88: numeric comparison failure
	   left: -1083244493 (0xbf6f0033); from: pid
	  right: 3 (0x3); from: target.pid
	tst-pldd.c:89: numeric comparison failure
	   left: -200 (0xffffff38); from: strcmp (basename (buffer), "tst-pldd")
	  right: 0 (0x0); from: 0
	tst-pldd.c:124: numeric comparison failure
	   left: 0 (0x0); from: interpreter_found
	  right: 1 (0x1); from: true
	tst-pldd.c:125: numeric comparison failure
	   left: 0 (0x0); from: libc_found
	  right: 1 (0x1); from: true
	error: 7 test failures

tst-ldconfig-bad-aux-cache also does not seem to have any known PR's. 
I get the following in the .out file:

	tst-ldconfig-bad-aux-cache.c:84: numeric comparison failure
	   left: 256 (0x100); from: status
	  right: 0 (0x0); from: 0
	error: support-xstat.c:29: stat64 ("/var/cache/ldconfig/aux-cache"): No such file or directory
	error: 2 test failures
	running post-clean rsync


Steve Ellcey
sellcey@marvell.com

Re: Aarch64 test results and questions

[Florian Weimer]

* Steve Ellcey:

> tst-pldd does not seem to have any known PR associated with it.  The
> .out file has:
>
> 	error: subprocess failed: pldd
> 	error:   expected exit status: 0
> 	error:   actual exit status: 1 [0x100]
> 	error: subprocess failed: pldd
> 	error:   unexpected error output from subprocess
> 	/extra/sellcey/test-glibc/install/bin/pldd: cannot attach to process 3: Operation not permitted

pldd may not work at all on this platform due to YAMA ptrace scope
enforcement.

Maybe this is something we could detect in the test and use the
information to mark it UNSUPPORTED.

> tst-ldconfig-bad-aux-cache also does not seem to have any known PR's. 
> I get the following in the .out file:
>
> 	tst-ldconfig-bad-aux-cache.c:84: numeric comparison failure
> 	   left: 256 (0x100); from: status
> 	  right: 0 (0x0); from: 0
> 	error: support-xstat.c:29: stat64 ("/var/cache/ldconfig/aux-cache"): No such file or directory
> 	error: 2 test failures
> 	running post-clean rsync

What's the path that ldconfig is actually creating on this target?

Thanks,
Florian

Re: Aarch64 test results and questions

[Steve Ellcey]

On Fri, 2019-07-26 at 18:52 +0200, Florian Weimer wrote:

> > tst-ldconfig-bad-aux-cache also does not seem to have any known
> > PR's. 
> > I get the following in the .out file:
> > 
> > 	tst-ldconfig-bad-aux-cache.c:84: numeric comparison failure
> > 	   left: 256 (0x100); from: status
> > 	  right: 0 (0x0); from: 0
> > 	error: support-xstat.c:29: stat64 ("/var/cache/ldconfig/aux-
> > cache"): No such file or directory
> > 	error: 2 test failures
> > 	running post-clean rsync
> 
> What's the path that ldconfig is actually creating on this target?

I built a toolchain (binutils, gcc, glibc) that are all in
/extra/sellcey/test-glibc/install.


% /extra/sellcey/test-glibc/install/sbin/ldconfig -p
/extra/sellcey/test-glibc/install/sbin/ldconfig: Can't open cache file
/extra/sellcey/test-glibc/install/etc/ld.so.cache
: No such file or directory

The directory /extra/sellcey/test-glibc/install/etc exists but
there is no ld.so.cache in it.  This may be related to how I build the
toolchain, I hack the GCC sources to change where it finds the dynamic
linker and I also hack library scripts like libc.so and libm.so (when
libmvec is built) after they are built to use the prefix I specify when
building.  Maybe I need some option when building glibc and/or some hack
to change where ld.so.cache is located?  I am using --prefix when building glibc.
I know the documentation says to always use --prefix=/usr but I am building
a native toolchain that looks for all of its libraries, headers, etc.
under a non-standard location when compiling/linking and when running and for
that I need to use --prefix=/something-other-than-usr when building glibc.

Steve Ellcey

Re: Aarch64 test results and questions

[Florian Weimer]

* Steve Ellcey:

> On Fri, 2019-07-26 at 18:52 +0200, Florian Weimer wrote:
>
>> > tst-ldconfig-bad-aux-cache also does not seem to have any known
>> > PR's. 
>> > I get the following in the .out file:
>> > 
>> > 	tst-ldconfig-bad-aux-cache.c:84: numeric comparison failure
>> > 	   left: 256 (0x100); from: status
>> > 	  right: 0 (0x0); from: 0
>> > 	error: support-xstat.c:29: stat64 ("/var/cache/ldconfig/aux-
>> > cache"): No such file or directory
>> > 	error: 2 test failures
>> > 	running post-clean rsync
>> 
>> What's the path that ldconfig is actually creating on this target?
>
> I built a toolchain (binutils, gcc, glibc) that are all in
> /extra/sellcey/test-glibc/install.
>
>
> % /extra/sellcey/test-glibc/install/sbin/ldconfig -p
> /extra/sellcey/test-glibc/install/sbin/ldconfig: Can't open cache file
> /extra/sellcey/test-glibc/install/etc/ld.so.cache
> : No such file or directory

I meant the aux-cache file.

> The directory /extra/sellcey/test-glibc/install/etc exists but
> there is no ld.so.cache in it.  This may be related to how I build the
> toolchain, I hack the GCC sources to change where it finds the dynamic
> linker and I also hack library scripts like libc.so and libm.so (when
> libmvec is built) after they are built to use the prefix I specify when
> building.  Maybe I need some option when building glibc and/or some hack
> to change where ld.so.cache is located?  I am using --prefix when building glibc.
> I know the documentation says to always use --prefix=/usr but I am building
> a native toolchain that looks for all of its libraries, headers, etc.
> under a non-standard location when compiling/linking and when running and for
> that I need to use --prefix=/something-other-than-usr when building glibc.

I think we can tweak the test so that it works with a non-standard
--prefix, too.

The test currently hard-codes /var:

  /* Create the needed directories. */
  xmkdirp ("/var/cache/ldconfig", 0777);

We can tweak this to something that suits your needs.  It will likely
require updating support/support_paths.c, though.  If localestatedir is
relevant to this, it will need capturing and exporting.  I'll gladly
help with patch review.

Thanks,
Florian

Re: Aarch64 test results and questions

[Steve Ellcey]

On Fri, 2019-07-26 at 19:37 +0200, Florian Weimer wrote:

> > 
> > % /extra/sellcey/test-glibc/install/sbin/ldconfig -p
> > /extra/sellcey/test-glibc/install/sbin/ldconfig: Can't open cache
> > file
> > /extra/sellcey/test-glibc/install/etc/ld.so.cache
> > : No such file or directory
> 
> I meant the aux-cache file.

I don't seem to have aux-cache in my toolchain.  I see
/var/cache/ldconfig/aux-cache on my system but in
the /extra/sellcey/test-glibc/install directory where
I installed gcc, binutils, and glibc there is no
such file.  install/var exists but there is no
ldconfig directory under that directory.

Steve Ellcey
sellcey@marvell.com

Re: Aarch64 test results and questions

[Szabolcs Nagy]

On 26/07/2019 17:43, Steve Ellcey wrote:
> tst-nss-files-hosts-long may be PR 24816, I am not running in a docker,
> I am on a native system.  The failure is:
> 
> 	error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
> 	error: 1 test failures

will be fixed once
https://sourceware.org/ml/libc-alpha/2019-07/msg00564.html
is committed.

i hope it will be committed before the release.

Re: Aarch64 test results and questions

[Carlos O'Donell]

On 7/29/19 4:18 AM, Szabolcs Nagy wrote:
> On 26/07/2019 17:43, Steve Ellcey wrote:
>> tst-nss-files-hosts-long may be PR 24816, I am not running in a docker,
>> I am on a native system.  The failure is:
>>
>> 	error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
>> 	error: 1 test failures
> 
> will be fixed once
> https://sourceware.org/ml/libc-alpha/2019-07/msg00564.html
> is committed.
> 
> i hope it will be committed before the release.
> 

Should be fixed now. Please re-test.

-- 
Cheers,
Carlos.

Re: Aarch64 test results and questions

[Szabolcs Nagy]

On 29/07/2019 14:42, Carlos O'Donell wrote:
> On 7/29/19 4:18 AM, Szabolcs Nagy wrote:
>> On 26/07/2019 17:43, Steve Ellcey wrote:
>>> tst-nss-files-hosts-long may be PR 24816, I am not running in a docker,
>>> I am on a native system.  The failure is:
>>>
>>>     error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
>>>     error: 1 test failures
>>
>> will be fixed once
>> https://sourceware.org/ml/libc-alpha/2019-07/msg00564.html
>> is committed.
>>
>> i hope it will be committed before the release.
>>
> 
> Should be fixed now. Please re-test.

the aarch64 buildbot is red because

FAIL: resolv/tst-resolv-search
FAIL: resolv/tst-resolv-trailing

both are timeouts (i haven't seen them before):

$ cat resolv/tst-resolv-search.out
Timed out: killed the child process
Termination time: 2019-07-29T15:09:55.891714
Last write to standard output: 2019-07-29T15:09:35.886436
$ cat resolv/tst-resolv-trailing.out
Timed out: killed the child process
Termination time: 2019-07-29T15:10:16.632413
Last write to standard output: 2019-07-29T15:09:56.626538

i'm not sure if these just intermittent failures or real,
but at least the nss/tst-nss-files-hosts-long failure is
gone now.

Re: Aarch64 test results and questions

[Carlos O'Donell]

On 7/29/19 11:20 AM, Szabolcs Nagy wrote:
> On 29/07/2019 14:42, Carlos O'Donell wrote:
>> On 7/29/19 4:18 AM, Szabolcs Nagy wrote:
>>> On 26/07/2019 17:43, Steve Ellcey wrote:
>>>> tst-nss-files-hosts-long may be PR 24816, I am not running in a docker,
>>>> I am on a native system.  The failure is:
>>>>
>>>>      error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
>>>>      error: 1 test failures
>>>
>>> will be fixed once
>>> https://sourceware.org/ml/libc-alpha/2019-07/msg00564.html
>>> is committed.
>>>
>>> i hope it will be committed before the release.
>>>
>>
>> Should be fixed now. Please re-test.
> 
> the aarch64 buildbot is red because
> 
> FAIL: resolv/tst-resolv-search
> FAIL: resolv/tst-resolv-trailing
> 
> both are timeouts (i haven't seen them before):
> 
> $ cat resolv/tst-resolv-search.out
> Timed out: killed the child process
> Termination time: 2019-07-29T15:09:55.891714
> Last write to standard output: 2019-07-29T15:09:35.886436
> $ cat resolv/tst-resolv-trailing.out
> Timed out: killed the child process
> Termination time: 2019-07-29T15:10:16.632413
> Last write to standard output: 2019-07-29T15:09:56.626538
> 
> i'm not sure if these just intermittent failures or real,
> but at least the nss/tst-nss-files-hosts-long failure is
> gone now.

Florian,

Any idea?

-- 
Cheers,
Carlos.

Re: Aarch64 test results and questions

[Florian Weimer]

* Carlos O'Donell:

> On 7/29/19 11:20 AM, Szabolcs Nagy wrote:
>> On 29/07/2019 14:42, Carlos O'Donell wrote:
>>> On 7/29/19 4:18 AM, Szabolcs Nagy wrote:
>>>> On 26/07/2019 17:43, Steve Ellcey wrote:
>>>>> tst-nss-files-hosts-long may be PR 24816, I am not running in a docker,
>>>>> I am on a native system.  The failure is:
>>>>>
>>>>>      error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
>>>>>      error: 1 test failures
>>>>
>>>> will be fixed once
>>>> https://sourceware.org/ml/libc-alpha/2019-07/msg00564.html
>>>> is committed.
>>>>
>>>> i hope it will be committed before the release.
>>>>
>>>
>>> Should be fixed now. Please re-test.
>>
>> the aarch64 buildbot is red because
>>
>> FAIL: resolv/tst-resolv-search
>> FAIL: resolv/tst-resolv-trailing
>>
>> both are timeouts (i haven't seen them before):
>>
>> $ cat resolv/tst-resolv-search.out
>> Timed out: killed the child process
>> Termination time: 2019-07-29T15:09:55.891714
>> Last write to standard output: 2019-07-29T15:09:35.886436
>> $ cat resolv/tst-resolv-trailing.out
>> Timed out: killed the child process
>> Termination time: 2019-07-29T15:10:16.632413
>> Last write to standard output: 2019-07-29T15:09:56.626538
>>
>> i'm not sure if these just intermittent failures or real,
>> but at least the nss/tst-nss-files-hosts-long failure is
>> gone now.
>
> Florian,
>
> Any idea?

Some test failures may be due to iptables/nftables issues.  In the past,
support_enter_network_namespace in support/resolv_test.c worked around
that because there was no connection tracking in the new namespace.

Does the kernel log anything about connection tracking table overflow?

These tests do not print anything on success.  You can run them with
--verbose to get some logging.

Thanks,
Florian

Re: [EXT] Re: Aarch64 test results and questions

[Steve Ellcey]

On Mon, 2019-07-29 at 09:42 -0400, Carlos O'Donell wrote:
> External Email
> 
> -------------------------------------------------------------------
> ---
> On 7/29/19 4:18 AM, Szabolcs Nagy wrote:
> > On 26/07/2019 17:43, Steve Ellcey wrote:
> > > tst-nss-files-hosts-long may be PR 24816, I am not running in a
> > > docker,
> > > I am on a native system.  The failure is:
> > > 
> > > 	error: tst-nss-files-hosts-long.c:35: ahostsv4 failed
> > > 	error: 1 test failures
> > 
> > will be fixed once
> > https://sourceware.org/ml/libc-alpha/2019-07/msg00564.html
> > is committed.
> > 
> > i hope it will be committed before the release.
> > 
> 
> Should be fixed now. Please re-test.

I reran the tests on a different machine (ThunderX instead of
ThunderX2).  This test now passes but I got 5 failures:

FAIL: elf/check-abi-libc
FAIL: nptl/tst-mutex10
FAIL: nptl/tst-rwlock18
FAIL: nptl/tst-rwlock9
FAIL: nptl/tst-stack4

elf/check-abi-libc is PR 14664.

mutex10, rwlock18, and rwlock9 were timeouts.

stack4 is something that I often see but I don't recall
what the issue is.  tst-stack4.out is empty.

I still had to edit nptl/tst-pthread-getattr.c to add
Florian's patch since it is not checked in yet.

Steve Ellcey
sellcey@marvell.com

Re: [EXT] Re: Aarch64 test results and questions

[Szabolcs Nagy]

On 29/07/2019 18:44, Steve Ellcey wrote:
> stack4 is something that I often see but I don't recall
> what the issue is.  tst-stack4.out is empty.

it can be
https://sourceware.org/bugzilla/show_bug.cgi?id=19329

Re: Aarch64 test results and questions

[Szabolcs Nagy]

On 29/07/2019 17:18, Florian Weimer wrote:
> * Carlos O'Donell:
>> On 7/29/19 11:20 AM, Szabolcs Nagy wrote:
>>> On 29/07/2019 14:42, Carlos O'Donell wrote:
>>>> On 7/29/19 4:18 AM, Szabolcs Nagy wrote:
>>>
>>> the aarch64 buildbot is red because
>>>
>>> FAIL: resolv/tst-resolv-search
>>> FAIL: resolv/tst-resolv-trailing
>>>
>>> both are timeouts (i haven't seen them before):
>>>
>>> $ cat resolv/tst-resolv-search.out
>>> Timed out: killed the child process
>>> Termination time: 2019-07-29T15:09:55.891714
>>> Last write to standard output: 2019-07-29T15:09:35.886436
>>> $ cat resolv/tst-resolv-trailing.out
>>> Timed out: killed the child process
>>> Termination time: 2019-07-29T15:10:16.632413
>>> Last write to standard output: 2019-07-29T15:09:56.626538
>>>
>>> i'm not sure if these just intermittent failures or real,
>>> but at least the nss/tst-nss-files-hosts-long failure is
>>> gone now.
>>
>> Florian,
>>
>> Any idea?
> 
> Some test failures may be due to iptables/nftables issues.  In the past,
> support_enter_network_namespace in support/resolv_test.c worked around
> that because there was no connection tracking in the new namespace.
> 
> Does the kernel log anything about connection tracking table overflow?
> 
> These tests do not print anything on success.  You can run them with
> --verbose to get some logging.

i could not reproduce the failures, i did not see anything weird
in kernel logs.

the aarch64 buildbot is now green.