Debugging containerized glibc tests with gdb (a developer use case for outside-of-container debugging).

Debugging containerized glibc tests with gdb (a developer use case for outside-of-container debugging).

[Carlos O'Donell]

Gabriel, DJ,

I have a question at the very end of this long email, please
bear with me while I think you for your good work. I have a follow-on
question that is harder, but I'll leave that for the new year.

I'm working on a new test that verifies running localedef and writing
to default paths. This is only possible with root in a container.

I am debugging this test using the excellent infrastructure that we have
now wired up via:

- WAIT_FOR_DEBUGGER=1, which causes the container-side process
  to save it's own PID and required debugging commands to a gdb script
  and then wait for the debugger to attach using the self-generated script.

- debugglibc.sh's new -c option which does all the work of starting
  the test in the container and then running the tests gdb script.

This works swimmingly. It's really easy to debug a containerized test that
you're writing.

With one command I'm right into the process running in the container and
I can debug what's going on.

[carlos@athas glibc-gr-localedef]$ ./debugglibc.sh -c locale/tst-localedef-path-norm

Debugging glibc...
Build directory  : /home/carlos/build/glibc-gr-localedef/
Source directory : /mnt/ssd/carlos/src/glibc-gr-localedef
GLIBC Testcase   : locale/tst-localedef-path-norm
GDB Commands     : /home/carlos/build/glibc-gr-localedef/debugglibc.gdb
Env vars         : 

Waiting for debugger, test process is pid 32512
gdb -x locale/tst-localedef-path-norm.gdb
GNU gdb (GDB) Fedora 8.3-7.fc30
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
warning: Could not load shared library symbols for /home/carlos/build/glibc-gr-localedef/elf/ld.so.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable.  Connect to gdbserver inside the container.
__GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffcfe31e590, 
    rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:75
75	  return (INTERNAL_SYSCALL_ERROR_P (r, err)

My question:

Why did gdb say "Could not load shared library symbols for .../ld.so"?

We fed it all the information it needed AFAIK.

In debugglibc.sh we effectively ran this:
~~~
# Use testrun.sh to start the test case with WAIT_FOR_DEBUGGER=1, then
# automatically attach GDB to it.
WAIT_FOR_DEBUGGER=1 /home/carlos/build/glibc-gr-localedef/testrun.sh --tool=container ${TESTCASE} &
gdb -x ${TESTCASE}.gdb
~~~
In ${TESTCASE}.gdb we ran this:
~~~
set sysroot /home/carlos/build/glibc-gr-localedef/testroot.root
file
file locale/tst-localedef-path-norm
symbol-file locale/tst-localedef-path-norm
exec-file locale/tst-localedef-path-norm
attach 32512
set wait_for_debugger = 0
~~~

Did we miss anything?

Any thoughts?

In test-container.c we mount $srcdir and $objdir into the same absolute paths
inside the container, so ld.so should be accessible from the same absolute paths.
If we fail those mounts we immediately fail the test with FAIL_EXIT1.

Note: You can also try './debugglibc.sh -c elf/tst-ldconfig-bad-aux-cache' to look at
      similar behaviour from an existing test.

-- 
Cheers,
Carlos.

Re: Debugging containerized glibc tests with gdb (a developer use case for outside-of-container debugging).

[Gabriel F. T. Gomes]

Hi, Carlos,

I tried this on several systems (varying archs and distros), but I wasn't
able to reproduce it (with elf/tst-ldconfig-bad-aux-cache)...

On Fri, 20 Dec 2019, Carlos O'Donell wrote:

>GNU gdb (GDB) Fedora 8.3-7.fc30
> [...]
>This GDB was configured as "x86_64-redhat-linux-gnu".

... even on Fedora 30 for x86_64, so I'm wondering if there's anything in
your configure options that I should try and mimic (I already tried some
combinations of --enable-profile --enable-addons --enable-multi-arch
--enable-tunables --enable-stack-protector=all, but I only tried that with
the system compiler (gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)).

>warning: Could not load shared library symbols for /home/carlos/build/glibc-gr-localedef/elf/ld.so.

I never got this.

>warning: Unable to find dynamic linker breakpoint function.

Nor this.

>warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable.  Connect to gdbserver inside the container.

But this one, I got, too.

Re: Debugging containerized glibc tests with gdb (a developer use case for outside-of-container debugging).

[Carlos O'Donell]

On 12/26/19 3:21 PM, Gabriel F. T. Gomes wrote:
> Hi, Carlos,
> 
> I tried this on several systems (varying archs and distros), but I wasn't
> able to reproduce it (with elf/tst-ldconfig-bad-aux-cache)...
> 
> On Fri, 20 Dec 2019, Carlos O'Donell wrote:
> 
>> GNU gdb (GDB) Fedora 8.3-7.fc30
>> [...]
>> This GDB was configured as "x86_64-redhat-linux-gnu".
> 
> ... even on Fedora 30 for x86_64, so I'm wondering if there's anything in
> your configure options that I should try and mimic (I already tried some
> combinations of --enable-profile --enable-addons --enable-multi-arch
> --enable-tunables --enable-stack-protector=all, but I only tried that with
> the system compiler (gcc (GCC) 9.2.1 20190827 (Red Hat 9.2.1-1)).
> 
>> warning: Could not load shared library symbols for /home/carlos/build/glibc-gr-localedef/elf/ld.so.
> 
> I never got this.

Did you try --enable-hardcoded-path-in-tests?

>> warning: Unable to find dynamic linker breakpoint function.
> 
> Nor this.
> 
>> warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable.  Connect to gdbserver inside the container.
> 
> But this one, I got, too.
> 


-- 
Cheers,
Carlos.

Re: Debugging containerized glibc tests with gdb (a developer use case for outside-of-container debugging).

[Gabriel F. T. Gomes]

Hi, Carlos,

On Fri, 27 Dec 2019, Carlos O'Donell wrote:
>
>Did you try --enable-hardcoded-path-in-tests?

I hadn't. ¬¬
That was it.

You wrote this in your first email:

>In test-container.c we mount $srcdir and $objdir into the same absolute paths
>inside the container, so ld.so should be accessible from the same absolute paths.
>If we fail those mounts we immediately fail the test with FAIL_EXIT1.

I think the paths are the same, but the contents are not identical.  On
the tests I did, the testroot.root/path/to/build/dir/elf directory is
missing the symlink between ld.so and ld<whatever>.so.<n>.  For instance:

  $ ls -l testroot.root/home/gabriel/build/x86_64/glibc/elf
  -rwxrwxr-x 1 gabriel gabriel 1342800 Dec 27 11:09 ld-linux-x86-64.so.2

  $ ls -l testroot.root/home/gabriel/build/powerpc64le/glibc/elf/
  -rwxr-xr-x 1 gabriel gabriel 1566232 Dec 27 09:05 ld64.so.2

Whereas on the actual build dir:

  $ ls -l elf/ld*.so*
  lrwxrwxrwx 1 gabriel gabriel       5 Dec 27 09:04 elf/ld64.so.2 -> ld.so
  -rwxr-xr-x 1 gabriel gabriel 1566232 Dec 27 09:04 elf/ld.so

>warning: Could not load shared library symbols for /home/carlos/build/glibc-gr-localedef/elf/ld.so.

If I create the ld.so symlink in the sysroot, the problem goes away.
Should there be a symlink in the sysroot?

Re: Debugging containerized glibc tests with gdb (a developer use case for outside-of-container debugging).

[Carlos O'Donell]

On 12/27/19 12:55 PM, Gabriel F. T. Gomes wrote:
> Hi, Carlos,
> 
> On Fri, 27 Dec 2019, Carlos O'Donell wrote:
>>
>> Did you try --enable-hardcoded-path-in-tests?
> 
> I hadn't. ¬¬
> That was it.

Interesting! Thank you for looking into this more deeply.

Reproducing these issues is half the battle!

> You wrote this in your first email:
> 
>> In test-container.c we mount $srcdir and $objdir into the same absolute paths
>> inside the container, so ld.so should be accessible from the same absolute paths.
>> If we fail those mounts we immediately fail the test with FAIL_EXIT1.
> 
> I think the paths are the same, but the contents are not identical.  On
> the tests I did, the testroot.root/path/to/build/dir/elf directory is
> missing the symlink between ld.so and ld<whatever>.so.<n>.  For instance:

Are you saying the contents of $objdir's mount in the sysroot are not the same
as the contents of $objdir as seen _outsite_ of the sysroot?

We have two paths:

- Normal $objdir
- $objdir as seen from within the sysroot's mount

 
>   $ ls -l testroot.root/home/gabriel/build/x86_64/glibc/elf
>   -rwxrwxr-x 1 gabriel gabriel 1342800 Dec 27 11:09 ld-linux-x86-64.so.2

A normal $objdir should have:

lrwxrwxrwx.  1 carlos carlos       5 Aug  1 00:51 ld-linux-x86-64.so.2 -> ld.so
-rwxrwxr-x.  1 carlos carlos 1458696 Aug  1 00:51 ld.so

>   $ ls -l testroot.root/home/gabriel/build/powerpc64le/glibc/elf/
>   -rwxr-xr-x 1 gabriel gabriel 1566232 Dec 27 09:05 ld64.so.2

Likewise.

> Whereas on the actual build dir:
 
>   $ ls -l elf/ld*.so*
>   lrwxrwxrwx 1 gabriel gabriel       5 Dec 27 09:04 elf/ld64.so.2 -> ld.so
>   -rwxr-xr-x 1 gabriel gabriel 1566232 Dec 27 09:04 elf/ld.so

Yes, this is correct.

>> warning: Could not load shared library symbols for /home/carlos/build/glibc-gr-localedef/elf/ld.so.
> 
> If I create the ld.so symlink in the sysroot, the problem goes away.
> Should there be a symlink in the sysroot?
 
Why would we need to create it? I thought this path was simply
a MS_BIND mount of the $objdir and should have identical contents?

-- 
Cheers,
Carlos.