Debugging containerized glibc tests with gdb (a developer use case for outside-of-container debugging).

[Carlos O'Donell <carlos@redhat.com>]

Gabriel, DJ,

I have a question at the very end of this long email, please
bear with me while I think you for your good work. I have a follow-on
question that is harder, but I'll leave that for the new year.

I'm working on a new test that verifies running localedef and writing
to default paths. This is only possible with root in a container.

I am debugging this test using the excellent infrastructure that we have
now wired up via:

- WAIT_FOR_DEBUGGER=1, which causes the container-side process
  to save it's own PID and required debugging commands to a gdb script
  and then wait for the debugger to attach using the self-generated script.

- debugglibc.sh's new -c option which does all the work of starting
  the test in the container and then running the tests gdb script.

This works swimmingly. It's really easy to debug a containerized test that
you're writing.

With one command I'm right into the process running in the container and
I can debug what's going on.

[carlos@athas glibc-gr-localedef]$ ./debugglibc.sh -c locale/tst-localedef-path-norm

Debugging glibc...
Build directory  : /home/carlos/build/glibc-gr-localedef/
Source directory : /mnt/ssd/carlos/src/glibc-gr-localedef
GLIBC Testcase   : locale/tst-localedef-path-norm
GDB Commands     : /home/carlos/build/glibc-gr-localedef/debugglibc.gdb
Env vars         : 

Waiting for debugger, test process is pid 32512
gdb -x locale/tst-localedef-path-norm.gdb
GNU gdb (GDB) Fedora 8.3-7.fc30
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
warning: Could not load shared library symbols for /home/carlos/build/glibc-gr-localedef/elf/ld.so.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
warning: Target and debugger are in different PID namespaces; thread lists and other data are likely unreliable.  Connect to gdbserver inside the container.
__GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffcfe31e590, 
    rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:75
75	  return (INTERNAL_SYSCALL_ERROR_P (r, err)

My question:

Why did gdb say "Could not load shared library symbols for .../ld.so"?

We fed it all the information it needed AFAIK.

In debugglibc.sh we effectively ran this:
~~~
# Use testrun.sh to start the test case with WAIT_FOR_DEBUGGER=1, then
# automatically attach GDB to it.
WAIT_FOR_DEBUGGER=1 /home/carlos/build/glibc-gr-localedef/testrun.sh --tool=container ${TESTCASE} &
gdb -x ${TESTCASE}.gdb
~~~
In ${TESTCASE}.gdb we ran this:
~~~
set sysroot /home/carlos/build/glibc-gr-localedef/testroot.root
file
file locale/tst-localedef-path-norm
symbol-file locale/tst-localedef-path-norm
exec-file locale/tst-localedef-path-norm
attach 32512
set wait_for_debugger = 0
~~~

Did we miss anything?

Any thoughts?

In test-container.c we mount $srcdir and $objdir into the same absolute paths
inside the container, so ld.so should be accessible from the same absolute paths.
If we fail those mounts we immediately fail the test with FAIL_EXIT1.

Note: You can also try './debugglibc.sh -c elf/tst-ldconfig-bad-aux-cache' to look at
      similar behaviour from an existing test.

-- 
Cheers,
Carlos.