OpenBSD debugging

["Štěpán Němec" <stepnem@smrk.net>]

I apologize for the late response.

On Mon, 23 Oct 2023 19:58:18 +0000
Eric Wong wrote:

> Thanks for the info.  Just curious, what HW specs (ncpus, RAM)
> is available on that system?  I wonder if that affects timing
> somehow...

dmesg:
https://dmesgd.nycbug.org/index.cgi?do=view&id=7357
ncpus = 2 (so it's running the MP (multiprocessor) kernel), 1GB RAM

>> (And again, if you want to have a look yourself, I'd be
>> happy to give you access to the machine; still the same
>> testing OC VM.)
>
> Unfortunately, my *BSD debugging knowledge is far behind my
> Linux; so I'm not sure how much help it'd be...

If nothing else, you could do some tests on an otherwise
idle machine with good Internet connectivity (unless the
connection issues you keep mentioning are mainly on your
end, that is).

> Some examples of things I miss on OpenBSD:
>
> * /proc/$PID/fdinfo/$FD_OF_EPOLL on Linux is immensely helpful
>   for knowing what and how epoll is watching target FDs.  I'm
>   not sure if there's a way to introspect kqueue like that

Yeah, most likely there isn't, though I'm not quite sure
what exactly "like that" entails.  Care to expand a bit upon
the immense usefulness mentioned, i.e., how this helps you
specifically?

OpenBSD fstat(1) prints the kqueue memory addresses, so I
suppose a sufficiently determined individual could get
arbitrary info from the running kernel based on that,
although at that point there are probably better ways to get
the address than running fstat...

As for existing tools I'm aware of, there's ddb(4) which can
dump structures etc. (it has access to kernel symbols), but
it's not very convenient for casual debugging/introspection,
as it stops everything until you continue from the kernel
debugger, so it will mess up the clock etc.

Then there's bt(5)/btrace(8), which is a bpftrace clone.
It's a work in progress and nowhere near Linux
feature-/coverage-wise, but when it works it's nice.

AFAIK the most you can currently get from it by default is
entry and return for syscalls and a couple dozen static
tracepoints.  It's possible to enable entry/return for all
kernel functions with a custom kernel (which, depending on
circumstances, isn't as bad as it sounds: compiling an
OpenBSD kernel from scratch is a matter of (tens of) minutes
even on a weak machine; it took about 40 minutes in the
above VM, single-threaded).

Unfortunately there's no support for arbitrary argument
access (though it seems to be on TODO), you need to add a
custom tracepoint for that (which can be easy enough,
e.g. <https://flak.tedunangst.com/post/probing-my-ssds-latency>,
but again requires a kernel compile).

> * Linux strace decodes more struct args info than kdump

strace is certainly more featureful, though in the specific
case of kqueue/kevent I think kdump does show everything one
would expect to see?

> * ability to control pathname of core dumps

Yeah, the way the OpenBSD knobs have evolved (i.e., sane
behavior attainable only for processes with altered U/GID)
seems pretty weird, and even though I suspect at least some
developers would be able to entertain the thought that the
situation isn't optimal (despite the way the recent misc@
thread you participated in turned out), I don't see a good
way to improve it without some redesign, i.e. breaking
backwards compatibility (adding further knobs on top of
kern.nosuidcoredump would make matters even messier IMO).

That said, if it's critical for your use case, I think we could
patch it locally in the VM easily enough, now that I've set it
up for kernel compilation anyway.

Same for any custom bt tracepoints or other adjustments I'd
be able to help with.

In summary, if you ever feel the VM could be of use, just
let me know; if you consider your resources better spent
elsewhere I certainly understand.

-- 
Štěpán