Date | Commit message (Collapse) |
|
There are still some places where on_destroy isn't suitable,
This gets rid of getpid() calls in most of those cases to
reduce syscall costs and cleanup syscall trace output.
|
|
BSD::Resource isn't packaged for Alpine (as of 3.19), but we
also have optional Inline::C support and already rely on calling
setrlimit(2) directly from the Inline::C version of pi_fork_exec.
|
|
None of our current code relies on it, and I can't imagine it's
something we'd need in the future, actually... This keeps the
door open for relying more on Spawn in TestCommon.
|
|
This fixes two major problems with the use of tie for filehandles:
* no way to do fcntl, stat, etc. calls directly on the tied handle,
forcing callers to use the `tied' perlop to access the underlying
IO::Handle
* needing separate classes to handle blocking and non-blocking I/O
As a result, Git->cleanup_if_unlinked, InputPipe->consume,
and Qspawn->_yield_start have fewer bizzare bits and we
can call `$io->blocking(0)' directly instead of
`(tied *$io)->{fh}->blocking(0)'
Having a PublicInbox::IO class will also allow us to support
custom read buffering which allows inspecting the current state.
|
|
This will open the door for us to drop `tie' usage from
ProcessIO completely in favor of OO method dispatch. While
OO method dispatches (e.g. `$fh->close') are slower than normal
subroutine calls, it hardly matters in this case since process
teardown is a fairly rare operation and we continue to use
`close($fh)' for Maildir writes.
|
|
This is similar to `backtick` but supports all our existing spawn
functionality (chdir, env, rlimit, redirects, etc.). It also
supports SCALAR ref redirects like run_script in our test suite
for std{in,out,err}.
We can probably use :utf8 by default for these redirects, even.
|
|
Specifying {cb_args} in the options hash felt awkward to me.
Instead, just use the Perl stack like we do with awaitpid()
and pass the list down directly.
|
|
Since we deal with pipes (of either direction) and bidirectional
stream sockets for this class, it's better to remove the `Pipe'
from the name and replace it with `IO' to communicate that it
works for any form of IO::Handle-like object tied to a process.
|
|
While forked processes inherit from the parent, exec-ed
processes need the `-w' flag passed to them. To determine
whether or not we should pass them, we must check the `$^W'
global perlvar, first.
We'll also favor `perl -e' over `perl -E' in places where
we don't rely on the latest features, since `-E' incurs
slightly more startup time overhead from loading feature.pm
(while `perl -Mv5.12' does not).
|
|
RLIMIT_CPU on OpenBSD doesn't work reliably with few syscalls or
on mostly idle systems. Even at its most accurate, it takes an
extra second to fire compared to FreeBSD or Linux due to
internal accounting differences, but worst case even the SIGKILL
can be 50s delayed.
So rewrite the CPU burner script in Perl where we can unblock
SIGXCPU and reliably use more syscalls.
Link: https://marc.info/?i=20230829010110.M269767@dcvr
|
|
I have no idea if mod_perl/mod_perl2 is used nowadays, but
we're stuck supporting it as long as mod_perl exists. So
add some tests and make minor updates to existing ones to
ensure it stays working.
|
|
awaitpid is the new API which will eventually replace dwaitpid.
It enables early registration of callback handlers. Eventually
(once dwaitpid is gone) it'll be able to use fewer waitpid
calls.
The avoidance of waitpid(-1) in our earlier days was driven by
the belief that threads may eventually become relevant for Perl 5,
but that's extremely unlikely at this stage. I will still
introduce optional threads via C, but they definitely won't be
spawning/reaping processes.
Argument order to callbacks is swapped (PID first) to allow
flattened multiple arguments more natrually. The previous API
(allowing only a single argument, as influenced by
pthread_create(3)) was more tedious as it involved packing
multiple arguments into yet another array.
|
|
In the container used to build packages of the GNU Guix distribution, PID 1
runs as the same user as the test so this spawn that should fail actually
succeeds.
Fix the problem by going through different PIDs and picking one that
either doesn't exist or we aren't allowed to signal.
|
|
Apparently this feature is only in Perl 5.12+, and we're
still on Perl 5.10.
|
|
We'll be using this to allow the "git clone" process hierarchy
to be killed via Ctrl-C. This also fixes a long-standing bug
in error reporting for the Inline::C version, because we're
actually testing for errors, now!
n.b. strlen(3) is officially async-signal-safe as of
POSIX.1-2016, but I can't think of a reason any previous
implementation prior to that wouldn't be.
|
|
While Perl tie is nice for some things, getting
IO::Handle->blocking to work transparently with it doesn't
seem possible at the moment.
Add some examples in t/spawn.t for future hackers.
Fixes: 22e51bd9da476fa9 ("qspawn: switch to ProcessPipe via popen_rd")
|
|
For another step in in syscall reduction, we'll support
transferring 3 FDs and a buffer with a single sendmsg/recvmsg
syscall using Socket::MsgHdr if available.
Beyond script/lei itself, this will be used for internal IPC
between search backends (perhaps with SOCK_SEQPACKET). There's
a chance this could make it to the public-facing daemons, too.
This adds an optional dependency on the Socket::MsgHdr package,
available as libsocket-msghdr-perl on Debian-based distros
(but not CentOS 7.x and FreeBSD 11.x, at least).
Our Inline::C version in PublicInbox::Spawn remains the last
choice for script/lei due to the high startup time, and
IO::FDPass remains supported for non-Debian distros.
Since the socket name prefix changes from 3 to 4, we'll also
take this opportunity to make the argv+env buffer transfer less
error-prone by relying on argc instead of designated delimiters.
|
|
This lets us call dwaitpid long before a process exits
and not have to wait around for it.
This is advantageous for lei where we can run dwaitpid on the
pager as soon as we spawn it, instead of waiting for a client
socket to go away on DESTROY.
|
|
We'll always be transferring stdin, stdout, and stderr together
for lei. Perhaps I lack imagination or foresight, but I can't
think of a reason to send more or less FDs.
|
|
IO::FDPass may be an extra installation burden I don't want to
impose on users. We only support Linux and *BSDs, however.
|
|
To get rid of the ugly $PublicInbox::DS::in_loop localization
in MboxReader, we'll distinguish between ->CLOSE and ->DESTROY
with ProcessPipe.
If we end up closing via ->DESTROY, we'll assume the caller will
want to deal with $? asynchronously via the event loop (or not
even care about $?).
If we hit ->CLOSE directly, we'll assume the caller called
close() and wants to check $? synchronously.
Note: wantarray doesn't seem to propagate into tied methods,
otherwise I'd be relying on that.
|
|
Using "make update-copyrights" after setting GNULIB_PATH in my
config.mak
|
|
Since Perl doesn't internally use a self-pipe for
sleep/select/poll/etc, wake up every 10ms to ensure
it can see the SIGCHLD; since neither signalfd nor EVFILT_SIGNAL
are always available.
Fixes: 761baa2a300e4268 ("spawn: unblock SIGCHLD in subprocess")
|
|
Subprocess we spawn may want to use SIGCHLD for themselves.
This also ensures we restore default signal handlers
in the pure Perl version.
|
|
I didn't wait until September to do it, this year!
|
|
We can save callers the trouble of {-hold} and {-dev_null}
refs as well as the trouble of calling fileno().
|
|
It's unnecessary code which I'm not sure we ever used. In
retrospect, completely clearing the environment doesn't make
sense for the processes we spawn. We don't need to clobber
individual environment variables in our code, either
(and if we did for tests, we can use 'local').
|
|
While I've never seen "git log" fail on its own, it could happen
one day and we should be prepared to abort indexing when it
happens.
Beef up tests for t/spawn.t to ensure close() behaves
on popen_rd the way we expect it to.
|
|
|
|
Instead, the O_NONBLOCK flag is set by PublicInbox::HTTPD::Async;
and we won't be setting it elsewhere.
|
|
Our high-level config already treats single limits as a
soft==hard limit for limiters; so stop handling that redundant
in the low-level spawn() sub.
|
|
We'll be spawning cgit and git-diff, which can take gigantic
amounts of CPU time and/or heap given the right (ermm... wrong)
input. Limit the damage that large/expensive diffs can cause.
|
|
Using update-copyrights from gnulib
While we're at it, use the SPDX identifier for AGPL-3.0+ to
ease mechanical processing.
|
|
While we only want to stop our daemons and gracefully destroy
subprocesses, it is common for 'Ctrl-C' from a terminal to kill
the entire pgroup.
Killing an entire pgroup nukes subprocesses like git-upload-pack
breaks graceful shutdown on long clones. Make a best effort to
ensure git-upload-pack processes are not broken when somebody
signals an entire process group.
Followup-to: commit 37bf2db81bbbe114d7fc5a00e30d3d5a6fa74de5
("doc: systemd examples should only kill one process")
|
|
We need to ensure $? is set properly for users.
|
|
This should reduce overhead of spawning git processes
from our long-running httpd and nntpd servers.
|
|
Under Linux, vfork maintains constant performance as
parent process size increases. fork needs to prepare pages
for copy-on-write, requiring a linear scan of the address
space.
|