about summary refs log tree commit homepage
path: root/t/spawn.t
DateCommit message (Collapse)
2024-04-03treewide: avoid getpid for more ownership checks
There are still some places where on_destroy isn't suitable, This gets rid of getpid() calls in most of those cases to reduce syscall costs and cleanup syscall trace output.
2024-01-30spawn: support some rlimit uses via Inline::C
BSD::Resource isn't packaged for Alpine (as of 3.19), but we also have optional Inline::C support and already rely on calling setrlimit(2) directly from the Inline::C version of pi_fork_exec.
2023-11-13spawn: don't append to scalarrefs on stdout/stderr
None of our current code relies on it, and I can't imagine it's something we'd need in the future, actually... This keeps the door open for relying more on Spawn in TestCommon.
2023-11-03replace ProcessIO with untied PublicInbox::IO
This fixes two major problems with the use of tie for filehandles: * no way to do fcntl, stat, etc. calls directly on the tied handle, forcing callers to use the `tied' perlop to access the underlying IO::Handle * needing separate classes to handle blocking and non-blocking I/O As a result, Git->cleanup_if_unlinked, InputPipe->consume, and Qspawn->_yield_start have fewer bizzare bits and we can call `$io->blocking(0)' directly instead of `(tied *$io)->{fh}->blocking(0)' Having a PublicInbox::IO class will also allow us to support custom read buffering which allows inspecting the current state.
2023-11-03treewide: use ->close to call ProcessIO->CLOSE
This will open the door for us to drop `tie' usage from ProcessIO completely in favor of OO method dispatch. While OO method dispatches (e.g. `$fh->close') are slower than normal subroutine calls, it hardly matters in this case since process teardown is a fairly rare operation and we continue to use `close($fh)' for Maildir writes.
2023-10-25spawn: support synchronous run_qx
This is similar to `backtick` but supports all our existing spawn functionality (chdir, env, rlimit, redirects, etc.). It also supports SCALAR ref redirects like run_script in our test suite for std{in,out,err}. We can probably use :utf8 by default for these redirects, even.
2023-10-08process_io: pass args to awaitpid as list
Specifying {cb_args} in the options hash felt awkward to me. Instead, just use the Perl stack like we do with awaitpid() and pass the list down directly.
2023-10-08rename ProcessPipe to ProcessIO
Since we deal with pipes (of either direction) and bidirectional stream sockets for this class, it's better to remove the `Pipe' from the name and replace it with `IO' to communicate that it works for any form of IO::Handle-like object tied to a process.
2023-10-01treewide: enable warnings in all exec-ed processes
While forked processes inherit from the parent, exec-ed processes need the `-w' flag passed to them. To determine whether or not we should pass them, we must check the `$^W' global perlvar, first. We'll also favor `perl -e' over `perl -E' in places where we don't rely on the latest features, since `-E' incurs slightly more startup time overhead from loading feature.pm (while `perl -Mv5.12' does not).
2023-08-29t/spawn.t: workaround OpenBSD RLIMIT_CPU delays
RLIMIT_CPU on OpenBSD doesn't work reliably with few syscalls or on mostly idle systems. Even at its most accurate, it takes an extra second to fire compared to FreeBSD or Linux due to internal accounting differences, but worst case even the SIGKILL can be 50s delayed. So rewrite the CPU burner script in Perl where we can unblock SIGXCPU and reliably use more syscalls. Link: https://marc.info/?i=20230829010110.M269767@dcvr
2023-01-30spawn_pp: use `which()' properly for pure-Perl spawn
I have no idea if mod_perl/mod_perl2 is used nowadays, but we're stuck supporting it as long as mod_perl exists. So add some tests and make minor updates to existing ones to ensure it stays working.
2023-01-18ds: introduce awaitpid, switch ProcessPipe users
awaitpid is the new API which will eventually replace dwaitpid. It enables early registration of callback handlers. Eventually (once dwaitpid is gone) it'll be able to use fewer waitpid calls. The avoidance of waitpid(-1) in our earlier days was driven by the belief that threads may eventually become relevant for Perl 5, but that's extremely unlikely at this stage. I will still introduce optional threads via C, but they definitely won't be spawning/reaping processes. Argument order to callbacks is swapped (PID first) to allow flattened multiple arguments more natrually. The previous API (allowing only a single argument, as influenced by pthread_create(3)) was more tedious as it involved packing multiple arguments into yet another array.
2022-06-11t/spawn: Find invalid PID to try to join its process group
In the container used to build packages of the GNU Guix distribution, PID 1 runs as the same user as the test so this spawn that should fail actually succeeds. Fix the problem by going through different PIDs and picking one that either doesn't exist or we aren't allowed to signal.
2021-02-24treewide: avoid "delete local" construct on hashes
Apparently this feature is only in Perl 5.12+, and we're still on Perl 5.10.
2021-02-07spawn: pi_fork_exec: support "pgid"
We'll be using this to allow the "git clone" process hierarchy to be killed via Ctrl-C. This also fixes a long-standing bug in error reporting for the Inline::C version, because we're actually testing for errors, now! n.b. strlen(3) is officially async-signal-safe as of POSIX.1-2016, but I can't think of a reason any previous implementation prior to that wouldn't be.
2021-02-05httpd/async: set O_NONBLOCK correctly
While Perl tie is nice for some things, getting IO::Handle->blocking to work transparently with it doesn't seem possible at the moment. Add some examples in t/spawn.t for future hackers. Fixes: 22e51bd9da476fa9 ("qspawn: switch to ProcessPipe via popen_rd")
2021-01-12cmd_ipc: send FDs with buffer payload
For another step in in syscall reduction, we'll support transferring 3 FDs and a buffer with a single sendmsg/recvmsg syscall using Socket::MsgHdr if available. Beyond script/lei itself, this will be used for internal IPC between search backends (perhaps with SOCK_SEQPACKET). There's a chance this could make it to the public-facing daemons, too. This adds an optional dependency on the Socket::MsgHdr package, available as libsocket-msghdr-perl on Debian-based distros (but not CentOS 7.x and FreeBSD 11.x, at least). Our Inline::C version in PublicInbox::Spawn remains the last choice for script/lei due to the high startup time, and IO::FDPass remains supported for non-Debian distros. Since the socket name prefix changes from 3 to 4, we'll also take this opportunity to make the argv+env buffer transfer less error-prone by relying on argc instead of designated delimiters.
2021-01-12ds: block signals when reaping
This lets us call dwaitpid long before a process exits and not have to wait around for it. This is advantageous for lei where we can run dwaitpid on the pager as soon as we spawn it, instead of waiting for a client socket to go away on DESTROY.
2021-01-03send and receive all 3 FDs at once
We'll always be transferring stdin, stdout, and stderr together for lei. Perhaps I lack imagination or foresight, but I can't think of a reason to send more or less FDs.
2021-01-03spawn: support send_fd+recv_fd w/o IO::FDPass
IO::FDPass may be an extra installation burden I don't want to impose on users. We only support Linux and *BSDs, however.
2021-01-02processpipe: allow synchronous close to set $?
To get rid of the ugly $PublicInbox::DS::in_loop localization in MboxReader, we'll distinguish between ->CLOSE and ->DESTROY with ProcessPipe. If we end up closing via ->DESTROY, we'll assume the caller will want to deal with $? asynchronously via the event loop (or not even care about $?). If we hit ->CLOSE directly, we'll assume the caller called close() and wants to check $? synchronously. Note: wantarray doesn't seem to propagate into tied methods, otherwise I'd be relying on that.
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-07-07t/spawn: fix test reliability
Since Perl doesn't internally use a self-pipe for sleep/select/poll/etc, wake up every 10ms to ensure it can see the SIGCHLD; since neither signalfd nor EVFILT_SIGNAL are always available. Fixes: 761baa2a300e4268 ("spawn: unblock SIGCHLD in subprocess")
2020-06-30spawn: unblock SIGCHLD in subprocess
Subprocess we spawn may want to use SIGCHLD for themselves. This also ensures we restore default signal handlers in the pure Perl version.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2019-12-30spawn: allow passing GLOB handles for redirects
We can save callers the trouble of {-hold} and {-dev_null} refs as well as the trouble of calling fileno().
2019-12-11spawn: remove support for clearing the env
It's unnecessary code which I'm not sure we ever used. In retrospect, completely clearing the environment doesn't make sense for the processes we spawn. We don't need to clobber individual environment variables in our code, either (and if we did for tests, we can use 'local').
2019-11-04index: "git log" failures are fatal
While I've never seen "git log" fail on its own, it could happen one day and we should be prepared to abort indexing when it happens. Beef up tests for t/spawn.t to ensure close() behaves on popen_rd the way we expect it to.
2019-09-09run update-copyrights from gnulib for 2019
2019-06-24spawn: remove `Blocking' flag handling
Instead, the O_NONBLOCK flag is set by PublicInbox::HTTPD::Async; and we won't be setting it elsewhere.
2019-04-04spawn: require soft and hard entries in RLIMIT_* handling
Our high-level config already treats single limits as a soft==hard limit for limiters; so stop handling that redundant in the low-level spawn() sub.
2019-04-04spawn: support RLIMIT_CPU, RLIMIT_DATA and RLIMIT_CORE
We'll be spawning cgit and git-diff, which can take gigantic amounts of CPU time and/or heap given the right (ermm... wrong) input. Limit the damage that large/expensive diffs can cause.
2018-02-07update copyrights for 2018
Using update-copyrights from gnulib While we're at it, use the SPDX identifier for AGPL-3.0+ to ease mechanical processing.
2016-06-18spawn: try to keep signals blocked in spawned child
While we only want to stop our daemons and gracefully destroy subprocesses, it is common for 'Ctrl-C' from a terminal to kill the entire pgroup. Killing an entire pgroup nukes subprocesses like git-upload-pack breaks graceful shutdown on long clones. Make a best effort to ensure git-upload-pack processes are not broken when somebody signals an entire process group. Followup-to: commit 37bf2db81bbbe114d7fc5a00e30d3d5a6fa74de5 ("doc: systemd examples should only kill one process")
2016-05-22t/spawn.t: additional tests for popen_rd
We need to ensure $? is set properly for users.
2016-02-27git: use built-in spawn implementation for vfork
This should reduce overhead of spawning git processes from our long-running httpd and nntpd servers.
2016-02-27initial spawn implementation using vfork
Under Linux, vfork maintains constant performance as parent process size increases. fork needs to prepare pages for copy-on-write, requiring a linear scan of the address space.