about summary refs log tree commit homepage
path: root/lib/PublicInbox/Daemon.pm
DateCommit message (Collapse)
2020-01-28daemon: provide TCP_DEFER_ACCEPT for Perl <5.14
Socket::TCP_DEFER_ACCEPT() did not appear in the Socket module distributed with Perl until 5.14, despite it being available since Linux 2.4.
2020-01-08daemon: ignore SIGPIPE while running
This is only needed for IO::Poll users, since users with (signalfd || EVFILT_SIGNAL) support run with SIGPIPE (and all other signals) blocked. Fixes: 81a9a43fb858d197 ("daemon: use sigprocmask to block signals at startup")
2020-01-06treewide: "require" + "use" cleanup and docs
There's a bunch of leftover "require" and "use" statements we no longer need and can get rid of, along with some excessive imports via "use". IO::Handle usage isn't always obvious, so add comments describing why a package loads it. Along the same lines, document the tmpdir support as the reason we depend on File::Temp 0.19, even though every Perl 5.10.1+ user has it. While we're at it, favor "use" over "require", since it it gives us extra compile-time checking.
2019-12-16daemon: drop listeners early in master on graceful shutdown
For users not relying on socket activation via systemd (or similar), we want to drop listeners ASAP so another process can bind to their address. While we're at it, disable TTIN and HUP handlers since we have no chance of starting usable workers without listeners.
2019-12-16daemon: shorten lifetime of listener_names mapping
Keeping a ref to the IO::Socket handle was preventing close(2) from being invoked on graceful shutdown of worker.
2019-12-14daemon: use DESTROY for unlinking --pid-file
This gets rid of the last "END{}" block in our code and cleans up a (temporary) circular reference. Furthermore, ensure the cleanup code still works in all configurations by adding tests and testing both the -W1 (default, 1 worker) and -W0 (no workers) code paths.
2019-12-14ds: move EvCleanup code into DS
EvCleanup only existed since Danga::Socket was a separate component, and cleanup code belongs with the event loop.
2019-11-27httpd|nntpd: avoid missed signal wakeups
Our attempt at using a self-pipe in signal handlers was ineffective, since pure Perl code execution is deferred and Perl doesn't use an internal self-pipe/eventfd. In retrospect, I actually prefer the simplicity of Perl in this regard... We can use sigprocmask() from Perl, so we can introduce signalfd(2) and EVFILT_SIGNAL support on Linux and *BSD-based systems, respectively. These OS primitives allow us to avoid a race where Perl checks for signals right before epoll_wait() or kevent() puts the process to sleep. The (few) systems nowadays without signalfd(2) or IO::KQueue will now see wakeups every second to avoid missed signals.
2019-11-24daemon: avoid race when quitting workers
While the master process has a self-pipe to avoid missing signals, worker processes lack that aside from a pipe to detect master death. That pipe doesn't exist when there's no master process, so it's possible DS::close never finishes because it never woke up from epoll_wait. So create a pipe on the worker_quit signal and force it into epoll/kevent so it wakes up right away.
2019-11-24daemon: use sigprocmask when respawning workers
We need to block signals in workers during respawns until they're ready to receive signals.
2019-11-24daemon: use sigprocmask to block signals at startup
`$SIG{FOO} = "IGNORE"' will cause the daemon to miss signals entirely. Instead, we can use sigprocmask to block signal delivery until we have our signal handlers setup. This closes a race where a PID file can be written for an init script and a signal to be dropped via "IGNORE".
2019-09-09run update-copyrights from gnulib for 2019
2019-07-08daemon: use POSIX and WNOHANG more idiomatically
No point in uglifying our code since we need the POSIX module in many places, anyways.
2019-06-30daemon: warn on inheriting blocking listeners
For users relying on socket activation via service manager (e.g. systemd) and running multiple service instances (@1, @2), we need to ensure configuration of the socket is NonBlocking. Otherwise, service managers such as systemd may clear the O_NONBLOCK flag for a small window where accept/accept4 blocks: public-inbox-nntpd@1 |systemd |public-inbox-nntpd@2 --------------------------+----------------+-------------------- F_SETFL,O_NONBLOCK|O_RDWR | | (not running, yet) |F_SETFL, O_RDWR | |fork+exec @2... | accept(...) # blocks! | |(started by systemd) | |F_SETFL,O_NONBLOCK|O_RDWR | |accept(...) non-blocking It's a very small window where O_NONBLOCK can be cleared, but it exists, and I finally hit it after many years.
2019-06-29parentpipe: make the ->close call more obvious
We can close directly in event_step without bad side effects, and then we also don't need to take a reason arg from worker_quit, since we weren't logging it anywhere.
2019-06-24daemon: use FreeBSD accept filters on non-NNTP
Similar to TCP_DEFER_ACCEPT on Linux, FreeBSD has a 'dataready' accept filter which we can use to reduce wakeups when doing TLS negotiation or plain HTTP. There's also a 'httpready' which we can use for plain HTTP connections.
2019-06-24daemon: set TCP_DEFER_ACCEPT on everything but NNTP
This Linux-specific option can save us some wakeups during the TLS negotiation phase, and it can help with ordinary HTTP, too. Plain NNTP (and in the future, POP3) are the only things which require the server send messages, first.
2019-06-24daemon: use SSL_MODE_RELEASE_BUFFERS
34K per idle connection adds up to large amounts of memory; especially with the speed of malloc nowadays compared to the cost of cache misses or worse, swapping.
2019-06-24daemon: map inherited sockets to well-known schemes
I don't want to specify "--listen" in my systemd .service files, so map 563 to NNTPS automatically (and 443 to HTTPS, but HTTPS support doesn't work, yet).
2019-06-24nntp: NNTPS and NNTP+STARTTLS working
It kinda, barely works, and I'm most happy I got it working without any modifications to the main NNTP::event_step callback thanks to the DS->write(CODE) support we inherited from Danga::Socket.
2019-06-24ds: use and export monotonic now()
All of our internal timing code should use monotonic clocks for consistency against system clock adjustments. This can be shared by our Daemon and NNTP packages.
2019-05-15remove hard Devel::Peek dependency and lazy load for daemons
It's only useful for a corner case in long-running daemons when an admin decides to compact or vacuum a Xapian or SQLite DB. As a result, other scripts should run slightly faster. For instance, this saves about 80ms (2.710s => 2.630s) in t/mda.t on my remote workstation. While we're at it, make sure EvCleanup is properly require'd in Daemon.pm and HTTP.pm and document our use of Devel::Peek.
2019-05-04bundle Danga::Socket and Sys::Syscall
These modules are unmaintained upstream at the moment, but I'll be able to help with the intended maintainer once/if CPAN ownership is transferred. OTOH, we've been waiting for that transfer for several years, now... Changes I intend to make: * EPOLLEXCLUSIVE for Linux * remove unused fields wasting memory * kqueue bugfixes e.g. https://rt.cpan.org/Ticket/Display.html?id=116615 * accept4 support And some lower priority experiments: * switch to EV_ONESHOT / EPOLLONESHOT (incompatible changes) * nginx-style buffering to tmpfile instead of string array * sendfile off tmpfile buffers * io_uring maybe?
2019-02-07daemon: favor Socket.pm in Perl 5.14+ for IPv6
Users on Perl 5.14+ are common, so we can try the bundled Socket (not "Socket6") module before attempting Socket6 for IPv6.
2019-02-07daemon: die properly when Socket6 is missing
We need to keep Unix-socket-only httpd instances working without Socket6. This fixes t/httpd-unix.t with Socket6 uninstalled.
2019-01-10daemon: make Socket6 optional, note about Net::Socket::IP
It looks like Net::Socket::IP comes with Perl 5.20 and later; so we won't have to hassle users with another package to install.
2019-01-05shrink low-bandwidth pipes under Linux
I've hit /proc/sys/fs/pipe-user-pages-* limits on some systems. So stop hogging resources on pipes which don't benefit from giant sizes. Some of these can use eventfd in the future to further reduce resource use.
2018-03-22fix syntax warnings
I keep forgetting to run "make syntax"
2018-03-02evcleanup: disable outside of daemon
We'll be using these in a more OO manner for V2Writable (which doesn't use Danga::Socket), so lets not unnecessarily register cleanup handlers intended for network daemons.
2018-02-07update copyrights for 2018
Using update-copyrights from gnulib While we're at it, use the SPDX identifier for AGPL-3.0+ to ease mechanical processing.
2016-12-12daemon: set $now time for NNTP shutdown
commit 6e238ee3396719e578d6a90e177a71ce9f8c1ca0 ("nntp: respect 3 minute idle time for shutdown") was incomplete, and needed this change to Daemon to be effective. In the future, there will be more common code between NNTP.pm and HTTP.pm
2016-08-02daemon: do not chdir unless daemonizing
As far as most process managers are concerned (e.g. systemd), they should already start in '/'. So avoid making our daemon more complex to run by requiring absolute paths during development.
2016-07-29daemon: re-enable SIGWINCH without setsid
This allows systemd users to use SIGWINCH to temporarily (and gracefully) stop an instance of a service without doing a code reload to bring it back up: # start temporary new service code systemctl start public-inbox-nntpd@2.service # momentarily paralyze original service systemctl kill -s WINCH public-inbox-nntpd@1.service if new_code_at_2_sucks then # restart original workers systemctl kill -s HUP public-inbox-nntpd@1.service else # new is better than old, replace original instance systemctl restart public-inbox-nntpd@1.service fi # cleanup the temporary service systemctl stop public-inbox-nntpd@2.service This partially reverts commit 73d274e83b7d300f31e0cc1ceeacbf73c6c2a1e4 ("daemon: disable SIGWINCH unless explicitly daemonized")
2016-07-06daemon: disable USR2/TTIN/TTOU/WINCH in workers
If using a master/worker setup, a careless user could be trying to signal all processes using "killall". This may trigger bad side-effects; but try to limit the side-effects as much as possible.
2016-06-21daemon: disable SIGWINCH unless explicitly daemonized
Checking stdin/stdout/stderr is not sufficient as the daemon without setsid can still be under the control of a terminal. Unfortunately this means systemd users cannot use SIGWINCH, either.
2016-06-18daemon: be less misleading about graceful shutdown
We do not need to count the httpd.async object against our running client count, that is tied to the socket of the actual client. This prevents misleading sysadmins about connected clients during shutdown.
2016-06-11daemon: reset unused signal handlers to default in child
They're effectively noops anyways, and we don't want to be holding a reference to the read end of the parent pipe.
2016-05-23daemon: ignore SIGWINCH when connected to terminal
Users may change terminal sizes if the process is connected to a terminal, so we can't reasonably expect SIGWINCH to work as intended.
2016-05-21daemon: simplify forking
We shouldn't need sigprocmask unless we're running multiple native threads or using vfork, neither of which is the case, here.
2016-05-21localize $/ in more places to avoid potential problems
This hopefully makes the intent of the code clearer, too. The the HTTP use of the numeric reference for getline caused problems in Git.pm, already.
2016-04-30daemon: graceful shutdown warning and limit removal
git clones may take longer than 30s, much longer... So prepare to wait almost indefinitely for sockets to timeout and document the second signal behavior for immediate shutdown. While we're at it, move parent death handling to a separate class to avoid Danga::Socket->AddOtherFds, since that does not allow proper handling the parent pipe being closed and would actually misterminate a worker prematurely. t/nntpd.t is update to illustrate the failure with workers enabled. We will work to keep memory usage low and let clients take their time without interrupting them.
2016-03-17daemon: expand @ARGV paths for running in '/'
We also require --stdout/--stderr/--pid-file to be absolute paths for USR2 usage. However, allow PSGI files for -httpd to be relative paths for ease-of-use.
2016-03-11daemon: fixup usage of the '-l' switch with IP/INET6 sockets
We need to ensure $sock_pkg is preserved outside of the loop. The variable passed to "for" or "foreach" is implicitly local and restores the previous value when the loop exits. This is documented in the perlsyn manpage in the "Foreach Loops" section. Fixes: ea1b6cbd422b ("daemon: allow using IO::Socket::IP over INET6")
2016-03-08daemon: allow using IO::Socket::IP over INET6
IO::Socket::IP is bundled with newer versions of Perl, so it is more likely to be available. There should be no differences between these with our use cases.
2016-03-05daemon: sockname detects listeners correctly
This means we can avoid false-positives when inheriting multiple Unix domain sockets.
2016-03-05daemon: document optional Net::Server dependency
Non-socket activation users will want to install Net::Server for daemonization, pid file writing, and user/group switching.
2016-03-05daemon: simplify parent death handling
No need to create a new sub which kill ourselves $$ when we can invoke worker_quit directly.
2016-03-05daemon: avoid cyclic references for once-used callbacks
Not that these subs are repeatedly created, but this makes the code easier-to-review and these callbacks are idempotent anyways.
2016-03-05daemon: drop listener sockets ASAP on termination
We do not want to be accepting connections during graceful shutdown because another new process is likely taking over. This also allows us to free up the listener case another (independent) process wants to claim it.
2016-03-04daemon: simplify socket inheriting, slightly
IO::Handle->new_from_fd has existed since at least 1996, so it should be safe to depend on at this point.