about summary refs log tree commit homepage
DateCommit message (Collapse)
2019-11-27httpd|nntpd: avoid missed signal wakeups
Our attempt at using a self-pipe in signal handlers was ineffective, since pure Perl code execution is deferred and Perl doesn't use an internal self-pipe/eventfd. In retrospect, I actually prefer the simplicity of Perl in this regard... We can use sigprocmask() from Perl, so we can introduce signalfd(2) and EVFILT_SIGNAL support on Linux and *BSD-based systems, respectively. These OS primitives allow us to avoid a race where Perl checks for signals right before epoll_wait() or kevent() puts the process to sleep. The (few) systems nowadays without signalfd(2) or IO::KQueue will now see wakeups every second to avoid missed signals.
2019-11-27dskqxs: fix missing EV_DISPATCH define
Oops, IO::KQueue support was broken due to this missing constant. Add a new ds-kqxs.t test case to ensure we test the IO::KQueue path if IO::KQueue is available.
2019-11-27msgtime: deal with strange minutes in TZ offsets
I'm not sure if TZ minute offsets aside from '00' or '30' exist, but lets just deal with them properly when negative. Examples taken from various inboxes on lore.kernel.org. These are mostly message from spammers, but some are legitimate messages.
2019-11-27t/msgtime: add more checks for known cases
Broken email clients sent the darndest things, make sure we can still support them when we make Date::Parse optional.
2019-11-27t/msgtime: show date in test descriptions
Otherwise it's hard to figure what fails.
2019-11-24tests: move giant inbox/git dependent tests to xt/
xt/ is typically reserved for "eXtended tests" intended for the maintainers and not ordinary users. Since these require special configuration and do nothing by waste cycles during startup, they qualify.
2019-11-24t/perf-*.t: use $ENV{GIANT_INBOX_DIR} consistently
It's more consistent with our current terminology and "PI_DIR" is already used to override ~/.public-inbox/ (which holds "config" and possibly other files which affect all inboxes for a particular user, but is not an inbox itself); so stop advertising GIANT_PI_DIR in skip messages.
2019-11-24tests: quiet down commit graph
Newer versions of git enable the commit graph by default. Since we blow away our temporary directories every test, generating graphis is a waste and clutters stderr with "Computing commit graph generation numbers" messages.
2019-11-24tests: use File::Temp->newdir instead of tempdir()
We'll also introduce a tmpdir() API to give tempdirs consistent names.
2019-11-24xapcmd: replace Xtmpdirs with File::Temp->newdir
Since we're using Perl 5.10.1 and File::Temp 0.19+, we don't need Xtmpdirs at all for cleaning up tempdirs on failure and can just rely on the DESTROY handler provided by File::Temp.
2019-11-24t/nntpd-validate: get rid of threads dependency
Threads are officially discouraged by perl5-porters and proves problematic with my Perl installation when using run_mode=1 to speed up tests. So just use fork() and pipes to share results from Net::NNTP.
2019-11-24t/common: start_script replaces spawn_listener
We can shave several hundred milliseconds off tests which spawn daemons by preloading and avoiding startup time for common modules which are already loaded in the parent process. This also gives ENV{TAIL} support to all tests which support daemons which log to stdout/stderr.
2019-11-24daemon: avoid race when quitting workers
While the master process has a self-pipe to avoid missing signals, worker processes lack that aside from a pipe to detect master death. That pipe doesn't exist when there's no master process, so it's possible DS::close never finishes because it never woke up from epoll_wait. So create a pipe on the worker_quit signal and force it into epoll/kevent so it wakes up right away.
2019-11-24daemon: use sigprocmask when respawning workers
We need to block signals in workers during respawns until they're ready to receive signals.
2019-11-24daemon: use sigprocmask to block signals at startup
`$SIG{FOO} = "IGNORE"' will cause the daemon to miss signals entirely. Instead, we can use sigprocmask to block signal delivery until we have our signal handlers setup. This closes a race where a PID file can be written for an init script and a signal to be dropped via "IGNORE".
2019-11-24.gitignore: ignore local prove(1) files
As described in prove(1), .prove is storage for --state=save and .proverc allows per-worktree customizations.
2019-11-24t/nntpd-tls: sometimes SSL_connect succeeds quickly
It seems caching can happen within OpenSSL or negotiation can be delayed in some cases. In any case, don't barf on PublicInbox::TLS::epollbit() when connect_SSL succeeds unexpectedly.
2019-11-24t/httpd-corner: wait for worker process death
We need to ensure the worker process is terminated before starting a new connection, so leave a persistent HTTP/1.1 connection open and wait for the SIGKILL to take effect and drop the client.
2019-11-24t/xcpdb-reshard: test xcpdb --compact
We did not have a test for this, and need to guard against regressions when changing Xapcmd to use File::Temp->newdir in future commits.
2019-11-24t/v1-add-remove-add: quiet down "git init"
Use the "-q" flag like everywhere else.
2019-11-24tests: use strict everywhere
The "strict" pragma makes code easier to debug, and we had undeclared variables as a result in t/watch_maildir_v2.t. So use it everywhere to be consistent with the rest of our code.
2019-11-24tests: disable daemon workers in a few more places
There were still a few places where we used worker processes unnecessarily in tests, causing a small amount of unnecessary overhead. Followup-to: ad221e9b2852f6c5 ("t/*.t: disable nntpd/httpd worker processes in most tests")
2019-11-24mboxgz: fix compiler parse error under under Perl 5.16.3
Perl 5.16.3 (and possibly older versions) fails with the following errors (from CentOS7): Use of ?PATTERN? without explicit operator is deprecated Search pattern not terminated
2019-11-24check for File::Temp 0.19 for ->newdir method
This is distributed with Perl 5.10.1 and onwards, so it should not be an installation burden for any users. I'm planning to move away from tempdir() entirely and use File::Temp->newdir to remove dependencies on END{} blocks.
2019-11-16t/xcpdb-reshard: use run_script for -xcpdb
This more than doubles the speed of the test, since we make many invocations of -xcpdb.
2019-11-16t/indexlevels-mirror*: switch to run_script
This more than doubles the speed of these tests
2019-11-16t/v2mda: switch to run_script in many places
This more than doubles the speed of the test.
2019-11-16t/watch_filter_rubylang: run_script for -init and -index
This nets us a 20% speedup or so.
2019-11-16t/nntpd: use run_script for -init
This only gives a 5% speedup or so, but anything helps.
2019-11-16t/watch_maildir_v2: use run_script for -init
This only gives a small 10% speedup or so, but anything helps.
2019-11-16t/httpd: use run_script for -init
This only gives a small ~10% speedup, since -httpd still needs execve, but any speedup is welcome.
2019-11-16t/convert-compact: convert to run_script
While this didn't use IPC::Run, having to reload several Perl modules and scripts is slow and inefficient, so roughly double the speed of this test.
2019-11-16t/v2mirror: switch to default run_mode for speedup
We need to be careful and explicitly close FDs before doing -index, since we can't rely on FD_CLOEXEC without execve(2) syscalls.
2019-11-16doc: remove IPC::Run as a dev and test dependency
It's no longer needed and we're able to speed up some of our tests as a result.
2019-11-16t/mda_filter_rubylang: drop IPC::Run dependency
This test runs more than twice as fast, now.
2019-11-16t/mda: switch to run_script for testing
Another noticeable speedup, this test is roughly ~3x faster now.
2019-11-16t/v2mirror: get rid of IPC::Run dependency
Not taking advantage of faster run modes in run_script, yet since some lifetime problems need to be sorted.
2019-11-16t/purge: convert to run_script
This nets us another sizeable speedup.
2019-11-16t/init: convert to using run_script
This gives a 2-3x speedup on the test with the default run_mode=1.
2019-11-16t/edit: switch to use run_script
Perl parsing is slow, and run_script default behavior allows this to speed up t/edit.t by over 100% in my case.
2019-11-16t/common: introduce run_script wrapper for t/cgi.t
This will give us a consistent interface for running test scripts in more performant ways while still giving us a consistent interface to recreate real-world behavior via spawn() (fork + execve), if needed. The default run_mode (1) is faster and can run within the test process with some minor adjustments to our code to avoid global state. This avoids the significante overhead of Perl code loading, parsing and compilation phases.
2019-11-16spawn: which: allow embedded slash for relative path
This makes the subroutine behave more like which(1) command and will make using spawn() in tests easier.
2019-11-16xapcmd: do not fire END and DESTROY handlers in child
We need to bypass whatever Test::More does with END/DESTROY handlers for use in lon-lived process. This doesn't affect any of our normal code since we don't use END/DESTROY for Xapcmd and its callers.
2019-11-16import: only pass Inbox object to SearchIdx->new
SearchIdx->new no longer accepts a GIT_DIR path as its argument since commit 585314673236d664729fe3ab2d4fb229d1c0f2d5 ("searchidx: require PublicInbox::Inbox (or InboxWritable) ref")
2019-11-16inboxwritable: add ->cleanup method
We've been using this in -edit, and will be using it in some more scripts and tests to optimize for run_mode=2 with run_script. Keeping this in the *Writable modules since I don't see it being useful for the WWW and NNTP read-only interfaces which use PublicInbox::Inbox.
2019-11-16learn: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub.
2019-11-16mda: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub.
2019-11-16init: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub. We also need to rely on ->DESTROY instead of END{} to unlink the lock file on sub exit.
2019-11-16index: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub.
2019-11-16admin: get rid of singleton $CFG var
PublicInbox::Admin::config() just adds an extra layer of indirection which we barely rely on. So get rid of this global variable and make it easier to run tests in the future without relying on global state.