about summary refs log tree commit homepage
path: root/lib/PublicInbox/TestCommon.pm
DateCommit message (Collapse)
2020-09-14tests: consistently check for xapian-compact
We may need to test against development versions of Xapian, which may rely on setting `XAPIAN_COMPACT=xapian-compact-1.5'. Ensure it's possible to do that. And add a missing check in t/xcpdb-reshard.t, too.
2020-09-10use "\&" where possible when referring to subroutines
"*foo" is ambiguous in that it may refer to a bareword file handle; so we'll use it where we can without triggering warnings. PublicInbox::TestCommon::run_script_exit required dropping the prototype, however. We'll also future-proof by dropping "use warnings" in Cgit.pm and use the less-ambiguous "//=" in Inbox.pm while we're in the area.
2020-06-28testcommon: $ENV{TAIL} supports non-@ARGV redirects
Existing use of the $ENV{TAIL} relied on parsing --std{out,err}, which was only usable for read-only daemons. However, -watch doesn't use PublicInbox::Daemon code(*), so attempt to figure out redirects. (*) -watch won't able to run as a daemon in cases when git-credential prompts for IMAP/NNTP passwords. PublicInbox::Daemon is also designed for read-only parallelism where all worker processes are the same. Any subprocesses spawned by -watch are to do specific tasks for a particular set of inboxes.
2020-06-28config: support ->urlmatch method for -watch
Since we have IMAP client support in -watch; make sure per-URL settings are familiar to git users by taking advantage of git's URL matching abilities. This requires git 1.8.5+, which most users ought to have (though base CentOS 7 is on 1.8.3).
2020-06-28watch: wire up IMAP IDLE reapers to DS
We can avoid synchronous `waitpid(-1, 0)' and save a process when simultaneously watching Maildirs. One DS bug is fixed: ->Reset needs to clear the DS $in_loop flag in forked children so dwaitpid() fails and allows git processes to be reaped synchronously. TestCommon also calls DS->Reset when spawning new processes, since t/imapd.t uses DS->EventLoop while waiting on -watch to write.
2020-06-28watch: remove Filesys::Notify::Simple dependency
Since we already use inotify and EVFILT_VNODE (kqueue) in -imapd, we might as well use them directly in -watch, too. This will allow public-inbox-watch to use PublicInbox::DS for timers to watch newsgroups/mailboxes and have saner signal handling in future commits.
2020-06-23testcommon: DS->Reset when using fork-only subprocess
This fixes a bug on FreeBSD 11 here -nntpd + TEST_RUN_MODE=2 (default) was occasionally causing failures in t/v2writable.t due to the kqueue descriptor being auto-closed by the OS on fork.
2020-06-21testcommon: require IO::Socket::SSL >= 2.007
Net::NNTP does not support older IO::Socket::SSL. 1.94 on CentOS 7.x fails HTTPS and IMAPS tests, too. cf. https://rt.cpan.org/Ticket/Display.html?id=100529
2020-06-15testcommon: allow OR-ing module dependencies
IMAP requires either the Email::Address::XS or Mail::Address package (part of perl-MailTools RPM or libmailtools-perl deb); and Email::Address::XS is not officially packaged for some older distros, most notably CentOS 7.x.
2020-06-13testcommon: tcp_(server|connect): BAIL_OUT on failure
None of our tests rely on this failing, so just bail out if the system is out of resources.
2020-05-31testcommon: speed up wait_for_tail() on GNU/Linux
Somewhat recent versions of GNU tail(1) use inotify(7) on Linux; so don't penalize hackers using TAIL='tail -F' to run their tests with extra delays. Ironically, we still need to busy loop on /proc/$TAIL_PID/{fd,fdinfo} since inotify doesn't seem to support procfs.
2020-05-17confine Email::MIME use even further
To avoid confusing future readers and users, recommend PublicInbox::Eml in our Import POD and refer to PublicInbox::Eml comments at the top of PublicInbox::MIME. mime_load() confined to t/eml.t, since we won't be using it anywhere else in our tests.
2020-05-09remove most internal Email::MIME usage
We no longer load or use Email::MIME outside of comparison tests.
2020-05-09replace most uses of PublicInbox::MIME with Eml
PublicInbox::Eml has enough functionality to replace the Email::MIME-based PublicInbox::MIME.
2020-05-09eml: pure-Perl replacement for Email::MIME
Email::MIME eats memory, wastes time parsing out all the headers, and some problems can't be fixed without breaking compatibility for other projects which depend on it. Informal benchmarks show a ~2x improvement in general stats gathering scripts and ~10% improvement in HTML view rendering. We also don't need the ability to create MIME messages, just parse them and maybe drop an attachment. While this isn't the zero-copy or streaming MIME parser of my dreams; it's still an improvement in that it doesn't keep a scalar copy of the raw body around along with subparts. It also doesn't parse subparts up front, so it can also replace our uses of Email::Simple.
2020-04-26testcommon: mime_load: drop extra $cb arg
We don't need the callback arg, anymore.
2020-04-26tests: remove Email::MIME->create use entirely
Replace them with .eml files generated with the help of Email::MIME, but without some extraneous and unnecessary headers, and strip mime_load down to just loading files. This will give us more freedom to experiment with other mail libraries which may be more correct, better maintained, use less memory and/or be faster than Email::MIME.
2020-04-26testcommon: introduce mime_load sub
We'll use this to create, memoize, and reuse .eml files. This will be used to reduce (and eventually eliminate) our dependency on Email::MIME in tests.
2020-04-20testcommon: spawn-aware system() and qx[] workalikes
Barely noticeable on Linux, but this gives a 1-2% speedup on a FreeBSD 11.3 VM and lets us use built-in redirects rather than relying on /bin/sh.
2020-04-15testcommon: DESTROY: wait for killed daemon
Otherwise, the waitpid(-1, 0) call in Xapcmd::process_queue() may reap it in a subsequent test when using t/run.perl to reuse processes for testing. While we're at it, make Xapcmd::process_queue warn about unknown PIDs in case other PIDs leak through to us in the future.
2020-04-09tests: document run_mode=1 as not implemented
It was implemented at some point, but it was more things to support and the worst of both worlds: both unrealistic compared to real-world use and slower than run_mode=2. Noticed while looking for speling erorrs.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2020-01-11spawn (and thus popen_rd) die on failure
Most spawn and popen_rd callers die on failure to spawn, anyways, and some are missing checks entirely. This saves us a bunch of verbose error-checking code in callers. This also makes popen_rd more consistent, since it already dies on pipe creation failures.
2020-01-05view: msg_html: reduce memory use on reused MIDs
In rare cases where Message-IDs get reused, we do not want to hold onto the large Email::MIME objects in memory after showing the first message. So discard each message as soon as we're done using it so we can save memory for the next message. The new and expensive xt/mem-msgview.t test shows a nearly 14MB reduction for two ~7MB messages. run_script() also gets upgraded to make it easier to pass large inputs via IO GLOBs.
2020-01-04tests: fix running without SQLite or Xapian
PublicInbox::Search always loads DBD::SQLite, so we can't blindly "use" it in t/xcpdb-reshard.t. We also need to account for that in TestCommon.
2019-12-30spawn: allow passing GLOB handles for redirects
We can save callers the trouble of {-hold} and {-dev_null} refs as well as the trouble of calling fileno().
2019-12-24search: support SWIG-generated Xapian.pm
Xapian upstream is slowly phasing out the XS-based Search::Xapian in favor of the SWIG-generated "Xapian" package. While Debian and both FreeBSD have Search::Xapian, OpenBSD only includes the "Xapian" binding. More information about the status of the "Xapian" Perl module here: https://trac.xapian.org/ticket/523
2019-12-24testcommon: add require_mods method and use it
This cuts down on lines of code in individual test cases and fixes some misnamed error messages by using "$0" consistently. This will also provide us with a method of swapping out dependencies which provide equivalent functionality (e.g "Xapian" SWIG can replace "Search::Xapian" XS bindings).
2019-12-22testcommon: require_git: use "plan" from Test::More
require_git is no longer in the "::main" namespace, so we must call Test::More::plan() explicitly.
2019-12-20testcommon: fix run_script for older Perls
Using Perl "open" to dup(2) and save the old handles is required since "local *STDIN = *STDIN" does not work on old Perls. Even worse, this was silently a no-op when tested with Perl 5.24.1 on Debian 9.x and led to confusing failures in the t/httpd-corner.t lsof(1) tests when run after t/v2mirror.t from the same worker process using t/run.perl.
2019-12-19t/run.perl: to avoid repeated process spawning for *.t
Spawning a new Perl interpreter for every test case means Perl has to reparse and recompile every single file it needs, costing us performance and development time. Now that we've modified our code to avoid global state, we can preload everything we need. The new "check-run" test target is now 20-30% faster than the original "check" target.
2019-12-19tests: move t/common.perl to PublicInbox::TestCommon
We want to be able to use run_script with *.t files, so t/common.perl putting subs into the top-level "main" namespace won't work. Instead, make it a module which uses Exporter like other libraries.