about summary refs log tree commit homepage
path: root/t/nntpd.t
DateCommit message (Collapse)
2021-09-13tests: add require_cmd, require curl when needed
t/v2mirror.t and t/lei-mirror.t are now skipped when curl is missing (instead of failing in appropriate places). A bunch of which() checks are updated to use require_cmd to avoid explicitly loading Spawn.
2021-03-25t/*: drop unnecessary v1-specific index calls
Outside of mirrors, we can -init with indexlevel to avoid calling -index explicitly.
2021-03-17tests: show lsof output on deleted-file-check failures
This may help track down some occasional test failures I'm seeing.
2021-03-15t/nntpd: use create_inbox
2021-02-24watch: switch IMAP and NNTP fetch loops to NetReader
NetReader::<imap|nntp>_each were based on the -watch code they now replace. v2: do not warn on EINTR if user quit to fix occasional test failure in t/imapd.t
2021-02-10tests: skip properly with git <2.6
Tested with git 1.8.3.1 on CentOS 7.x `plan skip_all => ...' doesn't work after some tests have run, we have to call skip() instead.
2021-02-08tests: favor IPv6
IPv4 gets plenty of real-world coverage, and apparently there's Debian buildd hosts which lack IPv4(*). So ensure everything can work on IPv6 and not cause problems for odd setups. (*) https://bugs.debian.org/979432
2021-01-01update copyrights for 2021
Using "make update-copyrights" after setting GNULIB_PATH in my config.mak
2020-09-14tests: consistently check for xapian-compact
We may need to test against development versions of Xapian, which may rely on setting `XAPIAN_COMPACT=xapian-compact-1.5'. Ensure it's possible to do that. And add a missing check in t/xcpdb-reshard.t, too.
2020-09-12t/nntpd: add test for the XPATH command
It's only in RFC 2980 (not 977 or 3977), but Net::NNTP has supported it since 2001, at least. We'll be making changes to avoid pathological behavior, so test it, first.
2020-09-10nntp: fix cross-newsgroup Message-ID lookups
We cannot blindly use the selected newsgroup for HEAD/ARTICLE/BODY requests using Message-ID, since those commands look across all newsgroups; not just the selected one (if any). So stuff a reference to the Inbox object into $smsg. We can reduce args passed into set_nntp_headers() and msg_hdr_write(), too. Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
2020-09-01rename WatchMaildir => Watch
This is no longer limited to Maildirs now that IMAP and NNTP support exist; so give it a shorter name.
2020-08-20init: support --newsgroup option
We can reduce the need to edit the config file for NNTP group names this way.
2020-08-03t/nntpd: do not fork on indexing, test v2
No need to waste resources when doing minimal work. With PI_TEST_VERSION=2, this fixes a test failure where Net::NNTP::DESTROY was getting called in the shard process. We'll also get rid of an unnecessary use_ok under v2, too.
2020-08-02nntp: fix STAT command
The return value of art_lookup changed but this command wasn't updated since it wasn't tested. Fixes: 0e6ceff37fc38f28 ("nntp: support slow blob retrievals")
2020-07-14over+msgmap: do not store filename after DBI->connect
SQLite already knows the filename internally, so avoid having it as a long-lived Perl SV to save some bytes when there's many inboxes and open DBs.
2020-07-14nntpd+imapd: detect unlinked msgmap
While it's even less common to experience a replaced msgmap.sqlite3 file, BOFHs may do the darndest things. This is another step towards reducing the number of needless wakeups we need to do in long-lived read-only daemons.
2020-07-02tests: add use/require statements for TEST_RUN_MODE=0
The default (and fast) TEST_RUN_MODE=2 preloads most modules, but TEST_RUN_MODE=0 is more realistic and can catch some problems which may show up in real-world use.
2020-06-28watch: add NNTP support
This is similar to IMAP support, but only supports polling. Automatic altid support is not yet supported, yet; but may be in the future. v2: small grammar fix by Kyle Meyer Link: https://public-inbox.org/meta/87sgeg5nxf.fsf@kyleam.com/
2020-06-21testcommon: require IO::Socket::SSL >= 2.007
Net::NNTP does not support older IO::Socket::SSL. 1.94 on CentOS 7.x fails HTTPS and IMAPS tests, too. cf. https://rt.cpan.org/Ticket/Display.html?id=100529
2020-06-21nntp: support slow blob retrievals
Having `git cat-file' as a separate process naturally lends itself to asynchronous dispatch. Our event loop for -nntpd no longer blocks on slow git storage. Pipelining in -imapd was tricky and bugs were exposed by mbsync(1). Update t/nntpd.t to support pipelining ARTICLE requests to ensure we don't have the same problems -imapd did during development.
2020-06-16imap: improve IDLE handling at graceful shutdown
Since IMAP IDLE users aren't expected to issue any commands, we can terminate their connections immediately on graceful shutdown. Furthermore, we need to drop the inotify FD from the epoll set to avoid warnings during global destruction. Embarassingly, this required fixing wacky test ordering from 2a717d13f10fcdc6 ("nntpd+imapd: detect replaced over.sqlite3")
2020-06-13nntpd+imapd: detect replaced over.sqlite3
For v1 inboxes (and possibly v2 in the future, for VACUUM), public-inbox-compact replaces over.sqlite3 with a new file. This currently doesn't need an extra inotify watch descriptor (or FD for kevent) at the moment, so it can coexist nicely for systems w/o IO::KQueue or Linux::Inotify2.
2020-06-13index: account for CRLF conversion when storing bytes
NNTP and IMAP both require CRLF conversions on the wire. They're also the only components which care about $smsg->{bytes}, so store the CRLF-adjusted value in over.sqlite3 and Xapian DBs.. This will allow us to optimize RFC822.SIZE fetch item in IMAP without triggering size mismatch errors in some clients' default configurations (e.g. Mail::IMAPClient), but not most others. It could also fix hypothetical problems with NNTP clients that report discrepancies between overview and article data.
2020-06-08index: v2: parallel by default
InboxWritable should only set $v2w->{parallel} if the $parallel flag is defined to 0 or 1. We want indexing a new inbox to utilize SMP, just like --reindex. -index once again allows -j0/--jobs=0 to force single-process use, and we'll be ensuring that works in tests to maintain performance on small systems. Fixes: 61a2fff5b34a3e32 ("admin: move index_inbox over")
2020-05-09remove most internal Email::MIME usage
We no longer load or use Email::MIME outside of comparison tests.
2020-04-22t/*.t: reduce dependency on Email::MIME APIs
Instead, favor PublicInbox::MIME->new for non-attachment emails. We may support alternatives to Email::MIME down the line. We'll still keep Email::MIME->create to deal with attachments, for now, but there's also a fair amount of test duplication we should eliminate, later.
2020-04-21t/nntpd: die if we can't open stderr output
We need to detect FS errors and bail out on the test if we can't open a file -nntpd was just writing to.
2020-04-21t/nntpd: reduce dependencies on internal API
Since the advent of run_script(), we can rely on it to simplify our test code. Changes like this will let us evolve the internal API more easily while preserving stable CLI interfaces, especially since we test the v2 path by default, now.
2020-04-21t/nntpd: fix lsof check w/ TEST_RUN_MODE=0
The `xqx' sub requires an absolute path for optional commands. Fixes: 6e07def560b211d9 ("testcommon: spawn-aware system() and qx[] workalikes")
2020-04-20testcommon: spawn-aware system() and qx[] workalikes
Barely noticeable on Linux, but this gives a 1-2% speedup on a FreeBSD 11.3 VM and lets us use built-in redirects rather than relying on /bin/sh.
2020-04-15t/*.t: localize $SIG{__WARN__} changes
We don't want to propagate %SIG changes to other tests when running multiple tests within the same process via t/run.perl.
2020-04-09tests: document run_mode=1 as not implemented
It was implemented at some point, but it was more things to support and the worst of both worlds: both unrealistic compared to real-world use and slower than run_mode=2. Noticed while looking for speling erorrs.
2020-02-06treewide: run update-copyrights from gnulib for 2019
I didn't wait until September to do it, this year!
2019-12-24testcommon: add require_mods method and use it
This cuts down on lines of code in individual test cases and fixes some misnamed error messages by using "$0" consistently. This will also provide us with a method of swapping out dependencies which provide equivalent functionality (e.g "Xapian" SWIG can replace "Search::Xapian" XS bindings).
2019-12-19t/run.perl: to avoid repeated process spawning for *.t
Spawning a new Perl interpreter for every test case means Perl has to reparse and recompile every single file it needs, costing us performance and development time. Now that we've modified our code to avoid global state, we can preload everything we need. The new "check-run" test target is now 20-30% faster than the original "check" target.
2019-12-19tests: move t/common.perl to PublicInbox::TestCommon
We want to be able to use run_script with *.t files, so t/common.perl putting subs into the top-level "main" namespace won't work. Instead, make it a module which uses Exporter like other libraries.
2019-11-24tests: use File::Temp->newdir instead of tempdir()
We'll also introduce a tmpdir() API to give tempdirs consistent names.
2019-11-24t/common: start_script replaces spawn_listener
We can shave several hundred milliseconds off tests which spawn daemons by preloading and avoiding startup time for common modules which are already loaded in the parent process. This also gives ENV{TAIL} support to all tests which support daemons which log to stdout/stderr.
2019-11-24tests: disable daemon workers in a few more places
There were still a few places where we used worker processes unnecessarily in tests, causing a small amount of unnecessary overhead. Followup-to: ad221e9b2852f6c5 ("t/*.t: disable nntpd/httpd worker processes in most tests")
2019-11-16t/nntpd: use run_script for -init
This only gives a 5% speedup or so, but anything helps.
2019-10-16config: support "inboxdir" in addition to "mainrepo"
"mainrepo" ws a bad name and artifact from the early days when I intended for there to be a "spamrepo" (now just the ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be especially confusing, since v2 needs at least two git repositories (epoch + all.git) to function and we shouldn't confuse users by having them point to a git repository for v2. Much of our documentation already references "INBOX_DIR" for command-line arguments, so use "inboxdir" as the git-config(1)-friendly variant for that. "mainrepo" remains supported indefinitely for compatibility. Users may need to revert to old versions, or may be referring to old documentation and must not be forced to change config files to account for this change. So if you're using "mainrepo" today, I do NOT recommend changing it right away because other bugs can lurk. Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
2019-09-09run update-copyrights from gnulib for 2019
2019-09-09tests: add tcp_connect() helper
IO::Socket::INET->new is rather verbose with the options hash, extract it into a standalone sub
2019-07-13nntp: support optional [range] arg in LISTGROUP
RFC3977 6.1.2.2 LISTGROUP allows a [range] arg after [group], and supporting it allows NNTP support in neomutt to work again. Tested with NeoMutt 20170113 (1.7.2) on Debian stretch (oldstable)
2019-07-13nntp: fix LIST OVERVIEW.FMT ordering and format
RFC3977 8.4.2 mandates the order of non-standard headers to be after the first seven standard headers/metadata; so "Xref:" must appear after "Lines:"|":lines". Additionally, non-required header names must be followed by ":full". Cc: Jonathan Corbet <corbet@lwn.net> Reported-by: Urs Jan├čen <E1hmKBw-0008Bq-8t@akw>
2019-07-06nntp: support COMPRESS DEFLATE per RFC 8054
This is only tested so far with my patches to Net::NNTP at: https://rt.cpan.org/Ticket/Display.html?id=129967 Memory use in C10K situations is disappointing, but that's the nature of compression. gzip compression over HTTPS does have the advantage of not keeping zlib streams open when clients are idle, at the cost of worse compression.
2019-07-05t/nntpd*.t: require IO::Socket::SSL 2.007 for Net::NNTP tests
Net::NNTP won't attempt to use older versions of IO::Socket::SSL because 2.007 is the "first version with default CA on most platforms" according to comments in Net::NNTP. But then again we don't make remote requests when testing...
2019-06-30nntp: add support for CAPABILITIES command
Some clients may rely on this for STARTTLS support.
2019-06-30tests: common tcp_server and unix_server helpers
IO::Socket:*->new options are verbose and we can save a bunch of code by putting this into t/common.perl, since the related spawn_listener stuff is already there.