about summary refs log tree commit homepage
path: root/lib/PublicInbox/TestCommon.pm
DateCommit message (Collapse)
2024-01-17tests: clarify Email::MIME is only for development
We moved to PublicInbox::Eml a while back and have no plans to go back to using Email::MIME, so don't tempt users and packagers to waste disk space on Email::MIME.
2024-01-10test_common: key2sub: don't require final ';' in scripts
I noticed this when I wrote a new (but probably unnecessary) *.t test and `make check-run' failed since I omitted the final semi-colon after `done_testing'.
2023-12-30lei: support reading MH for convert+import+index
The MH format is widely-supported and used by various MUAs such as mutt and sylpheed, and a MH-like format is used by mlmmj for archives, as well. Locking implementations for writes are inconsistent, so this commit doesn't support writes, yet. inotify|EVFILT_VNODE watches aren't supported, yet, but that'll have to come since MH allows packing unused integers and renaming files.
2023-12-16tests: quiet uninitialized warnings on CentOS 7.x
Test::More distributed with Perl 5.16.3 on CentOS 7.x expects the `$how_many' argument for `skip' and warns when its uninitialized, so quiet that warning down.
2023-12-13test_common: extract oct_is from search.t
And use it in convert-compact.t This gives us nicer errors for debugging a problem I noticed on Alpine Linux (tested 3.19.0)
2023-12-13tests: attempt compatibility w/ busybox lsof
BusyBox lsof(1) ignores the `-p PID' argument and shows the open files for every process it knows about. BusyBox lsof also lacks the `NODE' column of the non-BusyBox implementation, so we'll rely on /proc/PID/fd/ in those cases since the deleted file checks are Linux-only and it's common to have procfs is mounted on /proc on Linux.
2023-12-13tests: account for missing git-http-backend
Alpine Linux ships git-http-backend in the `git-daemon' package separately from `git', so we must test for its existence before attempting to test functionality which depends on it.
2023-12-13t/io: strace is optional on Linux
There are many Linux (GNU or otherwise) which do not have strace(1) installed.
2023-12-10imap: replace Mail::Address fallback with AddressPP
Our pure-Perl (PublicInbox::AddressPP) fallback is closer to the preferred Email::Address::XS (EAX) behavior than Mail::Address is for ->name support. EAX tends to be overkill with good spam filtering, and using our own fallback means life is easier for users with neither C/XS build tools nor a pre-built EAX package.
2023-12-01tests: note kevent+tmpfs failures on DragonFly <= 6.4
I forgot to set TMPDIR=/path/to/non-tmpfs again.
2023-11-29test_common: create_*: detect changes all parameters
Data::Dumper+B::Deparse seems fast enough to generate cache keys with, so this makes updating and developing tests easier (as opposed to forcing the developer to change the identifier). The main downside is we'll have to deal with cache expiration, but "make clean" seems overly aggressive already (it keeps blowing away the clones made by t/cindex-join.t :<)
2023-11-21cindex: rename --associate to --join, test w/ real repos
The association data is just stored as deflated JSON in Xapian metadata keys of shard[0] for now. It should be reasonably compact and fit in memory for now since we'll assume sane, non-malicious git coderepo history, for now. The new cindex-join.t test requires TEST_REMOTE_JOIN=1 to be set in the environment and tests the joins against the inboxes and coderepos of two small projects with a common history. Internally, we'll use `ibx_off', `root_off' instead of `ibx_id' and `root_id' since `_id' may be mistaken for columns in an SQL database which they are not.
2023-11-20test_common: fix excessive wait for GNU tail inotify
We want to use the filenames tail will watch, not the number of args passed to the `tail_f' subroutine. Fixes: 9231d2e7b93f (tests: map CLOFORK->FD_CLOEXEC temporarily for `tail -f')
2023-11-15treewide: more autodie safety fixes for older Perl
Avoid mixing autodie use in different scopes since it's likely to cause problems like it did in Gcf2. While none of these fix known problems with test cases, it's likely worthwhile to avoid it anyways to avoid future surprises. For Process::IO, we'll add some additional tests in t/io.t to ensure we don't get unintended exceptions for try_cat.
2023-11-14TestCommon: older strace does not have --version
The tests will check for strace >= 4.16, but version 4.24 that I have does not accept --version, only -V. This works for both older and newer strace, so switch to using "strace -V" for the check. Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
2023-11-13treewide: update read_all to avoid eof|close checks
read_all can be expanded to support FIFOs/pipes/sockets where read-until-EOF behavior is desired. We can also rely on wantarray to support splitting on EOL markers, but it's hard-coded to support only `$/ eq "\n"' since (AFAIK) it's the only way we use the wantarray form `readline'.
2023-11-11t/lei-import: skip strace for restricted systems
Systems with Yama can restrict ptrace(2) (the underlying syscall used by strace(1)) and make it difficult to test error handling via error injection. Just skip the tests on such systems since it's probably not worth the effort to start using prctl(2) to enable the test on such systems.
2023-11-09lei_input: always close single `eml' inputs
This matches the behavior we have for multi-message mbox files since we rely on ->close to detect errors on bad mboxes. This ensures we'll notice errors reading single messages from stdin. We'll also start relying more on strace error injection to test error handling.
2023-11-03move read_all, try_cat, and poll_in to PublicInbox::IO
The IO package seems like a better home for I/O subs than the Git package. We lose the 60 second read timeout for `git cat-file --batch-*' processes since it's probably not necessary given how reliable the code has proven and things would fall over hard in other ways if the storage device were completely hosed.
2023-10-18test_common: use $cwdfh for every run_script command
It's not lei-specific since we have `-C' to perform chdir for multiple admin commands.
2023-10-18test_common: only hide TCP port in messages
v2:// lei outputs are on the filesystem, so putting $HOST:$PORT is nonsensical. We'll also keep `127.0.0.1' or `[::1]' since it's harmless and can point out obvious errors in system configuration when testing with old Perls or libraries.
2023-10-18test_common: use autodie and read_all where possible
Noise reduction to help my aging eyes.
2023-10-18ds: introduce and use do_fork helper
This ensures we handle RNG reseeding and resetting the event loop properly in child processes after forking.
2023-10-06finalize DragonFlyBSD support
require_bsd and require_mods(':fcntl_lock') are now supported in TestCommon to make it easier to maintain than a big list of regexps. getsockopt for SO_ACCEPTFILTER seems to always succeed, even if the retrieved struct is all zeroes.
2023-10-04treewide: use PublicInbox::Lock->new
This gets rid of a few bare bless statements and helps ensure we properly load Lock.pm before using it.
2023-10-03daemon: enable SO_ACCEPTFILTER on NetBSD
NetBSD 5.0+ has accept filter support from FreeBSD; and I I think we can assume all NetBSD is 5.0+ (released in 2009) nowadays if we're already depending on Perl 5.12 from 2010.
2023-09-26spawn: add run_wait to simplify spawn+waitpid use
It's basically the `system' perlop with support for env overrides, redirects, chdir, rlimits, and setpgid support.
2023-09-25tests: add quit_waiter_pipe and wait_for_eof
These generalize the idiom from t/httpd-unix.t and allows them to be used for all the lei daemon shutdown in tests. This speeds up the serialized run of `prove -lvw t/lei.t t/lei-daemon.t' by roughly 8-9% or so since we're no longer sleeping and killing the daemon in a loop. Parallelized `make check' and `make check-run' don't show much difference, yet. Note that we do eliminate some kill(2) checks since those still require retries an delays. We assume the kernel auto-closing FDs on process exit is a strong-enough guarantee that the process will soon be reaped by PID:1. These will be useful for the FUSE daemons, as well. We'll also start introducing more uses of autodie to simplify our code.
2023-09-24test_common: force IPv4 with pre-IPv6 Net::POP3
This fixes t/pop3d-limit.t with the Net::POP3 shipped with Perl v5.16.3 on CentOS 7.x
2023-09-24test_common: fixes for TEST_RUN_MODE=0
This fixes t/httpd-https.t when running `make check TEST_RUN_MODE=0' to disable all code preloading and reuse. Fixes: b914e67f82ee (test_common: start_script: set default signals)
2023-09-24config: drop scalar ref support from internal API
It's a needless branch to maintain exclusively for our tests. The `git config -l' output isn't pleasant to write in tests, anyways. So just use heredocs to write git configs in their native format rather than emulate the output of `git config -l'. This does make the test suite do more work with temporary files and process invocations, but it doesn't seem very measurable when testing on tmpfs (TMPDIR=/dev/shm). We'll make a minor improvement to TestCommon::tmpdir by allowing it to return a single value (which I suspect we can rely on in more places since File::Temp::Dir overloads stringification).
2023-09-16Makefile: add `check-debris' target
This non-parallelized target is useful for noticing core dumps and preventing them from being clobbered as we run the test suite. It will also notice leftover temporary files and directories. This make target was used on OpenBSD 7.3 to develop at least two recent fixes: e281363ba937 (lei: ensure we run DESTROY|END at daemon exit w/ kqueue) 759885e60e59 (lei: ensure --stdin sets %ENV and $current_lei) I considered using a per-test TMPDIR for this to enable parallelization, but on-filesystem UNIX sockets won't work with excessively long path names.
2023-09-14test_common: eliminate call to non-existent cleanup_task
cleanup_task is long gone since all that cleanup work got moved into DS itself. Adding a DS->Reset here breaks t/imapd.t (and possibly other tests), and is actually unlikely to avoid any cleanup segfaults on OpenBSD since those are only happening after `END {}' blocks are called.
2023-09-12tests: allow more tests to pass without xapian-compact
2023-09-12tests: favor IO::Socket::IP for IPv6 listeners
IO::Socket::IP is bundled with Perl 5.24+ so it's more likely to be available than IO::Socket::INET6.
2023-09-11favor poll(2) for most daemons
public-inbox-watch, lei-daemon, the master process of public-inbox-(netd|httpd|imapd|nntpd|pop3d), and the (mostly) Perl implementation of XapHelper do not have many FDs to watch so epoll|kqueue end up being overkill. Of course, *BSDs already have separate kqueue FDs emulating signalfd and/or inotify, even. In other words, only the worker processes of public-inbox-(netd|httpd|imapd|nntpd|pop3d) are expected to see C10K (or C100K) types of traffic where epoll|kqueue shine. Perhaps lei could benefit from epoll/kqueue on some virtual users IMAP/JMAP system one day; as could -watch with many IMAP IDLE folders; but we'll probably add a knob if/when it comes to that.
2023-09-11tests: map CLOFORK->FD_CLOEXEC temporarily for `tail -f'
This fixes `TAIL="tail -F" prove -bvw t/lei-refresh-mail-sync.t' since that test relies on lacking FD_CLOEXEC to detect dead lei-daemons, but we still want FD_CLOEXEC when when relying on tail(1) to check -imapd output.
2023-09-11treewide: favor Xapian (SWIG binding) over Search::Xapian
The Xapian SWIG bindings are favored by Xapian upstream for ease-of-maintenance compared to the XS version. While Debian lags on this front, the SWIG bindings are widely available on all *BSDs.
2023-09-05test_common: start_script: set default signals
We need to ensure signal handlers in the child process aren't inherited from the parent. This change was originally intended to block signals all the way until PublicInbox::Daemon and PublicInbox::Watch were fully ready to handle them (preferably via EVFILT_SIGNAL or signalfd); but that proved unrealistic. Now, all signal handlers are restored to their default values before signals are unblocked. Drop a redundant DS->Reset while we're at it.
2023-09-05tests: add `+SCM_RIGHTS' as a require_mods target
We'll also ensure the existing `lei' target expands to depend on `+SCM_RIGHTS', and use require_mods in t/lei-import-nntp.t and t/lei.t so they can be skipped when Inline::C and Socket::MsgHdr are missing on OpenBSD.
2023-08-28t/xap_helper: skip test if missing SCM_RIGHTS support
xap_helper currently relies on FDs passed via SCM_RIGHTS for robustness against $TMPDIR failures and over-eager FS cleanup tasks. This depends on stable syscall numbers (Linux) or Inline::C||Socket::MsgHdr being available, though, as Perl5 itself doesn't support SCM_RIGHTS. We could probably add FIFO support to xap_helper for portability to systems where neither Inline::C nor Socket::MsgHdr are available, but that's for another day.
2023-04-12git: rename version() to git_version()
In case it gets confused with Inbox->version or similar.
2023-04-07switch git version comparisons to vstrings, too
There's too many require_git callsites in t/*.t to change, but we can make the rest of the code more readable and reuse PublicInbox::Git::version() in our test suite, too.
2023-03-25codesearch: initial cut w/ -cindex tool
It seems relying on root commits is a reasonable way to deduplicate and handle repositories with common history. I initially wanted to shoehorn this into extindex, but decided a separate Xapian index layout capable of being EITHER external to handle many forks or internal (in $GIT_DIR/public-inbox-cindex) for small projects is the right way to go. Unlike most existing parts of public-inbox, this relies on absolute paths of $GIT_DIR stored in the Xapian DB and does not rely on the config file. We'll be relying on the config file to map absolute paths to public URL paths for WWW.
2023-03-25test_common: create_inbox: use `$!' properly on mkdir failure
stat(2) may fail and set `$!', too, so we must stash it, first.
2023-03-07test_common: run_script: drop special-case for -clone
`make check' and `make check-run' actually work fine with it, and TMPDIR=/dev/shm prove -lvw t/clone-coderepo.t is 2-3x faster
2023-02-22treewide: simplify File::Path mkpath/make_path callers
File::Path already accounts for the existence of directories, handles races from redundant mkdir(2), and croaks on unrecoverable errors. So there's no point in doing any of that on our end. Furthermore, avoiding the overhead of loading File::Path doesn't seem worth it to save 20-60ms given the overhead of loading our other code. Instead, try to reduce optree overhead on our code, instead, since File::Path gets used in a bunch of places. We'll also favor the newer make_path for multi-directory invocations to avoid bloating our own optree to create an arrayref, but mkpath is one fewer subroutine call within File::Path itself, right now.
2023-02-12t/lei-refresh-mail-sync: avoid kill+sleep loop
While we can't waitpid() on daemonized process, we can abuse the lack of FD_CLOEXEC to detect a process death. This saves roughly 400ms for this slow test.
2023-01-31tests: make require_git and require_cmd easier-to-use
We'll rely on defined(wantarray) to implicitly skip subtests, and memoize these to reduce syscalls, since tests should be short-lived enough to not be affected by new installations or removals of git/xapian-compact/curl/etc...
2022-12-25test_common: avoid needless fcntl in start_script
POSIX::dup2 does not do anything in addition to dup2(2) and is thus immune to Perl automatically setting FD_CLOEXEC on FDs it makes into IO objects/globs. We only need to account for the case when both args for dup2 are identical, in which case the kernel treats it as a no-op and then thus we need to clear FD_CLOEXEC ourselves.