| Date | Commit message (Collapse) |
|---|
|
Repeatedly checking $PATH for `git' when we need to call it
multiple times in quick succession doesn't seem useful. So
avoid some expensive stat(2) syscalls to make things less bad
for systems which require expensive CPU vulnerability
mitigations.
This also saves a bunch of memory allocations since we do the
$PATH lookup in pure Perl to avoid doing the uncacheable lookup
in a vfork-ed child.
|
|
When read-only daemons reopen log files via SIGUSR1, be sure to
propagate it to Xapian helper processes to ensure old log files
can be closed and archived.
|
|
We should almost always be calling `check_build' instead of
`build'. Using ccache masked some of the overhead from
this, but various linker implementations are still slow.
|
|
Xapian helper processes are disabled by default once again.
However, they can be enabled via the new `-X INTEGER' parameter.
One big positive is the Xapian helpers being spawned by the
top-level daemon means they can be shared freely across all
workers for improved load balancing and memory reduction.
|
|
getpid() isn't cached by glibc nowadays and system calls are
more expensive due to CPU vulnerability mitigations. To
ensure we switch to the new semantics properly, introduce
a new `on_destroy' function to simplify callers.
Furthermore, most OnDestroy correctness is often tied to the
process which creates it, so make the new API default to
guarded against running in subprocesses.
For cases which require running in all children, a new
PublicInbox::OnDestroy::all call is provided.
|
|
We moved to PublicInbox::Eml a while back and have no plans
to go back to using Email::MIME, so don't tempt users and
packagers to waste disk space on Email::MIME.
|
|
I noticed this when I wrote a new (but probably unnecessary) *.t
test and `make check-run' failed since I omitted the final
semi-colon after `done_testing'.
|
|
The MH format is widely-supported and used by various MUAs such
as mutt and sylpheed, and a MH-like format is used by mlmmj for
archives, as well. Locking implementations for writes are
inconsistent, so this commit doesn't support writes, yet.
inotify|EVFILT_VNODE watches aren't supported, yet, but that'll
have to come since MH allows packing unused integers and
renaming files.
|
|
Test::More distributed with Perl 5.16.3 on CentOS 7.x expects
the `$how_many' argument for `skip' and warns when its
uninitialized, so quiet that warning down.
|
|
And use it in convert-compact.t This gives us nicer errors for
debugging a problem I noticed on Alpine Linux (tested 3.19.0)
|
|
BusyBox lsof(1) ignores the `-p PID' argument and shows
the open files for every process it knows about. BusyBox
lsof also lacks the `NODE' column of the non-BusyBox
implementation, so we'll rely on /proc/PID/fd/ in those
cases since the deleted file checks are Linux-only and
it's common to have procfs is mounted on /proc on Linux.
|
|
Alpine Linux ships git-http-backend in the `git-daemon'
package separately from `git', so we must test for its
existence before attempting to test functionality which
depends on it.
|
|
There are many Linux (GNU or otherwise) which do not have
strace(1) installed.
|
|
Our pure-Perl (PublicInbox::AddressPP) fallback is closer to the
preferred Email::Address::XS (EAX) behavior than Mail::Address
is for ->name support. EAX tends to be overkill with good spam
filtering, and using our own fallback means life is easier for
users with neither C/XS build tools nor a pre-built EAX package.
|
|
I forgot to set TMPDIR=/path/to/non-tmpfs again.
|
|
Data::Dumper+B::Deparse seems fast enough to generate cache keys
with, so this makes updating and developing tests easier (as
opposed to forcing the developer to change the identifier). The
main downside is we'll have to deal with cache expiration, but
"make clean" seems overly aggressive already (it keeps blowing
away the clones made by t/cindex-join.t :<)
|
|
The association data is just stored as deflated JSON in Xapian
metadata keys of shard[0] for now. It should be reasonably
compact and fit in memory for now since we'll assume sane,
non-malicious git coderepo history, for now.
The new cindex-join.t test requires TEST_REMOTE_JOIN=1 to be
set in the environment and tests the joins against the inboxes
and coderepos of two small projects with a common history.
Internally, we'll use `ibx_off', `root_off' instead of `ibx_id'
and `root_id' since `_id' may be mistaken for columns in an SQL
database which they are not.
|
|
We want to use the filenames tail will watch, not the number of
args passed to the `tail_f' subroutine.
Fixes: 9231d2e7b93f (tests: map CLOFORK->FD_CLOEXEC temporarily for `tail -f')
|
|
Avoid mixing autodie use in different scopes since it's likely
to cause problems like it did in Gcf2. While none of these
fix known problems with test cases, it's likely worthwhile to
avoid it anyways to avoid future surprises.
For Process::IO, we'll add some additional tests in t/io.t
to ensure we don't get unintended exceptions for try_cat.
|
|
The tests will check for strace >= 4.16, but version 4.24 that I have
does not accept --version, only -V. This works for both older and newer
strace, so switch to using "strace -V" for the check.
Signed-off-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
|
|
read_all can be expanded to support FIFOs/pipes/sockets where
read-until-EOF behavior is desired. We can also rely on
wantarray to support splitting on EOL markers, but it's
hard-coded to support only `$/ eq "\n"' since (AFAIK)
it's the only way we use the wantarray form `readline'.
|
|
Systems with Yama can restrict ptrace(2) (the underlying syscall
used by strace(1)) and make it difficult to test error handling
via error injection. Just skip the tests on such systems since
it's probably not worth the effort to start using prctl(2) to
enable the test on such systems.
|
|
This matches the behavior we have for multi-message mbox files
since we rely on ->close to detect errors on bad mboxes. This
ensures we'll notice errors reading single messages from stdin.
We'll also start relying more on strace error injection to test
error handling.
|
|
The IO package seems like a better home for I/O subs than the
Git package. We lose the 60 second read timeout for `git
cat-file --batch-*' processes since it's probably not necessary
given how reliable the code has proven and things would fall
over hard in other ways if the storage device were completely
hosed.
|
|
It's not lei-specific since we have `-C' to perform chdir
for multiple admin commands.
|
|
v2:// lei outputs are on the filesystem, so putting $HOST:$PORT
is nonsensical. We'll also keep `127.0.0.1' or `[::1]' since
it's harmless and can point out obvious errors in system
configuration when testing with old Perls or libraries.
|
|
Noise reduction to help my aging eyes.
|
|
This ensures we handle RNG reseeding and resetting the event
loop properly in child processes after forking.
|
|
require_bsd and require_mods(':fcntl_lock') are now
supported in TestCommon to make it easier to maintain
than a big list of regexps.
getsockopt for SO_ACCEPTFILTER seems to always succeed,
even if the retrieved struct is all zeroes.
|
|
This gets rid of a few bare bless statements and helps
ensure we properly load Lock.pm before using it.
|
|
NetBSD 5.0+ has accept filter support from FreeBSD; and I
I think we can assume all NetBSD is 5.0+ (released in 2009)
nowadays if we're already depending on Perl 5.12 from 2010.
|
|
It's basically the `system' perlop with support for env overrides,
redirects, chdir, rlimits, and setpgid support.
|
|
These generalize the idiom from t/httpd-unix.t and allows them
to be used for all the lei daemon shutdown in tests. This
speeds up the serialized run of `prove -lvw t/lei.t t/lei-daemon.t'
by roughly 8-9% or so since we're no longer sleeping and killing
the daemon in a loop.
Parallelized `make check' and `make check-run' don't show much
difference, yet.
Note that we do eliminate some kill(2) checks since those still
require retries an delays. We assume the kernel auto-closing
FDs on process exit is a strong-enough guarantee that the
process will soon be reaped by PID:1.
These will be useful for the FUSE daemons, as well.
We'll also start introducing more uses of autodie to simplify
our code.
|
|
This fixes t/pop3d-limit.t with the Net::POP3 shipped with Perl v5.16.3
on CentOS 7.x
|
|
This fixes t/httpd-https.t when running `make check TEST_RUN_MODE=0'
to disable all code preloading and reuse.
Fixes: b914e67f82ee (test_common: start_script: set default signals)
|
|
It's a needless branch to maintain exclusively for our tests.
The `git config -l' output isn't pleasant to write in tests,
anyways. So just use heredocs to write git configs in their
native format rather than emulate the output of `git config -l'.
This does make the test suite do more work with temporary files
and process invocations, but it doesn't seem very measurable
when testing on tmpfs (TMPDIR=/dev/shm).
We'll make a minor improvement to TestCommon::tmpdir by allowing
it to return a single value (which I suspect we can rely on in
more places since File::Temp::Dir overloads stringification).
|
|
This non-parallelized target is useful for noticing core dumps
and preventing them from being clobbered as we run the test
suite. It will also notice leftover temporary files and
directories.
This make target was used on OpenBSD 7.3 to develop at least two
recent fixes:
e281363ba937 (lei: ensure we run DESTROY|END at daemon exit w/ kqueue)
759885e60e59 (lei: ensure --stdin sets %ENV and $current_lei)
I considered using a per-test TMPDIR for this to enable
parallelization, but on-filesystem UNIX sockets won't work
with excessively long path names.
|
|
cleanup_task is long gone since all that cleanup work got moved
into DS itself. Adding a DS->Reset here breaks t/imapd.t (and
possibly other tests), and is actually unlikely to avoid any
cleanup segfaults on OpenBSD since those are only happening
after `END {}' blocks are called.
|
|
|
|
IO::Socket::IP is bundled with Perl 5.24+ so it's more likely
to be available than IO::Socket::INET6.
|
|
public-inbox-watch, lei-daemon, the master process of
public-inbox-(netd|httpd|imapd|nntpd|pop3d),
and the (mostly) Perl implementation of XapHelper do not
have many FDs to watch so epoll|kqueue end up being overkill.
Of course, *BSDs already have separate kqueue FDs emulating
signalfd and/or inotify, even.
In other words, only the worker processes of
public-inbox-(netd|httpd|imapd|nntpd|pop3d) are expected
to see C10K (or C100K) types of traffic where epoll|kqueue
shine.
Perhaps lei could benefit from epoll/kqueue on some virtual users
IMAP/JMAP system one day; as could -watch with many IMAP IDLE
folders; but we'll probably add a knob if/when it comes to that.
|
|
This fixes `TAIL="tail -F" prove -bvw t/lei-refresh-mail-sync.t'
since that test relies on lacking FD_CLOEXEC to detect dead
lei-daemons, but we still want FD_CLOEXEC when when relying
on tail(1) to check -imapd output.
|
|
The Xapian SWIG bindings are favored by Xapian upstream for
ease-of-maintenance compared to the XS version. While Debian
lags on this front, the SWIG bindings are widely available
on all *BSDs.
|
|
We need to ensure signal handlers in the child process aren't
inherited from the parent. This change was originally intended
to block signals all the way until PublicInbox::Daemon and
PublicInbox::Watch were fully ready to handle them (preferably
via EVFILT_SIGNAL or signalfd); but that proved unrealistic.
Now, all signal handlers are restored to their default values
before signals are unblocked.
Drop a redundant DS->Reset while we're at it.
|
|
We'll also ensure the existing `lei' target expands to depend on
`+SCM_RIGHTS', and use require_mods in t/lei-import-nntp.t and
t/lei.t so they can be skipped when Inline::C and Socket::MsgHdr
are missing on OpenBSD.
|
|
xap_helper currently relies on FDs passed via SCM_RIGHTS for
robustness against $TMPDIR failures and over-eager FS cleanup
tasks. This depends on stable syscall numbers (Linux) or
Inline::C||Socket::MsgHdr being available, though, as Perl5
itself doesn't support SCM_RIGHTS.
We could probably add FIFO support to xap_helper for portability
to systems where neither Inline::C nor Socket::MsgHdr are available,
but that's for another day.
|
|
In case it gets confused with Inbox->version or similar.
|
|
There's too many require_git callsites in t/*.t to change,
but we can make the rest of the code more readable and reuse
PublicInbox::Git::version() in our test suite, too.
|
|
It seems relying on root commits is a reasonable way to
deduplicate and handle repositories with common history.
I initially wanted to shoehorn this into extindex, but decided a
separate Xapian index layout capable of being EITHER external to
handle many forks or internal (in $GIT_DIR/public-inbox-cindex)
for small projects is the right way to go.
Unlike most existing parts of public-inbox, this relies on
absolute paths of $GIT_DIR stored in the Xapian DB and does not
rely on the config file. We'll be relying on the config file to
map absolute paths to public URL paths for WWW.
|
|
stat(2) may fail and set `$!', too, so we must stash it, first.
|