| Date | Commit message (Collapse) |
|---|
|
barrier (synchronous checkpoint) is better than ->done with
parallel lei commands being issued (via '&' or different
terminals), since repeatedly stopping and restarting processes
doesn't play nicely with expensive tasks like `lei reindex'.
This introduces a slight regression in maintaining more
processes (and thus resource use) when lei is idle, but that'll
be fixed in the next commit.
|
|
Noticed while working on other things...
Fixes: 299aac294ec3 (lei: do label/keyword parsing in optparse, 2023-10-02)
|
|
This adds support for the "POST /$INBOX/$MSGID/?x=m?q=..."
added last year to support per-thread searches
764035c83 (www: support POST /$INBOX/$MSGID/?x=m&q=, 2023-03-30)
This only supports instances of public-inbox since 764035c83,
but unfortunately there hasn't been a release since then.
|
|
getpid() isn't cached by glibc nowadays and system calls are
more expensive due to CPU vulnerability mitigations. To
ensure we switch to the new semantics properly, introduce
a new `on_destroy' function to simplify callers.
Furthermore, most OnDestroy correctness is often tied to the
process which creates it, so make the new API default to
guarded against running in subprocesses.
For cases which require running in all children, a new
PublicInbox::OnDestroy::all call is provided.
|
|
This should help us deal with MH sequence number packing and
invalidating mail_sync.sqlite3.
|
|
The MH format is widely-supported and used by various MUAs such
as mutt and sylpheed, and a MH-like format is used by mlmmj for
archives, as well. Locking implementations for writes are
inconsistent, so this commit doesn't support writes, yet.
inotify|EVFILT_VNODE watches aren't supported, yet, but that'll
have to come since MH allows packing unused integers and
renaming files.
|
|
`lei index' should be capable of indexing the the same way
`lei import' does, but without the indexing. I only noticed
this omission while developing a new feature.
|
|
--no-import-before skips importing entire messages, not just
keywords, so it can cause permanent data loss if -o is pointed
to precious data.
|
|
We only care about error checking when stdout is an mbox output
pointed to a pathname. This is noticeable with `lei up' with
multiple non-mbox* destinations. We'll also ensure throwing
exceptions to trigger lei->x_it from lei->do_env results in the
epoll/kqueue watch being discarded, otherwise commands may never
terminate (leading to stuck tests)
|
|
This is needed to support forking from already-forked lei workers
and $lei->{2} is already STDERR.
Fixes: e015c3742f91 (lei: use autodie where appropriate, 2023-10-17)
|
|
Avoid mixing autodie use in different scopes since it's likely
to cause problems like it did in Gcf2. While none of these
fix known problems with test cases, it's likely worthwhile to
avoid it anyways to avoid future surprises.
For Process::IO, we'll add some additional tests in t/io.t
to ensure we don't get unintended exceptions for try_cat.
|
|
Unlike modern Perls, Perl 5.16.3 on CentOS doesn't accept
negative string signals like "-TERM" .
This only became a problem since commit b231d91f42d7
(treewide: enable warnings in all exec-ed processes)
made our code stricter by enabling more warnings.
In both cases, the kill is probably unnecessary and safe
to remove since we can rely on closing sockets to drop
processes.
|
|
We must use a foreground process to read from terminals
on stdin, otherwise weird things like lost keystrokes and
EIO can happen. So take advantage of ->send_exec_cmd to
spawn `cat' in the same way we spawn MUAs, pagers,
`git config --edit' and `git credential' from script/lei
|
|
We'll avoid having a redundant STDERR FD open in lei workers,
and some explicit close() on `lei up' sockets reduces the
likelyhood of inadvertantly open FDs causing processes to
linger.
|
|
->lei_daemon_pid can only be called in the top-level daemon
process when $daemon_pid is valid, so avoid a getpid(2) syscall
in those cases.
|
|
This will open the door for us to drop `tie' usage from
ProcessIO completely in favor of OO method dispatch. While
OO method dispatches (e.g. `$fh->close') are slower than normal
subroutine calls, it hardly matters in this case since process
teardown is a fairly rare operation and we continue to use
`close($fh)' for Maildir writes.
|
|
This saves us some code, and is a small step towards getting
ProcessIO working with stat, fcntl and other perlops that don't
work with tied handles.
|
|
When ->ovv_begin is called in LeiXSearch->do_query in the top-level
lei-daemon process, $lei->{pkt_op_p} still exists. We must make
sure we're exiting the correct process since lei->out can call
lei->fail and lei->fail calls lei->x_it.
As to avoiding how I caused ->ovv_begin failures to begin with,
that's for a much bigger change...
|
|
This is intended to replace psgi_return and HTTPD/Async
entirely, hopefully making our code less convoluted while
maintaining the ability to handle slow clients on
memory-constrained systems
This was made possible by the philosophy shift in commit 21a539a2df0c
(httpd/async: switch to buffering-as-fast-as-possible, 2019-06-28).
We'll still support generic PSGI via the `pull' model with a
GetlineResponse class which is similar to the old GetlineBody.
|
|
We only need to write one byte at MUA start instead of a byte
for every LeiXSearch worker. Also, make sure it succeeds by
enabling autodie for syswrite.
When reading, we can rely on `:perlio' layer `read' semantics
to retry on EINTR to avoid looping and other error checking.
|
|
This makes us a bit harsher with misbehaving clients, but we
only have one client implementation at the moment.
|
|
We can share more code amongst stdin slurper (not streaming)
commands. This also fixes uninitialized variable warnings when
feeding an empty stdin to these commands.
|
|
We don't want to end up dumping nr_seen/nr_write when progress
is disabled, nor do we want forked off `lei note-event' workers
dump them when DS->Reset is called on fork.
|
|
Delayed commits allows users to trade off immediate safety for
throughput and reduced storage wear when running multiple
discreet commands.
This feature is currently useful for providing a way to make
t/lei-store-fail.t reliable and for ensuring `lei blob' can
retrieve messages which have not yet been committed.
In the future, it'll also be useful for the FUSE layer to batch
git activity.
|
|
It's safer against deadlocks and we still get proper error
reporting by passing stderr across in addition to the lei
socket.
|
|
Eric Wong <e@80x24.org> wrote:
> +++ b/t/lei-store-fail.t
> + my $cmd = [ qw(lei import -q -F mboxrd) ];
> + my $tp = start_script($cmd, undef, $opt);
Of course the lack of `-' or `--stdin' only worked on Linux and
NetBSD, but not other BSDs.
-------8<------
Subject: [PATCH] lei: fix implicit stdin support for pipes
st_mode permission bits can't be used to determine if a file or
pipe we have on stdin readable or not. Writable regular files
can be opened O_RDONLY, and permissions bits for pipes are
inconsistent across platforms.
On FreeBSD, OpenBSD, and Dragonfly, only the S_IFIFO bit is set
in st_mode with none of the permission bits are set. Linux and
NetBSD have both the read and write permission bits set for both
ends of a the pipe, so they're just as inaccurate but allowed
the feature to work before this change.
For now, we'll just assume our users know that stdin is intended
for input and consider any pipe or regular file to be readable.
If we were to be pedantic, we'd check O_RDONLY or O_RDWR
description flags via the F_GETFL fcntl(2) op to determine if a
pipe or socket is readable. However, I don't think it's worth
the code to do so.
|
|
This fixes attempts to use an undefined value as an ARRAY reference
in PublicInbox::IPC::wq_io_do
|
|
This ensures script/lei $send_cmd usage is EINTR-safe (since
I prefer to avoid loading PublicInbox::IPC for startup time).
Overall, it saves us some code, too.
|
|
This is more persistent than some of the others and we don't
swap it on use (unlike $nextq or $ToClose). In other words,
it's helpful for communicating its lifetime expectancy is
close to %DescriptorMap and not like to queue-type things
such as $ToClose.
|
|
This variable needs to be visible to a callback running inside
Getopt::Long, but we don't need to keep it around after
LEI->optparse runs.
|
|
This gets rid of a few bare bless statements and helps
ensure we properly load Lock.pm before using it.
|
|
Since we completely shut down all workers before exiting,
we no longer have to care about missing SIGCHLD wakeups
during shutdown.
|
|
No need to define our own empty `noop' sub when PublicInbox::Config
already has one and is loaded anyways.
|
|
We already have an ->incr callback we can enhance to support
multiple counters with a single request. Furthermore, we can
just flatten the object graph by storing counters directly in
the $lei object itself to reduce hash lookups.
|
|
This will make switching $lei contexts less error-prone
and hopefully save us from some suprising bugs in the future.
Followup-to: 759885e60e59 (lei: ensure --stdin sets %ENV and $current_lei, 2023-09-14)
|
|
We don't want FS activity to delay lei-daemon shutdown.
|
|
Compared to Danga::Socket, our @post_loop_do API is designed to
make it easier to avoid anonymous subs (and their potential for
leaks in buggy old versions of Perl).
|
|
It's not used by any post_loop_do callbacks anymore, and the
underlying FD map is a global `our' variable accessible from
anywhere, anyways.
|
|
It's shared by both by lei and public-facing daemons in using
the ->busy callback.
|
|
This will allow us to avoid unblocking signals during
shutdown to simplify our code.
|
|
We don't want workers continuing after their stdout has triggered
EPIPE or some other write error.
This fixes xt/lei-onion-convert.t to ensure the quit_waiter_pipe
is fully-closed at daemon teardown during tests. Using the
`exit' perlop still ensures OnDestroy callbacks will fire.
|
|
Calling vmd_mod_extract after optparse causes the implicit
stdin-as-input functionality to fail, as the implicit stdin
requires a lack of inputs remaining in argv after option
parsing (along with a regular file or pipe as stdin).
This allows commands such as `lei import -F eml +kw:seen'
to work without `--stdin', `-' or any path names when
importing a single message. This also ensures commands like
`lei import +kw:seen' without any inputs/locations will fail
reliably, as the extra +kw: arg won't be a false-positive.
|
|
lei/store can get stuck if lei-daemon is blocked, and lei-daemon
can get stuck when a clients stderr is redirected to a pager
that isn't consumed.
So start relying on Time::HiRes::alarm to generate SIGALRM to
break out of the `print' perlop. Unfortunately, this isn't easy
since Perl auto-restarts all writes, so we dup(2) the
destination FD and close the copy in the SIGALRM handler to
force `print' to return.
Most programs (MUAs, editors, etc.) aren't equipped to deal with
non-blocking STDERR, so we can't make the stderr file description
non-blocking.
Another way to solve this problem would be to have script/lei
send a non-blocking pipe to lei-daemon in the {2} slot and
make script/lei splice messages from the pipe to stderr.
Unfortunately, that requires more work and forces more
complexity into script/lei and slow down normal cases where
stderr doesn't get blocked.
|
|
We can't use floating point numbers nor Inf/-Inf as exit codes;
but we can allow `-1' as shorthand for 255.
|
|
The first argument passed to Perl signal handlers is a
signal name (e.g. "TERM") and not an integer that can
be passed to the `exit' perlop. Thus we must look up the
integer value from the POSIX module.
|
|
We can't use the `-O' switch since it conflicts with
--only|-O= to specify externals. Thus we'll introduce
a more verbose `--order-file=FILE' option when running
`git diff'.
|
|
Using and memoizing the usability of `--rsyncable' is unsafe
since pigz (or GNU gzip) can be uninstalled and leave a user
with a non-rsync-aware gzip implementation in the long-running
daemon. So we stop passing --rsyncable by default to pigz/gzip
and no longer attempt to check for it (since it was a TOCTTOU
error, anyways).
Specifying --rsyncable explicitly didn't work, either, and
ended up passing `1' to the gzip/pigz argv :x
Finally, we now test --rsyncable on the CLI by adding support
for it in `lei convert' and testing it in t/lei-convert.t
|
|
It's basically the `system' perlop with support for env overrides,
redirects, chdir, rlimits, and setpgid support.
|
|
Reaping children needs to keep the event_loop spinning another
round when the @post_loop_do callback may be used to check
on process exit during shutdown.
This allows us to get rid of the hacky SetLoopTimeout calls in
lei-daemon and XapHelper.pm during process shutdown if we're
trying to wait for all PIDs to exit before leaving the event
loop.
|
|
Perl v5.16.3 (and possibly some later versions) complain about
this, but newer (v5.32.1) are fine with it.
Fixes: e281363ba937 ("lei: ensure we run DESTROY|END at daemon exit w/ kqueue")
|