| Date | Commit message (Collapse) |
|---|
|
These can be used to temporarily disable using certain
externals in case of temporary network failure or mount point
unavailability.
|
|
Some error messages already include "\n" (w/ file+line info),
so don't add another one. (`warn' will automatically add its
caller location unless there's a final "\n").
|
|
We can still continue with some local externals, maybe;
but the error needs to be propagated to the calling process
for scripting purposes.
|
|
This allows "lei up" to continue processing unrelated externals
if on output fails.
|
|
This will help distinguish between mail outputs and external
public-inboxes.
|
|
This might speed up non-daemon-using tests.
|
|
Running "make test" on this project doesn't pass unless you've got an
existing PublicInbox.pm in your @INC, presumably nobody's set this up
on a fresh machine in a while.
This Makefile.PL trickery seems to do it, I've validated this with
this ad-hoc test of committing blib/ and Makefile to the repository:
git clean -dxf; perl Makefile.PL && make -j8 all && git add -f blib Makefile.PL Makefile && git commit -m"now"
Running that in interactive rebase before/after shows that only the
PublicInbox.pm file was added to blib/lib/. We use $(INST_LIB) instead
of a hardcoded 'blib/lib' now, but it's what ExtUtils::MakeMaker
recommends, so it's probably for the better.
As far as I can tell this broke with 1fae720d (build: generate
PublicInbox.pm with $VERSION, 2021-04-01), but I have not tested
that. See also 1fae720d (build: generate PublicInbox.pm with $VERSION,
2021-04-01) which made the PublicInbox.pm a generated file.
|
|
For odd messages with reused Message-IDs, the second message
showing up in a mirror (via git-fetch + -index) should never
clobber an entry with a different blob in over.
This is noticeable only if the messages arrive in-between
indexing runs.
Fixes: 4441a38481ed ("v2: index forwards (via `git log --reverse')")
|
|
There seems to be a bug in v2 inbox reindexing somewhere...
|
|
Check for graceful termination at every message since it's
a fairly inexpensive check.
|
|
I'm not sure if this is a bug or not (or it could be
an old bug in the v2 indexing code).
|
|
Ensure the num highwater mark of the target inbox is stable
before using it. Otherwise we may end up repeating work
done to index a message.
|
|
Since this is intended for use on the command-line,
include TZ offset in time and try to shorten the
message a bit so it wraps less on a terminal.
|
|
Caching the value doesn't seem necessary from a performance
perspective, and it adds a caveat for read-only users which
may lead to bugs in future code.
|
|
Our previous workaround didn't actually work around the leak in
<https://rt.cpan.org/Public/Bug/Display.html?id=139622> since
croak()-via-Perl was still invoked before the SV reference
count could be decremented.
Put in a proper workaround which saves warnings onto a temporary
variable and only croak after ->decode or ->encode returns; not
inside those methods.
|
|
|
|
Sigfd->event_step needs priority over script/lei clients,
LeiSelfSocket, and everything else.
|
|
Sigfd->event_step needs priority over InputPipe (and everything
else). We keep Edge Triggering here but use ->requeue instead
of looping inside event_step. This was necessary because
InputPipe can be used with regular files which can't be
monitored with epoll.
We'll also rid of the vestigial lei-oneshot support while we're
at it.
|
|
Sigfd->event_step needs priority over PktOp (and everything else).
We'll also add ECONNRESET checking, here, since it could see
bidirectional use in the future.
This is unlikely to have any sort of performance difference
since this is only for small, occasional packets, but the code
reduction is nice.
|
|
Sigfd->event_step needs priority over WQWorkers (and everything
else). Do that by running once per event_loop iteration rather
than looping inside event_step. This lowers throughput since it
requires more syscalls, but that's the price of fairness.
|
|
Running tests over a non-interactive ssh session fails,
otherwise.
|
|
At this point all of the current lei commands, aside from -help and
-sucks, should be covered.
|
|
Most the lei-related entries in txt2pre and Makefile.PL are in
alphabetical order. Reorder the few that aren't.
While at it, reflow the Makefile.PL entries in preparation for the
entries that will be added in the next commit.
|
|
We can't attempt to unref messages beyond the highwater mark of
an inbox. This bugfix was found by commit c485036d0b1ce7ed
(extindex: guard against buggy unrefs, 2021-10-14), which
actually did its intended job and guarded against a buggy unref.
|
|
We'll save ourselves some code here and let the kernel do more
work, instead.
|
|
Some yak-shaving while I try to track down other bugs...
|
|
We don't need to flood the terminal with "W: $oid is (!= blob)\n"
messages when somebody nukes a git cat-file process from under
us.
|
|
This will make our code more flexible in case it gets used in
non-lei things.
|
|
Seeing the same warning over and over again gets annoying.
|
|
Relying on $lei->fail is unsustainable since there'll always
be parts of our code and dependencies which can trigger die()
and break the event loop.
|
|
The extra FD shouldn't cause noticeable overhead in short-lived
workers, and it lets us simplify lei->rel2abs. Get rid of a
2-argument form of open() while we're at it, since it's been
considered for warning+deprecation by Perl for safety reasons.
|
|
More code means more bugs.
|
|
Most of the HTTP server code was written for Danga::Socket and
not fully-transitioned to take advantage of PublicInbox::DS.
This change brings it up-to-date with the style of pipeline
handling used for -imapd and -nntpd.
|
|
It's needlessly complex and O(n), so it doesn't scale well to a
high number of clients nor is it easy-to-scale with the data
structures available to us in pure Perl.
In any case, I see no evidence of either -imapd nor -nntpd
experiencing high connection loads on public-facing sites.
-httpd has never had its own timer-based expiration, either.
Fwiw, public-inbox.org itself has been running a public-facing
HTTP/HTTPS server with no userspace idle client expiration for
the past 8 years or with no ill effect. Clients can come and go
as they wish, and SO_KEEPALIVE takes care of truly broken
connections if they're gone for ~2 hours.
Internet connections drop all time, so it should be harmless to
drop connections w/o warning since both NNTP and IMAP protocols
have well-defined semantics for determining if a message was
truncated (as does HTTP/1.1+).
|
|
There's no savings in having two ways to add watches to an
inotify nor kqueue descriptor.
|
|
This makes some of our code less noisy by reducing the
amount of pack('H*', ...) use.
|
|
v2w->wq_do('done') may die on I/O errors, and likely other
places. Just guard the entire block with an eval and ->fail
as appropriate.
|
|
I've been testing a lot of searches which I don't want to keep
around, so make it easy to remove a bunch at once. We'll behave
like rm(1) and keep going in the face of failure.
|
|
We need to send the socket over to lei/store and wait for the
kernel to drop the socket refcount down to zero before
script/lei can exit.
This is not a new bug and only caused very sporadic test
failures. I only noticed it while simplifying IPC stuff.
|
|
Simplify our APIs and force dwaitpid() to work in async mode for
all lei workers. This avoids having lingering zombies for
parallel searches if one worker finishes soon before another.
The old distinction between "old" and "new" workers was
needlessly complex, error-prone, and embarrasingly bad.
We also never handled v2:// writers properly before on
Ctrl-C/Ctrl-Z (SIGINT/SIGTSTP), so add them to @WQ_KEYS
to ensure they get handled by $lei when appropropriate.
|
|
If lei up and edit-search work on something, so should forget-search.
|
|
When importing several sources in parallel via http(s) mboxrd,
we need to be able to get keywords of uncommitted documents
directly from shard workers. Otherwise, Xapian DocNotFound
errors happen because the read-only LeiSearch won't see
documents from uncomitted transactions. Keep in mind that it's
possible the keywords can be changed on-the-fly even for
uncommitted documents because of inotify watches from LeiNoteEvent.
|
|
`dt:' documentation is redundant with `d:' approxidate support;
so drop `dt:' since mairix uses `d:'. We'll also document
`rt:' since there are legit messages from senders with broken
clocks.
Reduce indentation level of help texts to be in 2-space
increments to using too much horizontal space.
We'll always place IMAP ahead of NNTP since it's alphabetical
and there's likely more IMAP clients out there.
Add "--ng NEWSGROUP" to -init instructions if configured.
There's also some minor wording changes throughout.
|
|
The redispatch mechanism wasn't routing signals and messages
between redispatched workers and script/lei properly. We now
rely on PktOp to do bidirectional message forwarding and
carefully avoiding circular references by using PktOp.
|
|
We need to use DESTROY here to ensure we wait for workers, too;
not just the initial dispatch.
Fixes: cafbd77b3c82167d ("lei up: avoid excessively parallel --all")
|
|
By relying more on pgroups for remaining remaining processes,
this lets us pause all curl+tail subprocesses with a single
kill(2) to avoid cluttering stderr.
We won't bother pausing the pigz/gzip/bzip2/xz compressor
process not cat-file processes, though, since those don't write
to the terminal and they idle soon after the workers react to
SIGSTOP.
AutoReap is hoisted out from TestCommon.pm. CLONE_SKIP
is gone since we won't be using Perl threads any time
soon (they're discouraged by the maintainers of Perl).
|
|
We want these long-lived processes to die naturally when their
parent dies. Hopefully this improves graceful shutdown for
-extindex because I'm interrupting a lot of reindexing...
|
|
While we try to invoke all pending callbacks to force error
handling, the current callback wasn't getting invoked on
invoked on async_abort if my_read/my_readline failed.
|
|
We make $req a reference upon retrying, but
"SCALAR(...)" in error messages isn't helpful, so
dereference the scalar ref.
|
|
This may save us a small bit of startup time since there's
fewer args and opcodes should be smaller.
|