| Date | Commit message (Collapse) |
|---|
|
This prevents the following problem logged to the webserver's error log:
E: Undefined subroutine &PublicInbox::WwwStream::code_footer called at /usr/share/perl5/PublicInbox/WwwListing.pm line 102.
in PublicInbox::ConfigIter=ARRAY(0x557aea68b1a8)::each_section at /usr/share/perl5/PublicInbox/ConfigIter.pm line 37.
Fixes: 7a3946ef122e ("www: support listing of inboxes")
|
|
|
|
While disk I/O is typically buffered for good scheduling,
git blob decoding uses a non-trivial amount of CPU time
and it helps to leave some CPU available for it.
|
|
We'll also support "-C" at the end of most commands to give
users a little more flexibility when building command-lines.
This conflicts with "lei daemon-kill -CHLD", so that's
special-cased since "-C" makes no sense with daemon-kill,
anyways.
Unlike "git show", the to-be-implemented "lei show" will diverge
and enable "--find-copies[=<n>]" by default, so "-C[<n>]" won't
be necessary.
|
|
The --format argument is redundant and may be dropped entirely.
Update the lei manpages to prefer the format prefix.
cf. https://public-inbox.org/meta/20210217044032.GA17934@dcvr/
|
|
LeiAuth is no longer a separate worker process. Instead, it's
used directly by LeiToMail and LeiImport for sharing auth info
from the first worker to the rest of the workers, using
lei-daemon as a message router. So drop the old code to reduce
human cognitive load and interpreter memory overhead.
|
|
Since we stopped using LeiAuth as a WQ worker, keeping this
around as a single-use sub makes no sense and wastes several
KB of memory.
|
|
We only enable {mic_cached} in WQ workers, and those
aren't expected to fork again going forward. So cache
here avoid a penalty for the non-augmenting (imap_delete_all)
call with "lei q"
|
|
We can rework the first lei2mail worker to authenticate, and
then share auth info with the rest of the lei2mail workers. As
with "lei import", this uses PktOp and lei-daemon to share
updated credentials between the first an subsequent l2m workers.
|
|
lei_to_mail will be able to use this, too.
|
|
We'll start sharing auth info from the first worker to the
rest of the workers via wq_broadcast.
This lays the groundwork for getting rid of LeiAuth workers for
authentication work and reducing network round trips required
for IMAP.
|
|
Since this only has one worker, we can auth directly in the
worker since the convert worker now has access to the script/lei
{sock} for running "git credential".
|
|
Existing callers in LeiExternal actually depend on this,
and LeiAuth shouldn't need to be creating a config file
just to do a conversion against an anonymous IMAP server.
|
|
For non-persistent workers, there's no harm in keeping the
client socket open. This means we can avoid dancing around
closing it in PublicInbox::LeiAuth::ipc_atfork_child.
Eventually, other WQ workers will trigger "git credential"
spawning in script/lei directly.
|
|
We're authing for both reads and writes, this makes it
clear that we support both NetReader and NetWriter.
|
|
Since eidx_init updates ALL.git/objects/info/alternates, we need
to ensure new epochs we create from LeiStore->importer exist
before eidx_init writes alternates.
Reported-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/8735xou0gq.fsf@kyleam.com/
|
|
lei() and lei_ok() are superior since they offer prototype
checks and lei_ok() adds another check + description DRY-ness.
The $lei sub was only bound to a variable since it was in
t/lei.t and named subs don't work well with the key2sub()
wrapper.
|
|
We'll keep the daemon alive as long as a a script/lei client
remains connected. This ought to improve user experience
and is in line with what -imapd/-httpd/-nntpd users have
expected over the years.
|
|
And update t/home2/README while we're at it.
Followup-to: 4ea3975dbed0a533 ("tests: setup_public_inboxes: use IMAP-friendly newsgroups")
|
|
The grok-pull-based tests in www_listing are incompatible with
Grokmirror v2 in two ways: the generated configuration format and the
expected exit codes. Update the tests to work with v2, and skip them
for earlier versions.
This was tested with the latest release of Grokmirror, v2.0.7. Note
that the "pull" and "fsck" sections are required even though they're
empty.
|
|
Make it clear that this skip is because grok-pull isn't available at
all because the next commit will add another skip for older versions
of Grokmirror.
|
|
|
|
This lets us make use of multiple cores on IMAP and Maildir
backed by SSD (or better) storage. This benefits IMAP stores
with high network latency, but may still penalize IMAP servers
with rotational storage.
|
|
This flexibility should save us some code down-the-line.
|
|
We can use this to ensure sharded work doesn't do unexpected
things if workers are added/removed. We currently don't
increase/decrease workers once a workqueue is started, but
non-lei code (-httpd/imapd) may start doing so.
This also fixes a bug where lei2mail workers could not
be adjusted via --jobs on the command-line.
|
|
This is a step which will allow us to parallelize augment
on Maildir and IMAP.
|
|
We'll give workqueues a broadcast mechanism to ensure all
workers see a certain message. We'll also tag each worker
with {-wq_worker_nr} in preparation for work distribution.
This is intended to avoid extra connection and fork() costs
from LeiAuth in a future commit.
|
|
Augment (and dedupe) aren't parallel, yet, so its more sensitive to
high-latency networks.
|
|
This wasn't causing known failures, but maybe it was or will in
the future.
|
|
PublicInbox::Config isn't loaded elsewhere by this file.
|
|
This interpolation is used by the upstream URI package
and we rely on it elsewhere for HTTP(S) URIs, so save
ourselves some surprises down the line.
|
|
Requiring TEST_IMAP_WRITE_URL to be set to a writable IMAP
server URL isn't ideal, but it works for now until we have time
to setup a mock dovecot/cyrus/etc... instance for testing.
|
|
Due to an off-by-one error, we were unable to read mailboxes
with only a single message of UID:1. Without this fix, the
message with UID:1 could only be read after UID:2 was created;
so there's no permanent data loss as long as a new message
showed up.
This affects all releases of public-inbox-watch with IMAP
support, though it probably went unnoticed because single
message inboxes are rare.
|
|
All of our current IMAP code relies on Mail::IMAPClient
at the moment, so ensure we skip those tests on systems
without that module.
|
|
We won't have _post_augment_imap when we add IMAP support,
either.
_pre_augment_imap will not exist, either, since opening an
IMAP(S) connection can be time consuming so we'll roll that
into imap_common_init.
|
|
It'll be less ambiguous for inputs with "lei convert" and "lei import"
cf. https://public-inbox.org/meta/20210217044032.GA17934@dcvr/
|
|
As with LeiToMail, we'll exclusively rely on O_EXCL and EEXIST
instead of "-f" (stat(2)) for file name collision checking.
Furthermore, we can rely on link(2) error handling instead of
using stat(2) to check the result of link(2).
We'll still keep the hostname in these filenames, but memoize it
on a per-instance basis since hostname changes are rare and we
can assume it won't change between "tmp" and "cur".
We'll also start embedding the PID as {"tmp.$$"} into the fiel
name to guard against accidental deletion in child processes,
instead of requiring an extra hash lookup.
Finally, avoid multiple getpid(2) syscalls in internal subs
since glibc no longer caches in getpid(3).
We'll also favor constant comparison of $! against EEXIST for
inlining. and stop doing ->autoflush when we only have a single
print + flush.
|
|
link(2) may fail with errors other than EEXIST; just bail out
since something is likely seriously wrong.
|
|
We need to ensure authentication failures and error codes get
propagated to the parent process(es) properly.
v2: update MANIFEST
v3: LeiAuth.pm ->_lei_cfg bit moved to a previous commit
|
|
The backends for "lei add-external --mirror", "lei convert", and
"lei import" all share a similar pattern for spawning background
workers. Hoist out the common parts to slim down our code base
a bit.
The LeiXSearch and LeiToMail workers for "lei q" remains a the
odd duck due to the deep pipelining and parallelization.
|
|
This makes "lei import" more similar to "lei convert" and
allows importing from disparate sources simultaneously.
We'll also fix some ->child_error usage errors and make
the style of the code more similar to the "lei convert"
code.
v2: fix missing requires
|
|
This will make testing IMAP support for other commands easier, as
it doesn't write to lei/store at all. Like the pager and MUA,
"git credential" is always spawned by script/lei (and not
lei-daemon) so it has a controlling terminal for password
prompts.
v2: fix missing requires, correct test ordering
v3: ensure config exists for IMAP auth
|
|
We'll be completing more options with ':', '//' and '=' in the
future, so make it easier to disable trailing spaces on
completions.
|
|
{zpipe} is contained entirely within the $l2m object, now.
|
|
-imapd won't support newsgroups ending with /\.[0-9]+\z/ since
it reserves those for partitioning inboxes into 50K slices.
So bump the home[0-9]+ version and switch to IMAP-friendly
newsgroup names.
|
|
We'll be supporting "lei convert" in a future change; so it
makes sense to share a common internal API for common error
messages.
|
|
More to come in a later commit; some error handling and failure
modes will be trickier with IMAP due to authentication.
|
|
This is hopefully less surprising to users when they're prompted
for credentials.
|
|
We'll use this in LeiImport and likely other places.
|
|
We'll be needing ->url_match from PublicInbox::Config
|