| Date | Commit message (Collapse) |
|---|
|
Outside of mirrors, we can -init with indexlevel to avoid
calling -index explicitly.
|
|
While we were exiting with a error code, showing a successful
"# mirrored $URL" message is misleading and wrong. Don't show
success until everything is complete and the config is written.
|
|
There's no need to create or write lei/store when adding
an external, we just need to write to the config file.
|
|
We hide paths by default to reduce noise, but we want
noise with loud errors.
|
|
We need to rely on this to keep our config cache (and lei_store
pipes) under control with tests each creating a new config and
directory.
|
|
This stops us from leaking some more file handles across
test cases.
|
|
If a client passes us the incorrect number of FDs, we'll vivify
them into PerlIO objects so they can be auto-closed. Using
POSIX::close was considered, but it would've been more code to
handle an uncommon case.
|
|
All of our $lei->workers_start callers can simply rely on
that wrapper to do the right thing and pass fields to
->wq_worker_start children, only.
This could manifest as a unbound memory growth if somebody is
constantly mirroring, and was causing tests to get stuck when
experimenting with a persistent lei-daemon for the entire
test suite.
|
|
I'm not sure exactly why this is needed with run_script
localizing %SIG and everything else, but explictly cleaning up
seems to fix the occasional test failures I see.
Followup-to: 4c6c853494b49368 ("tests: show lsof output on deleted-file-check failures")
|
|
We'll prioritize the last two components of the path name
("lei/store") since that's how I often refer to the on-disk
location. Then, show the XDG_DATA_HOME it belongs to in case
a user changes HOME or XDG_* for testing purposes.
|
|
We need to consistently ensure pkt_op_c doesn't lead to a
long-lived circular reference if an exception is thrown in
pre_augment. Maybe the API could be better, but this fixes an
FD leak when attempting to --augment a FIFO.
Followup-to: b9524082ba39e665 ("lei_xsearch: cleanup {pkt_op_p} on exceptions")
|
|
This is necessary for lei->rel2abs correctness, and may
eventually be useful if we can use *at syscalls.
|
|
I'm not sure if this was causing real problems, but it's sure ugly.
|
|
Since lei-daemon will fchdir on every request, we must ensure
we're in the correct directory before unlink(2) is called,
since we can't use unlinkat(2) from pure Perl.
|
|
None of these fixes affect current public-inbox-* code, or even
normal uses of lei. However, lei users wanting to switch
between $HOME directories or use alternate store paths may
notice strange behavior and this fixes some of it.
We'll also loop to account for DESTROY callbacks inserting into
container objects and retry appropriately.
|
|
Instead of creating a short-lived circular reference,
ensure they don't exist in the first place.
Note the following changes to hold an extra ref to $sto:
- $self->_lei_store(1)->write_prepare($self);
+ my $sto = $self->_lei_store(1);
+ $sto->write_prepare($self);
I'm not a perlguts expert, but I actually wanted to switch
to the one-line version for LeiImport, but xt/lei-auth-fail.t
was getting stuck for some reason. It seems the extra ref
to the LeiStore ($sto) object is necessary.
|
|
"lei convert" is actually a bit of the odd one, since
it uses lei2mail for auth, unlike the others.
|
|
Since each lei->event_step can change the directory of
lei-daemon, we need to ensure the lei_store runs in a
directory that is stable.
|
|
This will make it easier to diagnose some large internal
rewrites.
|
|
We'll use `undef' to denote keywords are unknown/unsupported,
instead of an empty arrayref.
This will let callers use the same callback and args for
imap_each. Passing an empty arrayref to set_eml in LeiStore
causes keywords to be cleared completely, which is not desired
behavior when "lei import" is importing already-seen messages
from NNTP.
|
|
Otherwise we could get non-sensical results if somebody tries
running "lei atfork_child" from the command-line.
|
|
Only lightly tested, this seems to suffer from the same
problem as external completions for network URLs with
colons in them. In any case, its usable enough for me.
The core LEI module now supports completions for lazy-loaded
commands, too, so we'll be able to do completions for other
commands more easily.
|
|
Only tested for keywords and labels with file inputs, so far;
but it seems to do what it needs to do. There's a bit more
redundant code than I'd like, and more opportunities for code
sharing in the future
"lei import" will be expanded to support +kw:$KEYWORD and
+L:$LABEL in the future.
|
|
Those headers only have meaning with for mboxes. Don't surprise
users by trying to make sense of a header that is defined for mboxes.
It's possible to send email with (Status|X-Status) headers and
have those headers show up in a recipient's IMAP mailbox.
This was bad because an IMAP user may want to import a single
message through their MUA and pipe its contents to "lei import"
without noticing a mischievious sender stuck "X-Status: F"
(flagged/important) in there.
|
|
This matches the long-standing behavior of public-inbox-mda,
public-inbox-learn and our other tools. It is useful because
mutt, "git format-patch", and likely other tools will
pipe a single message with a "From " header line, but with
no further "From " escaping or Content-Length: header.
|
|
This improve code regularity, and will let us deal with
the "RFC822" messages with "From " line that mutt pipes
to.
|
|
Relying on UNIVERSAL::can may cause internal helper methods
to be used, which can lead to failures or nonsensical results.
|
|
Since workers_start is in the common PublicInbox::LEI
package, we can just use \&METHOD_NAME instead of relying
on UNIVERSAL->can to avoid a method dispatch.
Most of our worker code can just use lei->dclose, so default
to doing that unless it's been overridden.
|
|
These commands accept mail the same way, and this forces
us to maintain consistent input format support between
commands.
We'll be using this for "lei mark", too.
|
|
Net::Cmd::message (used by Net::NNTP) does no escaping at all,
so "\r" was causing confusing/nonsensical error messages when
I tried to import from the wrong group.
|
|
It's a bit nasty, but seems to mostly work for debugging
IMAP and NNTP commands.
|
|
This makes it slightly easier to implement future commands,
since there'll be a couple more relatively self-contained
ones.
|
|
And then test the contents of $lei_err to ensure it doesn't
happen again.
We'll also make MboxLock emit nicer warnings without the line
number, since the line number is irrelevant to the user fixing
an mbox lock contention problem.
Finally, we'll also allow showing loud warnings via
TEST_LEI_ERR_LOUD=1
|
|
This will let us tie keywords from remote externals
to those which only exist in local externals.
|
|
Keyword storage for external-only messages was preventing
messages from being explicitly imported. Teach lei_store
to vivify keyword-only entries into fully-indexed messages
on import.
|
|
There's no point in showing duplicate links to the same
Message-ID in summary view. The per-message page will
note the duplication (if any) separately.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210317132723.xx4klonordhsb6ve@chatter.i7.local/
|
|
Stop showing `docid' since it's not useful with shards.
`bytes' and `lines' are probably noise, but maybe could be
visible in some "fuller" view.
v2: t/lei_xsearch: fix warnings from {docid} removal
|
|
This should make a future change to "lei import" work more
nicely, since we'll be needing ALE to vivify external-only
messages upon explicit "lei import".
|
|
These changes may make it easier to do byte-for-byte comparisons
with mail copied out of mutt, a popular MUA for our target
audience.
mutt currently outputs the 'R' (seen) flag before the 'O'
character in the Status: header. We'll assume that stays
the case (it has been for a while).
Status now comes before X-Status, also matching mutt behavior.
|
|
Don't waste precious terminal space when there are only a small
number of possible keywords supported/reserved for JMAP. In the
future, we may implement more sophisticated wrapping for labels,
but it we'll cross tha bridge when we come to it.
|
|
"lei q" now preserves changes per-message keywords across
invocations when it's --output (Maildir or mbox) is reused
(with or without --augment).
In the future, these changes will be monitored via inotify,
EVFILT_VNODE or IMAP IDLE, too.
Unfortunately, this currently prevents "lei import" from ever
importing a message that's in an external. That will be fixed
in a future change.
|
|
This will be used for keyword (and label) storage for externals.
We'll be using this to ensure we don't redundantly auto-import
messages into lei/store if they're already in a local external
(they can still be imported explicitly via "lei import").
|
|
This was causing errors in a mass keyword import patch
I'm working on.
|
|
Redundant slashes look ugly in strace(1) output.
|
|
Assume that anybody using -I/--include for external locations
will want to override --no-$FOO if they're explicitly including
a location.
With some effort, we could make it order-dependent (e.g.
"-I $LOCATION --no-$FOO" and "--no-$FOO -I $LOCATION"
behave differently). However that's not straightforward
when using Getopt::Long to parse command-line options into
a hashref.
I'm also not sure if order-dependent switches are a desirable
UI/UX quality.
|
|
git 2.11 and earlier could not handle git directories with
newlines in them, nor does libgit2 support them.
Followup-to: d87dd0e679587043 ("config: reject `\n' in `inboxdir'")
|
|
git 2.11 and earlier could not handle git directories with
newlines in them, nor does libgit2 support them.
Followup-to: d87dd0e679587043 ("config: reject `\n' in `inboxdir'")
|
|
detect_nproc is in the IPC module, now; and we can safely
disable fsync when creating test data.
And "modernize" up to 5.10.1 while we're at it.
The use fsync was causing this to run for hours instead
of minutes since I forgot to use eatmydata.
|
|
Oops :x
|
|
Nothing like the -Wunused C compiler flag in perl, AFAIK...
|