| Date | Commit message (Collapse) |
|---|
|
We can share more code amongst stdin slurper (not streaming)
commands. This also fixes uninitialized variable warnings when
feeding an empty stdin to these commands.
|
|
This will make switching $lei contexts less error-prone
and hopefully save us from some suprising bugs in the future.
Followup-to: 759885e60e59 (lei: ensure --stdin sets %ENV and $current_lei, 2023-09-14)
|
|
--stdin usage means the current request can be delayed
indefinitely while other requests with different %ENV
come in. So make sure our warnings and %ENV can match
non-stdin behavior.
This probably fix segfaults during process cleanup on OpenBSD
since _lei_atfork_child use non-localized assignment of
$current_lei. But it could be another red herring. Either way,
it's the right thing to do from an environment replication
perspective.
|
|
Due to historic reasons, LeiQuery.pm gets loaded with LEI.pm and
-clone depends on LEI. So delay loading any DBI-dependent
modules until querying is actually required.
|
|
This fixes completions of labels (`+L:' for `lei import' and
`L:' for `lei q') so they can appear anywhere in the
command-line.
I mainly wanted this for `lei import $URL +L:label', but
this also fixes `lei forget-external' completions for URLs
(which involve colons).
|
|
While users may specify relative paths for convenience on the
command-line, absolute paths are required for `lei up' since
that (especially `lei up --all') could run from anywhere.
Note that we need to do this when parsing the command-line
options, since shortcuts for URL matching on URL path components
are allowed for `lei q', and those same shortcuts may remain
in effect across to `lei up' as the underlying external may
be moved to a different URI host.
|
|
Eric Wong <e@80x24.org> wrote:
> Thanks for confirming things work as intended. I think the
> default should be clamped, though... 15 seems a bit high for
> smaller IMAP servers *shrug*
--------8<-------
Subject: [PATCH] lei q|up: limit default write --jobs for IMAP(S)
IMAP(S) servers often limit per-user connections, so avoid
bumping into limits to improve the out-of-the-box experience.
4 seems like a conservative default, since we already chose
that number for remote HTTP(S) endpoints.
Link: https://public-inbox.org/meta/20220910201958.GA12212@dcvr/
|
|
SQLite prior to 3.8.3 did not reset its PRNG for generating
unique temporary file names, so it would barf on t/lei-up.t
occasionally due to O_EXCL -> EEXIST conflicts.
This fixes occasional test failures under CentOS 7.x which ships
SQLite 3.7.17.
|
|
This was pointed out by the Debian package linter "lintian".
|
|
I don't expect this to be hit in real-world use via normal
interactive shells. However, somebody could accidentally add
"\n" in languages (e.g. Perl, C) where it's easy to pass "\n"
in argv[].
|
|
"text" and "reply" outputs are intended for the pager, so
parallelizing them is a waste of resources.
v2 has shards, of course, so parallelizing writes to it
is also a waste since the deduplication work is a bit
more complex.
|
|
No need to go through the lei/store process when we write
mail_sync.sqlite3. This ought to reduce ENOBUFS errors (and the
sleep workaround) on RAM-starved systems.
|
|
These can be used to temporarily disable using certain
externals in case of temporary network failure or mount point
unavailability.
|
|
This allows "lei up" to continue processing unrelated externals
if on output fails.
|
|
Relying on $lei->fail is unsustainable since there'll always
be parts of our code and dependencies which can trigger die()
and break the event loop.
|
|
This was written before we had auto-loading, and forget-external
should be a rarely-used command that's not worth loading at
startup. Do some golfing while we're in the area, too.
|
|
Avoid slurping gigantic (e.g. 100000) result sets into a single
response if a giant limit is specified, and instead use 10000
as a window for the mset with a given offset. We'll also warn
and hint towards about the --limit= switch when the estimated
result set is larger than the default limit.
|
|
Just a typo.
|
|
PublicInbox::Import never imports @UNWANTED_HEADERS, so ensure
our mock blob OIDs do the same. This ought to prevent
duplicates if the PSGI mboxrd download starts setting
"X-Status: F" like "lei q -tt .."
|
|
As documented, File::Spec->canonpath does not canonicalize
"/../". While we want to do our best to preserve symlinks in
pathnames, leaving "/../" can mislead our inotify|kqueue usage.
|
|
Saved searches rely on (reverse) docid ordering for efficient
incremental results, and sorting any other way prevents that.
Update comment description in LeiQuery while we're at it:
"ls-query" and "rm-query" are "ls-search" and "forget-search",
respectively, and "mv-query" is implicit with "edit-search"
|
|
Every tick of the event loop can change the working directory,
so we need to restore it for every client if they operate
in different directories.
This would be easier if we had openat(2) and friends in Perl;
but Inline::C is practically required for lei, now.
|
|
This will allow keyword updates from other folders to propagate
to folders where search results may be duplicated.
|
|
POSIX.pm shipped with Perl 5.16.3 did not support lround,
at least. So just rely on built-in core functions.
|
|
Users may wish to update several saved searches at once. We can
support parallel updates in lei-daemon so users won't have to do
it themselves via xargs or similar.
Supporting IMAP outputs would be significantly more involved
since we'd have to pre-authenticate for every single IMAP
output before entering the redispatch loop.
|
|
Since --stdin could be waiting on user keyboard input or
something else slow, we handle it in the event loop. That
means other commands can change the working directory of
lei-daemon while a query is being trickled to us via stdin.
Rearranging query handling internals to delay opening the
--output destination in commit 26e0fe73de93f451 meant
another command could throw off our --output pathname if
it is relative.
Fixes: 26e0fe73de93f451 ("lei_query: rearrange internals to capture query early")
|
|
Somebody may want a saved search which consistently asks for
messages within a rolling time period window. In other words,
we want to support using "lei q --save dt:last.week.." and keeps
the "dt:last.week.." relative to whenever "lei up" is run. This
ensures relative date-time specifications get used in the future
rather than converting into an absolute date-time from the
initial "lei q" invocation.
|
|
The command isn't finalized, yet, but it's intended to update
an existing saved search.
|
|
This will have a over.sqlite3 for content-based deduplication.
It may exhibit ibxish methods, so serving a read-only (or even
R/W) IMAP or instance or displaying HTML isn't outside the realm
of possibility.
|
|
To support saved search, we need the query string available to
us before we setup LeiDedupe via (LeiOverview || LeiToMail).
|
|
AFAIK that was only used for nproc detection, and nproc
is handled by PublicInbox::IPC, nowadays.
|
|
No point in munging user-supplied $lei->{opt} when %mset_opt
exists. We'll be depending on docid being in descending order
for saved search support.
|
|
We'll be reusing it for "lei blob", as it makes sense
to keep handling of --only, --include, etc. switches
consistent.
|
|
JSON outputs won't write to lei/store at all, so there's
no point in forking the store worker if it's not already
running.
LeiSearch object ($lse) is also fork-safe until it opens a
persistent FD for Xapian/SQLite so we can unconditionally
carry it across fork.
|
|
Instead of creating a short-lived circular reference,
ensure they don't exist in the first place.
Note the following changes to hold an extra ref to $sto:
- $self->_lei_store(1)->write_prepare($self);
+ my $sto = $self->_lei_store(1);
+ $sto->write_prepare($self);
I'm not a perlguts expert, but I actually wanted to switch
to the one-line version for LeiImport, but xt/lei-auth-fail.t
was getting stuck for some reason. It seems the extra ref
to the LeiStore ($sto) object is necessary.
|
|
This should make a future change to "lei import" work more
nicely, since we'll be needing ALE to vivify external-only
messages upon explicit "lei import".
|
|
This will be used for keyword (and label) storage for externals.
We'll be using this to ensure we don't redundantly auto-import
messages into lei/store if they're already in a local external
(they can still be imported explicitly via "lei import").
|
|
Assume that anybody using -I/--include for external locations
will want to override --no-$FOO if they're explicitly including
a location.
With some effort, we could make it order-dependent (e.g.
"-I $LOCATION --no-$FOO" and "--no-$FOO -I $LOCATION"
behave differently). However that's not straightforward
when using Getopt::Long to parse command-line options into
a hashref.
I'm also not sure if order-dependent switches are a desirable
UI/UX quality.
|
|
commit 6c551bffd75afb41d9b5e4774068abe7e06ed0e7
("lei q: --import-augment for mbox and mbox.gz") added a check to
in _pre_augment_mbox for the option being a ref() to distinguish
between default values and user-supplied values (which are
non-ref SCALARs from Getopt::Long).
However, LeiQuery failed to use a SCALAR ref as the default
value, making the check in _pre_augment_mbox useless. We
now update LeiQuery to use \1 instead of 1 as the default
value so "lei q -f mboxrd ..." to stdout works once again.
Unfortunately, testing with redirects pointed to regular
files didn't trigger the code paths being updated. Testing
with a FIFO revealed further bugs in the FIFO handling code
which are also fixed in this commit.
We'll also update the $lei->out error message to be
less-specific about "stdout" and use the term "output", instead,
since LeiToMail replaces stdout for all mbox outputs.
|
|
Since this importing of keywords is active even when --augment
isn't specified, calling it --import-before seems more
appropriate.
In the future, this will likely default to adding unseen emails
to lei/store, not just updating keywords.
Link: https://public-inbox.org/meta/20210303222930.GA18597@dcvr/T/
|
|
This will eventually be supported for other mail stores,
but Maildir is the easiest to test and support, here.
This lets us avoid a situation where flag changes get
lost between search results.
|
|
This lets users avoid network traffic on subsequent searches at
the expense of local disk space. --no-import-remote may be
specified to reverse this trade-off for users with little
storage.
|
|
While disk I/O is typically buffered for good scheduling,
git blob decoding uses a non-trivial amount of CPU time
and it helps to leave some CPU available for it.
|
|
LeiAuth is no longer a separate worker process. Instead, it's
used directly by LeiToMail and LeiImport for sharing auth info
from the first worker to the rest of the workers, using
lei-daemon as a message router. So drop the old code to reduce
human cognitive load and interpreter memory overhead.
|
|
We can rework the first lei2mail worker to authenticate, and
then share auth info with the rest of the lei2mail workers. As
with "lei import", this uses PktOp and lei-daemon to share
updated credentials between the first an subsequent l2m workers.
|
|
We can use this to ensure sharded work doesn't do unexpected
things if workers are added/removed. We currently don't
increase/decrease workers once a workqueue is started, but
non-lei code (-httpd/imapd) may start doing so.
This also fixes a bug where lei2mail workers could not
be adjusted via --jobs on the command-line.
|
|
Augment (and dedupe) aren't parallel, yet, so its more sensitive to
high-latency networks.
|
|
This greatly improves the usability of d:, dt:, and rt: search
prefixes for users already familiar git's "approxidate" feature.
That is, users familiar with the --(since|after|until|before)=
options in git-log(1) and similar commands will be able to use
those dates in the WWW UI.
|
|
Instead of having --(sent|received)-(before|after)=s
command-line switches, we'll just try to make sense of argv so
it's usable within parenthesized statements and such.
Given the negligible performance penalty with Inline::C
process spawning, we'll probably wire this up to the
WWW interface, too.
"d:" is for mairix compatibility. I don't know if "dt:" and
"rt:" will be too useful, but they exist because of IMAP
(and JMAP).
|
|
Nobody is expected to use long options, but for consistency
with mairix(1), we'll use the pluralized option throughout
(including existing PublicInbox::{Search,SearchView}).
Link: https://public-inbox.org/meta/20210206090119.GA14519@dcvr/
|