| Date | Commit message (Collapse) |
|---|
|
Obfuscation has been available since v1.0.0. Help those that want to
use the feature figure out how.
|
|
Going forward, we'll probably support JSON for all the "ls-*"
subcommands. This also provides the basis for "lei up" shell
completion.
|
|
We'll keep it in PublicInbox::Config for now, since I'm not
sure if there's a better place to put it.
|
|
Repeated vfork+execv costing us around 20ms on t/lei-q-save.t,
so just learn to quote git-config values and write directly to
the config file.
|
|
We want to be able to use "lei up ." when inside a Maildir.
We'll also relax Maildir/mbox basenames to be any non-'/'
character after converting relative paths to absolute. The
old restriction on allowed characters was unnecessary and made
it impossible to reliably map "." when used as the sole argument
for "lei up".
|
|
We always represent --output destination directories with a
trailing slash to disambiguate directories from mbox filenames.
Therefore, we must use the trailing slash when mapping the
destination beck from the lei/saved-search/* directory.
"lei up" now relies exclusively on the users --output pathname
or URL for updates. This ought to be less confusing since
pathnames in ~/.local/store/lei/saved-searches aren't ideal.
|
|
Since "lei q" may read queries from stdin, we must reconnect a
known terminal before spawning terminal MUAs. Attempt to use
stdout as stdin for this purpose, since terminal MUAs tend to
expect stdout to be a terminal.
Reported-By: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87v98klxg3.fsf@kyleam.com/
|
|
A user may wish to clobber/refine existing search parameters
by issuing "lei q --save" again. Support that by overwriting
the lei.saved-search state file entirely.
We continue to preserve over.sqlite3 for deduplication purposes.
This way, we don't get something redundant like:
[lei]
q = term1
q = term2
q = term1
q = term2
q = term3
...whenever a user wants to refine their search. Instead,
we'll just have:
[lei]
q = term1
q = term2
q = term3
On the second go.
|
|
It is redundant since we stuff everything into the lei.q.output
config key.
|
|
Specifying a directory in ~/.local/share/lei/saved-searches/
is painful, so support (and start encouraging) the use of
the output.
|
|
We don't want pathnames with "GLOB(0xADD12355)" in them.
|
|
Since --stdin could be waiting on user keyboard input or
something else slow, we handle it in the event loop. That
means other commands can change the working directory of
lei-daemon while a query is being trickled to us via stdin.
Rearranging query handling internals to delay opening the
--output destination in commit 26e0fe73de93f451 meant
another command could throw off our --output pathname if
it is relative.
Fixes: 26e0fe73de93f451 ("lei_query: rearrange internals to capture query early")
|
|
We use it in t/lei-q-save.t, and were inadvertently writing
to the worktree.
v2: fix -C $DIR with TEST_RUN_MODE=0
|
|
NetReader->add_url supports URI-like objects, now. We'll be
relying on the canonicalization for LeiSavedSearch.
|
|
We want users to be able to edit and refine the query over
time while using the same output destination.
|
|
Since saved-searches aren't a part of lei/store, nor
could it be considered cache data... (or can it? it
is discardable, after all).
|
|
Somebody may want a saved search which consistently asks for
messages within a rolling time period window. In other words,
we want to support using "lei q --save dt:last.week.." and keeps
the "dt:last.week.." relative to whenever "lei up" is run. This
ensures relative date-time specifications get used in the future
rather than converting into an absolute date-time from the
initial "lei q" invocation.
|
|
If a user specifies "d:" with a higher precision than it was
traditionally able to handle, switch transparently to "dt:".
This lowers the learning curve and improves DWIM-ness.
v2: fix "d:YYYYMMDD..$NEEDS_APPROXIDATE" case
|
|
It looks like stability and compatibility will prevail, after all.
|
|
The command isn't finalized, yet, but it's intended to update
an existing saved search.
|
|
This will have a over.sqlite3 for content-based deduplication.
It may exhibit ibxish methods, so serving a read-only (or even
R/W) IMAP or instance or displaying HTML isn't outside the realm
of possibility.
|
|
To support saved search, we need the query string available to
us before we setup LeiDedupe via (LeiOverview || LeiToMail).
|
|
LeiSavedSearch will use a LeiDedupe-like internal API,
so we won't have to make as many changes to callsites
between saved and unsaved searches.
|
|
We only need the combined mset query when we care about sort
order. When writing to --output destinations intended for MUA
consumption, sort order is irrelevant as MUAs are expected to
offer their own sorting, so run queries to each external in
parallel.
This prepares us for docid-sort-based saved search support.
It will also become faster than the combined mset query for
users with many externals due to current Xapian exhibiting poor
performance with many shards (the same reason -extindex exists)
|
|
This seemed to be causing occasional "make check-run" failures
with errors bleeding into other tests.
|
|
As they are likely Message-IDs. If an email address ends up in
a URL, then it's likely public, so there's even less reason to
obfuscate that particular address.
[km: add xt/perf-obfuscate.t]
[ew: modernize perf test (5.10.1), use diag instead of print]
This version of the patch avoids the massive slowdown noted by Kyle in
<https://public-inbox.org/meta/87wnt9or6t.fsf@kyleam.com/>.
Performance remains roughly the same, if not slightly faster
(which may be due to me testing this on a busy server). Results
from xt/perf-obfuscate.t against 6078 messages on a local mirror
of <https://public-inbox.org/meta/>:
before: 6.67 usr + 0.04 sys = 6.71 CPU
after: 6.64 usr + 0.04 sys = 6.68 CPU
Reported-by: Kyle Meyer <kyle@kyleam.com>
Helped-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87a6q8p5qa.fsf@kyleam.com/
|
|
init.defaultBranch expects a branch name, not a fully qualified ref.
git-init prepends "refs/heads/" automatically and unconditionally.
PublicInbox::Import::default_branch, however, incorrectly passes on
the init.defaultBranch value as is, leading to it being used in spots
where a fully qualified ref is required. For example, with an
init.defaultBranch value of "master", public-inbox-index for a v2
repository would lead to an all.git repository where HEAD's content is
"ref: master" instead of "ref: refs/heads/master".
Prepend "refs/heads/" to the incoming init.defaultBranch value.
Fixes: 7c2f36de2fb49dd7 (import: respect init.defaultBranch)
|
|
It's nicer in case a user transfers lei/store across machines
and wants a way to track when/where they imported something.
|
|
We need to ensure we reap things we spawn.
|
|
IMAP authentication info is only shared amongst lei2mail workers,
so we must ensure all IMAP writes go through lei2mail workers
even if we don't have to access the mail through git.
This allows us to decouple the latency of the remote mboxrd from
the latency of the IMAP --output at the expense of extra IPC
overhead within our own processes.
|
|
We don't need to waste LoC on corner cases, single-use internal
subs, or restoring SIG{__WARN__} when a process exits. All that
extra code contributes to memory use and startup time, especially
for users who can't use FD passing.
|
|
We'll eventually want lei_input users like "lei import" and
"lei tag" to support parallel reads.
|
|
RFC 8621 registers $flagged, $answered, $seen, $draft which
map to IMAP, Maildir, and mbox Status/X-Status flags.
$forwarded is noted in JMAP, but only Maildir and and the
"Lemonade" IMAP profile (RFC 5550) support it
|
|
We don't need to import so many things. None of the Errno
constants are in common paths so unlikely to benefit from
constant folding.
|
|
It currently conflicts with the way OverIdx and SearchIdx
index messages, ultimately leading to violating a NOT NULL
constraint on id2num.id in over.sqlite3.
We may allow searching Resent-* fields separately, though I'm
not sure how useful it'll be.
|
|
Since every command that writes to lei/store calls ->done
to commit its output, we can rely on that to return a
pathname for a readable file with errors in it.
Errors can still get crossed up if multiple lei commands
are writing to the store at once, but reduces the delay
in seeing them and ensures it won't get seen when somebody
is attempting to use shell completion.
|
|
We need a stable fallback time for digest2mid in the presence
of messages without Received/Date headers. Furthermore, we
must avoid using uninitialized smsg->{mid} when parsing
References for draft replies.
|
|
Just exiting with a failure code and no error message is
confusing :x
|
|
$? is useful, as is labeling lei_err since I'm easily-confused :x
|
|
We'll just let the ExtSearchIdx code handle this uncommon case
by doing a full commit.
|
|
Sometimes I want to save debug info to a file or pipe even when
spawning an MUA.
|
|
Remote results can safely use the same mset progress reporting
as local results, despite not knowing the size of the result
set. We're assuming terminal MUAs, for now.
|
|
Because failures are often overlooked, unfortunately.
|
|
"convert" is actually a bit more complicated than "lei import"
since it may need auth for either input or output.
|
|
No reason for the hash key to differ from the subroutine
name, here.
|
|
We need net_merge_all and to lock the number of worker jobs.
Parallel inputs are not supported, yet (is it needed?, I don't
expect this to be used for multiple files very often...).
|
|
Leaving workers running on after auth failures is bad and messy,
cleanup our process management to have consistent worker
teardowns. Improve error reporting, too, instead of letting
Mail::IMAPClient->exists fail due to undef.
|
|
I wanted to say 2031, but that's probably too aggressive a
removal timeline.
|
|
I completely forgot about git-credential prompting when
making lei background the client process for MUA.
Now it backgrounds itself only for the MUA when no FDs are
passed, since the MUA is the final command run. Otherwise, it
relies on FD passing as before.
Fixes: c790a75439f3a1db ("script/lei: background ourselves on MUA/pager exec")
|
|
mbox and IMAP seem to have no way of describing this keyword.
but Maildir does with the "P" flagged (for "passed").
|