Date | Commit message (Collapse) |
|
This fixes t/lei-q-save.t getting stuck since $self->{ale} is
already gone by the time DESTROY gets called.
|
|
I have no idea how badly my brain malfunctioned here when I
wrote the code to create a temporary file without O_EXCL :x
I'm still not sure if users have enough saved searches for
justifying a cache, here.
|
|
On my x86-64 machine, OpenSSL SHA-256 is nearly twice as fast as
the Digest::SHA implementation from Perl, most likely due to an
optimized assembly implementation. SHA-1 is a few percent
faster, too.
|
|
It's a needless wrapper, nowadays. Originally, ->over was added
on experimental basis to optimize for /$INBOX/ where Xapian
->search is slower on gigantic (LKML-sized) inboxes.
Nowadays with extindex, ->over is here to stay given NNTP and
IMAP both benefit from it. So reduce the interpreter stack
overhead and just access ->over directly.
lxs->recent was never used outside of tests, anyways.
And while we're in the area, avoid needlessly bumping the
refcount of $ctx->{ibx} in View::paginate_recent.
|
|
`"' (double-quote) needs to be quoted for stdin searches.
We also need to differentiate between "lei q --stdin" usage
when calling "lei up", do it by setting an internal "rawstr"
knob to ensure we can parse the config properly regardless
of whether the initial search used --stdin or not.
|
|
This allows "lei up" to continue processing unrelated externals
if on output fails.
|
|
Avoid slurping gigantic (e.g. 100000) result sets into a single
response if a giant limit is specified, and instead use 10000
as a window for the mset with a given offset. We'll also warn
and hint towards about the --limit= switch when the estimated
result set is larger than the default limit.
|
|
I wanted to try --dedupe=none for something, but it failed
since I forgot --no-save :x So hint users towards --no-save
if necessary.
|
|
As with "lei edit-search", "lei config --edit" may
spawn an interactive editor which works best from
the terminal running script/lei.
So implement LeiConfig as a superclass of LeiEditSearch
so the two commands can share the same verification
hooks and retry logic.
|
|
Since we can't use maxuid for remote externals, automatically
maintaining the last time we got results and appending a dt:
range to the query will prevent HTTP(S) responses from getting
too big.
We could be using "rt:", but no stable release of public-inbox
supports it, yet, so we'll use dt:, instead.
By default, there's a two day fudge factor to account for MTA
downtime and delays; which is hopefully enough. The fudge
factor may be changed per-invocation with the
--remote-fudge-factor=INTERVAL option
Since different externals can have different message transport
routes, "lastresult" entries are stored on a per-external basis.
|
|
We failed to account for IMAP mailboxes containing `/'
characters when creating saved search files for them.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210915123347.knr4qpaei73tjc5q@meerkat.local/
|
|
lei shouldn't become unusable if a config file is invalid.
Instead, show the "git config" stderr and attempt to continue
gracefully.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210910141157.6u5adehpx7wftkor@meerkat.local/
|
|
Since "lei up" is more often useful than not and incurs neglible
overhead; enable --save by default and allow --no-save to work.
This also fixes a long-standing when overwriting --output
destinations with saved searches: dedupe data from previous
searches are reset and no longer influences the new (changed)
search, so results no longer go missing if two sequential
invocations of "lei q --save" point to the same --output.
|
|
Storing relative paths with '..' in them can be expensive to
resolve when running 'lei up', so prefer storing canonicalized
absolute paths. We only do this for paths with '..' in them,
though, since this can lose symlink info.
|
|
SQLite COUNT() is a slow operation that does a full table scan
with no conditions. There's no need for it, since lei dedupe
only needs to know if it's empty or not to decide between
new/ and cur/ for Maildir outputs.
|
|
This allows "lei-managed pseudo mailing lists" as described
by Konstantin.
Alternates use is optional and can be enables via --shared.
This doesn't manage or edit ~/.public-inbox/config; presumably
there'll need to be some tweaking of search parameters before
finalizing and making the inbox publicly accessible via HTTP/NNTP.
Link: https://public-inbox.org/meta/20210426164454.5zd5kgugfhfwfkpo@nitro.local/T/
|
|
Despite JMAP not supporting the equivalent of the IMAP \Recent
flag, it is useful for "lei q --augment", and "lei up" users to
be able to distinguish new results from old-but-unread messages
in an mbox or Maildir.
For mbox family messages, we'll drop the "O" status flag when
appending to mboxes, and we'll write to the "new" subdirectory
of Maildirs.
Behavior when writing to initially empty Maildirs and mboxes
remains unchanged since there's no need to distinguish between
new and old results in the initial case. Having users wait
for a rename(2) storm or complete mbox rewrite hurts UX.
With IMAP mailboxes, \Recent is already enforced by the IMAP
server and IMAP clients have no way of changing it(*)
(*) mutt uses the "Old" IMAP flag which isn't part of RFC 3501,
other MUAs may do similar things.
|
|
We use trailing slashes internally, but should not increase
visual noise for users by exposing them in config files or
DB storage (and shell completion/listings).
This fixes a long-standing bug in $lei->rel2abs that prevented
absolute paths from being canonicalized.
|
|
LeiToMail Maildir and IMAP write callbacks need to account for
the caller-supplied smsg. We'll also make better use of the
user-supplied smsg object by ensuring blob deduplication happens
ASAP.
Fixes: e76683309ca4f254 ("lei <q|up>: distinguish between mset and l2m counts")
|
|
The contents of the old lei.q.output will not be removed,
but will be converted into the new one.
|
|
This was harmless, but ugly and possibly confusing to
users who run "lei edit-search".
|
|
Use a separate git process when calling xoids_for to prevent
reentrancy in ->is_dup. Reentrancy happens since LeiToMail will
call ->is_dup when inside callbacks when writing mail.
This fixes --dedupe=mid test failures in t/lei-q-save.t
I could only reproduce this consistently on a uniprocessor VM.
"schedtool -a 0x1 -e ..." could not reproduce the problem on
2 and 4-core systems.
|
|
On my default FreeBSD 11.x system, "/home" is a symlink to
"/usr/home", which causes "lei up" path resolution to fail when
I use outputs in $HOME. Fall back to a slow path of globbing
and matching pathnames based on st_ino+st_dev.
|
|
This is less surprising in case users are used to using --dedupe=
without --save.
|
|
This necessitated fixing pause_dedupe to release the handle
used by ->lock_for_scope_fast, but otherwise no changes to
the LeiToMail package.
|
|
They're more different than alike, and having two separate
methods seems less confusing to me.
|
|
I don't know if it's worth it to sub (or super)class
PublicInbox::Config into something more generic for
lei, but this change simplifies a good chunk of lei
code that reuses the public-inbox config parsing.
|
|
While perl (5.28) doesn't complain about this, it's confusing to
my easily-confused mind.
|
|
It's conceivable a user will want to erase all previous
results but still rerun/refresh a search to get new results.
We probably won't support prune functionality, here, and
instead require explicit removal of saved searches.
|
|
Going forward, we'll probably support JSON for all the "ls-*"
subcommands. This also provides the basis for "lei up" shell
completion.
|
|
We'll keep it in PublicInbox::Config for now, since I'm not
sure if there's a better place to put it.
|
|
Repeated vfork+execv costing us around 20ms on t/lei-q-save.t,
so just learn to quote git-config values and write directly to
the config file.
|
|
We want to be able to use "lei up ." when inside a Maildir.
We'll also relax Maildir/mbox basenames to be any non-'/'
character after converting relative paths to absolute. The
old restriction on allowed characters was unnecessary and made
it impossible to reliably map "." when used as the sole argument
for "lei up".
|
|
We always represent --output destination directories with a
trailing slash to disambiguate directories from mbox filenames.
Therefore, we must use the trailing slash when mapping the
destination beck from the lei/saved-search/* directory.
"lei up" now relies exclusively on the users --output pathname
or URL for updates. This ought to be less confusing since
pathnames in ~/.local/store/lei/saved-searches aren't ideal.
|
|
A user may wish to clobber/refine existing search parameters
by issuing "lei q --save" again. Support that by overwriting
the lei.saved-search state file entirely.
We continue to preserve over.sqlite3 for deduplication purposes.
This way, we don't get something redundant like:
[lei]
q = term1
q = term2
q = term1
q = term2
q = term3
...whenever a user wants to refine their search. Instead,
we'll just have:
[lei]
q = term1
q = term2
q = term3
On the second go.
|
|
It is redundant since we stuff everything into the lei.q.output
config key.
|
|
Specifying a directory in ~/.local/share/lei/saved-searches/
is painful, so support (and start encouraging) the use of
the output.
|
|
We want users to be able to edit and refine the query over
time while using the same output destination.
|
|
Since saved-searches aren't a part of lei/store, nor
could it be considered cache data... (or can it? it
is discardable, after all).
|
|
Somebody may want a saved search which consistently asks for
messages within a rolling time period window. In other words,
we want to support using "lei q --save dt:last.week.." and keeps
the "dt:last.week.." relative to whenever "lei up" is run. This
ensures relative date-time specifications get used in the future
rather than converting into an absolute date-time from the
initial "lei q" invocation.
|
|
The command isn't finalized, yet, but it's intended to update
an existing saved search.
|
|
This will have a over.sqlite3 for content-based deduplication.
It may exhibit ibxish methods, so serving a read-only (or even
R/W) IMAP or instance or displaying HTML isn't outside the realm
of possibility.
|