about summary refs log tree commit homepage
path: root/lib/PublicInbox/LeiSavedSearch.pm
DateCommit message (Collapse)
2023-11-22lei_saved_search: don't create Git object during ->DESTROY
This fixes t/lei-q-save.t getting stuck since $self->{ale} is already gone by the time DESTROY gets called.
2023-09-22lei: use File::Temp for listing saved searches
I have no idea how badly my brain malfunctioned here when I wrote the code to create a temporary file without O_EXCL :x I'm still not sure if users have enough saved searches for justifying a cache, here.
2023-01-30use Net::SSLeay (OpenSSL) for SHA-(1|256) if installed
On my x86-64 machine, OpenSSL SHA-256 is nearly twice as fast as the Digest::SHA implementation from Perl, most likely due to an optimized assembly implementation. SHA-1 is a few percent faster, too.
2022-08-29treewide: ditch inbox->recent method
It's a needless wrapper, nowadays. Originally, ->over was added on experimental basis to optimize for /$INBOX/ where Xapian ->search is slower on gigantic (LKML-sized) inboxes. Nowadays with extindex, ->over is here to stay given NNTP and IMAP both benefit from it. So reduce the interpreter stack overhead and just access ->over directly. lxs->recent was never used outside of tests, anyways. And while we're in the area, avoid needlessly bumping the refcount of $ctx->{ibx} in View::paginate_recent.
2021-11-09lei q|up: fix saved searches for single-phrase search
`"' (double-quote) needs to be quoted for stdin searches. We also need to differentiate between "lei q --stdin" usage when calling "lei up", do it by setting an internal "rawstr" knob to ensure we can parse the config properly regardless of whether the initial search used --stdin or not.
2021-10-19lei: use die for external and query handling
This allows "lei up" to continue processing unrelated externals if on output fails.
2021-09-21lei q: improve --limit behavior and progress
Avoid slurping gigantic (e.g. 100000) result sets into a single response if a giant limit is specified, and instead use 10000 as a window for the mset with a given offset. We'll also warn and hint towards about the --limit= switch when the estimated result set is larger than the default limit.
2021-09-21lei q: update messages to reflect --save default
I wanted to try --dedupe=none for something, but it failed since I forgot --no-save :x So hint users towards --no-save if necessary.
2021-09-19lei config --edit: use controlling terminal
As with "lei edit-search", "lei config --edit" may spawn an interactive editor which works best from the terminal running script/lei. So implement LeiConfig as a superclass of LeiEditSearch so the two commands can share the same verification hooks and retry logic.
2021-09-18lei up: automatically use dt: for remote externals
Since we can't use maxuid for remote externals, automatically maintaining the last time we got results and appending a dt: range to the query will prevent HTTP(S) responses from getting too big. We could be using "rt:", but no stable release of public-inbox supports it, yet, so we'll use dt:, instead. By default, there's a two day fudge factor to account for MTA downtime and delays; which is hopefully enough. The fudge factor may be changed per-invocation with the --remote-fudge-factor=INTERVAL option Since different externals can have different message transport routes, "lastresult" entries are stored on a per-external basis.
2021-09-15lei_saved_search: fix prefix for IMAP folders w/ slash
We failed to account for IMAP mailboxes containing `/' characters when creating saved search files for them. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210915123347.knr4qpaei73tjc5q@meerkat.local/
2021-09-11lei: fix handling of broken lei.saved-search config files
lei shouldn't become unusable if a config file is invalid. Instead, show the "git config" stderr and attempt to continue gracefully. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210910141157.6u5adehpx7wftkor@meerkat.local/
2021-08-19lei q: make --save the default
Since "lei up" is more often useful than not and incurs neglible overhead; enable --save by default and allow --no-save to work. This also fixes a long-standing when overwriting --output destinations with saved searches: dedupe data from previous searches are reset and no longer influences the new (changed) search, so results no longer go missing if two sequential invocations of "lei q --save" point to the same --output.
2021-08-11lei_saved_search: canonicalized relative save paths
Storing relative paths with '..' in them can be expensive to resolve when running 'lei up', so prefer storing canonicalized absolute paths. We only do this for paths with '..' in them, though, since this can lose symlink info.
2021-07-25lei: avoid SQLite COUNT() for dedupe
SQLite COUNT() is a slow operation that does a full table scan with no conditions. There's no need for it, since lei dedupe only needs to know if it's empty or not to decide between new/ and cur/ for Maildir outputs.
2021-05-28lei q|up: support v2:/path/to/inboxdir destination
This allows "lei-managed pseudo mailing lists" as described by Konstantin. Alternates use is optional and can be enables via --shared. This doesn't manage or edit ~/.public-inbox/config; presumably there'll need to be some tweaking of search parameters before finalizing and making the inbox publicly accessible via HTTP/NNTP. Link: https://public-inbox.org/meta/20210426164454.5zd5kgugfhfwfkpo@nitro.local/T/
2021-05-23lei <q|up>: set \Recent on non-empty mbox and Maildir
Despite JMAP not supporting the equivalent of the IMAP \Recent flag, it is useful for "lei q --augment", and "lei up" users to be able to distinguish new results from old-but-unread messages in an mbox or Maildir. For mbox family messages, we'll drop the "O" status flag when appending to mboxes, and we'll write to the "new" subdirectory of Maildirs. Behavior when writing to initially empty Maildirs and mboxes remains unchanged since there's no need to distinguish between new and old results in the initial case. Having users wait for a rename(2) storm or complete mbox rewrite hurts UX. With IMAP mailboxes, \Recent is already enforced by the IMAP server and IMAP clients have no way of changing it(*) (*) mutt uses the "Old" IMAP flag which isn't part of RFC 3501, other MUAs may do similar things.
2021-05-06lei_mail_sync: Maildir canonicalization omits trailing slash
We use trailing slashes internally, but should not increase visual noise for users by exposing them in config files or DB storage (and shell completion/listings). This fixes a long-standing bug in $lei->rel2abs that prevented absolute paths from being canonicalized.
2021-05-04lei up: fix dedupe with remote externals on Maildir + IMAP
LeiToMail Maildir and IMAP write callbacks need to account for the caller-supplied smsg. We'll also make better use of the user-supplied smsg object by ensuring blob deduplication happens ASAP. Fixes: e76683309ca4f254 ("lei <q|up>: distinguish between mset and l2m counts")
2021-05-01lei edit-search: support relocating lei.q.output
The contents of the old lei.q.output will not be removed, but will be converted into the new one.
2021-05-01lei_saved_search: fix excess indent for first lei.q entry
This was harmless, but ugly and possibly confusing to users who run "lei edit-search".
2021-04-24lei_saved_search: avoid reentrancy in ->is_dup
Use a separate git process when calling xoids_for to prevent reentrancy in ->is_dup. Reentrancy happens since LeiToMail will call ->is_dup when inside callbacks when writing mail. This fixes --dedupe=mid test failures in t/lei-q-save.t I could only reproduce this consistently on a uniprocessor VM. "schedtool -a 0x1 -e ..." could not reproduce the problem on 2 and 4-core systems.
2021-04-23lei up: support symlinked pathnames
On my default FreeBSD 11.x system, "/home" is a symlink to "/usr/home", which causes "lei up" path resolution to fail when I use outputs in $HOME. Fall back to a slow path of globbing and matching pathnames based on st_ino+st_dev.
2021-04-23lei: saved searches support --dedupe=<mid|oid>
This is less surprising in case users are used to using --dedupe= without --save.
2021-04-19lei q: --save and --augment may be combined
This necessitated fixing pause_dedupe to release the handle used by ->lock_for_scope_fast, but otherwise no changes to the LeiToMail package.
2021-04-19lei_saved_search: split "lei q --save" and "lei up" init paths
They're more different than alike, and having two separate methods seems less confusing to me.
2021-04-19config: git_config_dump blesses
I don't know if it's worth it to sub (or super)class PublicInbox::Config into something more generic for lei, but this change simplifies a good chunk of lei code that reuses the public-inbox config parsing.
2021-04-19lei_saved_search: avoid needless var shadowing
While perl (5.28) doesn't complain about this, it's confusing to my easily-confused mind.
2021-04-19lei: support unlinked/missing saved searches
It's conceivable a user will want to erase all previous results but still rerun/refresh a search to get new results. We probably won't support prune functionality, here, and instead require explicit removal of saved searches.
2021-04-18lei ls-search: command to list saved searches
Going forward, we'll probably support JSON for all the "ls-*" subcommands. This also provides the basis for "lei up" shell completion.
2021-04-18avoid wrong/needless squote_maybe shortcuts
We'll keep it in PublicInbox::Config for now, since I'm not sure if there's a better place to put it.
2021-04-18lei_saved_search: avoid git-config
Repeated vfork+execv costing us around 20ms on t/lei-q-save.t, so just learn to quote git-config values and write directly to the config file.
2021-04-17lei up: further improve Maildir canonicalization
We want to be able to use "lei up ." when inside a Maildir. We'll also relax Maildir/mbox basenames to be any non-'/' character after converting relative paths to absolute. The old restriction on allowed characters was unnecessary and made it impossible to reliably map "." when used as the sole argument for "lei up".
2021-04-17lei up: fix canonicalization of Maildirs
We always represent --output destination directories with a trailing slash to disambiguate directories from mbox filenames. Therefore, we must use the trailing slash when mapping the destination beck from the lei/saved-search/* directory. "lei up" now relies exclusively on the users --output pathname or URL for updates. This ought to be less confusing since pathnames in ~/.local/store/lei/saved-searches aren't ideal.
2021-04-17lei q --save: clobber config file on repeats
A user may wish to clobber/refine existing search parameters by issuing "lei q --save" again. Support that by overwriting the lei.saved-search state file entirely. We continue to preserve over.sqlite3 for deduplication purposes. This way, we don't get something redundant like: [lei] q = term1 q = term2 q = term1 q = term2 q = term3 ...whenever a user wants to refine their search. Instead, we'll just have: [lei] q = term1 q = term2 q = term3 On the second go.
2021-04-17lei q --save: avoid lei.q.format
It is redundant since we stuff everything into the lei.q.output config key.
2021-04-17lei up: support output destination as arg
Specifying a directory in ~/.local/share/lei/saved-searches/ is painful, so support (and start encouraging) the use of the output.
2021-04-16lei: saved searches keyed only by path/URL and format
We want users to be able to edit and refine the query over time while using the same output destination.
2021-04-16lei: expose share_path as a method
Since saved-searches aren't a part of lei/store, nor could it be considered cache data... (or can it? it is discardable, after all).
2021-04-16lei q: --save preserves relative time queries
Somebody may want a saved search which consistently asks for messages within a rolling time period window. In other words, we want to support using "lei q --save dt:last.week.." and keeps the "dt:last.week.." relative to whenever "lei up" is run. This ensures relative date-time specifications get used in the future rather than converting into an absolute date-time from the initial "lei q" invocation.
2021-04-13lei: add "lei up" to complement "lei q --save"
The command isn't finalized, yet, but it's intended to update an existing saved search.
2021-04-13lei q: start wiring up saved search
This will have a over.sqlite3 for content-based deduplication. It may exhibit ibxish methods, so serving a read-only (or even R/W) IMAP or instance or displaying HTML isn't outside the realm of possibility.