about summary refs log tree commit homepage
DateCommit message (Collapse)
2021-04-27lei q + lcat: support --format=text output
This is mainly for "lei lcat" where it's the default, but I find it useful anyways compared to the JSON view. Colors are loaded from ~/.config/lei/config, and fall back to using diff colors from a normal git config (e.g. ~/.gitconfig).
2021-04-27lei lcat: extract Message-IDs from URLs and show them
It's a wrapper around "lei q" which extracts Message-IDs from URLs, "<$MSGID>", "id:$MSGID" and attempts to display the local version of the message. Its main purpose is to extract Message-IDs out of commonly-understood URLs to save users bandwidth and time by displaying the message locally. When reading from stdin, it will discard things it doesn't understand, so you can just pipe an entire "Link: $URL" line to it and it'll attempt to pluck the Message-ID out of the URL.
2021-04-27lei: standardize on _lei_wq_eof callback for workers
Simplify our internals a little bit.
2021-04-27lei blob: support retrieving attachments via $OID:$IDX
We'll be supporting some sort of text view for pager or piping to an $EDITOR buffer.
2021-04-27lei: add "ls-sync" command for listing sync folders
This will be useful, later.
2021-04-26URIimap: canonicalize uppercases UIDVALIDITY parameter
And the UID one, too, as well. This simplifies future comparison and matching code once case is no longer taken into account.
2021-04-26lei p2q: exit with failure if format-patch fails
Merely redirecting the failure message from git to our stderr is insufficient.
2021-04-26lei_input: support PublicInbox::WWW mboxrd URLs
This gives "lei import", "lei tag", and similar commands the ability to use URLs recognized by our PSGI frontend directly. This is more convenient than an equivalent shell pipeline since "set -o pipefail" is not portable and errors may be lost.
2021-04-26www: missing /$INBOX/$MSGID/raw returns 404
Don't attempt to return HTTP 300 via Extmsg on it, since whoever uses /raw is likely piping it to some other command.
2021-04-25lei_xsearch: show --output location with match count
This makes "lei up --all=local" output easier-to-understand when it's updating multiple saved searches.
2021-04-24lei_saved_search: avoid reentrancy in ->is_dup
Use a separate git process when calling xoids_for to prevent reentrancy in ->is_dup. Reentrancy happens since LeiToMail will call ->is_dup when inside callbacks when writing mail. This fixes --dedupe=mid test failures in t/lei-q-save.t I could only reproduce this consistently on a uniprocessor VM. "schedtool -a 0x1 -e ..." could not reproduce the problem on 2 and 4-core systems.
2021-04-24test_common: don't attempt to seek on pipes and terminals
This allows proper error reporting on daemon failure when using "check-run".
2021-04-24extindex: --gc: use escape pathnames for SQL LIKE properly
This allows us to handle odd inboxes w/o a newsgroup configured if they also make the strange choice of having backslashes in their path name. Also, ensure we use case-sensitive LIKE, since case-insensitive FSes are not worth supporting.
2021-04-24lei import: keep sync info for Maildir and IMAP folders
We aren't using it, yet, but the plan is to be able to use this information to propagate keyword changes back to IMAP and Maildir folders using some to-be-implemented command. "lei inspect" is a half-baked new command to make testing this change easier. It will be updated to support more SQLite+Xapian introspection duties in the future, including public-inbox things independent of lei.
2021-04-24lei_mail_sync: for bidirectional keyword sync
We'll be using the new class to efficiently propagate keyword changes from lei/store back to Maildir or IMAP folders.
2021-04-24doc: lei_design_notes: add a bit on WAL usage
2021-04-24net_reader: imap_each: add UIDVALIDITY to URL arg
This will allow the callback to reliably maintain OID <=> UID mappings between lei/store and the IMAP folder.
2021-04-24URIimap: support ->uidvalidity and ->iuid
These will be useful for keyword synchronization, and perhaps importing a single IMAP message with ->iuid.
2021-04-24t/lei_to_mail: split "lei import" test $HOME directory
"lei import" behavior will may change w.r.t. keyword handling. Use separate $HOME between different test_lei to ensure isolation between the tests.
2021-04-24lei_input: drop outdated comment w.r.t. compression
Followup-to: 49b036771ef3bf45 ("lei_input: support compressed mboxes")
2021-04-23doc: INSTALL: add Guix to distribution list
public-inbox has been available in Guix since 2019.
2021-04-23lei import: support adding keywords and labels on import
This saves some work and makes it easier to set volatile metadata on a message at import time.
2021-04-23lei_to_mail: cwd-agnostic Maildir wakeup
Since we don't have *at() syscalls readily available to us, lei-daemon may call ->poke_dst in the wrong relative directory. Despite not having *at() syscalls, we can still capture the "$MAILDIR/cur" directory handle at pre_augment time so we can reliably call futimes(2) on it using the `utime' perlop.
2021-04-23net_reader: restart on first UID when UIDVALIDITY changes
In other words, treat the same IMAP folder with a different UIDVALIDITY as a completely different folder. If the UIDVALIDITY changes, we can start from UID=1 without falling behind or losing data. If the UIDVALIDITY gets reset to a previously known-good message, we can still resume where we left off before the first UIDVALIDITY change. This affects public-inbox-watch and "lei import" One potential downside of this is for rare altid users, but that's mainly intended for NNTP article numbers which are/were often publicized; not IMAP UIDs which are rarely publicized. The other potential downside is bandwidth waste in in the rare case UIDVALIDITY changes while IMAP folder contents remain unchanged. There's no extra storage used due to existing (v1|v2|lei/store) deduplication mechanisms. Before this change, we were matching offlineimap behavior and stopped synching an IMAP folder when its UIDVALIDITY changed. offlineimap behavior made sense for IMAP <=> Maildir synchronization since Maildirs had no sense of UIDVALIDITY and could only rely on name mapping.
2021-04-23doc: INSTALL: update with more distro info, symlink-install
2021-04-23lei up: support symlinked pathnames
On my default FreeBSD 11.x system, "/home" is a symlink to "/usr/home", which causes "lei up" path resolution to fail when I use outputs in $HOME. Fall back to a slow path of globbing and matching pathnames based on st_ino+st_dev.
2021-04-23lei: saved searches support --dedupe=<mid|oid>
This is less surprising in case users are used to using --dedupe= without --save.
2021-04-22lei: XDG_RUNTIME_DIR=/dev/null disables daemon mode
We'll support this mode of operation for now to quiet down testing of oneshot mode where the daemon doesn't persist.
2021-04-22lei import|convert: drop --no-kw aliases
Supporting --no-keywords and --no-flags aliases is harmful if users end up assuming "keywords:" and "flags:" are valid search prefixes (they're not).
2021-04-22lei import: --incremental default for NNTP and IMAP
No point in burning through bandwidth to import stuff we already saw. All this logic is shared with -watch but uses a different pathname for lei since it's tied to lei/store (and not a public-inbox).
2021-04-22imap_tracker: prepare for use with lei
We'll support nodatacow as we do in other SQLite DBs
2021-04-22lei: flesh out `forwarded' kw support for Maildir and IMAP
Maildir and IMAP can both handle `forwarded'. Ensure we don't lose `forwarded' when reading from stores which do not support it, but ensure we can set it when reading from IMAP and Maildir stores.
2021-04-21doc: add lei_design_notes.txt and lei-store-format(5)
lei itself is a somewhat weird design, but lei/store is a fairly normal hybrid of extindex with v2-style storage.
2021-04-21lei: share common *done_wait callbacks
Code is the enemy, and there's no need to duplicate things, here. There may be further opportunities along these lines to further deduplicate things...
2021-04-21t/lei-daemon: skip inaccessible socket test as root
"chmod 0000" on a Unix socket can't stop root from connecting to it; so just skip the test for rare cases when testing as root. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210420213712.qfpftr2r543cqg7l@nitro.local/
2021-04-20lei_query: avoid POSIX::lround for older Perls
POSIX.pm shipped with Perl 5.16.3 did not support lround, at least. So just rely on built-in core functions.
2021-04-20test_common: fix xbail for Perl 5.16
Our use of `ref' was triggering ambiguity in older versions of the Perl parser. Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org> Link: https://public-inbox.org/meta/20210420174912.h6d2yv7zu5xr4yfc@nitro.local/
2021-04-20lei edit-search: command to tweak search parameters
This may be useful for users to tweak search parameters. This command is also the reason lei.saved-search is a git-config file rather than JSON.
2021-04-20lei forget-search: new command to forget saved searches
Readers may lose interest in subscription topics. This lets them avoid clutter by forgetting a saved search. This does not and will not destroy the contents of an --output mailbox. In other words, this is similar to unsubscribing from an Atom/RSS feed or NNTP group. I've also decided we won't support 'mv-search', since it'll probably be rarely used and "lei convert" can be used, instead.
2021-04-20lei up: support --all=local
Users may wish to update several saved searches at once. We can support parallel updates in lei-daemon so users won't have to do it themselves via xargs or similar. Supporting IMAP outputs would be significantly more involved since we'd have to pre-authenticate for every single IMAP output before entering the redispatch loop.
2021-04-20lei up: more error checking for config loading
We'll support editing the saved search config file, so user errors may happen and we need to throw sensible errors in that case.
2021-04-20config: favor ->get_all when possible
It's slightly less code.
2021-04-20lei up: fix help output and ARGV handling
We don't support changing search terms once "lei q --save" is used.
2021-04-20lei-sigpipe: update and move test from xt => t
We have "lei import" and better test infrastructure for lei, now, so we can more easily test SIGPIPE without relying on an already-configured instance.
2021-04-19lei q: --save and --augment may be combined
This necessitated fixing pause_dedupe to release the handle used by ->lock_for_scope_fast, but otherwise no changes to the LeiToMail package.
2021-04-19lei_saved_search: split "lei q --save" and "lei up" init paths
They're more different than alike, and having two separate methods seems less confusing to me.
2021-04-19config: git_config_dump blesses
I don't know if it's worth it to sub (or super)class PublicInbox::Config into something more generic for lei, but this change simplifies a good chunk of lei code that reuses the public-inbox config parsing.
2021-04-19lei_saved_search: avoid needless var shadowing
While perl (5.28) doesn't complain about this, it's confusing to my easily-confused mind.
2021-04-19lei q: implement import-before default for --save
This makes "lei q --save" as safe as "lei q" to prevent against accidental data loss when clobbering an existing output,
2021-04-19lei: support unlinked/missing saved searches
It's conceivable a user will want to erase all previous results but still rerun/refresh a search to get new results. We probably won't support prune functionality, here, and instead require explicit removal of saved searches.