about summary refs log tree commit homepage
path: root/t/lei-import-imap.t
DateCommit message (Collapse)
2021-07-25t/lei*: check error messages on failures
I just hit an unreproducible failure in t/lei-p2q.t and lacked $lei_err information to diagnose it. Hopefully this helps track down odd failures in the future.
2021-06-12lei ls-mail-source: list IMAP folders and NNTP groups
While other tools can provide the same functionality, having integration with git-credential is convenient, here. Caching and completion will be implemented separately.
2021-05-30lei lcat: allow IMAP folder URLs w/o UIDVALIDITY
Requiring UIDVALIDITY on the command-line is of course unreasonable.
2021-05-30lei import|lcat: improve+fix single message IMAP support
lcat can now dump the memoized contents of entire IMAP folders, not just a single UID. It's now parallelized and pipelined for multiple lei2mail workers. Furthemore, various forms of JSON output work consistently with blob-only output, now. While working on this, I noticed NetReader was passing UID URLs to imap_each callbacks, which was causing mail_sync.sqlite3 to store UIDs in `folders' and clearly wrong so it's now fixed.
2021-05-28lei: handle a single IMAP message in most places
"lei import" can now import a single IMAP message via <imaps://example.com/MAILBOX/;UID=$UID> Likewise, "lei inspect" can show the blob information for UID URLs and "lei lcat" can display the blob without network access if imported. "lei lcat" also gets rid of some unused code and supports "blob:$OIDHEX" syntax as described in the comments (and used by our "text" output format). v2: enforce UID in URL, fail without v3: fix error reporting (s/fail/child_error/)
2021-05-25lei forget-mail-sync: new command to drop sync information
Sometimes a user stops caring to sync an IMAP or Maildir folder, or wants to force a resync. Let them run this command to have lei forget all the sync information about the mail folder. This won't delete any stored messages in git, but will leave "lei index" users with dangling references.
2021-05-24lei inspect: use LeiMailSync->match_imap_url
Move match_imap_url into LeiMailSync so it can be used in more places, such as "lei inspect". Upcoming commands such as "lei forget-mail-sync" and {add,forget,pause,resume}-watch will also support relaxed IMAP matching rules since there's no reasonable way to expect users use ";UIDVALIDITY=" on the command-line.
2021-05-23lei import: store IMAP user+auth in mail_sync folder URI
Just having UIDVALIDITY in the URI isn't enough, since a single lei user may have multiple IMAP logins on the same server. This leads to compatibility problems and forces a reimport for the few users already using this lei functionality, but it's not stable nor released, yet.
2021-05-01lei import: fix --mail-sync handling in LeiInput
"lei inspect" also shows "mail-sync" as a field name
2021-05-01lei: rename ls-sync to ls-mail-sync
This allows tab-completion for "ls-search" to work with fewer characters ("ls-s<TAB>" instead of "ls-se<TAB>"), and I expect "ls-search" to be used more frequently than "ls-mail-sync". This also matches the --mail-sync switch of "lei import"
2021-04-30lei import: support UIDVALIDITY in IMAP URL
Specifying a UIDVALIDITY value allows the user to enforce a strict match and force failure. This necessitated changes to NetReader to allow die() and make error reporting more suitable for CLI usage rather than daemonized usage of -watch.
2021-04-30lei import: avoid IMAPTracker, use LeiMailSync more
IMAPTracker has a UNIQUE constraint on the `url' column, which may cause compatibility and/or rollback problems in attempting to deal with UIDVALIDITY changes. Having multiple sources of truth leads to confusion and bugs, so relying on LeiMailSync exclusively ought to simplify things. Furthermore, since LeiMailSync is only written to by LeiStore, it is safer in that it won't mark a UID or article as imported until git-fast-import has seen it, and the SQLite commit always happens after "done\n" is sent to fast-import. This mostly reverts recent commits to IMAPTracker to support lei, those are: 1) commit 7632d8f7590daf70c65d4270e750c36552fa9389 ("net_reader: restart on first UID when UIDVALIDITY changes") 2) commit 311a5d37ad275cd75b1e64d87827c4d13fe4bfab ("imap_tracker: prepare for use with lei"). This means public-inbox-watch will not change between 1.6 and 1.7: -watch stops synching a folder when UIDVALIDITY changes.
2021-04-27lei blob: support retrieving attachments via $OID:$IDX
We'll be supporting some sort of text view for pager or piping to an $EDITOR buffer.
2021-04-27lei: add "ls-sync" command for listing sync folders
This will be useful, later.
2021-04-24lei import: keep sync info for Maildir and IMAP folders
We aren't using it, yet, but the plan is to be able to use this information to propagate keyword changes back to IMAP and Maildir folders using some to-be-implemented command. "lei inspect" is a half-baked new command to make testing this change easier. It will be updated to support more SQLite+Xapian introspection duties in the future, including public-inbox things independent of lei.
2021-04-22lei import: --incremental default for NNTP and IMAP
No point in burning through bandwidth to import stuff we already saw. All this logic is shared with -watch but uses a different pathname for lei since it's tied to lei/store (and not a public-inbox).
2021-04-03lei tag: fix tagging of IMAP inputs
We need net_merge_all and to lock the number of worker jobs. Parallel inputs are not supported, yet (is it needed?, I don't expect this to be used for multiple files very often...).
2021-03-28test_common: require_mods bundles
This makes it easier to manage test dependencies on systems where optional stuff isn't installed. This fixes some lei tests which didn't check for Plack before starting -httpd, and ensures Parse::RecDescent is available for -imapd in case Mail::IMAPClient stops using it.
2021-03-05search: use "z:" instead of "bytes:" prefix
So far, searching by size has never been publicly documented, and IMHO, of questionable utility. In any case, "z:" is what mairix(1) uses, so it may be familiar to existing mairix users (I've never used this prefix myself). So far, this prefix is only used internally in tests and in auto-translated queries from IMAP; thus this incompatible change is unlikely to affect anyone.
2021-02-19tests: require Mail::IMAPClient for IMAP tests
All of our current IMAP code relies on Mail::IMAPClient at the moment, so ensure we skip those tests on systems without that module.
2021-02-18lei import: add IMAP and (maildir|mbox*):$PATHNAME support
This makes "lei import" more similar to "lei convert" and allows importing from disparate sources simultaneously. We'll also fix some ->child_error usage errors and make the style of the code more similar to the "lei convert" code. v2: fix missing requires