about summary refs log tree commit homepage
path: root/t/lei-import.t
DateCommit message (Collapse)
2021-04-23lei import: support adding keywords and labels on import
This saves some work and makes it easier to set volatile metadata on a message at import time.
2021-04-05lei_search: ignore Resent-Message-ID for indexing
It currently conflicts with the way OverIdx and SearchIdx index messages, ultimately leading to violating a NOT NULL constraint on id2num.id in over.sqlite3. We may allow searching Resent-* fields separately, though I'm not sure how useful it'll be.
2021-04-03lei: improve handling of Message-ID-less draft messages
We need a stable fallback time for digest2mid in the presence of messages without Received/Date headers. Furthermore, we must avoid using uninitialized smsg->{mid} when parsing References for draft replies.
2021-03-29lei_input: treat ".eml" and ".patch" suffix as "eml"
".eml" is a suffix supported by (/usr/local)/etc/mime.types on Debian and FreeBSD systems using the "mime-support" package. ".patch" is what "git format-patch" generates by default since git v1.5.0 in 2007.
2021-03-27lei blob: aka "git-show-harder" for blobs
This implements blob reconstruction via SolverGit, emulating the functionality of /$INBOX/$OID/s/ endpoint in PublicInbox::WWW. It uses the current working tree as a coderepo, and accepts any number of --git-dir=$PATH args. Remote externals are not yet supported. v2: use absolute path for git repos
2021-03-26lei: support /dev/fd/[0-2] inputs and outputs in daemon
Since lei-daemon won't have the same FDs as the client, we need to special-case thse mappings and won't be able to open arbitrary, non-standard FDs. We also won't attempt to support /proc/self/fd/[0-2] since that's a Linux-ism. /dev/fd/[0-2] and /dev/std{in,out,err} are portable to FreeBSD, at least. mawk(1) also supports /dev/std{out,err}, as does gawk(1) (which supports everything we can support, and arbitrary /dev/fd/$FD).
2021-03-23lei import: ignore Status headers in "eml" messages
Those headers only have meaning with for mboxes. Don't surprise users by trying to make sense of a header that is defined for mboxes. It's possible to send email with (Status|X-Status) headers and have those headers show up in a recipient's IMAP mailbox. This was bad because an IMAP user may want to import a single message through their MUA and pipe its contents to "lei import" without noticing a mischievious sender stuck "X-Status: F" (flagged/important) in there.
2021-03-23lei_input: drop "From " line on single "eml" (message/rfc822)
This matches the long-standing behavior of public-inbox-mda, public-inbox-learn and our other tools. It is useful because mutt, "git format-patch", and likely other tools will pipe a single message with a "From " header line, but with no further "From " escaping or Content-Length: header.
2021-03-21lei q: trim JSON output
Stop showing `docid' since it's not useful with shards. `bytes' and `lines' are probably noise, but maybe could be visible in some "fuller" view. v2: t/lei_xsearch: fix warnings from {docid} removal
2021-02-26lei import: use --in-format/-F for consistency
Since we recommend $IN_FORMAT:$LOCATION, this is hopefully not intrusive (not that this is released software, yet). This is to be consistent with "lei convert" usage. We'll keep "-f" only for output formats, since that is used for "lei q" and "lei convert" for outputs
2021-02-22lei_store: populate ALL.git/alternates with new epochs
Since eidx_init updates ALL.git/objects/info/alternates, we need to ensure new epochs we create from LeiStore->importer exist before eidx_init writes alternates. Reported-by: Kyle Meyer <kyle@kyleam.com> Link: https://public-inbox.org/meta/8735xou0gq.fsf@kyleam.com/
2021-02-22t/lei*: drop $lei->(...) sub
lei() and lei_ok() are superior since they offer prototype checks and lei_ok() adds another check + description DRY-ness. The $lei sub was only bound to a variable since it was in t/lei.t and named subs don't work well with the key2sub() wrapper.
2021-02-10lei: split out MdirReader package, lazy-require earlier
We'll do more requires in the top-level lei-daemon process to save work in workers. We can also work towards aborting on user errors in lei-daemon rather than worker processes. "lei import -f mbox*" is finally tested inside t/lei_to_mail.t
2021-02-07tests: add test_lei wrapper, split out t/lei-import.t
This will make it easier to maintain and test lei going forward, we need to be testing against existing read-only daemons. We'll also save ourselves some boilerplate by exporting all the Test::More methods directly in TestCommon We'll start using this by splitting out the latest "lei import" tests into its own file.