about summary refs log tree commit homepage
path: root/lib/PublicInbox/NetReader.pm
DateCommit message (Collapse)
2021-03-24net_reader: nntp_each: pass keywords as `undef'
We'll use `undef' to denote keywords are unknown/unsupported, instead of an empty arrayref. This will let callers use the same callback and args for imap_each. Passing an empty arrayref to set_eml in LeiStore causes keywords to be cleared completely, which is not desired behavior when "lei import" is importing already-seen messages from NNTP.
2021-03-23net_reader: escape nasty chars from Net::NNTP->message
Net::Cmd::message (used by Net::NNTP) does no escaping at all, so "\r" was causing confusing/nonsensical error messages when I tried to import from the wrong group.
2021-03-10watch: IMAP: ignore \Deleted and \Draft messages
This matches existing Maildir behavior, as trash and draft messages have little reason to be exposed publicly.
2021-03-04lei q: support --import-augment for IMAP
IMAP is similar to Maildir and we can now preserve keyword updates done on IMAP folders.
2021-02-24net_reader: trim exports and remove unused uri_new
More network things for -watch are isolated in NetReader, now, so fewer exports are necessary.
2021-02-24watch: switch IMAP and NNTP fetch loops to NetReader
NetReader::<imap|nntp>_each were based on the -watch code they now replace. v2: do not warn on EINTR if user quit to fix occasional test failure in t/imapd.t
2021-02-24lei <import|convert>: support NNTP sources
We can read NNTP in -watch and Net::NNTP is shipped with Perl5, so lei import and convert have no excuse not to support NNTP as a client. Authentication is not tested, yet; but should be close to what IMAP is like...
2021-02-22net_reader: mic_get: reuse connections if cache enabled
We only enable {mic_cached} in WQ workers, and those aren't expected to fork again going forward. So cache here avoid a penalty for the non-augmenting (imap_delete_all) call with "lei q"
2021-02-22lei convert: auth directly from worker process
Since this only has one worker, we can auth directly in the worker since the convert worker now has access to the script/lei {sock} for running "git credential".
2021-02-21lei2mail: parallel augment for lock-free stores
This lets us make use of multiple cores on IMAP and Maildir backed by SSD (or better) storage. This benefits IMAP stores with high network latency, but may still penalize IMAP servers with rotational storage.
2021-02-21net_reader: use and accept URIimap objects in more places
This flexibility should save us some code down-the-line.
2021-02-21lei q: support IMAP/IMAPS --output destinations
Augment (and dedupe) aren't parallel, yet, so its more sensitive to high-latency networks.
2021-02-19net_writer: start implementing IMAP write support
Requiring TEST_IMAP_WRITE_URL to be set to a writable IMAP server URL isn't ideal, but it works for now until we have time to setup a mock dovecot/cyrus/etc... instance for testing.
2021-02-19net_reader: handle single-message IMAP mailboxes
Due to an off-by-one error, we were unable to read mailboxes with only a single message of UID:1. Without this fix, the message with UID:1 could only be read after UID:2 was created; so there's no permanent data loss as long as a new message showed up. This affects all releases of public-inbox-watch with IMAP support, though it probably went unnoticed because single message inboxes are rare.
2021-02-18lei: check for IMAP auth errors
We need to ensure authentication failures and error codes get propagated to the parent process(es) properly. v2: update MANIFEST v3: LeiAuth.pm ->_lei_cfg bit moved to a previous commit
2021-02-18lei convert: mail format conversion sub-command
This will make testing IMAP support for other commands easier, as it doesn't write to lei/store at all. Like the pager and MUA, "git credential" is always spawned by script/lei (and not lei-daemon) so it has a controlling terminal for password prompts. v2: fix missing requires, correct test ordering v3: ensure config exists for IMAP auth
2021-02-18lei import: start rearranging code for IMAP support
More to come in a later commit; some error handling and failure modes will be trickier with IMAP due to authentication.
2021-02-18watch: connect to NNTP and IMAP in config order
This is hopefully less surprising to users when they're prompted for credentials.
2021-02-18watch: move imap_common_init to NetReader
We'll use this in LeiImport and likely other places.
2021-02-10net_reader: new package split from -watch
We'll be using some of this for IMAP and NNTP support in lei, too. More will need to be done to improve code sharing and reusability, soon, but this is a start.