about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2020-06-28watch: use our own "git credential" wrapper
Git.pm may not be installed on some systems; or some users have multiple Perl installations and Git.pm is not available to the Perl running -watch. Accomodate both those types of users by providing our own "git credential" wrapper.
2020-06-28watch: show user-specified URL consistently.
Since we use the non-ref scalar URL in many error messages, favor keeping the unblessed URL in the long-lived process. This avoids showing "snews://" to users who've specified "nntps://" URLs, since "nntps" is IANA-registered nowadays and what we show in our documentation, while "snews" was just a draft the URI package picked up decades ago.
2020-06-28watch: add NNTP support
This is similar to IMAP support, but only supports polling. Automatic altid support is not yet supported, yet; but may be in the future. v2: small grammar fix by Kyle Meyer Link: https://public-inbox.org/meta/87sgeg5nxf.fsf@kyleam.com/
2020-06-28testcommon: $ENV{TAIL} supports non-@ARGV redirects
Existing use of the $ENV{TAIL} relied on parsing --std{out,err}, which was only usable for read-only daemons. However, -watch doesn't use PublicInbox::Daemon code(*), so attempt to figure out redirects. (*) -watch won't able to run as a daemon in cases when git-credential prompts for IMAP/NNTP passwords. PublicInbox::Daemon is also designed for read-only parallelism where all worker processes are the same. Any subprocesses spawned by -watch are to do specific tasks for a particular set of inboxes.
2020-06-28watch: just use ->urlmatch
We may just modify PublicInbox::Config->urlmatch in the future to support git <1.8.5, but I wonder if there's enough users on git <1.8.5 to justify it.
2020-06-28watch: remove {mdir} array
Since we store all watched directory names as keys in %mdmap, there should be no need to keep an array of those directories around. t/watch_maildir*.t required changes to remove trained spam. Once we've trained something as spam, there shouldn't be a need to rescan it.
2020-06-28watch: support multiple watch: directives per-inbox
Some users will find it useful to merge several Maildir or IMAP mailboxes into one public-inbox. Let them do it, since we've always supported multi-address inboxes.
2020-06-28watch: imap: be quiet about disconnecting on quit
If ->idle_done was handled successfully, we can just let normal ->DESTROY disconnect and avoid ugly backtraces when a user hits Ctrl-C to take down the process group.
2020-06-28watch: support imap.fetchBatchSize parameter
IMAP allows retrieving multiple messages with a single command, and Mail::IMAPClient supports that. Unfortunately, it means we slurp multiple messages into memory at once. This option allows users to trade off memory usage to reduce network round-trips. Ideally, we'd support pipelining; but AFAIK no widely installed Perl IMAP library supports it.
2020-06-28watch: avoid long transaction to IMAPTracker
With different polling intervals, multiple processes may simultaneously write to IMAPtracker. This ought to reduce SQLite busy waiting and contention issues when importing many inboxes in parallel.
2020-06-28imaptracker: drop {dbname} field
It's not used anywhere since the IMAPTracker object doesn't disconnect and reconnect. If we ever need the filename, {dbh}->sqlite_db_filename may be used. Cc: Eric W. Biederman <ebiederm@xmission.com>
2020-06-28imaptracker: add {url} field to reduce args
Passing a $url parameter to every function was error-prone and having {url} field for a short-lived object is appropriate. This matches the version of IMAPTracker posted by Eric W. Biederman on 2020-05-15 at: https://public-inbox.org/meta/87ftc0c3r4.fsf_-_@x220.int.ebiederm.org/ The version I originally imported was based on the one posted on 2019-10-09: https://public-inbox.org/meta/874l0i9vhc.fsf_-_@x220.int.ebiederm.org/ Cc: Eric W. Biederman <ebiederm@xmission.com>
2020-06-28ds: add_timer: allow passing arg to callback.
This allows callers to avoid creating expensive closures. We no longer pass the `$now' value to callers, as none of the callers used it.
2020-06-28watch: use UID SEARCH to avoid empty UID FETCH
For mailboxes with many gaps in the UID sequence, performing a UID SEARCH beforehand can reduce the number of articles to fetch. However, the downside to this is we may end up with an arbitrarly large list of UIDs from the server.
2020-06-28watch: stop importers before forking
This fixes cases where watch is handling both Maildirs and IMAP connections. While we're at it, close open directories in the IMAP children to save FDs.
2020-06-28config: support ->urlmatch method for -watch
Since we have IMAP client support in -watch; make sure per-URL settings are familiar to git users by taking advantage of git's URL matching abilities. This requires git 1.8.5+, which most users ought to have (though base CentOS 7 is on 1.8.3).
2020-06-28watch: support IMAP polling
Not all IMAP servers support IDLE, and IDLE may be prohibitively expensive for some IMAP servers with many inboxes. So allow configuring a imap.$IMAP_URL.pollInterval=SECONDS to poll mailboxes. We'll also need to poll for NNTP servers in the future.
2020-06-28watch: wire up IMAP IDLE reapers to DS
We can avoid synchronous `waitpid(-1, 0)' and save a process when simultaneously watching Maildirs. One DS bug is fixed: ->Reset needs to clear the DS $in_loop flag in forked children so dwaitpid() fails and allows git processes to be reaped synchronously. TestCommon also calls DS->Reset when spawning new processes, since t/imapd.t uses DS->EventLoop while waiting on -watch to write.
2020-06-28ds: remove fields.pm usage
Since the removal of pseudo-hash support in Perl 5.10, the "fields" module no longer provides the space or speed benefits it did in 5.8. It also does not allow for compile-time checks, only run-time checks. To me, the extra developer overhead in maintaining "use fields" args has become a hassle. None of our non-DS-related code uses fields.pm, nor do any of our current dependencies. In fact, Danga::Socket (which DS was originally forked from) and its subclasses are the only fields.pm users I've ever encountered in the wild. Removing fields may make our code more approachable to other Perl hackers. So stop using fields.pm and locked hashes, but continue to document what fields do for non-trivial classes.
2020-06-28watch: use signalfd for Maildir watching
We can get rid of the janky wannabe self-using-a-directory-instead-of-pipe thing we needed to workaround Filesys::Notify::Simple being blocking. For existing Maildir users, this should be more robust and immune to missed wakeups for signalfd and kqueue-enabled systems; as well as being immune to BOFHs clearing $TMPDIR and preventing notifications from firing. The IMAP IDLE code still uses normal Perl signals, so it's still vulnerable to missed wakeups. That will be addressed in future commits.
2020-06-28watch: remove Filesys::Notify::Simple dependency
Since we already use inotify and EVFILT_VNODE (kqueue) in -imapd, we might as well use them directly in -watch, too. This will allow public-inbox-watch to use PublicInbox::DS for timers to watch newsgroups/mailboxes and have saner signal handling in future commits.
2020-06-28kqnotify|fake_inotify: detect Maildir write ops
We need to detect link(2) and rename(2) in other apps writing to the Maildir. We'll be removing the Filesys::Notify::Simple from -watch in favor of using IO::KQueue or Linux::Inotify2 directly. Ensure non-inotify emulations can support everything we expect for Maildir writers.
2020-06-28watch: preliminary IMAP support
Only servers with IDLE are supported, for now. Polling will be needed since users may need to watch many inboxes with a few active connections due to IMAP server limitations.
2020-06-28URI IMAP support
We'll be supporting the IMAP URL scheme described in RFC 5092 for -watch, so add this module to fill in what the `URI' package lacks.
2020-06-28watchmaildir: fix check for spam vs ham inbox conflicts
The old check was ineffective since we process the spam folder config before ham inboxes; and would only fail when attempting to treat the scalar "watchspam" string as an array ref.
2020-06-28watchmaildir: hoist out compile_watchheaders
It's too deeply indented, and we will be using it for IMAP, too.
2020-06-28imaptracker: use ~/.local/share/public-inbox/imap.sqlite3
Respect XDG_DATA_HOME to avoid cluttering ~/.public-inbox/. Existing users of ~/.public-inbox/imap.sqlite3 will remain supported, but the preference for new data is to use ~/.local/share and other paths standardized by XDG. Cc: "Eric W. Biederman" <ebiederm@xmission.com>
2020-06-28IMAPTracker: Add a helper to track our place in reading imap mailboxes
This removes the need to delete from an imap mailbox when downloading it's messages. [ew: minor style changes] Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2020-06-28inbox: warn on ->on_inbox_unlock exception
Otherwise, we may never know what went wrong.
2020-06-28inboxwritable: ensure ssoma.lock exists on init
This will allow us to use InboxIdle on empty/unindexed v1 inboxes.
2020-06-28eml: header_str_set: correctly encode UTF-8 headers
Apparently, using $1 from an octet string still results in a multi-byte string. Thus we need to perform utf8::encode after the regexp character match to ensure wide characters don't get passed to encode_base64. This fixes a bug in which caused -watch to crash when using PublicInbox::Filter::SubjectTag to remove "[list prefix]" tags from Subject: lines. I only found this bug because the proposed -watch updates for NNTP/IMAP support introduced a possible bug which triggered a full rescan of old archives: https://public-inbox.org/meta/20200627100400.9871-1-e@yhbt.net/
2020-06-27linkify: support imap, imaps, news, and snews URIs
Since we'll have an IMAP server released soon, maybe imaps:// and imap:// URLs can become popular. news:// is defined with nntp:// in RFC 5538, and we can at least support the news:// form in rendered HTML. snews:// may appear in old mail archives, too, so we'll attempt to support it in case clients do.
2020-06-27imap: EXAMINE: avoid potential race conditions
We need to rely on num_highwater for UIDNEXT since the highest `num' stored in over.sqlite3 may be rolled back if the most recent messages were spam. We also need to load the uo2m immediately on EXAMINE to ensure EXISTS responses are always consistent with regard to future updates.
2020-06-27imap: always send EXISTS on uo2m_extend
Clients which are NOT in an IDLE state still need to be notified of message existence. Unlike the EXPUNGE response, untagged EXISTS responses seem to be allowed at any time according to RFC 3501. We'll also perform uo2m_extend on the NOOP command, since NOOP is the recommended command for message polling.
2020-06-25ds: flush_write: early return on closed socket
This quiets warnings from IMAP::fetch_blob (called via long_response) failing to access `$self->{ibx}->git' because ->{ibx} gets deleted by IMAP::close.
2020-06-25git_async_cat: remove circular reference
While this circular reference was carefully managed to not leak memory; it was still triggering a warning at -imapd/-nntpd shutdown due to the EPOLL_CTL_DEL op failing after the $Epoll FD gets closed. So remove the circular reference by providing a ref to `undef', instead.
2020-06-25lock: reduce inotify wakeups
We can reduce the amount of platform-specific code by always relying on IN_MODIFY/NOTE_WRITE notifications from lock release. This reduces the number of times our read-only daemons will need to wake up when -watch sees no-op message changes (e.g. replied, seen, recent flag changes).
2020-06-23testcommon: DS->Reset when using fork-only subprocess
This fixes a bug on FreeBSD 11 here -nntpd + TEST_RUN_MODE=2 (default) was occasionally causing failures in t/v2writable.t due to the kqueue descriptor being auto-closed by the OS on fork.
2020-06-23init: add --skip-artnum parameter
For archivists with only newer mail archives, this option allows reserving reserve NNTP article numbers for yet-to-be-archived old messages. Indexers will need to be updated to support this feature in future commits. -V1 inboxes will now be initialized with SQLite and Xapian support if this option is used, or if --indexlevel= is specified.
2020-06-23imap: refill_xap: remove needless loop
There's no need to loop when the first iteration guarantees a `return'.
2020-06-21testcommon: require IO::Socket::SSL >= 2.007
Net::NNTP does not support older IO::Socket::SSL. 1.94 on CentOS 7.x fails HTTPS and IMAPS tests, too. cf. https://rt.cpan.org/Ticket/Display.html?id=100529
2020-06-21nntp: support slow blob retrievals
Having `git cat-file' as a separate process naturally lends itself to asynchronous dispatch. Our event loop for -nntpd no longer blocks on slow git storage. Pipelining in -imapd was tricky and bugs were exposed by mbsync(1). Update t/nntpd.t to support pipelining ARTICLE requests to ensure we don't have the same problems -imapd did during development.
2020-06-21nntp: event_step: prepare for async git reads
This matches PublicInbox::IMAP::event_step and will allow us to handle blob retrievals from git asynchronously without falling over on pipelined requests.
2020-06-21daemon: use ->can to check for IO::Socket::SSL
Doing a ref($obj) string comparison ties us to IO::Socket::SSL (and OpenSSL) In the future, we may support GnuTLS or other TLS implementations. This was already done in the IMAP code.
2020-06-16imap: fix UID-offset-to-MSN mapping bugs
We need to clear the UID-offset-to-MSN mapping when leaving mailboxes via EXAMINE/SELECT/CLOSE. Furthermore, uo2m_last_uid() needs to account for tiny mailboxes where the scalar representation of {uo2m} may be evaluated to `false' in a boolean context.
2020-06-16imap: *SEARCH: reinstate "TEXT" search-key
I accidentally dropped "TEXT" handling while porting the IMAP search query parser to Parse::RecDescent. This reinstates it and adds a test to prevent future regression, and the additional test fixes a counting error for non-Xapian-enabled systems.
2020-06-16imap: *SEARCH: fix CHARSET handling
We no longer pass an arrayref to search_common() or parse_query(), so handle the CHARSET directive in the Parse::RecDescent-generated parser directly.
2020-06-16imap: *SEARCH: use Parse::RecDescent
For properly parsing IMAP search requests, it's easier to use a recursive descent parser generator to deal with subqueries and the "OR" statement. Parse::RecDescent was chosen since it's mature, well-known, widely available and already used by our optional dependencies: Inline::C and Mail::IMAPClient. While it's possible to build Xapian queries without using the Xapian string query parser; this iteration of the IMAP parser still builds a string which is passed to Xapian's query parser for ease-of-diagnostics. Since this is a recursive descent parser dealing with untrusted inputs, subqueries have a nesting limit of 10. I expect that is more than adequate for real-world use.
2020-06-16imap: reinstate non-UID SEARCH
Since we support MSNs properly, now, it seems acceptable to support regular SEARCH requests in case there are any clients which still use non-UID SEARCH.
2020-06-16imap: stop_idle: fix parameter parsing :x
stop_idle was a noop when the client issues a "DONE" continuation or just disconnects. This would not have led to a long term memory leak since FDs get closed and reused, anyways, and all of our InboxIdle mappings are keyed by FD.