Date | Commit message (Collapse) |
|
IMAP allows retrieving multiple messages with a single command,
and Mail::IMAPClient supports that. Unfortunately, it means we
slurp multiple messages into memory at once. This option allows
users to trade off memory usage to reduce network round-trips.
Ideally, we'd support pipelining; but AFAIK no widely installed
Perl IMAP library supports it.
|
|
With different polling intervals, multiple processes may
simultaneously write to IMAPtracker. This ought to reduce
SQLite busy waiting and contention issues when importing
many inboxes in parallel.
|
|
It's not used anywhere since the IMAPTracker object doesn't
disconnect and reconnect. If we ever need the filename,
{dbh}->sqlite_db_filename may be used.
Cc: Eric W. Biederman <ebiederm@xmission.com>
|
|
Passing a $url parameter to every function was error-prone
and having {url} field for a short-lived object is appropriate.
This matches the version of IMAPTracker posted by
Eric W. Biederman on 2020-05-15 at:
https://public-inbox.org/meta/87ftc0c3r4.fsf_-_@x220.int.ebiederm.org/
The version I originally imported was based on the one
posted on 2019-10-09:
https://public-inbox.org/meta/874l0i9vhc.fsf_-_@x220.int.ebiederm.org/
Cc: Eric W. Biederman <ebiederm@xmission.com>
|
|
This allows callers to avoid creating expensive closures.
We no longer pass the `$now' value to callers, as none of
the callers used it.
|
|
For mailboxes with many gaps in the UID sequence,
performing a UID SEARCH beforehand can reduce the
number of articles to fetch.
However, the downside to this is we may end up with
an arbitrarly large list of UIDs from the server.
|
|
This fixes cases where watch is handling both Maildirs and IMAP
connections. While we're at it, close open directories in the
IMAP children to save FDs.
|
|
Since we have IMAP client support in -watch; make sure per-URL
settings are familiar to git users by taking advantage of git's
URL matching abilities.
This requires git 1.8.5+, which most users ought to have
(though base CentOS 7 is on 1.8.3).
|
|
Not all IMAP servers support IDLE, and IDLE may be prohibitively
expensive for some IMAP servers with many inboxes. So allow
configuring a imap.$IMAP_URL.pollInterval=SECONDS to poll
mailboxes.
We'll also need to poll for NNTP servers in the future.
|
|
We can avoid synchronous `waitpid(-1, 0)' and save a process
when simultaneously watching Maildirs.
One DS bug is fixed: ->Reset needs to clear the DS $in_loop flag
in forked children so dwaitpid() fails and allows git processes
to be reaped synchronously. TestCommon also calls DS->Reset
when spawning new processes, since t/imapd.t uses DS->EventLoop
while waiting on -watch to write.
|
|
Since the removal of pseudo-hash support in Perl 5.10, the
"fields" module no longer provides the space or speed benefits
it did in 5.8. It also does not allow for compile-time checks,
only run-time checks.
To me, the extra developer overhead in maintaining "use fields"
args has become a hassle. None of our non-DS-related code uses
fields.pm, nor do any of our current dependencies. In fact,
Danga::Socket (which DS was originally forked from) and its
subclasses are the only fields.pm users I've ever encountered in
the wild. Removing fields may make our code more approachable
to other Perl hackers.
So stop using fields.pm and locked hashes, but continue to
document what fields do for non-trivial classes.
|
|
We can get rid of the janky wannabe
self-using-a-directory-instead-of-pipe thing we needed to
workaround Filesys::Notify::Simple being blocking.
For existing Maildir users, this should be more robust and
immune to missed wakeups for signalfd and kqueue-enabled
systems; as well as being immune to BOFHs clearing $TMPDIR
and preventing notifications from firing.
The IMAP IDLE code still uses normal Perl signals, so it's still
vulnerable to missed wakeups. That will be addressed in future
commits.
|
|
Since we already use inotify and EVFILT_VNODE (kqueue)
in -imapd, we might as well use them directly in -watch,
too.
This will allow public-inbox-watch to use PublicInbox::DS
for timers to watch newsgroups/mailboxes and have saner
signal handling in future commits.
|
|
We need to detect link(2) and rename(2) in other apps
writing to the Maildir.
We'll be removing the Filesys::Notify::Simple from -watch
in favor of using IO::KQueue or Linux::Inotify2 directly.
Ensure non-inotify emulations can support everything we
expect for Maildir writers.
|
|
Only servers with IDLE are supported, for now. Polling will
be needed since users may need to watch many inboxes with
a few active connections due to IMAP server limitations.
|
|
We'll be supporting the IMAP URL scheme described in RFC 5092
for -watch, so add this module to fill in what the `URI' package
lacks.
|
|
The old check was ineffective since we process the spam folder
config before ham inboxes; and would only fail when attempting
to treat the scalar "watchspam" string as an array ref.
|
|
It's too deeply indented, and we will be using it for IMAP, too.
|
|
Respect XDG_DATA_HOME to avoid cluttering ~/.public-inbox/.
Existing users of ~/.public-inbox/imap.sqlite3 will remain
supported, but the preference for new data is to use
~/.local/share and other paths standardized by XDG.
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
This removes the need to delete from an imap mailbox when
downloading it's messages.
[ew: minor style changes]
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Otherwise, we may never know what went wrong.
|
|
This will allow us to use InboxIdle on empty/unindexed v1 inboxes.
|
|
Apparently, using $1 from an octet string still results in a
multi-byte string. Thus we need to perform utf8::encode after
the regexp character match to ensure wide characters don't get
passed to encode_base64.
This fixes a bug in which caused -watch to crash when using
PublicInbox::Filter::SubjectTag to remove "[list prefix]"
tags from Subject: lines.
I only found this bug because the proposed -watch updates for
NNTP/IMAP support introduced a possible bug which triggered a
full rescan of old archives:
https://public-inbox.org/meta/20200627100400.9871-1-e@yhbt.net/
|
|
Since we'll have an IMAP server released soon, maybe imaps://
and imap:// URLs can become popular.
news:// is defined with nntp:// in RFC 5538, and we can at least
support the news:// form in rendered HTML. snews:// may appear
in old mail archives, too, so we'll attempt to support it in
case clients do.
|
|
We need to rely on num_highwater for UIDNEXT since the
highest `num' stored in over.sqlite3 may be rolled back
if the most recent messages were spam.
We also need to load the uo2m immediately on EXAMINE to ensure
EXISTS responses are always consistent with regard to future
updates.
|
|
Clients which are NOT in an IDLE state still need to be
notified of message existence. Unlike the EXPUNGE response,
untagged EXISTS responses seem to be allowed at any time
according to RFC 3501.
We'll also perform uo2m_extend on the NOOP command, since
NOOP is the recommended command for message polling.
|
|
This quiets warnings from IMAP::fetch_blob (called via
long_response) failing to access `$self->{ibx}->git'
because ->{ibx} gets deleted by IMAP::close.
|
|
While this circular reference was carefully managed to not leak
memory; it was still triggering a warning at -imapd/-nntpd
shutdown due to the EPOLL_CTL_DEL op failing after the $Epoll FD
gets closed.
So remove the circular reference by providing a ref to `undef',
instead.
|
|
We can reduce the amount of platform-specific code by always
relying on IN_MODIFY/NOTE_WRITE notifications from lock release.
This reduces the number of times our read-only daemons will
need to wake up when -watch sees no-op message changes
(e.g. replied, seen, recent flag changes).
|
|
This fixes a bug on FreeBSD 11 here -nntpd + TEST_RUN_MODE=2
(default) was occasionally causing failures in t/v2writable.t
due to the kqueue descriptor being auto-closed by the OS on fork.
|
|
I used find(1) here for debugging. The "make check-run" test
target needs to be updated to make stderr spew more obvious.
|
|
For archivists with only newer mail archives, this option allows
reserving reserve NNTP article numbers for yet-to-be-archived
old messages. Indexers will need to be updated to support this
feature in future commits.
-V1 inboxes will now be initialized with SQLite and Xapian
support if this option is used, or if --indexlevel= is
specified.
|
|
Since V2 uses multiple git repositories, stop using
the word "repo" when referring to inboxes.
|
|
On a powerful (by my standards) machine with 16GB RAM and an
7200 RPM HDD marketed for "enterprise" use, indexing a 8.1G (in
git) LKML snapshot from Sep 2019 did not finish after 7 days
with the default number (3) of Xapian shards (`--jobs=4') and
`--batch-size=10m'.
Indexing starts off fast, but progressively get slower as
contents of the inbox (including Xapian + SQLite DBs) could no
longer be cached by the kernel. Once the on-disk size
increased, HDD seek contention between the Xapian shard workers
slowed the process down to a crawl.
With a single shard, it still took around 3.5 days to index on
the HDD. That's not good, but it's far better than not
finishing after 7 days. So allow unfortunate HDD users to
easily specify a single shard on public-inbox-init.
For reference, a freshly TRIM-ed low-end TLC SSD on the SATA II
bus on the same machine indexes that same snapshot of LKML in
~7 hours with 3 shards and the same 10m batch size. In the past,
a higher-end consumer grade MLC SSDs on similar hardware indexed
a similarly sized-data set in ~4 hours.
|
|
There's no need to loop when the first iteration guarantees
a `return'.
|
|
We also need to check for git 2.6 earlier in each test case,
before any other TAP output is emitted to avoid confusing the
TAP consumers.
|
|
Net::NNTP does not support older IO::Socket::SSL. 1.94 on
CentOS 7.x fails HTTPS and IMAPS tests, too.
cf. https://rt.cpan.org/Ticket/Display.html?id=100529
|
|
Having `git cat-file' as a separate process naturally lends
itself to asynchronous dispatch. Our event loop for -nntpd no
longer blocks on slow git storage.
Pipelining in -imapd was tricky and bugs were exposed by
mbsync(1). Update t/nntpd.t to support pipelining ARTICLE
requests to ensure we don't have the same problems -imapd
did during development.
|
|
This matches PublicInbox::IMAP::event_step and will allow us to
handle blob retrievals from git asynchronously without falling
over on pipelined requests.
|
|
Doing a ref($obj) string comparison ties us to IO::Socket::SSL
(and OpenSSL) In the future, we may support GnuTLS or other TLS
implementations. This was already done in the IMAP code.
|
|
We need to clear the UID-offset-to-MSN mapping when
leaving mailboxes via EXAMINE/SELECT/CLOSE.
Furthermore, uo2m_last_uid() needs to account for tiny mailboxes
where the scalar representation of {uo2m} may be evaluated to
`false' in a boolean context.
|
|
I accidentally dropped "TEXT" handling while porting
the IMAP search query parser to Parse::RecDescent.
This reinstates it and adds a test to prevent future
regression, and the additional test fixes a counting
error for non-Xapian-enabled systems.
|
|
We no longer pass an arrayref to search_common() or
parse_query(), so handle the CHARSET directive in
the Parse::RecDescent-generated parser directly.
|
|
For properly parsing IMAP search requests, it's easier to use a
recursive descent parser generator to deal with subqueries and
the "OR" statement.
Parse::RecDescent was chosen since it's mature, well-known,
widely available and already used by our optional dependencies:
Inline::C and Mail::IMAPClient. While it's possible to build
Xapian queries without using the Xapian string query parser;
this iteration of the IMAP parser still builds a string which is
passed to Xapian's query parser for ease-of-diagnostics.
Since this is a recursive descent parser dealing with untrusted
inputs, subqueries have a nesting limit of 10. I expect that is
more than adequate for real-world use.
|
|
Since we support MSNs properly, now, it seems acceptable
to support regular SEARCH requests in case there are any
clients which still use non-UID SEARCH.
|
|
|
|
stop_idle was a noop when the client issues a "DONE"
continuation or just disconnects. This would not have
led to a long term memory leak since FDs get closed and
reused, anyways, and all of our InboxIdle mappings are
keyed by FD.
|
|
Since IMAP IDLE users aren't expected to issue any commands, we
can terminate their connections immediately on graceful
shutdown.
Furthermore, we need to drop the inotify FD from the epoll set
to avoid warnings during global destruction. Embarassingly,
this required fixing wacky test ordering from 2a717d13f10fcdc6
("nntpd+imapd: detect replaced over.sqlite3")
|
|
"DONE" is a continuation and not a normal IMAP command, so
ensure it can't be called like a normal IMAP command which
has a tag.
|
|
Mail::IMAPClient understandably stumbles into a warning
by our bogus test request. Just silence it on our end
since it's not normal operation for Mail::IMAPClient.
|