| Date | Commit message (Collapse) |
|---|
|
Since it seems somewhat common for IMAP clients to limit
searches by sent Date: or INTERNALDATE, we can rely on
the NNTP/WWW-optimized overview DB.
For other queries, we'll have to depend on the Xapian DB.
|
|
We won't support searching across mailboxes, just yet;
but maybe in the future.
|
|
None of the new cases are wired up, yet, but existing cases
still work.
|
|
No point in spewing "uninitialized" warnings into logs when
the cat jumps on the Enter key.
|
|
We can share code between them and account for each 50K
mailbox slice. However, we must overreport these for
non-zero slices and just return lots of empty data for
high-numbered slices because some MUAs still insist
on non-UID fetches.
|
|
Some clients insist on sending "INBOX" in all caps,
since it's special in RFC 3501.
|
|
Having two large numbers separated by a dash can make visual
comparisons difficult when numbers are in the 3,000,000 range
for LKML. So avoid the $UID_END value, since it can be
calculated from $UID_MIN. And we can avoid large values of
$UID_MIN, too, by instead storing the block index and just
multiplying it by 50000 (and adding 1) on the server side.
Of course, LKML still goes up to 72, at the moment.
|
|
Finish up the IMAP-only portion of iterative config reloading,
which allows us to create all sub-ranges of an inbox up front.
The InboxIdler still uses ->each_inbox which will struggle with
100K inboxes.
Having messages in the top-level newsgroup name of an inbox will
still waste bandwidth for clients which want to do full syncs
once there's a rollover to a new 50K range. So instead, make
every inbox accessible exclusively via 50K slices in the form of
"$NEWSGROUP.$UID_MIN-$UID_END".
This introduces the DummyInbox, which makes $NEWSGROUP
and every parent component a selectable, empty inbox.
This aids navigation with mutt and possibly other MUAs.
Finally, the xt/perf-imap-list maintainer test is broken, now,
so remove it. The grep perlfunc is already proven effective,
and we'll have separate tests for mocking out ~100k inboxes.
|
|
This limit on mailbox size should keep users of tools like
mbsync (isync) and offlineimap happy, since typical filesystems
struggle with giant Maildirs.
I chose 50K since it's a bit more than what LKML typically sees
in a month and still manages to give acceptable performance on
my ancient Centrino laptop.
There were also no responses to my original proposal at:
<https://public-inbox.org/meta/20200519090000.GA24273@dcvr/>
so no objections, either :>
|
|
"$UID_START:*" needs to return at least one message according
to RFC 3501 section 6.4.8.
While we're in the area, coerce ranges to (unsigned) integers by
adding zero ("+ 0") to reduce memory overhead.
|
|
Trying to avoid a circular reference by relying on $ibx object
here makes no sense, since skipping GitCatAsync::close will
result in an FD leak, anyways. So keep GitAsyncCat contained to
git-only operations, since we'll be using it for Solver in the
distant feature.
|
|
Since IMAP yields control to GitAsyncCat, IMAP->event_step may
be invoked with {long_cb} still active. We must be sure to
bail out of IMAP->event_step if that happens and continue to let
GitAsyncCat drive IMAP.
This also improves fairness by never processing more than one
request per ->event_step.
|
|
The RFC 3501 `sequence-set' definition allows comma-delimited
ranges, so we'll support it in case clients send them.
Coalescing overlapping ranges isn't required, so we won't
support it as such an attempt to save bandwidth would waste
memory on the server, instead.
|
|
Since we only support read-only operation, we can't save
subscriptions requested by clients. So just list no inboxes as
subscribed, some MUAs may blindly try to fetch everything its
subscribed to.
|
|
This ought to improve overall performance with multiple clients.
Single client performance suffers a tiny bit due to extra
syscall overhead from epoll.
This also makes the existing async interface easier-to-use,
since calling cat_async_begin is no longer required.
|
|
While we can't memoize the regexp forever like we do with other
Eml users, we can still benefit from caching regexp compilation
on a per-request basis.
A FETCH request from mutt on a 4K message inbox is around 8%
faster after this. Since regexp compilation via qr// isn't
unbearably slow, a shared cache probably isn't worth the
trouble of implementing. A per-request cache seems enough.
|
|
It seems worthless to support CLOSE for read-only inboxes, but
mutt sends it, so don't return a BAD error with proper use.
|
|
They're not specified in RFC 3501 for responses, and at least
mutt fails to handle it.
|
|
We'll return dummy messages for now when sequence numbers go
missing, in case clients can't handle missing messages.
|
|
While the contents of normal %want hash keys are bounded in
size, %partial can cause more overhead and lead to repeated sort
calls on multi-message fetches. So sort it once and use
arrayrefs to make the data structure more compact.
|
|
We must keep the contents of {-partial} around when handling
a request to fetch multiple messages.
|
|
Mail::IMAPClient doesn't seem to mind the lack of `resp-text';
but it's required by RFC 3501. Preliminary tests with
offlineimap(1) indicates the presence of `resp-text' is
necessary, even if it's just the freeform `text'.
And make the `text' more consistent, favoring "done" over
"complete" or "completed"; while we're at it.
|
|
IMAP supports a high level of granularity when it comes to
fetching, but fortunately Perl makes it fairly easy to support.
|
|
I'm not sure which clients use these, but it could be useful
down the line.
|
|
We'll optimize for the common case of: $TAG LIST "" *
and rely on the grep perlfunc to handle trickier cases.
|
|
IMAP clients may quote args and escape similar to POSIX shell,
so attempt to handle them properly using this standard library
module.
|
|
I'm not sure if there's much use for this command, but it's
part of RFC3501 and works read-only.
|
|
InboxIdle should not be holding onto Inbox objects after the
Config object they came from expires, and Config objects may
expire on SIGHUP.
Old Inbox objects still persist due to IMAP clients holding onto
them, but that's a concern we'll deal with at another time, or
not at all, since all clients expire, eventually.
Regardless, stale inotify watch descriptors should not be left
hanging after SIGHUP refreshes.
|
|
There's enough places where we only care about the max NNTP
article number to warrant avoiding a call into SQLite.
Using ->num_highwater in read-only packages such as
PublicInbox::IMAP is also incorrect, since that memoizes
and won't pick up changes made by other processes.
|
|
It seems to be working as far as Mail::IMAPClient is concerned.
|
|
It shares a bit of code with NNTP. It's copy+pasted for now
since this provides new ground to experiment with APIs for
dealing with slow storage and many inboxes.
|