| Date | Commit message (Collapse) |
|---|
|
This should be able to replace multiple `git cat-file' for blob
retrieval, but adjustments may be needed.
|
|
Having tens of thousands of inboxes and associated git processes
won't work well, so we'll use libgit2 to access the object DB
directly. We only care about OID lookups and won't need to rely
on per-repo revision names or paths.
The Git::Raw XS package won't be used since its manpages don't
promise a stable API. Since we already use Inline::C and have
experience with I::C when it comes to compatibility, this only
introduces libgit2 itself as a source of new incompatibilities.
This also provides an excuse for me to writev(2) to reduce
syscalls, but liburing is on the horizon for next year.
|
|
We will need to allow simultaneous iterators on the same
config object, since we'll need this for ExtMsg, NNTPD,
WwwListing, NewsWWW, and other places.
|
|
It's still as slow as before with hundreds/thousands of inboxes,
but at least it's fair. Future changes will allow it to be
cached and memoized with persistent HTTP servers.
|
|
While it's not a known problem, our deduplicating logic may
change in the future; or a BOFH could be manually injecting
duplicate messages directly into the git epoch repositories.
Ensure indexing in mirrors doesn't break when there's
duplicates. This is in preparation for detached indices
for multi-inbox search.
|
|
ParentPipe was a subset of EOFpipe, except EOFpipe correctly
accounts for theoretical(*) spurious wakeups on the pipe.
(*) AFAIK, spurious wakeups are/were more likely on TCP sockets
due to checksum failures, something that's not a problem on
local pipes. We're also not sharing pipes like we do with
listen sockets on accept(2), so there's no chance of another
process grabbing bytes (unless we have bugs in our code).
|
|
It's a bit inefficient to use a pipe, here. However, using
dwaitpid() on a process that's not expected to exit soon is
also inefficient as it causes excessive wakeups as most of
our inbox-writing code expects synchronous waitpid().
This only affects -watch instances configured for NNTP and IMAP
clients.
|
|
This is no longer limited to Maildirs now that IMAP and NNTP
support exist; so give it a shorter name.
|
|
We've got examples for all the other daemons, too!
|
|
Since this was already a separate package, split it off
into its own file since SearchView may not handle inbox
groups.
|
|
Determining storage device speed and latencies doesn't
seem portable or even possible with the wide variety
of storage layers in use.
This means we need to write a tuning document and hope
users read and improve on it :P
|
|
No need to waste resources when doing minimal work. With
PI_TEST_VERSION=2, this fixes a test failure where
Net::NNTP::DESTROY was getting called in the shard process.
We'll also get rid of an unnecessary use_ok under v2, too.
|
|
SQLite and Xapian files are written randomly, thus they become
fragmented under btrfs with copy-on-write. This leads to
noticeable performance problems (and probably ENOSPC) as these
files get big.
lore/git (v2, <1GB) indexes around 20% faster with this on an
ancient SSD. lore/lkml seems to be taking forever and I'll
probably cancel it to save wear on my SSD.
Unfortunately, disabling CoW also means disabling checksumming
(and compression), so we'll be careful to only set the No_COW
attribute on regeneratable data. We want to keep CoW (and
checksums+compression) on git storage because current ref
storage is neither checksummed nor compressed, and git streams
pack output.
|
|
This avoids pinning a potentially large chunk of memory from
`git-log --reverse' into RAM (or triggering less predictable
swap behavior). Instead it uses a contiguous temporary file
with a fixed-size record for every blob we'll need to index.
|
|
Since we'll need to expose THREADID to JMAP and IMAP users,
index all messages in the order they were committed to ensure
our `tid' (thread ID) column ascends in mirrors the same way
they do in the source inbox.
This drastically simplifies our code but increases memory
usage of `git-log'. The next commit will bring memory use
back down at the expense of $TMPDIR usage.
|
|
Virtually all of our responses are going to be gzipped, anyways.
This will allow us to utilize zlib as a buffering layer and
share common code for async blob retrieval responses.
To streamline this and allow GzipFilter to be a parent class,
we'll replace the NoopFilter with a similar CompressNoop class
which emulates the two Compress::Raw::Zlib::Deflate methods we
use.
This drops a bunch of redundant code and will hopefully make
upcoming WwwStream changes easier to reason about.
|
|
This lets the -httpd worker process make better use of time
instead of waiting for git-cat-file to respond. With 4 jobs in
the new test case against a clone of
<https://public-inbox.org/meta/>, a speedup of 10-12% is shown.
Even a single job shows a 2-5% improvement on an SSD.
|
|
The changes to GzipFilter here may be beneficial for building
HTML and XML responses in other places, too.
|
|
Git.pm may not be installed on some systems; or some users have
multiple Perl installations and Git.pm is not available to the
Perl running -watch. Accomodate both those types of users by
providing our own "git credential" wrapper.
|
|
This is similar to IMAP support, but only supports polling.
Automatic altid support is not yet supported, yet; but may
be in the future.
v2: small grammar fix by Kyle Meyer
Link: https://public-inbox.org/meta/87sgeg5nxf.fsf@kyleam.com/
|
|
Since we already use inotify and EVFILT_VNODE (kqueue)
in -imapd, we might as well use them directly in -watch,
too.
This will allow public-inbox-watch to use PublicInbox::DS
for timers to watch newsgroups/mailboxes and have saner
signal handling in future commits.
|
|
We need to detect link(2) and rename(2) in other apps
writing to the Maildir.
We'll be removing the Filesys::Notify::Simple from -watch
in favor of using IO::KQueue or Linux::Inotify2 directly.
Ensure non-inotify emulations can support everything we
expect for Maildir writers.
|
|
Only servers with IDLE are supported, for now. Polling will
be needed since users may need to watch many inboxes with
a few active connections due to IMAP server limitations.
|
|
We'll be supporting the IMAP URL scheme described in RFC 5092
for -watch, so add this module to fill in what the `URI' package
lacks.
|
|
Respect XDG_DATA_HOME to avoid cluttering ~/.public-inbox/.
Existing users of ~/.public-inbox/imap.sqlite3 will remain
supported, but the preference for new data is to use
~/.local/share and other paths standardized by XDG.
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
This removes the need to delete from an imap mailbox when
downloading it's messages.
[ew: minor style changes]
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
For properly parsing IMAP search requests, it's easier to use a
recursive descent parser generator to deal with subqueries and
the "OR" statement.
Parse::RecDescent was chosen since it's mature, well-known,
widely available and already used by our optional dependencies:
Inline::C and Mail::IMAPClient. While it's possible to build
Xapian queries without using the Xapian string query parser;
this iteration of the IMAP parser still builds a string which is
passed to Xapian's query parser for ease-of-diagnostics.
Since this is a recursive descent parser dealing with untrusted
inputs, subqueries have a nesting limit of 10. I expect that is
more than adequate for real-world use.
|
|
|
|
Since we limit our mailboxes slices to 50K and can guarantee a
contiguous UID space for those mailboxes, we can store a mapping
of "UID offsets" (not full UIDs) to Message Sequence Numbers as
an array of 16-bit unsigned integers in a 100K scalar.
For UID-only FETCH responses, we can momentarily unpack the
compact 100K representation to a ~1.6M Perl array of IV/UV
elements for a slight speedup.
Furthermore, we can (ab)use hash key deduplication in Perl5 to
deduplicate this 100K scalar across all clients with the same
mailbox slice open.
Technically we can increase our slice size to 64K w/o increasing
our storage overhead, but I suspect humans are more accustomed
to slices easily divisible by 10.
|
|
Finish up the IMAP-only portion of iterative config reloading,
which allows us to create all sub-ranges of an inbox up front.
The InboxIdler still uses ->each_inbox which will struggle with
100K inboxes.
Having messages in the top-level newsgroup name of an inbox will
still waste bandwidth for clients which want to do full syncs
once there's a rollover to a new 50K range. So instead, make
every inbox accessible exclusively via 50K slices in the form of
"$NEWSGROUP.$UID_MIN-$UID_END".
This introduces the DummyInbox, which makes $NEWSGROUP
and every parent component a selectable, empty inbox.
This aids navigation with mutt and possibly other MUAs.
Finally, the xt/perf-imap-list maintainer test is broken, now,
so remove it. The grep perlfunc is already proven effective,
and we'll have separate tests for mocking out ~100k inboxes.
|
|
imapd-validate is a beefed up version of our nntpd-validate test
which hammers the server with parallel connections over regular
IMAP, IMAPS, IMAP+STARTTLS; and COMPRESS=DEFLATE variants of
each of those. It uses $START_UID:$END_UID fetch ranges to
reduce requests and slurp many responses at once to saturate
"git cat-file --batch" processes.
mbsync(1) also uses pipelining extensively (but IMHO
unnecessarily), so it was able to shake out some bugs in
the async git code.
Finally, we remove xt/cmp-imapd-compress.t since it's
redundant now that we have PublicInbox::IMAPClient to work
around bugs in Mail::IMAPClient.
|
|
We'll be using this wrapper class to workaround some upstream
bugs in Mail::IMAPClient. There may also be experiments with
new APIs for more performance.
|
|
Include a test for Mail::IMAPTalk, here, since Mail::IMAPClient
stalls with compression enabled:
https://rt.cpan.org/Ticket/Display.html?id=132720
|
|
This ought to improve overall performance with multiple clients.
Single client performance suffers a tiny bit due to extra
syscall overhead from epoll.
This also makes the existing async interface easier-to-use,
since calling cat_async_begin is no longer required.
|
|
This makes the test code easier-to-manage and allows us to run
faster unit tests which don't involve loading Mail::IMAPClient.
|
|
IMAP supports a high level of granularity when it comes to
fetching, but fortunately Perl makes it fairly easy to support.
|
|
This will be used to implement IMAP IDLE, first.
Eventually, it may be used to trigger other things:
* incremental internal updates for manifest.js.gz
* restart `git cat-file' processes on pack index unlink
* IMAP IDLE-like long-polling HTTP endpoint
And maybe more things we haven't thought of, yet.
It uses Linux::Inotify2 or IO::KQueue depending on what packages
are installed and what the kernel supports. It falls back to
nanosecond-aware Time::HiRes::stat() (available with Perl 5.10.0+)
on systems lacking Linux::Inotify2 and IO::KQueue.
In the future, a pure Perl alternative to Linux::Inotify2 may be
supplied for users of architectures we already support signalfd
and epoll on.
v2 changes:
- avoid O_TRUNC on lock file
- change ctime on Linux systems w/o inotify
- fix naming of comments and fields
|
|
It shares a bit of code with NNTP. It's copy+pasted for now
since this provides new ground to experiment with APIs for
dealing with slow storage and many inboxes.
|
|
We'll be adding support to descend into message/rfc822 (and
legacy message/news) attachments. First, we must ensure
existing message/rfc822 attachments can be downloaded and remain
downloadable in future commits.
|
|
The old name may be confused with "Content-ID" as described in
RFC 2392, so use an alternate name to avoid confusing future
readers.
|
|
This allows maintainers to easily check limits against the
contents of existing inboxes. This script covers most of
the new limits enforced by PublicInbox::Eml.
Usage is similar to most xt/*.t scripts:
GIANT_INBOX_DIR=/path/to/inbox prove -bvw xt/eml_check_limits.t
Setting `TEST_CLASS=PublicInbox::MIME' allows us to check
performance and memory use against the old subclass of
Email::MIME.
|
|
While our codebase can still work with either MIME
implementation, add comparison tests to ensure we
handle corner cases in existing archives.
|
|
Since we're getting rid of Email::MIME, get rid of
Email::MIME::ContentType, too; since we may introduce
speedups down the line specific to our codebase.
|
|
Email::MIME eats memory, wastes time parsing out all the
headers, and some problems can't be fixed without breaking
compatibility for other projects which depend on it.
Informal benchmarks show a ~2x improvement in general
stats gathering scripts and ~10% improvement in HTML
view rendering.
We also don't need the ability to create MIME messages, just
parse them and maybe drop an attachment.
While this isn't the zero-copy or streaming MIME parser of my
dreams; it's still an improvement in that it doesn't keep a
scalar copy of the raw body around along with subparts. It also
doesn't parse subparts up front, so it can also replace our uses
of Email::Simple.
|
|
Since some client tools exist for dealing with public-inbox
specifically, it seems like a good idea to list some of them.
Cc: Danh Doan <congdanhqx@gmail.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: Leah Neukirchen <leah@vuxu.org>
|
|
Replace them with .eml files generated with the help of
Email::MIME, but without some extraneous and unnecessary
headers, and strip mime_load down to just loading files.
This will give us more freedom to experiment with other mail
libraries which may be more correct, better maintained, use
less memory and/or be faster than Email::MIME.
|
|
The watchheader key supports only a single value. Supporting multiple
watchheader values was mentioned in discussion [1] of 8d3e3bd8 (doc:
explain publicinbox.<name>.watchheader, 2019-10-09), and it wasn't
clear if there was a need.
One scenario in which matching multiple headers would be convenient is
when someone wants to set up public-inbox archives for some small
projects but does _not_ want to run mailing lists for them, instead
allowing others to follow the project by any of the pull mechanisms.
Using a common underlying address, an address alias for each project
is configured via a third-party email provider, with messages for each
alias being exposed as a separate public-inbox archive. In this
setup, messages for an inbox cannot be selected by a List-ID header
but can be identified by the inbox's address in either the To or Cc
header.
To support such a use case, update the watchheader handling to
consider multiple values, accepting a message if it matches any value.
While selecting a message based on matching _any_ rather than _all_
values is motivated by the above scenario, it's worth noting that the
"any" behavior is consistent with how multiple listid config values
are handled.
[1] https://public-inbox.org/meta/20191010085118.r3amey4cayazfycb@dcvr/
|
|
I don't consider Perl's memory management "automatic". Instead,
having an extra bit of control as a hacker is nice and there's
no need to burden ordinary users with GC tuning knobs.
|
|
There's nothing Maildir-specific about the function, so
`maildir_path_load' was a bad name. So give it a more
appropriate name and use it in our tests.
This save ourselves some code and inconsistency by reusing an
existing internal library routine in more places. We can drop
the "From_" line in some of our (formerly) mbox sample files.
|
|
|