| Date | Commit message (Collapse) |
|---|
|
None of our tests rely on this failing, so just bail out
if the system is out of resources.
|
|
The RFC 3501 `sequence-set' definition allows comma-delimited
ranges, so we'll support it in case clients send them.
Coalescing overlapping ranges isn't required, so we won't
support it as such an attempt to save bandwidth would waste
memory on the server, instead.
|
|
Since we only support read-only operation, we can't save
subscriptions requested by clients. So just list no inboxes as
subscribed, some MUAs may blindly try to fetch everything its
subscribed to.
|
|
We do this for the C10K-oriented HTTP/NNTP/IMAP processes, and
we may support thousands of git-cat-file processes in the
future.
|
|
This ought to improve overall performance with multiple clients.
Single client performance suffers a tiny bit due to extra
syscall overhead from epoll.
This also makes the existing async interface easier-to-use,
since calling cat_async_begin is no longer required.
|
|
To work with our event loop, we must perform read buffering
ourselves or risk starvation, as there doesn't appear to be
a way to check the amount of data buffered in userspace by
by the PerlIO layers without resorting to C or XS.
This lets us perform fewer syscalls at the expense of more Perl
ops. As it stands, there seems to be a tiny performance
improvement, but more will be possible in the future.
|
|
Small array refs have considerable overhead in Perl, so reduce
AV/SV overhead and instead allow the inflight array to grow
twice as large.
|
|
While we can't memoize the regexp forever like we do with other
Eml users, we can still benefit from caching regexp compilation
on a per-request basis.
A FETCH request from mutt on a 4K message inbox is around 8%
faster after this. Since regexp compilation via qr// isn't
unbearably slow, a shared cache probably isn't worth the
trouble of implementing. A per-request cache seems enough.
|
|
It seems worthless to support CLOSE for read-only inboxes, but
mutt sends it, so don't return a BAD error with proper use.
|
|
They're not specified in RFC 3501 for responses, and at least
mutt fails to handle it.
|
|
We'll return dummy messages for now when sequence numbers go
missing, in case clients can't handle missing messages.
|
|
While the contents of normal %want hash keys are bounded in
size, %partial can cause more overhead and lead to repeated sort
calls on multi-message fetches. So sort it once and use
arrayrefs to make the data structure more compact.
|
|
We must keep the contents of {-partial} around when handling
a request to fetch multiple messages.
|
|
Mail::IMAPClient doesn't seem to mind the lack of `resp-text';
but it's required by RFC 3501. Preliminary tests with
offlineimap(1) indicates the presence of `resp-text' is
necessary, even if it's just the freeform `text'.
And make the `text' more consistent, favoring "done" over
"complete" or "completed"; while we're at it.
|
|
IMAP supports a high level of granularity when it comes to
fetching, but fortunately Perl makes it fairly easy to support.
|
|
Instead of counts starting at 0, we start the single-part
message at 1 like we do with subparts of a multipart message.
This will make it easier to map offsets for "BODY[$SECTION]"
when using IMAP FETCH, since $SECTION must contain non-zero
numbers according to RFC 3501.
This doesn't make any difference for WWW URLs, since single part
messages cannot have downloadable attachments.
|
|
I'm not sure which clients use these, but it could be useful
down the line.
|
|
We can fill in some missing pieces from the emulation APIs
to enable IMAP IDLE tests on non-Linux platforms.
|
|
We'll optimize for the common case of: $TAG LIST "" *
and rely on the grep perlfunc to handle trickier cases.
|
|
IMAP clients may quote args and escape similar to POSIX shell,
so attempt to handle them properly using this standard library
module.
|
|
I'm not sure if there's much use for this command, but it's
part of RFC3501 and works read-only.
|
|
InboxIdle should not be holding onto Inbox objects after the
Config object they came from expires, and Config objects may
expire on SIGHUP.
Old Inbox objects still persist due to IMAP clients holding onto
them, but that's a concern we'll deal with at another time, or
not at all, since all clients expire, eventually.
Regardless, stale inotify watch descriptors should not be left
hanging after SIGHUP refreshes.
|
|
There's enough places where we only care about the max NNTP
article number to warrant avoiding a call into SQLite.
Using ->num_highwater in read-only packages such as
PublicInbox::IMAP is also incorrect, since that memoizes
and won't pick up changes made by other processes.
|
|
It seems to be working as far as Mail::IMAPClient is concerned.
|
|
This will be used to implement IMAP IDLE, first.
Eventually, it may be used to trigger other things:
* incremental internal updates for manifest.js.gz
* restart `git cat-file' processes on pack index unlink
* IMAP IDLE-like long-polling HTTP endpoint
And maybe more things we haven't thought of, yet.
It uses Linux::Inotify2 or IO::KQueue depending on what packages
are installed and what the kernel supports. It falls back to
nanosecond-aware Time::HiRes::stat() (available with Perl 5.10.0+)
on systems lacking Linux::Inotify2 and IO::KQueue.
In the future, a pure Perl alternative to Linux::Inotify2 may be
supplied for users of architectures we already support signalfd
and epoll on.
v2 changes:
- avoid O_TRUNC on lock file
- change ctime on Linux systems w/o inotify
- fix naming of comments and fields
|
|
It shares a bit of code with NNTP. It's copy+pasted for now
since this provides new ground to experiment with APIs for
dealing with slow storage and many inboxes.
|
|
We'll be using newsgroup names as mailbox names for IMAP,
too, so ensure we don't send wonky characters in responses.
I doubt this affects any real-world instances, but a BOFH could
choose strange names to cause grief for clients.
|
|
InboxWritable should only set $v2w->{parallel} if the $parallel
flag is defined to 0 or 1. We want indexing a new inbox to
utilize SMP, just like --reindex.
-index once again allows -j0/--jobs=0 to force single-process
use, and we'll be ensuring that works in tests to maintain
performance on small systems.
Fixes: 61a2fff5b34a3e32 ("admin: move index_inbox over")
|
|
We forcibly stop git-log here, so erroring out on git-log close
failures is wrong since it sees SIGPIPE. Noticed while
reindexing a large v1 inbox for IMAP changes.
Fixes: b32b47fb12a3043d ("index: "git log" failures are fatal")
|
|
It's no longer necessary to populate the smsg->{mid} field now
that ->smsg_eml calls smsg->populate in rare cases where the
smsg did not originate from SQLite.
|
|
We'll continue to favor simpler data models that can be
used directly rather than wasting time and memory with
accessor APIs.
The ->from, ->to, -cc, ->mid, ->subject, >references methods can
all be trivially replaced by hash lookups since all their values
are stored in doc_data. Most remaining callers of those methods
were test cases, anyways.
->from_name is only used in the PSGI code, so we can just
use ->psgi_cull to take care of populating the {from_name}
field.
|
|
They're stored directly in Xapian and SQLite document data.
NNTP accesses those fields directly to avoid method invocation
overhead so there's no reason to waste several kilobytes for
each sub.
|
|
We'll let $smsg->populate take care of everything all at once
without hanging onto the header object for too long.
|
|
PublicInbox::Smsg::date remains the only exception which
requires any subroutine calls, here, so we'll just have
a branch just for that.
|
|
To further simplify callers and avoid embarrasing memory
explosions[1], we can finally eliminate this method in
favor of smsg_eml.
[1] commit 7d02b9e64455831d3bda20cd2e64e0c15dc07df5
("view: stop storing all MIME objects on large threads")
fixed a huge memory blowup.
|
|
None of our current callers care about the size of the blob
we're retrieving, so stop wasting stack space and code for
it.
|
|
We'll just use `bless' like most current PublicInbox::Smsg callers.
|
|
This will eventually replace the __hdr() calling methods and
eradicate {mime} usage from Smsg. For now, we can eliminate
PublicInbox::Smsg->new since most callers already rely on an
open `bless' to avoid the old {mime} arg.
|
|
First, prefer the leaner "parent" module over the heavy "base"
module to establish ISA relationships, since "base" is only
needed for "fields".
The "//" and "//=" operators allow us simplify our code and fix
minor bugs where a value of "0" was disallowed. Yes, we'll
allow "0" as an email address, too, since some twisted BOFH
could theoretically use it as a local user name.
Going forward, we'll also be avoiding "use warnings" and
instead rely on `-w' in the shebang.
|
|
No point in attempting to print the value of an undefined
variable if there's a bug. Fortunately, (AFAIK) we've never hit
that bug check :>
|
|
We can simplify WwwAtomStream callbacks by performing ->smsg_eml
calls in the `feed_entry' sub itself. This simplifies callers,
by reducing the number of places which can load an Eml object
into memory.
|
|
The goal of this is to eventually remove the $smsg->{mime} field
which is easy-to-misuse and cause memory explosions which
necessitated fixes like commit 7d02b9e64455831d
("view: stop storing all MIME objects on large threads").
|
|
Assisted by commit a73957b5b05f2a00f7a85353b1658b6d8cde05ae
("testcommon: speed up wait_for_tail() on GNU/Linux")
Fixes: 846161e3d1207d59 ("treat $INBOX_DIR/description and gitweb.owner as UTF-8")
|
|
Somewhat recent versions of GNU tail(1) use inotify(7) on Linux;
so don't penalize hackers using TAIL='tail -F' to run their tests
with extra delays.
Ironically, we still need to busy loop on /proc/$TAIL_PID/{fd,fdinfo}
since inotify doesn't seem to support procfs.
|
|
gitweb does the same with $GIT_DIR/description and gitweb.owner.
Allowing UTF-8 description should not cause problems when used
in responses for to the NNTP "LIST NEWSGROUPS" request, either,
since RFC 3977 section 7.6.6 recommends the description be UTF-8
(but does not require it).
Link: https://public-inbox.org/meta/20200528151216.l7vmnmrs4ojw372g@sourcephile.fr/
|
|
This prevents $TMPDIR from being littered with *-journal files
after running the test suite.
This shouldn't cause excessive memory use since $v2w->{mm_tmp}
doesn't see big transactions. There's no need to worry about
data loss, here,either, since this is just a temporary clone
we've even disabled fsync on.
Fixes: 78888d36fb80889f ("msgmap: use TRUNCATE for journal_mode, for now")
|
|
Offering links to download 0-byte files is useless. We could
waste memory by preserving $eml->{bdy} during iteration, but
offering attachments of type "multipart" is not very useful,
as users are usually interested in decoded attachments or
the entire raw message.
Fixes: e60231148eb604a3 ("descend into message/(rfc822|news|global) parts")
|
|
We don't need to load Xapian until we have a directory
which looks like a shard, otherwise we're wasting cycles
on memory when running short-lived processes.
|
|
Older versions of Inline (e.g. 0.53 in CentOS 7) did not accept
the `directory' parameter, so use conditional assignment to set
a default value on $ENV{PERL_INLINE_DIRECTORY}, instead.
|
|
The <EXPR> perlop, `readline', and `read' functions will all
retry on EINTR, so there's no need to retry and loop ourselves
with `sysread'.
|