| Date | Commit message (Collapse) |
|---|
|
Since over.sqlite3 seems here to stay, we no longer need to do
Message-ID lookups against Xapian and can simply rely on the
docid <=> NNTP article number equivalancy SCHEMA_VERSION=15
gave us.
This rids us of the closure-using batch_do sub in the v1
code path and vastly simplifies both v1 and v2 unindexing.
|
|
The eval in key2sub via t/run.perl ("make check-run") won't
trigger the warning, but running "prove -bvw t/import.t"
directly, does. In any case, ensure the contents of this
variable doesn't linger across runs.
|
|
"\n" and other characters requiring quoting and/or escaping in
in $GIT_DIR/objects/info/alternates was not supported in git 2.11
and earlier; nor does it seem supported at all in libgit2.
This will allow us to support sharing git-cat-file or similar
endpoints across multiple inboxes via alternates.
This breaks an existing use case for anybody wacky
enough to put `\n' in the `inboxdir' pathname; but I doubt
this affects anybody.
|
|
SQLite already knows the filename internally, so avoid having it
as a long-lived Perl SV to save some bytes when there's many
inboxes and open DBs.
|
|
While it's even less common to experience a replaced
msgmap.sqlite3 file, BOFHs may do the darndest things. This is
another step towards reducing the number of needless wakeups
we need to do in long-lived read-only daemons.
|
|
Non-slice mailboxes never have messages themselves,
so we must not assume a message exists when sending
untagged EXISTS messages.
|
|
Returning an empty string for a filename makes no sense,
so instead return `undef' so the caller can setup a fallback
using the "//" operator.
This fixes uninitialized variable warnings because split()
on an empty string returns `undef', which caused to_filename
to warn on s// and tr// ops.
|
|
Since Perl doesn't internally use a self-pipe for
sleep/select/poll/etc, wake up every 10ms to ensure
it can see the SIGCHLD; since neither signalfd nor EVFILT_SIGNAL
are always available.
Fixes: 761baa2a300e4268 ("spawn: unblock SIGCHLD in subprocess")
|
|
We can reuse some of the GzipFilter infrastructure used by other
WWW components to handle slow blob retrieval, here. The
difference from previous changes is we don't decide on the 200
status code until we've retrieved the blob and found the
attachment.
While we're at it, ensure we can compress text attachment
responses once again, since all text attachments are served
as text/plain.
|
|
This allows -httpd to handle other requests while waiting
for git to retrieve and decode blobs. We'll also break
apart t/psgi_v2.t further to ensure tests run against
-httpd in addition to generic PSGI testing.
Using xt/httpd-async-stream.t to test against clones of meta@public-inbox.org
shows a 10-12% performance improvement with the following env:
TEST_JOBS=1000 TEST_CURL_OPT=--compressed TEST_ENDPOINT=new.atom
|
|
This restores gzip-by-default behavior for /$INBOX/$MSGID/raw
endpoints for all indexed inboxes. Unindexed v1 inboxes will
remain uncompressed, for now.
|
|
This will allow us to gzip responses generated by cgit
and any other CGI programs or long-lived streaming
responses we may spawn.
|
|
This will allow others to mimic our award-winning homepage
design without needing to rely on Plack::Middleware::Deflater
or varnish to compress responses.
|
|
The changes to GzipFilter here may be beneficial for building
HTML and XML responses in other places, too.
|
|
It'll give us a nicer HTML header and footer.
|
|
Most of our plain-text responses are config files
big enough to warrant compression.
|
|
Spammers may send emails with nasty characters which can throw
off git-fast-import. Users with non-existent or weaker spam
filters may be susceptible to corruption in the fast-import
stream as a result.
This was actually quietly fixed in git on 2020-06-01 by
commit 9ab886546cc89f37819e1ef09cb49fd9325b3a41
("smsg: introduce ->populate method"), but no test case
was created.
Reported-by: Eric W. Biederman <ebiederm@xmission.com>
Link: https://public-inbox.org/meta/87imf4qn87.fsf@x220.int.ebiederm.org/
Link: https://public-inbox.org/meta/20200601100657.14700-6-e@yhbt.net/
|
|
Network connections fail and need to be detected sooner rather
than later during IDLE to avoid backtrace floods. In case the
IDLE process dies completely, don't respawn right away, either,
to avoid entering a respawn loop.
There's also a typo fix :P
|
|
The default (and fast) TEST_RUN_MODE=2 preloads most modules,
but TEST_RUN_MODE=0 is more realistic and can catch some
problems which may show up in real-world use.
|
|
Anonymous subs cost over 5K each on x86-64. So prefer the
less-recommended-but-still-documented way of using
Linux::Inotify2::watch to register watchers.
This also updates FakeInotify to detect modifications correctly
when used on systems with neither IO::KQueue nor
Linux::Inotify2.
|
|
This will help us catch warnings in new code and notice
inadvertantly skipped tests.
|
|
Subprocess we spawn may want to use SIGCHLD for themselves.
This also ensures we restore default signal handlers
in the pure Perl version.
|
|
SQLite only issues non-blocking F_SETLK ops (not F_SETLKW) and
retries failures using a configurable busy_timeout. SQLite's
busy loop sleeps for a millisecond and retries the lock until
the configured busy_timeout is hit.
Trying to set ->sqlite_busy_timeout to larger values (e.g. 30000
milliseconds) still leads to failure when running the new stress
test with 8 processes with TMPDIR on a 7200 RPM HDD.
Inspection of SQLite source reveals there's no built-in way to
use F_SETLKW, so tack on the existing flock(2) support we use to
synchronize git + SQLite + Xapian for inbox writing. We use
flock(2) instead of POSIX fcntl(2) locks since Perl doesn't
provide a way to manipulate "struct flock" portably.
|
|
Since we use the non-ref scalar URL in many error messages,
favor keeping the unblessed URL in the long-lived process.
This avoids showing "snews://" to users who've specified
"nntps://" URLs, since "nntps" is IANA-registered nowadays and
what we show in our documentation, while "snews" was just a
draft the URI package picked up decades ago.
|
|
This is similar to IMAP support, but only supports polling.
Automatic altid support is not yet supported, yet; but may
be in the future.
v2: small grammar fix by Kyle Meyer
Link: https://public-inbox.org/meta/87sgeg5nxf.fsf@kyleam.com/
|
|
We may just modify PublicInbox::Config->urlmatch in the future
to support git <1.8.5, but I wonder if there's enough users on
git <1.8.5 to justify it.
|
|
Since we store all watched directory names as keys in %mdmap,
there should be no need to keep an array of those directories
around.
t/watch_maildir*.t required changes to remove trained spam.
Once we've trained something as spam, there shouldn't be
a need to rescan it.
|
|
Since we have IMAP client support in -watch; make sure per-URL
settings are familiar to git users by taking advantage of git's
URL matching abilities.
This requires git 1.8.5+, which most users ought to have
(though base CentOS 7 is on 1.8.3).
|
|
Not all IMAP servers support IDLE, and IDLE may be prohibitively
expensive for some IMAP servers with many inboxes. So allow
configuring a imap.$IMAP_URL.pollInterval=SECONDS to poll
mailboxes.
We'll also need to poll for NNTP servers in the future.
|
|
We can get rid of the janky wannabe
self-using-a-directory-instead-of-pipe thing we needed to
workaround Filesys::Notify::Simple being blocking.
For existing Maildir users, this should be more robust and
immune to missed wakeups for signalfd and kqueue-enabled
systems; as well as being immune to BOFHs clearing $TMPDIR
and preventing notifications from firing.
The IMAP IDLE code still uses normal Perl signals, so it's still
vulnerable to missed wakeups. That will be addressed in future
commits.
|
|
Since we already use inotify and EVFILT_VNODE (kqueue)
in -imapd, we might as well use them directly in -watch,
too.
This will allow public-inbox-watch to use PublicInbox::DS
for timers to watch newsgroups/mailboxes and have saner
signal handling in future commits.
|
|
We need to detect link(2) and rename(2) in other apps
writing to the Maildir.
We'll be removing the Filesys::Notify::Simple from -watch
in favor of using IO::KQueue or Linux::Inotify2 directly.
Ensure non-inotify emulations can support everything we
expect for Maildir writers.
|
|
Only servers with IDLE are supported, for now. Polling will
be needed since users may need to watch many inboxes with
a few active connections due to IMAP server limitations.
|
|
We'll be supporting the IMAP URL scheme described in RFC 5092
for -watch, so add this module to fill in what the `URI' package
lacks.
|
|
The old check was ineffective since we process the spam folder
config before ham inboxes; and would only fail when attempting
to treat the scalar "watchspam" string as an array ref.
|
|
Respect XDG_DATA_HOME to avoid cluttering ~/.public-inbox/.
Existing users of ~/.public-inbox/imap.sqlite3 will remain
supported, but the preference for new data is to use
~/.local/share and other paths standardized by XDG.
Cc: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
Apparently, using $1 from an octet string still results in a
multi-byte string. Thus we need to perform utf8::encode after
the regexp character match to ensure wide characters don't get
passed to encode_base64.
This fixes a bug in which caused -watch to crash when using
PublicInbox::Filter::SubjectTag to remove "[list prefix]"
tags from Subject: lines.
I only found this bug because the proposed -watch updates for
NNTP/IMAP support introduced a possible bug which triggered a
full rescan of old archives:
https://public-inbox.org/meta/20200627100400.9871-1-e@yhbt.net/
|
|
I used find(1) here for debugging. The "make check-run" test
target needs to be updated to make stderr spew more obvious.
|
|
For archivists with only newer mail archives, this option allows
reserving reserve NNTP article numbers for yet-to-be-archived
old messages. Indexers will need to be updated to support this
feature in future commits.
-V1 inboxes will now be initialized with SQLite and Xapian
support if this option is used, or if --indexlevel= is
specified.
|
|
On a powerful (by my standards) machine with 16GB RAM and an
7200 RPM HDD marketed for "enterprise" use, indexing a 8.1G (in
git) LKML snapshot from Sep 2019 did not finish after 7 days
with the default number (3) of Xapian shards (`--jobs=4') and
`--batch-size=10m'.
Indexing starts off fast, but progressively get slower as
contents of the inbox (including Xapian + SQLite DBs) could no
longer be cached by the kernel. Once the on-disk size
increased, HDD seek contention between the Xapian shard workers
slowed the process down to a crawl.
With a single shard, it still took around 3.5 days to index on
the HDD. That's not good, but it's far better than not
finishing after 7 days. So allow unfortunate HDD users to
easily specify a single shard on public-inbox-init.
For reference, a freshly TRIM-ed low-end TLC SSD on the SATA II
bus on the same machine indexes that same snapshot of LKML in
~7 hours with 3 shards and the same 10m batch size. In the past,
a higher-end consumer grade MLC SSDs on similar hardware indexed
a similarly sized-data set in ~4 hours.
|
|
We also need to check for git 2.6 earlier in each test case,
before any other TAP output is emitted to avoid confusing the
TAP consumers.
|
|
Net::NNTP does not support older IO::Socket::SSL. 1.94 on
CentOS 7.x fails HTTPS and IMAPS tests, too.
cf. https://rt.cpan.org/Ticket/Display.html?id=100529
|
|
Having `git cat-file' as a separate process naturally lends
itself to asynchronous dispatch. Our event loop for -nntpd no
longer blocks on slow git storage.
Pipelining in -imapd was tricky and bugs were exposed by
mbsync(1). Update t/nntpd.t to support pipelining ARTICLE
requests to ensure we don't have the same problems -imapd
did during development.
|
|
I accidentally dropped "TEXT" handling while porting
the IMAP search query parser to Parse::RecDescent.
This reinstates it and adds a test to prevent future
regression, and the additional test fixes a counting
error for non-Xapian-enabled systems.
|
|
We no longer pass an arrayref to search_common() or
parse_query(), so handle the CHARSET directive in
the Parse::RecDescent-generated parser directly.
|
|
For properly parsing IMAP search requests, it's easier to use a
recursive descent parser generator to deal with subqueries and
the "OR" statement.
Parse::RecDescent was chosen since it's mature, well-known,
widely available and already used by our optional dependencies:
Inline::C and Mail::IMAPClient. While it's possible to build
Xapian queries without using the Xapian string query parser;
this iteration of the IMAP parser still builds a string which is
passed to Xapian's query parser for ease-of-diagnostics.
Since this is a recursive descent parser dealing with untrusted
inputs, subqueries have a nesting limit of 10. I expect that is
more than adequate for real-world use.
|
|
Since we support MSNs properly, now, it seems acceptable
to support regular SEARCH requests in case there are any
clients which still use non-UID SEARCH.
|
|
Since IMAP IDLE users aren't expected to issue any commands, we
can terminate their connections immediately on graceful
shutdown.
Furthermore, we need to drop the inotify FD from the epoll set
to avoid warnings during global destruction. Embarassingly,
this required fixing wacky test ordering from 2a717d13f10fcdc6
("nntpd+imapd: detect replaced over.sqlite3")
|
|
Mail::IMAPClient understandably stumbles into a warning
by our bogus test request. Just silence it on our end
since it's not normal operation for Mail::IMAPClient.
|
|
->has_capability on Mail::IMAPClient 3.37 (tested on CentOS 7.x)
only returned boolean values, and not the value of the capability.
|