Date | Commit message (Collapse) |
|
Older versions of public-inbox < 1.3.0 had subtly
different semantics around threading in some corner
cases. This switch (when combined with --reindex)
allows us to fix them by regenerating associations.
|
|
Noticed while reindexing a largish v2 inbox in parallel on an
SSD which required checkpointing and respawning shard workers.
Fixes: f06e84220e5566e7 ("over+msgmap: do not store filename after DBI->connect")
|
|
We can rely on FD_CLOEXEC being set by default (since Perl 5.6+)
on pipes to avoid FS/page-cache traffic, here. We also know
"git hash-object" won't output anything until it's consumed all
of its standard input; so there's no danger of a deadlock even
in the the unlikely case git uses a hash that can't fit into
PIPE_BUF :P
|
|
Since over.sqlite3 seems here to stay, we no longer need to do
Message-ID lookups against Xapian and can simply rely on the
docid <=> NNTP article number equivalancy SCHEMA_VERSION=15
gave us.
This rids us of the closure-using batch_do sub in the v1
code path and vastly simplifies both v1 and v2 unindexing.
|
|
Prefer "parent" to "base" since the former is lighter and part
of Perl 5.10+. We'll also rely on warnings from "-w" globally
(or not) instead of via "use".
|
|
OO method dispatch was 10-15% slower when I was implementing the
NNTP server. It also serves as a helpful reminder to the reader
at the callsite as to whether a sub is likely in the same
package as the caller or not.
|
|
This saves runtime allocations and reduces the likelyhood of
memory leaks either from cycles or buggy old Perl versions.
|
|
While it makes the code flow slightly less well in some places,
it saves us runtime allocations and indentation.
|
|
In case this ends up in the same process as Mbox::msg_hdr,
it can reduce memory use by sharing the cache key in
PublicInbox::Eml::re_memo
|
|
We only support Unix-like platforms where binmode (":raw") is
the default anyways, and v5.10 semantics means it won't do
unicode_strings (unlike v5.12). So save some lines of code.
|
|
The "5.010_001" form was for Perl 5.6, which I doubt anybody
would attempt; so favor "v5.10.1" as it is more readable to
humans. Prefer "parent" to "base" since the former is lighter.
We'll also rely on warnings from "-w" globally (or not) instead
of via "use".
We'll also update "use" statements to reflect what's actually
used by V2Writable.
|
|
"\n" and other characters requiring quoting and/or escaping in
in $GIT_DIR/objects/info/alternates was not supported in git 2.11
and earlier; nor does it seem supported at all in libgit2.
This will allow us to support sharing git-cat-file or similar
endpoints across multiple inboxes via alternates.
This breaks an existing use case for anybody wacky
enough to put `\n' in the `inboxdir' pathname; but I doubt
this affects anybody.
|
|
SQLite already knows the filename internally, so avoid having it
as a long-lived Perl SV to save some bytes when there's many
inboxes and open DBs.
|
|
While it's even less common to experience a replaced
msgmap.sqlite3 file, BOFHs may do the darndest things. This is
another step towards reducing the number of needless wakeups
we need to do in long-lived read-only daemons.
|
|
None of the human-readable strings stored in over.sqlite3
require UTF-8. Message-IDs do not, nor do the compressed
Subject IDs (sid) we use for Subject-based threading. And the
`ddd' (doc-data-deflated) column is of course binary data.
This frees us of having to use SQL_BLOB for the `ddd', column,
and will open the door for us to use dbh_new for Msgmap, too.
|
|
We must not trigger wakeups on InboxIdle users until after we've
renamed all files into place. Otherwise, the InboxIdle caller
may just reopen the old (soon-to-be-unlinked) file.
This fixes occasional test failures in t/nntpd.t
Fixes: f977826a17f8735e ("lock: reduce inotify wakeups")
|
|
Instead of returning "BAD program fault", just give the
standard "BAD search not available"... message we show
for mailbox slices.
|
|
As in Import, we'll fall back to Sender: if From: is missing,
and use the primary_address of the inboxes to indicate the total
absence of those fields.
|
|
Non-slice mailboxes never have messages themselves,
so we must not assume a message exists when sending
untagged EXISTS messages.
|
|
The resulting OID ("oid_b") is a required arg and part of
$env->{PATH_INFO}, instead; so it's never part of an optional
query parameter.
|
|
Returning an empty string for a filename makes no sense,
so instead return `undef' so the caller can setup a fallback
using the "//" operator.
This fixes uninitialized variable warnings because split()
on an empty string returns `undef', which caused to_filename
to warn on s// and tr// ops.
|
|
This means we need to filter out "" from query parameters.
While we're at it, update comments for the WWW endpoint.
|
|
I'm not sure why this wasn't done in Jun/July 2016 when I was
working on PublicInbox::Address to replace the DoS-vulnerable
Email::Address.
Nowadays, PublicInbox::Address allows using Email::Address::XS
which should be fast and robust.
|
|
parent.pm is leaner than base.pm, and Time::HiRes::stat is
more accurate, so take advantage of these Perl 5.10+-isms
since it's been over a year since we left 5.8 behind.
|
|
We actually don't do anything with {env} or {'psgix.io'}
on client aborts, so checking the truthiness of '{forward}'
is necessary.
|
|
Since -edit and -purge should be rare and TOCTOU around them
rarer still; missing {blobs} could be indicative of a real bug
elsewhere. Warn on them.
And I somehow ended up with 3 different field names for Inbox
objects. Perhaps they'll be made consistent in the future.
|
|
While all the {async_next} callbacks needed eval guards anyways
because of DS->write, {async_eml} callbacks did not.
Ensure any bugs in our code or data corruption result in
termination of the HTTP connection, so as not to leave clients
hanging on a response which never comes or is mangled in some
way.
|
|
We can reuse some of the GzipFilter infrastructure used by other
WWW components to handle slow blob retrieval, here. The
difference from previous changes is we don't decide on the 200
status code until we've retrieved the blob and found the
attachment.
While we're at it, ensure we can compress text attachment
responses once again, since all text attachments are served
as text/plain.
|
|
gzf_maybe always returns a GzipFilter object, even if it uses
CompressNoop. We can also use ->zflush instead of
->translate(undef) here for the final bit.
|
|
This simplifies the primary callers of eml_entry while only making
mknews.perl worse.
|
|
We no longer favor getline+close for streaming PSGI responses
when using public-inbox-httpd. We still support it for other
PSGI servers, though.
|
|
All of our streaming responses use ::aresponse, now, and our
synchronous responses use html_oneshot. So there's no need
for the old WwwStream::response.
|
|
We can build and buffer the HTML <head> section once the first
non-ghost message in a thread is loaded, so there's no need to
perform an extra check on $ctx->{nr} once the $eml is ready.
|
|
We can save stack space and simplify subroutine calls, here.
|
|
Another 10% or so speedup when displaying full messages off
search results.
|
|
Once again this speeds another endpoint up 10% or so.
|
|
$ctx->{msgs} won't ever contain undef values.
|
|
Another 10% or so speedup in a frequently-hit endpoint.
|
|
Once again, this shows a ~10% speedup with multi-message
threads in xt/httpd-async-stream.t regardless of whether
TEST_JOBS is 1 or 100.
|
|
This will allow -httpd to handle other requusts if waiting on
an HDD seek or git to decode a blob.
|
|
This makes WwwStream closer to MboxGz and WwwAtomStream
and will eventually allow us to follow the same patterns.
|
|
parent.pm is leaner than base and we'll rely on `-w' for
warnings during development.
|
|
Z_FINISH is the default for Compress::Raw::Zlib::Deflate->flush,
anyways, so there's no reason to import it. And none of C::R::Z
is needed in WwwText now that gzf_maybe handles it all.
|
|
Virtually all of our responses are going to be gzipped, anyways.
This will allow us to utilize zlib as a buffering layer and
share common code for async blob retrieval responses.
To streamline this and allow GzipFilter to be a parent class,
we'll replace the NoopFilter with a similar CompressNoop class
which emulates the two Compress::Raw::Zlib::Deflate methods we
use.
This drops a bunch of redundant code and will hopefully make
upcoming WwwStream changes easier to reason about.
|
|
This will make it easier to support asynchronous blob
retrievals. The `$ctx->{nr}' counter is no longer implicitly
supplied since many users didn't care for it, so stack overhead
is slightly reduced.
|
|
Like with WwwAtomStream and MboxGz, we can bless the existing
$ctx object directly to avoid allocating a new hashref. We'll
also switch from "->" to "::" to reduce stack utilization.
|
|
This allows -httpd to handle other requests while waiting
for git to retrieve and decode blobs. We'll also break
apart t/psgi_v2.t further to ensure tests run against
-httpd in addition to generic PSGI testing.
Using xt/httpd-async-stream.t to test against clones of meta@public-inbox.org
shows a 10-12% performance improvement with the following env:
TEST_JOBS=1000 TEST_CURL_OPT=--compressed TEST_ENDPOINT=new.atom
|
|
No need to deepen our object graph, here.
|
|
stat(2) on the inboxdir is unlikely to be correct, now that
msgmap truncates its journal (rather than unlinking it).
|
|
We always return Z (UTC) times, anyways, so we'll always
use gmtime() on the seconds-after-the-epoch.
|