| Date | Commit message (Collapse) |
|---|
|
In case people were running old buggy versions from 2016...
(and -convert should probably clean those up, eventually)
|
|
First off, decode text portions of messages since some archived
mail I got was converted from quoted-printable or base-64 to
8bit by the original recipient. Attempting to merge them with
my own archives (which had no conversion done) led to
unnecessary duplicates showing up.
Then, normalize CRLF line endings in text portions to LF.
In the headers, we relax the content_id hashing to ignore quotes
and lower-case domain names in To, Cc, and From headers since
some mail processors will alter them.
Finally, I've discovered Email::MIME->new($mime->as_string)
does not always round-trip reliably, so we calculate the
content_id twice on user-supplied messages.
|
|
While hunting duplicates, I noticed a leading '-' in some
Message-IDs as a result of RFC4648 encoding. While '-' seems
allowed by RFC5322 and URL-friendly (RFC4648), they are uncommon
and make using Message-IDs as arguments for command-line tools
more difficult. So prefix them with a datestamp to at least
give readers some sense of the age. And shorten the "localhost"
hostname to "z" to save space.
|
|
I'm not sure how useful this view is, but it exists for now.
|
|
git fast-import and the main V2Writable process combined takes
about one CPU, so avoid having too many Xapian partitions which
cause unnecessary I/O contention.
|
|
--no-renumber does not allow merging, and merging is not ideal
for reindexing, either.
|
|
Otherwise articles show up again...
|
|
Gigantic feeds probably make some clients unhappy,
clamp it to what it was in the past.
Fixes: b9534449ecce2c59 ("view: avoid offset during pagination")
|
|
This significantly improves the performance of the NNTP GROUP
command with 2.7 million messages from over 250ms to 700us.
SQLite is weird about this, but at least there's a way to
optimize it.
|
|
Since we only query the SQLite over DB for OVER/XOVER; do not
need to waste space storing fields To/Cc/:bytes/:lines or the
XNUM term. We only use From/Subject/References/Message-ID/:blob
in various places of the PSGI code.
For reindexing, we will take advantage of docid stability
in "xapian-compact --no-renumber" to ensure duplicates do not
show up in search results. Since the PSGI interface is the
only consumer of Xapian at the moment, it has no need to
search based on NNTP article number.
|
|
public-inbox-convert ought to be 100% lossless, now
|
|
Not everybody needs multiprocess support.
|
|
Since we handle the overview info synchronously, we only need
barriers in tests, now. We will use asynchronous checkpoints
to sync less-important Xapian data.
For data deduplication, this requires us to hoist out the
cat-blob support in ::Import for reading uncommitted data
in git.
|
|
Since the overview stuff is a synchronization point anyways,
move it into the main V2Writable process and allow us to
drop a bunch of code. This is another step towards making
Xapian optional for v2.
In other words, the fan-out point is moved and the Xapian
partitions no longer need to synchronize against each other:
Before:
/-------->\
/---------->\
v2writable -->+----parts----> over
\---------->/
\-------->/
After:
/---------->
/----------->
v2writable --> over-->+----parts--->
\----------->
\---------->
Since the overview/threading logic needs to run on the same core
that feeds git-fast-import, it's slower for small repos but is
not noticeable in large imports where I/O wait in the partitions
dominates.
|
|
No need to read what we don't need into the Perl process.
Fix some broken capitalization while we're at it.
|
|
We only need to call get_thread beyond 1000 messages for
fetching entire mboxes. It's probably too much for the HTML
display otherwise.
|
|
Xapian is size-intensive and SQLite is not strictly necessary for v1.
|
|
Favor simpler internal APIs this time around, this cuts
a fair amount of code out and takes another step towards
removing Xapian as a dependency for v2 repos.
|
|
Noted by Jonathan Corbet in https://lwn.net/Articles/748184/
|
|
Oops :x
|
|
Dscho found this useful for finding matching git commits based
on AuthorDate in git. Add it to the overview DB format, too;
so in the future we can support v2 repos without Xapian.
https://public-inbox.org/git/nycvar.QRO.7.76.6.1804041821420.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz
https://public-inbox.org/git/alpine.DEB.2.20.1702041206130.3496@virtualbox/
|
|
Some of this jankiness was from early performance problems
and they turned out to be unnecessary measures.
|
|
This hopefully helps for people who try to understand
this design.
|
|
For upgrades, this will let users keep an old version
running while performing "public-inbox-index" on the
newest version.
|
|
The Xapian partitions will trigger the removal anyways.
Test this and fix some description/spelling errors
while we're at it.
|
|
Lets not scare users when they encounter files that are supposed
to be there. Then, preserve the journal and pipe.lock, even if
they're supposedly unused due to us holding the inbox-wide lock.
|
|
There's enough gmane links out there in wild that it makes sense
to maintain support for these mappings.
|
|
$mset->size is probably more obvious than relying on a tied
array and saves us a line.
|
|
Sorting large msets is a waste when it comes to mboxes
since MUAs should thread and sort them as the user desires.
This forces us to rework each of the mbox download mechanisms
to be more independent of each other, but might make things
easier to reason about.
|
|
This was vestigial code from the switch to the overview DB
|
|
These internal attributes are not exposed and no longer
used in our APIs.
|
|
The partition count can change if public-inbox-compact runs
while public-inbox-watch or public-inbox-index is running.
|
|
Xapian may become unhappy if a DB is modified during iteration:
nntp://news.gmane.org/20180228004400.GU12724@survex.com
|
|
This is important for people running mirrors via "git fetch",
as they need to be kept up-to-date. Purging is also now
supported in mirrors.
The short-lived "--regenerate" option is gone and is now
implicitly enabled as a result. It's still cheap when
article number regeneration is unnecessary, as we track
the range for each git repository.
|
|
We do not need to rewrite old commits unaffected by the object_id
purge, only newer commits. This was a state management bug :x
We will also return the new commit ID of rewritten history to
aid in incremental indexing of mirrors for the next change.
|
|
|
|
This allows us to emulate the display of thread-aware MUAs when
multiple messages share the same Message-ID. This also is a
place where "public-inbox-index --reindex" is useful to fix
existing messages and no schema version bump is necessary.
|
|
We we worked around the default range/termination conditions of
long_response in many cases to reduce calls to SQLite or Xapian.
So continue that trend and become more like the PSGI API
which doesn't force callers to specify an article range or
work inside a loop.
|
|
id_batch had a an overly complicated interface, replace it
with id_batch which is simpler and takes advantage of
selectcol_arrayref in DBI. This allows simplification of
callers and the diffstat agrees with me.
|
|
We can use id_batch in the common case to speed up full mbox
retrievals. Gigantic msets are still a problem, but will
be fixed in future commits.
|
|
OFFSET in SQLite gets painful to deal with. Instead,
rely on timestamps (from Received:) for pagination.
This also sets us up for more precise Date searching
in case we want it.
|
|
While SQLite is faster than Xapian for some queries we
use, it sucks at handling OFFSET. Fortunately, we do
not need offsets when retrieving sorted results and
can bake it into the query.
For inbox.comp.version-control.git (v1 Xapian),
XOVER and XHDR are over 20x faster.
|
|
There'll be more performance-related tests in the future.
|
|
We'll be adding more tests in the same vein as this
to improve NNTP performance.
|
|
JOIN operations on SQLite can be disasterously slow.
This reduces per-message pages with the thread overview
at the bottom of those pages from over 800ms to ~60ms.
In comparison, the v1 code took around 70-80ms using
Xapian on my machine.
|
|
In many cases, we do not care about the total number of
messages. It's a rather expensive operation in SQLite
(Xapian only provides an estimate).
For LKML, this brings top-level /$INBOX/ loading time from
~375ms to around 60ms on my system. Days ago, this operation
was taking 800-900ms(!) for me before introducing the SQLite
overview DB.
|
|
We need to ensure we don't match NULL 'sid' columns in the
`over' table.
|
|
searchidx_checkpoint was too convoluted and confusing.
Since barrier is mostly the same thing; use that instead
and add an fsync option for the overview DB.
|
|
This ought to provide better performance and scalability
which is less dependent on inbox size. Xapian does not
seem optimized for some queries used by the WWW homepage,
Atom feeds, XOVER and NEWNEWS NNTP commands.
This can actually make Xapian optional for NNTP usage,
and allow more functionality to work without Xapian
installed.
Indexing performance was extremely bad at first, but
DBI::Profile helped me optimize away problematic queries.
|
|
We can store :bytes and :lines in doc_data since we never
sort or search by them. We don't have much use for the Date:
stamp at the moment, either.
|