| Date | Commit message (Collapse) |
|---|
|
This significantly improves the performance of the NNTP GROUP
command with 2.7 million messages from over 250ms to 700us.
SQLite is weird about this, but at least there's a way to
optimize it.
|
|
Since we only query the SQLite over DB for OVER/XOVER; do not
need to waste space storing fields To/Cc/:bytes/:lines or the
XNUM term. We only use From/Subject/References/Message-ID/:blob
in various places of the PSGI code.
For reindexing, we will take advantage of docid stability
in "xapian-compact --no-renumber" to ensure duplicates do not
show up in search results. Since the PSGI interface is the
only consumer of Xapian at the moment, it has no need to
search based on NNTP article number.
|
|
Since we handle the overview info synchronously, we only need
barriers in tests, now. We will use asynchronous checkpoints
to sync less-important Xapian data.
For data deduplication, this requires us to hoist out the
cat-blob support in ::Import for reading uncommitted data
in git.
|
|
Since the overview stuff is a synchronization point anyways,
move it into the main V2Writable process and allow us to
drop a bunch of code. This is another step towards making
Xapian optional for v2.
In other words, the fan-out point is moved and the Xapian
partitions no longer need to synchronize against each other:
Before:
/-------->\
/---------->\
v2writable -->+----parts----> over
\---------->/
\-------->/
After:
/---------->
/----------->
v2writable --> over-->+----parts--->
\----------->
\---------->
Since the overview/threading logic needs to run on the same core
that feeds git-fast-import, it's slower for small repos but is
not noticeable in large imports where I/O wait in the partitions
dominates.
|
|
No need to read what we don't need into the Perl process.
Fix some broken capitalization while we're at it.
|
|
We only need to call get_thread beyond 1000 messages for
fetching entire mboxes. It's probably too much for the HTML
display otherwise.
|
|
Favor simpler internal APIs this time around, this cuts
a fair amount of code out and takes another step towards
removing Xapian as a dependency for v2 repos.
|
|
Noted by Jonathan Corbet in https://lwn.net/Articles/748184/
|
|
Oops :x
|
|
Dscho found this useful for finding matching git commits based
on AuthorDate in git. Add it to the overview DB format, too;
so in the future we can support v2 repos without Xapian.
https://public-inbox.org/git/nycvar.QRO.7.76.6.1804041821420.55@ZVAVAG-6OXH6DA.rhebcr.pbec.zvpebfbsg.pbz
https://public-inbox.org/git/alpine.DEB.2.20.1702041206130.3496@virtualbox/
|
|
Some of this jankiness was from early performance problems
and they turned out to be unnecessary measures.
|
|
This hopefully helps for people who try to understand
this design.
|
|
For upgrades, this will let users keep an old version
running while performing "public-inbox-index" on the
newest version.
|
|
The Xapian partitions will trigger the removal anyways.
Test this and fix some description/spelling errors
while we're at it.
|
|
There's enough gmane links out there in wild that it makes sense
to maintain support for these mappings.
|
|
$mset->size is probably more obvious than relying on a tied
array and saves us a line.
|
|
Sorting large msets is a waste when it comes to mboxes
since MUAs should thread and sort them as the user desires.
This forces us to rework each of the mbox download mechanisms
to be more independent of each other, but might make things
easier to reason about.
|
|
This was vestigial code from the switch to the overview DB
|
|
These internal attributes are not exposed and no longer
used in our APIs.
|
|
The partition count can change if public-inbox-compact runs
while public-inbox-watch or public-inbox-index is running.
|
|
Xapian may become unhappy if a DB is modified during iteration:
nntp://news.gmane.org/20180228004400.GU12724@survex.com
|
|
This is important for people running mirrors via "git fetch",
as they need to be kept up-to-date. Purging is also now
supported in mirrors.
The short-lived "--regenerate" option is gone and is now
implicitly enabled as a result. It's still cheap when
article number regeneration is unnecessary, as we track
the range for each git repository.
|
|
We do not need to rewrite old commits unaffected by the object_id
purge, only newer commits. This was a state management bug :x
We will also return the new commit ID of rewritten history to
aid in incremental indexing of mirrors for the next change.
|
|
This allows us to emulate the display of thread-aware MUAs when
multiple messages share the same Message-ID. This also is a
place where "public-inbox-index --reindex" is useful to fix
existing messages and no schema version bump is necessary.
|
|
We we worked around the default range/termination conditions of
long_response in many cases to reduce calls to SQLite or Xapian.
So continue that trend and become more like the PSGI API
which doesn't force callers to specify an article range or
work inside a loop.
|
|
id_batch had a an overly complicated interface, replace it
with id_batch which is simpler and takes advantage of
selectcol_arrayref in DBI. This allows simplification of
callers and the diffstat agrees with me.
|
|
We can use id_batch in the common case to speed up full mbox
retrievals. Gigantic msets are still a problem, but will
be fixed in future commits.
|
|
OFFSET in SQLite gets painful to deal with. Instead,
rely on timestamps (from Received:) for pagination.
This also sets us up for more precise Date searching
in case we want it.
|
|
While SQLite is faster than Xapian for some queries we
use, it sucks at handling OFFSET. Fortunately, we do
not need offsets when retrieving sorted results and
can bake it into the query.
For inbox.comp.version-control.git (v1 Xapian),
XOVER and XHDR are over 20x faster.
|
|
JOIN operations on SQLite can be disasterously slow.
This reduces per-message pages with the thread overview
at the bottom of those pages from over 800ms to ~60ms.
In comparison, the v1 code took around 70-80ms using
Xapian on my machine.
|
|
In many cases, we do not care about the total number of
messages. It's a rather expensive operation in SQLite
(Xapian only provides an estimate).
For LKML, this brings top-level /$INBOX/ loading time from
~375ms to around 60ms on my system. Days ago, this operation
was taking 800-900ms(!) for me before introducing the SQLite
overview DB.
|
|
searchidx_checkpoint was too convoluted and confusing.
Since barrier is mostly the same thing; use that instead
and add an fsync option for the overview DB.
|
|
This ought to provide better performance and scalability
which is less dependent on inbox size. Xapian does not
seem optimized for some queries used by the WWW homepage,
Atom feeds, XOVER and NEWNEWS NNTP commands.
This can actually make Xapian optional for NNTP usage,
and allow more functionality to work without Xapian
installed.
Indexing performance was extremely bad at first, but
DBI::Profile helped me optimize away problematic queries.
|
|
We can store :bytes and :lines in doc_data since we never
sort or search by them. We don't have much use for the Date:
stamp at the moment, either.
|
|
We need to stop ghost messages from generating longer
Message-IDs than Xapian can handle with terms.
|
|
I was too aggressively disabling parallelization to speed up
the test suite and broke this :x Re-enable parallelization
for the v2reindex test so we can catch it later.
|
|
We need to ensure there is only one file in the top-level tree
at any commit so the "add; remove; add;" sequence on the same
message is detected properly.
Otherwise, git will not detect the second "add" unless
a second message is added to history.
Deletes are now stored in "d" (and not "D" or "_/D") at the
top-level, now. There's no need to have a "_" to reduce churn
as "m" and "d" should never co-exist. It's now lowercased to
make it easier-to-distinguish from "D" in git-log output.
|
|
We will vivify multiple ghosts if a message has multiple
Message-IDs.
|
|
Ensure -convert and -compact do not make repositories
unreadable on live servers.
|
|
We'll be making sure V2Writable uses this.
|
|
Some folks had bad mail clients which generated 3-digit years
around Y2K...
|
|
This is a smaller improvement than the landing /$INBOX/ page
because full message bodies are shown; but still saves around
100ms for my system with LKML.
|
|
It's no longer necessary to have this since load_expand
now populates $smsg->mid with the "preferred" Message-ID.
This saves around 10ms on the homepage for me.
|
|
This saves over 400ms on my system with the full LKML
with over 2.8 million messages.
|
|
This is consistent with how we internally generate new
Message-IDs to break conflicts and allows ->reindex to
succeed while walking backwards through history
|
|
Relying solely on git for v2 repos is probably not
so useful, so add pointers to public-inbox-init/index
commands.
|
|
By supporting purge and allowing users to delete git partitions,
we can open up ourselves to gaps and un-reindexible data. Let
that be.
|
|
Somebody may only care about the most recent history,
so allow -init and -index to operate quietly on missing
partitions.
|
|
-watch on a busy/giant Maildir caused too many Xapian
errors while attempting to browse.
|
|
I mainly focus on -watch for mirroring busy mailing lists, but
using -mda should remain an option.
|