| Date | Commit message (Collapse) |
|---|
|
This is needed to limit the RSS of processes and ensure the
stored data in over.sqlite3 and Xapian DBs are consistent if
interrupted. Without checkpoints, indexing lore causes shard
workers to take several GB of memory and thrash/OOM smaller
systems.
|
|
This bit is duplicated with per-Inbox indexing in Admin,
undecided if it's the right place for it.
|
|
This seems necessary for some cross-posted messages (and we did
it historically before we used over.sqlite3).
|
|
This lets us pretend an ExtSearch object is an Inbox object
in most of the existing WWW code.
|
|
We can now handle cases where messages are edited in one inbox
but not another, bifurcating the message.
V2Writable::log_range handles some edge-cases which could happen
in v2-only code paths, as well, but weren't usually triggered
due to default git-gc knobs not pruning immediately
|
|
We'll probably still need synchronous message retrieval
in a few places (tests, at least).
|
|
We don't actually use it anywhere, and may not need it in
the future.
|
|
We want NNTP clients to see consistent Xref: headers to ensure
client-side caches don't get confused.
|
|
It doesn't seem worth storing xref3 data in Xapian now that
the same info is in over.sqlite3.
|
|
We may not end up storing xref3 data in Xapian, actually.
This will make indexlevel=basic possible, and along with
--sequential-shard indexing support for slow storage.
Making oidmap a separate table seems unnecessary, too, so
fold it into the xref3 table since it's unlikely a git blob
will be responsible for multiple xref3 rows.
|
|
This doesn't do anything, yet, but it will once the rest
of the eindex stuff works.
|
|
Not documented, yet, but it runs...
|
|
But warn on it, this lets us test new or throwaway commands more
easily if we don't have to start a new POD for everything we
want to dump in script/.
|
|
In case we want to reuse code with ExtSearchIdx or V2Writable.
|
|
This will let us work consistently with both existing inboxes
and external indices.
|
|
A couple of more things to prepare us to run syncs on
both v1 and v2 inboxes.
|
|
We'll be needing it in ExtSearchIdx for the next commit.
|
|
Now that the V2Writable code is more generic, we can
sync with it to use `units' which represent either
a v2 epoch or an entire v1 inbox.
|
|
We'll be validating against this in the future to stop
bugs from creeping in.
|
|
Moved to per-epoch "units".
|
|
And clearly label it. We may try to reuse some of this for v1
indexing code paths.
|
|
We'll use `index_oid' and `unindex_oid' as our method names
so V2Writable methods may use `$self->can' to access them.
|
|
This will let us use it from ExtSearchIdx.
|
|
This will allow ExtSearchIdx to override or reuse them more
easily. Unfortunately we lose prototype validation, but that
seems to be discouraged anyways given the 'signatures' feature
in Perl 5.20+.
|
|
This will make it easier to reuse some indexing code for ExtSearchIdx.
|
|
Using `->can(method)' allows subclasses to override `index_oid'
and `unindex_oid' methods.
|
|
We want to reuse this code for ExtSearchIdx, eventually.
|
|
Since we store {ibx} in $sync state, we no longer have to
pass it as an argument to log2stack.
|
|
This will allow reusability with ExtSearchIdx
|
|
Having a special init path for external indices is probably
easier than further overloading SearchIdx->new initialization
to work without an Inbox object.
|
|
Not yet tested, but Perl compiles it!
|
|
Using `O' (owner) here (according Xapian omega's
termprefixes.rst) since we could say the newsgroup or inbox is
the owner of the given message.
|
|
It compiles...
|
|
ExtSearchIdx will not have Msgmap, since it may index
non email blobs in the future (it'll still be usable
with IMAP, but not NNTP).
|
|
"remote" used to imply "child process on the same machine" which
was somewhat non-sensical, anyways. And OverIdx has been in the
same process since v2 was finalized. So use the suffix "aux"
for "auxiliary" since it can be safely jettisoned without
breaking URLs.
|
|
This is preferable to open-coding "newsgroup // inboxdir" everywhere.
|
|
We'll be using per-sync-state {ibx} refs instead, so make parts
of the v2 indexing code less-dependent on $self->{ibx} where
$self is a V2Writable object.
|
|
Since external indices won't have msgmap.sqlite3, we'll need to
store last_commit-* metadata in over.sqlite3 instead. This
has a longer limits to account for path names or newsgroup names
stored in keys.
We'll also rely on built-in counters for Xapian document IDs,
since msgmap.sqlite3 no longer provides an AUTOINCREMENT column.
|
|
This will be needed for ExtSearchIdx which doesn't have a
persistent PublicInbox::Inbox object.
|
|
This will make it easier-to-use in ExtSearchIdx.
|
|
We don't need to keep it in code paths which are guaranteed to
only see PublicInbox::Eml (and not Email::MIME or PublicInbox::MIME
which did not round-trip properly). However, we must set
{raw_bytes} since PublicInbox::Eml may add an extra "\n" for
rare messages with no bodies.
|
|
We'll be reusing this for external indices and possibly
other places.
|
|
External indices won't have $self->{ibx} since it needs to
deal with multiple inboxes. We can also hoist out
->parallel_init to make it easier to distinguish the
non-parallel control flow.
|
|
This will be used to track cross-posted messages in the
external/detached index.
|
|
We can simplify callers by using $self->{xpfx} instead of
passing another arg on the stack.
|
|
We'll try to reuse as much V2Writable code as possible for
external indices, but the way "last_commit" info is stored
must be different as external indices will deal with last_commit
info for multiple inboxes.
|
|
This will make it easier to share code with ExtSearchIdx.
|
|
This will be used by external/detached indices, too.
|
|
This will provide a similar API to PublicInbox::Inbox for
read-only WWW, -imapd, and -nntpd interfaces.
|
|
We'll be using this in detached (ext) Xapian indexes
in cross inbox search.
|