| Date | Commit message (Collapse) |
|---|
|
Emit information about reindexing git revision ranges when used
with xcpdb. Additionally, distinguish Xapian copy output from
v2 git epoch counting by increasing directory context info.
For now, v1 batches batches are emitted. v2 indexing is still
missing progress reporting for batches, as the data structures
for reindexing would benefit from a refactoring, first.
This does not currently affect the use of public-inbox-index,
but may in the future.
|
|
`warn' is reserved for actual warnings, as it respects
$SIG{__WARN__} and we rely on that override to print
message context information when we are indexing.
|
|
By creating temporary directories as deep as possible,
we can allow v2 repositories to have `xap$SCHEMA_VERSION'
(e.g. `xap15') reside on a separate FS.
We also check st_dev ahead-of-time to avoid doing work which
will fail with EXDEV. Of course, another process may still
move/change things around.
|
|
Running a full "public-inbox-index --reindex" in parallel
with "public-inbox-xcpdb" on the same inbox can still cause
problems, though.
|
|
Copying an entire Xapian DB is horribly slow whether it's done
via Perl or copydatabase(1). So displaying some progress
indication is good for user experience.
While we're at it, prefix xapian-compact output, too; since
parallel processes end up clobbering each other.
|
|
Copying an entire Xapian DB takes a long time, so update our
reindexing code to support partial reindexing, snapshot the
pre-copydatabase git revisions, perform the lengthy copy,
and do a partial reindex when the copy + renames are done.
|
|
To minimize the delay on active inboxes, it's actually ideal to
run xapian-compact at the end of the per-partition cpdb process;
since the new DB isn't accessible yet and so we don't have to
deal with lock contention with -mda or -watch processes. The
downside is temporary file overhead (3x instead of 2x) required.
|
|
By avoid copydatabase(1) entirely, we can make further changes
to avoid locking the entire inbox for a long operation and
switch to fine-grained locking.
|
|
We move the old directory into the new directory, so avoid the
situation where a bug or error could cause the tempdir cleanup to run
and destroy both our old and new directories.
|
|
copydatabase(1) is exceptionally noisy and it's output is
confusing when run in parallel. Support redirects at least, and
env while we're at it to give us future options.
We can also stuff a -jobs parameter into the options to limit
parallelism since it can be useful for low-priority upgrade
jobs.
|
|
Port public-inbox-compact(1) over to using it, and we will need
to wrap copydatabase(1) to ease glass migrations, too.
|