* [PATCH 3/4] xcpdb: use destination shard as progress prefix
2019-06-14 3:03 6% [PATCH 0/4] xcpdb: support resharding Xapian DBs Eric Wong
@ 2019-06-14 3:03 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-06-14 3:03 UTC (permalink / raw)
To: meta
For M:N resharding, we'll want to display the number from
the new shard number.
---
lib/PublicInbox/Xapcmd.pm | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/Xapcmd.pm b/lib/PublicInbox/Xapcmd.pm
index 7204a91..5d8c35f 100644
--- a/lib/PublicInbox/Xapcmd.pm
+++ b/lib/PublicInbox/Xapcmd.pm
@@ -198,10 +198,11 @@ sub cpdb_retryable ($$) {
}
sub progress_pfx ($) {
- my @p = split('/', $_[0]);
+ my ($wip) = @_; # tempdir v2: ([0-9])+-XXXXXXXX
+ my @p = split('/', $wip);
# return "xap15/0" for v2, or "xapian15" for v1:
- ($p[-1] =~ /\A[0-9]+\z/) ? "$p[-2]/$p[-1]" : $p[-1];
+ ($p[-1] =~ /\A([0-9]+)/) ? "$p[-2]/$1" : $p[-1];
}
# xapian-compact wrapper
@@ -268,7 +269,7 @@ sub cpdb ($$) {
my ($it, $end);
my ($nr, $tot, $fmt); # progress output
my $pr = $opt->{-progress};
- my $pfx = $opt->{-progress_pfx} = progress_pfx($old);
+ my $pfx = $opt->{-progress_pfx} = progress_pfx($new);
do {
eval {
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] xcpdb: support resharding Xapian DBs
@ 2019-06-14 3:03 6% Eric Wong
2019-06-14 3:03 7% ` [PATCH 3/4] xcpdb: use destination shard as progress prefix Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-06-14 3:03 UTC (permalink / raw)
To: meta
Defaulting the number of Xapian shards based on the number
of CPUs can be detrimental to performance given the lack of
speed in common storage systems; since NVMe speeds are not
yet common.
To help public-inbox users recover from this inefficiency while
allowing continuous email archival, we can support arbitrary
resharding to have fewer shards (or more, if doing HW upgrades).
Note: I'm also going to move the documentation towards using the
word "shard" (instead of "partition") to be consistent with
current Xapian documentation (1.4+, and "master").
Xapian 1.2 did not use the word "shard" at all, but IME from my
interactions with non-Xapian search engine folks, the word
"shard" is pretty common.
Eric Wong (4):
v2writable: use a smaller default for Xapian partitions
xapcmd: preserve indexlevel based on the destination
xcpdb: use destination shard as progress prefix
xcpdb: support resharding v2 repos
Documentation/public-inbox-xcpdb.pod | 11 ++
MANIFEST | 1 +
lib/PublicInbox/V2Writable.pm | 18 ++-
lib/PublicInbox/Xapcmd.pm | 222 +++++++++++++++++++++------
script/public-inbox-xcpdb | 4 +-
t/xcpdb-reshard.t | 83 ++++++++++
6 files changed, 286 insertions(+), 53 deletions(-)
create mode 100644 t/xcpdb-reshard.t
--
EW
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-06-14 3:03 6% [PATCH 0/4] xcpdb: support resharding Xapian DBs Eric Wong
2019-06-14 3:03 7% ` [PATCH 3/4] xcpdb: use destination shard as progress prefix Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).