user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 5/6] xcpdb: wire up new index options and --help
Date: Wed, 12 Aug 2020 09:17:18 +0000
Message-ID: <20200812091719.13739-6-e@yhbt.net> (raw)
In-Reply-To: <20200812091719.13739-1-e@yhbt.net>

--sequential-shard also disables the copy parallelism (--jobs),
so it can be useful for systems unable to handle parallel random
I/O but still want many shards.

There was a missing "use strict", too, which is fixed.
---
 Documentation/public-inbox-xcpdb.pod | 19 +++++++-
 lib/PublicInbox/Xapcmd.pm            |  3 +-
 script/public-inbox-xcpdb            | 66 +++++++++++++++++++++++-----
 3 files changed, 75 insertions(+), 13 deletions(-)

diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod
index 2ed4c5821..62a28c0a1 100644
--- a/Documentation/public-inbox-xcpdb.pod
+++ b/Documentation/public-inbox-xcpdb.pod
@@ -19,7 +19,7 @@ L<public-inbox-learn(1)>, and L<public-inbox-index(1)>.
 
 =over
 
-=item --compact
+=item -c, --compact
 
 In addition to performing the copy operation, run L<xapian-compact(1)>
 on each Xapian shard after copying but before finalizing it.
@@ -52,6 +52,23 @@ Disable L<fsync(2)> and L<fdatasync(2)>.
 
 Available in public-inbox 1.6.0 (PENDING).
 
+=item --sequential-shard
+
+Copy each shard sequentially, ignoring C<--jobs>.  This also
+affects indexing done at the end of a run.
+
+=item --batch-size=BYTES
+
+=item --max-size=BYTES
+
+See L<public-inbox-index(1)> for a description of these options.
+
+These indexing options indexing at the end of a run.
+C<public-inbox-xcpdb> may run in parallel with with
+L<public-inbox-index(1)>, and C<public-inbox-xcpdb> needs to
+reindex changes made to the old Xapian DBs by
+L<public-inbox-index(1)> while it was running.
+
 =back
 
 =head1 ENVIRONMENT
diff --git a/lib/PublicInbox/Xapcmd.pm b/lib/PublicInbox/Xapcmd.pm
index b6279218c..46548a948 100644
--- a/lib/PublicInbox/Xapcmd.pm
+++ b/lib/PublicInbox/Xapcmd.pm
@@ -82,7 +82,8 @@ sub commit_changes ($$$$) {
 				$im->{shards} = $n;
 			}
 		}
-
+		my $env = $opt->{-idx_env};
+		local %ENV = (%ENV, %$env) if $env;
 		PublicInbox::Admin::index_inbox($ibx, $im, $opt);
 	}
 }
diff --git a/script/public-inbox-xcpdb b/script/public-inbox-xcpdb
index 2c91598cb..718a34b77 100755
--- a/script/public-inbox-xcpdb
+++ b/script/public-inbox-xcpdb
@@ -1,20 +1,64 @@
-#!/usr/bin/perl -w
+#!perl -w
 # Copyright (C) 2019-2020 all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-# xcpdb: Xapian copy database, a wrapper around Xapian's copydatabase(1)
+use strict;
+use v5.10.1;
 use Getopt::Long qw(:config gnu_getopt no_ignore_case auto_abbrev);
-use PublicInbox::InboxWritable;
-use PublicInbox::Xapcmd;
+my $usage = 'Usage: public-inbox-xcpdb [options] INBOX_DIR';
+my $help = <<EOF; # the following should fit w/o scrolling in 80x24 term:
+usage: $usage
+
+  upgrade or reshard Xapian DB(s) used by public-inbox
+
+options:
+
+  --compact | -c      run public-inbox-compact(1) after indexing
+  --reshard=NUM       change number the number of shards
+  --jobs=NUM          limit parallelism to JOBS count
+  --verbose | -v      increase verbosity (may be repeated)
+  --sequential-shard  copy+index Xapian shards sequentially (for slow HDD)
+  --help | -?         show this help
+
+index options (see public-inbox-index(1) man page for full description):
+
+  --no-fsync          speed up indexing, risk corruption on power outage
+  --batch-size=BYTES  flush changes to OS after a given number of bytes
+  --max-size=BYTES    do not index messages larger than the given size
+
+See public-inbox-xcpdb(1) man page for full documentation.
+EOF
+my $opt = { quiet => -1, compact => 0, fsync => 1 };
+GetOptions($opt, qw(
+	fsync|sync! compact|c reshard|R=i
+	max_size|max-size=s batch_size|batch-size=s
+	sequential_shard|seq-shard|sequential-shard
+	jobs|j=i quiet|q verbose|v
+	blocksize|b=s no-full|n fuller|F
+	help|?)) or die "bad command-line args\n$usage";
+if ($opt->{help}) { print $help; exit 0 };
+
 use PublicInbox::Admin;
 PublicInbox::Admin::require_or_die('-search');
-my $usage = "Usage: public-inbox-xcpdb [--compact] INBOX_DIR\n";
-my $opt = { fsync => 1 };
-my @opt = (qw(fsync|sync! compact reshard|R=i),
-	@PublicInbox::Xapcmd::COMPACT_OPT);
-GetOptions($opt, @opt) or die "bad command-line args\n$usage";
-my @ibxs = PublicInbox::Admin::resolve_inboxes(\@ARGV) or die $usage;
+
+require PublicInbox::Config;
+my $cfg = PublicInbox::Config->new;
+my @ibxs = PublicInbox::Admin::resolve_inboxes(\@ARGV, undef, $cfg) or
+	die $usage;
+my $idx_env = PublicInbox::Admin::index_prepare($opt, $cfg);
+
+# we only set XAPIAN_FLUSH_THRESHOLD for index, since cpdb doesn't
+# know sizes, only doccounts
+$opt->{-idx_env} = $idx_env;
+
+if ($opt->{sequential_shard} && ($opt->{jobs} // 1) > 1) {
+	warn "W: --jobs=$opt->{jobs} ignored with --sequential-shard\n";
+	$opt->{jobs} = 0;
+}
+
+require PublicInbox::InboxWritable;
+require PublicInbox::Xapcmd;
 foreach (@ibxs) {
 	my $ibx = PublicInbox::InboxWritable->new($_);
-	# we rely on --no-renumber to keep docids synched to NNTP
+	# we rely on --no-renumber to keep docids synched for NNTP
 	PublicInbox::Xapcmd::run($ibx, 'cpdb', $opt);
 }

  parent reply	other threads:[~2020-08-12  9:17 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-08-12  9:17 [PATCH 0/6] xcpdb -index improvements Eric Wong
2020-08-12  9:17 ` [PATCH 1/6] xapcmd: simplify sub reference Eric Wong
2020-08-12  9:17 ` [PATCH 2/6] xcpdb: support --no-fsync from CLI Eric Wong
2020-08-12  9:17 ` [PATCH 3/6] xapcmd: reduce CPU idling when shards exceeds job count Eric Wong
2020-08-12  9:17 ` [PATCH 4/6] admin: don't warn when --jobs exceeds shards Eric Wong
2020-08-12  9:17 ` Eric Wong [this message]
2020-08-12  9:17 ` [PATCH 6/6] v2writable: remove IdxStack import Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200812091719.13739-6-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git