user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <>
Subject: [PATCH 1/3] init: add -j / --jobs parameter
Date: Sun, 21 Jun 2020 00:21:31 +0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

On a powerful (by my standards) machine with 16GB RAM and an
7200 RPM HDD marketed for "enterprise" use, indexing a 8.1G (in
git) LKML snapshot from Sep 2019 did not finish after 7 days
with the default number (3) of Xapian shards (`--jobs=4') and

Indexing starts off fast, but progressively get slower as
contents of the inbox (including Xapian + SQLite DBs) could no
longer be cached by the kernel.  Once the on-disk size
increased, HDD seek contention between the Xapian shard workers
slowed the process down to a crawl.

With a single shard, it still took around 3.5 days to index on
the HDD.  That's not good, but it's far better than not
finishing after 7 days.  So allow unfortunate HDD users to
easily specify a single shard on public-inbox-init.

For reference, a freshly TRIM-ed low-end TLC SSD on the SATA II
bus on the same machine indexes that same snapshot of LKML in
~7 hours with 3 shards and the same 10m batch size.  In the past,
a higher-end consumer grade MLC SSDs on similar hardware indexed
a similarly sized-data set in ~4 hours.
 Documentation/public-inbox-init.pod | 14 ++++++++++++++
 script/public-inbox-init            |  8 ++++++++
 t/v2mirror.t                        |  4 +++-
 3 files changed, 25 insertions(+), 1 deletion(-)

diff --git a/Documentation/public-inbox-init.pod b/Documentation/public-inbox-init.pod
index 4744da96..495a258f 100644
--- a/Documentation/public-inbox-init.pod
+++ b/Documentation/public-inbox-init.pod
@@ -48,6 +48,20 @@ added-after-the-fact (without affecting "git clone" followers).
 Default: unset, no epochs are skipped
+=item -j, --jobs=JOBS
+Control the number of Xapian index shards in a
+C<-V2> (L<public-inbox-v2-format(5)>) inbox.
+It is useful to use a single shard (C<-j1>) for inboxes on
+high-latency storage (e.g. rotational HDD) unless the system has
+enough RAM to cache 5-10x the size of the git repository.
+It is generally not useful to specify higher values than the
+default due to contention in the top-level producer process.
+Default: the number of online CPUs, up to 4
diff --git a/script/public-inbox-init b/script/public-inbox-init
index 10d3ad45..00147db5 100755
--- a/script/public-inbox-init
+++ b/script/public-inbox-init
@@ -27,10 +27,12 @@ use Cwd qw/abs_path/;
 my $version = undef;
 my $indexlevel = undef;
 my $skip_epoch;
+my $jobs;
 my %opts = (
 	'V|version=i' => \$version,
 	'L|indexlevel=s' => \$indexlevel,
 	'S|skip|skip-epoch=i' => \$skip_epoch,
+	'j|jobs=i' => \$jobs,
 GetOptions(%opts) or usage();
 PublicInbox::Admin::indexlevel_ok_or_die($indexlevel) if defined $indexlevel;
@@ -144,6 +146,12 @@ my $ibx = PublicInbox::Inbox->new({
 my $creat_opt = {};
+if (defined $jobs) {
+	die "--jobs is only supported for -V2 inboxes\n" if $version == 1;
+	die "--jobs=$jobs must be >= 1\n" if $jobs <= 0;
+	$creat_opt->{nproc} = $jobs;
 PublicInbox::InboxWritable->new($ibx, $creat_opt)->init_inbox(0, $skip_epoch);
 # needed for git prior to v2.1.0
diff --git a/t/v2mirror.t b/t/v2mirror.t
index fc03c3d7..b24528fe 100644
--- a/t/v2mirror.t
+++ b/t/v2mirror.t
@@ -80,9 +80,11 @@ foreach my $i (0..$epoch_max) {
 	ok(-d "$tmpdir/m/git/$i.git", "mirror $i OK");
-@cmd = ("-init", '-V2', 'm', "$tmpdir/m", '',
+@cmd = ("-init", '-j1', '-V2', 'm', "$tmpdir/m", '',
 ok(run_script(\@cmd), 'initialized public-inbox -V2');
+my @shards = glob("$tmpdir/m/xap*/?");
+is(scalar(@shards), 1, 'got a single shard on init');
 ok(run_script([qw(-index -j0), "$tmpdir/m"]), 'indexed');

  reply	other threads:[~2020-06-21  0:21 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-21  0:21 [PATCH 0/3] -init updates Eric Wong
2020-06-21  0:21 ` Eric Wong [this message]
2020-06-21  0:21 ` [PATCH 2/3] init: refer to inboxes as "inbox" or "inboxes" in errors Eric Wong
2020-06-21  0:21 ` [PATCH 3/3] init: add --skip-artnum parameter Eric Wong
2020-06-23 18:34   ` [PATCH] t/init: remove leftover find(1) call Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).