diff options
author | Eric Wong <e@yhbt.net> | 2020-08-07 01:14:04 +0000 |
---|---|---|
committer | Eric Wong <e@yhbt.net> | 2020-08-07 23:45:38 +0000 |
commit | 06a2418fd053c9a5b80217e74d1b47b8e1ca85e1 (patch) | |
tree | 37dc120e64b6f2114164a3e4d2d358373b1b1eb5 /t/config.t | |
parent | 32f6a1f9498f759041b72d6f4d5cb959088a3dec (diff) | |
download | public-inbox-06a2418fd053c9a5b80217e74d1b47b8e1ca85e1.tar.gz |
This gives better page cache utilization for Xapian indexing on slow storage by improving locality for random I/O activity on the Xapian DB. Instead of doing a single-pass to index both SQLite and Xapian; this indexes them separately. The first pass is identical to indexlevel=basic: it indexes both over.sqlite3 and msgmap.sqlite3. Subsequent passes only operate on a single Xapian shard for documents belonging to that shard. Given enough shards, each individual shard can be made small enough to fit into the kernel page cache and avoid HDD seeks for read activity. Doing rough tests with a busy system with a 7200 RPM HDD with ext4, full indexing of LKML (9 epochs) goes from ~80 hours (-j0) to ~30 hours (-j8) with 16GB RAM with 7 shards configured and fsync(2) disabled (--no-sync) and `--batch-size=10m'.
Diffstat (limited to 't/config.t')
-rw-r--r-- | t/config.t | 6 |
1 files changed, 3 insertions, 3 deletions
@@ -220,18 +220,18 @@ EOF { for my $t (qw(TRUE true yes on 1 +1 -1 13 0x1 0x12 0X5)) { - is(PublicInbox::Config::_git_config_bool($t), 1, "$t is true"); + is(PublicInbox::Config::git_bool($t), 1, "$t is true"); is(xqx([qw(git -c), "test.val=$t", qw(config --bool test.val)]), "true\n", "$t matches git-config behavior"); } for my $f (qw(FALSE false no off 0 +0 +000 00 0x00 0X0)) { - is(PublicInbox::Config::_git_config_bool($f), 0, "$f is false"); + is(PublicInbox::Config::git_bool($f), 0, "$f is false"); is(xqx([qw(git -c), "test.val=$f", qw(config --bool test.val)]), "false\n", "$f matches git-config behavior"); } - is(PublicInbox::Config::_git_config_bool('bogus'), undef, + is(PublicInbox::Config::git_bool('bogus'), undef, 'bogus is undef'); } |