* [PATCH 2/9] v2writable: nproc_shards: subtract 1 from given value
2020-02-02 6:52 5% [PATCH 0/9] convert: more fixes and tests Eric Wong
@ 2020-02-02 6:52 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-02-02 6:52 UTC (permalink / raw)
To: meta
This is to be consistent with the `nproc(1)' code path. It also
quiets down a warning from Admin when "-j $JOBS" is specified,
since the master process (which distributes work to shards and
handles OverIdx and Msgmap) is considered a job on its own.
---
lib/PublicInbox/V2Writable.pm | 9 ++-------
t/xcpdb-reshard.t | 2 +-
2 files changed, 3 insertions(+), 8 deletions(-)
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index 8ad71b54..37d27302 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -34,13 +34,8 @@ our $NPROC_MAX_DEFAULT = 4;
sub nproc_shards ($) {
my ($creat_opt) = @_;
- if (ref($creat_opt) eq 'HASH') {
- if (defined(my $n = $creat_opt->{nproc})) {
- return $n
- }
- }
-
- my $n = $ENV{NPROC};
+ my $n = $creat_opt->{nproc} if ref($creat_opt) eq 'HASH';
+ $n //= $ENV{NPROC};
if (!$n) {
chomp($n = `nproc 2>/dev/null`);
# assume 2 cores if GNU nproc(1) is not available
diff --git a/t/xcpdb-reshard.t b/t/xcpdb-reshard.t
index 9f0034f1..5fce5ead 100644
--- a/t/xcpdb-reshard.t
+++ b/t/xcpdb-reshard.t
@@ -40,7 +40,7 @@ for my $i (1..$ndoc) {
}
$im->done;
my @shards = grep(m!/\d+\z!, glob("$ibx->{inboxdir}/xap*/*"));
-is(scalar(@shards), $nproc, 'got expected shards');
+is(scalar(@shards), $nproc - 1, 'got expected shards');
my $orig = $ibx->over->query_xover(1, $ndoc);
my %nums = map {; "$_->{num}" => 1 } @$orig;
^ permalink raw reply related [relevance 7%]
* [PATCH 0/9] convert: more fixes and tests
@ 2020-02-02 6:52 5% Eric Wong
2020-02-02 6:52 7% ` [PATCH 2/9] v2writable: nproc_shards: subtract 1 from given value Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-02-02 6:52 UTC (permalink / raw)
To: meta
Nothing major, but SMP systems should be better utilized
now that -convert can take advantage of parallel shard
indexing in Xapian. The switches for convert are also
now documented in the manpage.
Eric Wong (9):
t/multi-mid.t: extra test for -convert highwater mark
v2writable: nproc_shards: subtract 1 from given value
v2writable: do not clobber {shards} or {parallel} if unset
convert: remove unused variables capturing :from
searchidxshard: rely on autoflush instead of ->flush
convert: shift @ARGV explicitly
convert: fix --no-index switch
doc: -convert: document switches
v2writable: more ways to detect online CPU count
Documentation/public-inbox-convert.pod | 41 ++++++++++++++++-
MANIFEST | 1 +
lib/PublicInbox/SearchIdxShard.pm | 4 +-
lib/PublicInbox/V2Writable.pm | 38 +++++++++++-----
script/public-inbox-convert | 16 ++-----
t/convert-compact.t | 3 ++
t/multi-mid.t | 61 ++++++++++++++++++++++++++
t/xcpdb-reshard.t | 2 +-
8 files changed, 137 insertions(+), 29 deletions(-)
create mode 100644 t/multi-mid.t
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-02-02 6:52 5% [PATCH 0/9] convert: more fixes and tests Eric Wong
2020-02-02 6:52 7% ` [PATCH 2/9] v2writable: nproc_shards: subtract 1 from given value Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).