* [PATCH 9/9] v2writable: more ways to detect online CPU count
2020-02-02 6:52 5% [PATCH 0/9] convert: more fixes and tests Eric Wong
@ 2020-02-02 6:52 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-02-02 6:52 UTC (permalink / raw)
To: meta
OpenBSD and FreeBSD support `getconf NPROCESSORS_ONLN` (no
leading underscore). They may also have GNU nproc installed as
"gnproc".
We may also encounter Linux systems w/o GNU coreutils, but able
to use `getconf _NPROCESSORS_ONLN` (with leading underscore).
---
lib/PublicInbox/V2Writable.pm | 22 +++++++++++++++++++---
1 file changed, 19 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index 72d8d5af..82843241 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -7,6 +7,7 @@ package PublicInbox::V2Writable;
use strict;
use warnings;
use base qw(PublicInbox::Lock);
+use 5.010_001;
use PublicInbox::SearchIdxShard;
use PublicInbox::MIME;
use PublicInbox::Git;
@@ -32,14 +33,29 @@ my $PACKING_FACTOR = 0.4;
# to increase Xapian shards
our $NPROC_MAX_DEFAULT = 4;
+sub detect_nproc () {
+ for my $nproc (qw(nproc gnproc)) { # GNU coreutils nproc
+ `$nproc 2>/dev/null` =~ /^(\d+)$/ and return $1;
+ }
+
+ # getconf(1) is POSIX, but *NPROCESSORS* vars are not
+ for (qw(_NPROCESSORS_ONLN NPROCESSORS_ONLN)) {
+ `getconf $_ 2>/dev/null` =~ /^(\d+)$/ and return $1;
+ }
+
+ # should we bother with `sysctl hw.ncpu`? Those only give
+ # us total processor count, not online processor count.
+ undef
+}
+
sub nproc_shards ($) {
my ($creat_opt) = @_;
my $n = $creat_opt->{nproc} if ref($creat_opt) eq 'HASH';
$n //= $ENV{NPROC};
if (!$n) {
- chomp($n = `nproc 2>/dev/null`);
- # assume 2 cores if GNU nproc(1) is not available
- $n = 2 if !$n;
+ # assume 2 cores if not detectable or zero
+ state $NPROC_DETECTED = detect_nproc() || 2;
+ $n = $NPROC_DETECTED;
$n = $NPROC_MAX_DEFAULT if $n > $NPROC_MAX_DEFAULT;
}
^ permalink raw reply related [relevance 7%]
* [PATCH 0/9] convert: more fixes and tests
@ 2020-02-02 6:52 5% Eric Wong
2020-02-02 6:52 7% ` [PATCH 9/9] v2writable: more ways to detect online CPU count Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-02-02 6:52 UTC (permalink / raw)
To: meta
Nothing major, but SMP systems should be better utilized
now that -convert can take advantage of parallel shard
indexing in Xapian. The switches for convert are also
now documented in the manpage.
Eric Wong (9):
t/multi-mid.t: extra test for -convert highwater mark
v2writable: nproc_shards: subtract 1 from given value
v2writable: do not clobber {shards} or {parallel} if unset
convert: remove unused variables capturing :from
searchidxshard: rely on autoflush instead of ->flush
convert: shift @ARGV explicitly
convert: fix --no-index switch
doc: -convert: document switches
v2writable: more ways to detect online CPU count
Documentation/public-inbox-convert.pod | 41 ++++++++++++++++-
MANIFEST | 1 +
lib/PublicInbox/SearchIdxShard.pm | 4 +-
lib/PublicInbox/V2Writable.pm | 38 +++++++++++-----
script/public-inbox-convert | 16 ++-----
t/convert-compact.t | 3 ++
t/multi-mid.t | 61 ++++++++++++++++++++++++++
t/xcpdb-reshard.t | 2 +-
8 files changed, 137 insertions(+), 29 deletions(-)
create mode 100644 t/multi-mid.t
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-02-02 6:52 5% [PATCH 0/9] convert: more fixes and tests Eric Wong
2020-02-02 6:52 7% ` [PATCH 9/9] v2writable: more ways to detect online CPU count Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).