* [PATCH 4/5] searchidx: use Perl truthiness to detect XAPIAN_FLUSH_THRESHOLD
2020-08-07 10:52 5% [PATCH 0/5] more indexing improvements Eric Wong
@ 2020-08-07 10:52 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-08-07 10:52 UTC (permalink / raw)
To: meta
XAPIAN_FLUSH_THRESHOLD is a C string in the environment, so
users may be tempted to assign an empty string in in their
shell, e.g. `XAPIAN_FLUSH_THRESHOLD= <command>' instead of using
`unset' POSIX shell built-in.
With either a value of "0" or "" (empty string), Xapian will
fall back to its default (10000 documents), which causes grief
for memory-starved users.
---
lib/PublicInbox/SearchIdx.pm | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 22489731..01b9f52d 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -25,8 +25,7 @@ our @EXPORT_OK = qw(crlf_adjust log2stack is_ancestor check_size nodatacow_dir);
my $X = \%PublicInbox::Search::X;
my ($DB_CREATE_OR_OPEN, $DB_OPEN);
our $DB_NO_SYNC = 0;
-our $BATCH_BYTES = defined($ENV{XAPIAN_FLUSH_THRESHOLD}) ?
- 0x7fffffff : 1_000_000;
+our $BATCH_BYTES = $ENV{XAPIAN_FLUSH_THRESHOLD} ? 0x7fffffff : 1_000_000;
use constant DEBUG => !!$ENV{DEBUG};
my $xapianlevels = qr/\A(?:full|medium)\z/;
^ permalink raw reply related [relevance 7%]
* [PATCH 0/5] more indexing improvements
@ 2020-08-07 10:52 5% Eric Wong
2020-08-07 10:52 7% ` [PATCH 4/5] searchidx: use Perl truthiness to detect XAPIAN_FLUSH_THRESHOLD Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-08-07 10:52 UTC (permalink / raw)
To: meta
VERY big batch sizes seem helpful on HDDs.. And I also blew up
a run because --compact ran in parallel with 32 shards :x
And --help should exist for all commands users may run from
the CLI.
Eric Wong (5):
v2writable: fix batch size accounting
index: --compact respects --sequential-shard
index: max out XAPIAN_FLUSH_THRESHOLD if using --batch-size
searchidx: use Perl truthiness to detect XAPIAN_FLUSH_THRESHOLD
index: add built-in --help / -?
Documentation/public-inbox-index.pod | 4 +-
lib/PublicInbox/SearchIdx.pm | 3 +-
lib/PublicInbox/V2Writable.pm | 10 ++++-
script/public-inbox-index | 58 ++++++++++++++++++++++------
4 files changed, 58 insertions(+), 17 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-08-07 10:52 5% [PATCH 0/5] more indexing improvements Eric Wong
2020-08-07 10:52 7% ` [PATCH 4/5] searchidx: use Perl truthiness to detect XAPIAN_FLUSH_THRESHOLD Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).