From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH] searchidx: respect XAPIAN_FLUSH_THRESHOLD env if set
Date: Thu, 19 Jul 2018 03:21:38 +0000 [thread overview]
Message-ID: <20180719032138.6329-1-e@80x24.org> (raw)
Xapian documents and respect XAPIAN_FLUSH_THRESHOLD to define
the interval in documents to flush, so don't override it with
our own BATCH_BYTES. This is helpful for initial indexing for
those on slower storage but enough RAM.
It is unnecessary for -watch and frequent incremental indexing;
and it increases transaction times if -watch is playing "catch-up"
if it was stopped for a while.
The original BATCH_BYTES was tuned for a machine with little
memory as the default XAPIAN_FLUSH_THRESHOLD of 10000 documents
was causing swap storms. Using document counts also proved an
innaccurate estimator of RAM usage compared to the actual bytes
processed.
---
lib/PublicInbox/SearchIdx.pm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 0e0796c..5ba326b 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -22,7 +22,8 @@ require PublicInbox::Git;
use Compress::Zlib qw(compress);
use constant {
- BATCH_BYTES => 1_000_000,
+ BATCH_BYTES => defined($ENV{XAPIAN_FLUSH_THRESHOLD}) ?
+ 0x7fffffff : 1_000_000,
DEBUG => !!$ENV{DEBUG},
};
--
EW
reply other threads:[~2018-07-19 3:21 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180719032138.6329-1-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).