From 7faaa2ab3464880aabbba909acd48a4b11b67452 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Thu, 19 Jul 2018 03:21:38 +0000 Subject: searchidx: respect XAPIAN_FLUSH_THRESHOLD env if set Xapian documents and respect XAPIAN_FLUSH_THRESHOLD to define the interval in documents to flush, so don't override it with our own BATCH_BYTES. This is helpful for initial indexing for those on slower storage but enough RAM. It is unnecessary for -watch and frequent incremental indexing; and it increases transaction times if -watch is playing "catch-up" if it was stopped for a while. The original BATCH_BYTES was tuned for a machine with little memory as the default XAPIAN_FLUSH_THRESHOLD of 10000 documents was causing swap storms. Using document counts also proved an innaccurate estimator of RAM usage compared to the actual bytes processed. --- lib/PublicInbox/SearchIdx.pm | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm index 04e85306..bb60506c 100644 --- a/lib/PublicInbox/SearchIdx.pm +++ b/lib/PublicInbox/SearchIdx.pm @@ -22,7 +22,8 @@ require PublicInbox::Git; use Compress::Zlib qw(compress); use constant { - BATCH_BYTES => 1_000_000, + BATCH_BYTES => defined($ENV{XAPIAN_FLUSH_THRESHOLD}) ? + 0x7fffffff : 1_000_000, DEBUG => !!$ENV{DEBUG}, }; -- cgit v1.2.3-24-ge0c7