diff options
| author | Eric Wong <e@80x24.org> | 2018-07-19 03:21:38 +0000 |
|---|---|---|
| committer | Eric Wong <e@80x24.org> | 2018-07-19 03:22:15 +0000 |
| commit | 7faaa2ab3464880aabbba909acd48a4b11b67452 (patch) | |
| tree | 5fcd3b0592c05d53bc6921de6c9c4cc37d0aa311 /lib/PublicInbox | |
| parent | 917a23b7879b8cbda3b6b2cac5d60f40575b0ee8 (diff) | |
| download | public-inbox-7faaa2ab3464880aabbba909acd48a4b11b67452.tar.gz | |
Xapian documents and respect XAPIAN_FLUSH_THRESHOLD to define the interval in documents to flush, so don't override it with our own BATCH_BYTES. This is helpful for initial indexing for those on slower storage but enough RAM. It is unnecessary for -watch and frequent incremental indexing; and it increases transaction times if -watch is playing "catch-up" if it was stopped for a while. The original BATCH_BYTES was tuned for a machine with little memory as the default XAPIAN_FLUSH_THRESHOLD of 10000 documents was causing swap storms. Using document counts also proved an innaccurate estimator of RAM usage compared to the actual bytes processed.
Diffstat (limited to 'lib/PublicInbox')
| -rw-r--r-- | lib/PublicInbox/SearchIdx.pm | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm index 04e85306..bb60506c 100644 --- a/lib/PublicInbox/SearchIdx.pm +++ b/lib/PublicInbox/SearchIdx.pm @@ -22,7 +22,8 @@ require PublicInbox::Git; use Compress::Zlib qw(compress); use constant { - BATCH_BYTES => 1_000_000, + BATCH_BYTES => defined($ENV{XAPIAN_FLUSH_THRESHOLD}) ? + 0x7fffffff : 1_000_000, DEBUG => !!$ENV{DEBUG}, }; |