user/dev discussion of public-inbox itself
 help / Atom feed
* [PATCH] searchidx: respect XAPIAN_FLUSH_THRESHOLD env if set
@ 2018-07-19  3:21 Eric Wong
  0 siblings, 0 replies; 1+ messages in thread
From: Eric Wong @ 2018-07-19  3:21 UTC (permalink / raw)
  To: meta

Xapian documents and respect XAPIAN_FLUSH_THRESHOLD to define
the interval in documents to flush, so don't override it with
our own BATCH_BYTES.  This is helpful for initial indexing for
those on slower storage but enough RAM.

It is unnecessary for -watch and frequent incremental indexing;
and it increases transaction times if -watch is playing "catch-up"
if it was stopped for a while.

The original BATCH_BYTES was tuned for a machine with little
memory as the default XAPIAN_FLUSH_THRESHOLD of 10000 documents
was causing swap storms.  Using document counts also proved an
innaccurate estimator of RAM usage compared to the actual bytes
processed.
---
 lib/PublicInbox/SearchIdx.pm | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 0e0796c..5ba326b 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -22,7 +22,8 @@ require PublicInbox::Git;
 use Compress::Zlib qw(compress);
 
 use constant {
-	BATCH_BYTES => 1_000_000,
+	BATCH_BYTES => defined($ENV{XAPIAN_FLUSH_THRESHOLD}) ?
+			0x7fffffff : 1_000_000,
 	DEBUG => !!$ENV{DEBUG},
 };
 
-- 
EW


^ permalink raw reply	[flat|nested] 1+ messages in thread

only message in thread, back to index

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-19  3:21 [PATCH] searchidx: respect XAPIAN_FLUSH_THRESHOLD env if set Eric Wong

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox