* [PATCH 06/10] extsearchidx: close SQLite handles after attaching
2020-12-23 8:38 5% [PATCH 00/10] start optimizing startup w/ ALL->misc Eric Wong
@ 2020-12-23 8:38 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-12-23 8:38 UTC (permalink / raw)
To: meta
This is needed to prevent us from running out of FDs when
indexing many inboxes. Perhaps checking these on attach_inbox
is unnecessary and may be removed entirely down the line.
---
lib/PublicInbox/ExtSearchIdx.pm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/ExtSearchIdx.pm b/lib/PublicInbox/ExtSearchIdx.pm
index 9d64ff5a..fb627089 100644
--- a/lib/PublicInbox/ExtSearchIdx.pm
+++ b/lib/PublicInbox/ExtSearchIdx.pm
@@ -65,11 +65,14 @@ sub attach_inbox {
my $misc = $self->{misc};
if ($misc && $misc->inbox_data($ibx)) { # all good if already indexed
} else {
- if (!$ibx->over || !$ibx->mm) {
+ my @sqlite = ($ibx->over, $ibx->mm);
+ my $uidvalidity = $ibx->uidvalidity;
+ $ibx->{mm} = $ibx->{over} = undef;
+ if (scalar(@sqlite) != 2) {
warn "W: skipping $ekey (unindexed)\n";
return;
}
- if (!defined($ibx->uidvalidity)) {
+ if (!defined($uidvalidity)) {
warn "W: skipping $ekey (no UIDVALIDITY)\n";
return;
}
^ permalink raw reply related [relevance 7%]
* [PATCH 00/10] start optimizing startup w/ ALL->misc
@ 2020-12-23 8:38 5% Eric Wong
2020-12-23 8:38 7% ` [PATCH 06/10] extsearchidx: close SQLite handles after attaching Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-12-23 8:38 UTC (permalink / raw)
To: meta
-nntpd [PATCH 5/10] is the single most significant improvements.
And some cleanups, and some general improvements independent of
indexing in patches 7-10 (patch 8 is already superceded by 10,
but kept separate for documentation purposes).
PublicInbox::Config->new is over twice as fast, now.
Eric Wong (10):
miscsearch: load Xapian at initialization
xt: add create-many-inboxes helper test
inbox: git_epoch: correct false comment
inboxwritable: _init_v1: set created_at ASAP
miscsearch: index UIDVALIDITY, use as startup cache
extsearchidx: close SQLite handles after attaching
config: _fill: inbox name extraction optimization
config: git_config_dump: pre-compile RE for split
config: config_fh_parse: micro-optimize
config: config_fh_parse: micro-optimize even harder
MANIFEST | 1 +
lib/PublicInbox/Config.pm | 26 ++++-----
lib/PublicInbox/ExtSearchIdx.pm | 25 +++++---
lib/PublicInbox/Inbox.pm | 2 +-
lib/PublicInbox/InboxWritable.pm | 3 +-
lib/PublicInbox/MiscIdx.pm | 26 ++++++---
lib/PublicInbox/MiscSearch.pm | 57 ++++++++++++++++--
lib/PublicInbox/NNTPD.pm | 4 +-
lib/PublicInbox/Search.pm | 9 ++-
lib/PublicInbox/SearchIdx.pm | 7 ---
t/search.t | 4 +-
xt/create-many-inboxes.t | 99 ++++++++++++++++++++++++++++++++
12 files changed, 213 insertions(+), 50 deletions(-)
create mode 100644 xt/create-many-inboxes.t
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-12-23 8:38 5% [PATCH 00/10] start optimizing startup w/ ALL->misc Eric Wong
2020-12-23 8:38 7% ` [PATCH 06/10] extsearchidx: close SQLite handles after attaching Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).