* [PATCH 1/7] search: hoist out shards_dir for future use
2023-08-24 1:22 5% [PATCH 0/7] cindex: optional C++ Xapian helper Eric Wong
@ 2023-08-24 1:22 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-08-24 1:22 UTC (permalink / raw)
To: meta
This will be useful for internal tooling and APIs.
---
lib/PublicInbox/Search.pm | 34 +++++++++++++++++++---------------
1 file changed, 19 insertions(+), 15 deletions(-)
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index b2de3450..1559d9b3 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -202,28 +202,32 @@ sub xdir ($;$) {
}
}
-# returns all shards as separate Xapian::Database objects w/o combining
-sub xdb_shards_flat ($) {
+# returns shard directories as an array of strings, does not verify existence
+sub shard_dirs ($) {
my ($self) = @_;
my $xpfx = $self->{xpfx};
- my (@xdb, $slow_phrase);
- load_xapian();
- $self->{qp_flags} //= $QP_FLAGS;
- if ($xpfx =~ m!/xapian[0-9]+\z!) { # v1
- @xdb = ($X{Database}->new($xpfx));
- $self->{qp_flags} |= FLAG_PHRASE() if !-f "$xpfx/iamchert";
- } else { # v2, eidx, cidx
+ if ($xpfx =~ m!/xapian[0-9]+\z!) { # v1 inbox
+ ($xpfx);
+ } else { # v2 inbox, eidx, cidx
opendir(my $dh, $xpfx) or return (); # not initialized yet
# We need numeric sorting so shard[0] is first for reading
# Xapian metadata, if needed
my $last = max(grep(/\A[0-9]+\z/, readdir($dh))) // return ();
- @xdb = map {
- my $shard_dir = "$xpfx/$_";
- $slow_phrase ||= -f "$shard_dir/iamchert";
- $X{Database}->new($shard_dir);
- } (0..$last);
- $self->{qp_flags} |= FLAG_PHRASE() if !$slow_phrase;
+ map { "$xpfx/$_" } (0..$last);
}
+}
+
+# returns all shards as separate Xapian::Database objects w/o combining
+sub xdb_shards_flat ($) {
+ my ($self) = @_;
+ load_xapian();
+ $self->{qp_flags} //= $QP_FLAGS;
+ my $slow_phrase;
+ my @xdb = map {
+ $slow_phrase ||= -f "$_/iamchert";
+ $X{Database}->new($_); # raises if missing
+ } shard_dirs($self);
+ $self->{qp_flags} |= FLAG_PHRASE() if !$slow_phrase;
@xdb;
}
^ permalink raw reply related [relevance 7%]
* [PATCH 0/7] cindex: optional C++ Xapian helper
@ 2023-08-24 1:22 5% Eric Wong
2023-08-24 1:22 7% ` [PATCH 1/7] search: hoist out shards_dir for future use Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-08-24 1:22 UTC (permalink / raw)
To: meta
Associating inboxes with coderepos is an extremely expensive
operation, especially for Perl (even with XS or SWIG) as Perl's
method dispatch overhead to dump data out of Xapian becomes
noticeable.
The actual association is fast with POSIX sort(1) and join(1);
but getting the necessary data out of Xapian to join on is
expensive as neither quest(1) nor xapian-delve(1) are suitable
for this task.
The actual association data isn't stored or usable anywhere,
yet, and some of them are too loose to be useful. More
work is required on that point....
The association could probably be faster with rculfhash (from
Userspace-RCU), but I don't think it's worth the maintenance and
installation overhead for this (though I intend to use rculfhash
for the FUSE shim).
These performance problems weren't as noticeable in the past
since our other Xapian uses spent significant amounts of time
when retrieving document data from SQLite and blobs from git.
Using the C++ implementation of xap_helper.h allows a full join
(without limits or date ranges) of lore + git.kernel.org repos
within one hour on my ancient system while the Perl+(XS|SWIG)
implementation took roughly 8 hours.
Eric Wong (7):
search: hoist out shards_dir for future use
cindex: read-only association dump
cindex: add --show-roots switch
introduce optional C++ xap_helper
cindex: fix sorting and uniqueness
cindex: implement dump_roots in C++
xap_helper: reopen+retry in MSetIterator loops
MANIFEST | 7 +
lib/PublicInbox/CidxRecvIbx.pm | 46 ++
lib/PublicInbox/CidxXapHelperAux.pm | 44 ++
lib/PublicInbox/CodeSearch.pm | 54 +-
lib/PublicInbox/CodeSearchIdx.pm | 349 ++++++++--
lib/PublicInbox/Config.pm | 2 +-
lib/PublicInbox/Isearch.pm | 5 +
lib/PublicInbox/Search.pm | 92 ++-
lib/PublicInbox/XapClient.pm | 50 ++
lib/PublicInbox/XapHelper.pm | 226 +++++++
lib/PublicInbox/XapHelperCxx.pm | 93 +++
lib/PublicInbox/xap_helper.h | 947 ++++++++++++++++++++++++++++
script/public-inbox-cindex | 4 +-
t/xap_helper.t | 175 +++++
14 files changed, 2021 insertions(+), 73 deletions(-)
create mode 100644 lib/PublicInbox/CidxRecvIbx.pm
create mode 100644 lib/PublicInbox/CidxXapHelperAux.pm
create mode 100644 lib/PublicInbox/XapClient.pm
create mode 100644 lib/PublicInbox/XapHelper.pm
create mode 100644 lib/PublicInbox/XapHelperCxx.pm
create mode 100644 lib/PublicInbox/xap_helper.h
create mode 100644 t/xap_helper.t
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-08-24 1:22 5% [PATCH 0/7] cindex: optional C++ Xapian helper Eric Wong
2023-08-24 1:22 7% ` [PATCH 1/7] search: hoist out shards_dir for future use Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).