user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* [PATCH 3/7] cindex: add --show-roots switch
  2023-08-24  1:22  7% [PATCH 0/7] cindex: optional C++ Xapian helper Eric Wong
@ 2023-08-24  1:22  7% ` Eric Wong
  0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-08-24  1:22 UTC (permalink / raw)
  To: meta

This aids in development, but I'm not sure it's going to stay
or be moved into another interface.
---
 lib/PublicInbox/CodeSearchIdx.pm | 32 ++++++++++++++++++++++++++++++++
 script/public-inbox-cindex       |  2 +-
 2 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/CodeSearchIdx.pm b/lib/PublicInbox/CodeSearchIdx.pm
index 2480dbd2..e795c2b3 100644
--- a/lib/PublicInbox/CodeSearchIdx.pm
+++ b/lib/PublicInbox/CodeSearchIdx.pm
@@ -1058,6 +1058,37 @@ sub _prep_ibx { # each_inbox callback
 		push @{$self->{IBX}}, $ibx;
 }
 
+sub show_roots { # for diagnostics
+	my ($self) = @_;
+	local $self->{xdb};
+	my $cur = $self->xdb->allterms_begin('G');
+	my $end = $self->{xdb}->allterms_end('G');
+	my $qrepo = $PublicInbox::Search::X{Query}->new('T'.'r');
+	my $enq = $PublicInbox::Search::X{Enquire}->new($self->{xdb});
+	$enq->set_weighting_scheme($PublicInbox::Search::X{BoolWeight}->new);
+	$enq->set_docid_order($PublicInbox::Search::ENQ_ASCENDING);
+	for (; $cur != $end; $cur++) {
+		my $G_oidhex = $cur->get_termname;
+		my $qry = $PublicInbox::Search::X{Query}->new(
+				PublicInbox::Search::OP_FILTER(),
+				$qrepo, $G_oidhex);
+		$enq->set_query($qry);
+		my ($off, $lim) = (0, 10000);
+		say 'commit ',substr($G_oidhex, 1), ' appears in:';
+		while (1) {
+			my $mset = $enq->get_mset($off, $lim);
+			my $size = $mset->size or last;
+			for my $x ($mset->items) {
+				my $doc = $x->get_document;
+				for (xap_terms('P', $x->get_document)) {
+					say '- /', substr($_, 1);
+				}
+			}
+			$off += $size;
+		}
+	}
+}
+
 sub cidx_run { # main entry point
 	my ($self) = @_;
 	my $restore_umask = prep_umask($self);
@@ -1150,6 +1181,7 @@ sub cidx_run { # main entry point
 	PublicInbox::DS::event_loop($MY_SIG, $SIGSET) if shards_active();
 	PublicInbox::DS->Reset;
 	$self->lock_release(!!$NCHANGE);
+	show_roots($self) if $self->{-opt}->{'show-roots'} # for diagnostics
 }
 
 sub ipc_atfork_child { # @IDX_SHARDS
diff --git a/script/public-inbox-cindex b/script/public-inbox-cindex
index 888c8b10..0526434c 100755
--- a/script/public-inbox-cindex
+++ b/script/public-inbox-cindex
@@ -29,7 +29,7 @@ GetOptions($opt, qw(quiet|q verbose|v+ reindex jobs|j=i fsync|sync! dangerous
 		indexlevel|index-level|L=s associate associate-max=i
 		associate-date-range=s associate-prefixes=s@
 		batch_size|batch-size=s max_size|max-size=s
-		include|I=s@ only=s@ all
+		include|I=s@ only=s@ all show-roots
 		project-list=s exclude=s@
 		sort-parallel=s sort-compress-program=s sort-buffer-size=s
 		d=s update|u scan! prune dry-run|n C=s@ help|h))

^ permalink raw reply related	[relevance 7%]

* [PATCH 0/7] cindex: optional C++ Xapian helper
@ 2023-08-24  1:22  7% Eric Wong
  2023-08-24  1:22  7% ` [PATCH 3/7] cindex: add --show-roots switch Eric Wong
  0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-08-24  1:22 UTC (permalink / raw)
  To: meta

Associating inboxes with coderepos is an extremely expensive
operation, especially for Perl (even with XS or SWIG) as Perl's
method dispatch overhead to dump data out of Xapian becomes
noticeable.

The actual association is fast with POSIX sort(1) and join(1);
but getting the necessary data out of Xapian to join on is
expensive as neither quest(1) nor xapian-delve(1) are suitable
for this task.

The actual association data isn't stored or usable anywhere,
yet, and some of them are too loose to be useful.  More
work is required on that point....

The association could probably be faster with rculfhash (from
Userspace-RCU), but I don't think it's worth the maintenance and
installation overhead for this (though I intend to use rculfhash
for the FUSE shim).

These performance problems weren't as noticeable in the past
since our other Xapian uses spent significant amounts of time
when retrieving document data from SQLite and blobs from git.

Using the C++ implementation of xap_helper.h allows a full join
(without limits or date ranges) of lore + git.kernel.org repos
within one hour on my ancient system while the Perl+(XS|SWIG)
implementation took roughly 8 hours.

Eric Wong (7):
  search: hoist out shards_dir for future use
  cindex: read-only association dump
  cindex: add --show-roots switch
  introduce optional C++ xap_helper
  cindex: fix sorting and uniqueness
  cindex: implement dump_roots in C++
  xap_helper: reopen+retry in MSetIterator loops

 MANIFEST                            |   7 +
 lib/PublicInbox/CidxRecvIbx.pm      |  46 ++
 lib/PublicInbox/CidxXapHelperAux.pm |  44 ++
 lib/PublicInbox/CodeSearch.pm       |  54 +-
 lib/PublicInbox/CodeSearchIdx.pm    | 349 ++++++++--
 lib/PublicInbox/Config.pm           |   2 +-
 lib/PublicInbox/Isearch.pm          |   5 +
 lib/PublicInbox/Search.pm           |  92 ++-
 lib/PublicInbox/XapClient.pm        |  50 ++
 lib/PublicInbox/XapHelper.pm        | 226 +++++++
 lib/PublicInbox/XapHelperCxx.pm     |  93 +++
 lib/PublicInbox/xap_helper.h        | 947 ++++++++++++++++++++++++++++
 script/public-inbox-cindex          |   4 +-
 t/xap_helper.t                      | 175 +++++
 14 files changed, 2021 insertions(+), 73 deletions(-)
 create mode 100644 lib/PublicInbox/CidxRecvIbx.pm
 create mode 100644 lib/PublicInbox/CidxXapHelperAux.pm
 create mode 100644 lib/PublicInbox/XapClient.pm
 create mode 100644 lib/PublicInbox/XapHelper.pm
 create mode 100644 lib/PublicInbox/XapHelperCxx.pm
 create mode 100644 lib/PublicInbox/xap_helper.h
 create mode 100644 t/xap_helper.t

^ permalink raw reply	[relevance 7%]

Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-08-24  1:22  7% [PATCH 0/7] cindex: optional C++ Xapian helper Eric Wong
2023-08-24  1:22  7% ` [PATCH 3/7] cindex: add --show-roots switch Eric Wong

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).