From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 06/12] miscsearch: implement ->newsgroup_matches
Date: Fri, 27 Nov 2020 09:52:48 +0000 [thread overview]
Message-ID: <20201127095254.21624-7-e@80x24.org> (raw)
In-Reply-To: <20201127095254.21624-1-e@80x24.org>
This may be used to speed up newsgroup searches down-the-line,
but the grep perlop isn't too shabby, at the moment.
---
lib/PublicInbox/MiscSearch.pm | 40 +++++++++++++++++++++++++++++++++++
t/extsearch.t | 4 ++++
2 files changed, 44 insertions(+)
diff --git a/lib/PublicInbox/MiscSearch.pm b/lib/PublicInbox/MiscSearch.pm
index 48ef6914..f2e31443 100644
--- a/lib/PublicInbox/MiscSearch.pm
+++ b/lib/PublicInbox/MiscSearch.pm
@@ -76,6 +76,46 @@ sub mset {
retry_reopen($self, \&misc_enquire_once, $qr, $opt);
}
+sub ibx_matches_once { # retry_reopen callback
+ my ($self, $qr, $by_newsgroup) = @_;
+ # double in case no newsgroups are configured:
+ my $limit = scalar(keys %$by_newsgroup) * 2;
+ my $opt = { limit => $limit, offset => 0, relevance => -1 };
+ my $ret = {}; # newsgroup => $ibx of matches
+ while (1) {
+ my $mset = misc_enquire_once($self, $qr, $opt);
+ for my $mi ($mset->items) {
+ my $doc = $mi->get_document;
+ my $end = $doc->termlist_end;
+ my $cur = $doc->termlist_begin;
+ $cur->skip_to('Q');
+ if ($cur != $end) {
+ my $ng = $cur->get_termname; # eidx_key
+ $ng =~ s/\AQ// or warn "BUG: no `Q': $ng";
+ if (my $ibx = $by_newsgroup->{$ng}) {
+ $ret->{$ng} = $ibx;
+ }
+ } else {
+ warn <<EOF;
+W: docid=${\$mi->get_docid} has no `Q' (eidx_key) term
+EOF
+ }
+ }
+ my $nr = $mset->size;
+ return $ret if $nr < $limit;
+ $opt->{offset} += $nr;
+ }
+}
+
+# returns a newsgroup => PublicInbox::Inbox mapping
+sub newsgroup_matches {
+ my ($self, $qs, $pi_cfg) = @_;
+ my $qp = $self->{qp} //= mi_qp_new($self);
+ $qs .= ' type:inbox';
+ my $qr = $qp->parse_query($qs, $PublicInbox::Search::QP_FLAGS);
+ retry_reopen($self, \&ibx_matches_once, $qr, $pi_cfg->{-by_newsgroup});
+}
+
sub ibx_data_once {
my ($self, $ibx) = @_;
my $xdb = $self->{xdb};
diff --git a/t/extsearch.t b/t/extsearch.t
index 0045294b..85cdf74a 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -79,5 +79,9 @@ my @it = $misc->mset('')->items;
is(scalar(@it), 2, 'two inboxes');
like($it[0]->get_document->get_data, qr/v2test/, 'docdata matched v2');
like($it[1]->get_document->get_data, qr/v1test/, 'docdata matched v1');
+my $pi_cfg = PublicInbox::Config->new;
+$pi_cfg->fill_all;
+my $ret = $misc->newsgroup_matches('', $pi_cfg);
+is_deeply($pi_cfg->{-by_newsgroup}, $ret, '->newsgroup_matches');
done_testing;
next prev parent reply other threads:[~2020-11-27 9:52 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-27 9:52 [PATCH 00/12] some NNTP-related fixes + speedups Eric Wong
2020-11-27 9:52 ` [PATCH 01/12] nntp: use Inbox->uidvalidity instead of ->mm->created_at Eric Wong
2020-11-27 9:52 ` [PATCH 02/12] nntpd: share {groups} hash with {-by_newsgroup} in Config Eric Wong
2020-11-27 9:52 ` [PATCH 03/12] mm: min/max: return 0 instead of undef Eric Wong
2020-11-27 9:52 ` [PATCH 04/12] nntp: use grep operation for wildmat matching Eric Wong
2020-11-27 9:52 ` [PATCH 05/12] nntp: NEWNEWS: speed up filtering Eric Wong
2020-11-27 9:52 ` Eric Wong [this message]
2020-11-27 9:52 ` [PATCH 07/12] nntp: LIST ACTIVE.TIMES use angle brackets around address Eric Wong
2020-11-27 9:52 ` [PATCH 08/12] nntp: move LIST iterators to long_response Eric Wong
2020-11-27 9:52 ` [PATCH 09/12] t/extsearch: show a more realistic case Eric Wong
2020-11-27 9:52 ` [PATCH 10/12] nntp: some minor golfing Eric Wong
2020-11-27 9:52 ` [PATCH 11/12] nntp: xref: simplify sub signature Eric Wong
2020-11-27 9:52 ` [PATCH 12/12] nntp: xref: use ->ALL extindex if available Eric Wong
2020-11-30 19:42 ` xref3 + NNTP problems Eric Wong
2020-11-30 23:37 ` [PATCH] nntp: make ->ALL Xref generation more fuzzy Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201127095254.21624-7-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).