* [PATCH 0/5] nntp: round 2 of ->ALL extindex speedups
@ 2020-11-28 5:09 7% Eric Wong
2020-11-28 5:09 6% ` [PATCH 1/5] nntp: NEWGROUPS uses long_response Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-11-28 5:09 UTC (permalink / raw)
To: meta
All of the O(n) iterations through newsgroups can either go
through the ->ALL extindex or is broken out into long_response
to not monopolize event loops.
So 50-100K newsgroups ought to be usable...
Further speeding up NEWGROUPS will require some MiscIdx
indexing additions, but that no longer hogs up an event
loop iteration.
One remaining problems is startup time with 50-100K newsgroups;
and that affects everything (especially -mda...)
Eric Wong (5):
nntp: NEWGROUPS uses long_response
nntp: speed up mid_lookup() using ->ALL extindex
nntp: art_lookup: use mid_lookup and simplify
nntp: XPATH uses ->ALL extindex, too
nntpd: remove redundant {groups} shortcut
lib/PublicInbox/NNTP.pm | 125 +++++++++++++++++++++++++--------------
lib/PublicInbox/NNTPD.pm | 8 +--
t/extsearch.t | 4 +-
t/nntp.t | 1 -
4 files changed, 87 insertions(+), 51 deletions(-)
^ permalink raw reply [relevance 7%]
* [PATCH 1/5] nntp: NEWGROUPS uses long_response
2020-11-28 5:09 7% [PATCH 0/5] nntp: round 2 of ->ALL extindex speedups Eric Wong
@ 2020-11-28 5:09 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-11-28 5:09 UTC (permalink / raw)
To: meta
We can amortize the cost of NEWGROUPS time filtering using the
long_response API. This lets us handle hundreds/thousands of
inboxes without monopolizing the event loop for this command.
Further speedup is possible using MiscSearch, but that requires
not-yet-done indexing changes to MiscIdx.
---
lib/PublicInbox/NNTP.pm | 21 +++++++++++++++------
lib/PublicInbox/NNTPD.pm | 5 +----
t/nntp.t | 1 -
3 files changed, 16 insertions(+), 11 deletions(-)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 8eec6b91..cc6534b9 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -263,6 +263,19 @@ sub group_line ($$) {
more($self, "$ng->{newsgroup} $max $min n");
}
+sub newgroups_i {
+ my ($self, $ts, $i, $groupnames) = @_;
+ my $end = $$i + 100;
+ my $groups = $self->{nntpd}->{pi_config}->{-by_newsgroup};
+ while ($$i < $end) {
+ my $ngname = $groupnames->[$$i++] // return;
+ my $ibx = $groups->{$ngname} or next; # expired on reload
+ next unless (eval { $ibx->uidvalidity } // 0) > $ts;
+ group_line($self, $ibx);
+ }
+ 1;
+}
+
sub cmd_newgroups ($$$;$$) {
my ($self, $date, $time, $gmt, $dists) = @_;
my $ts = eval { parse_time($date, $time, $gmt) };
@@ -270,12 +283,8 @@ sub cmd_newgroups ($$$;$$) {
# TODO dists
more($self, '231 list of new newsgroups follows');
- foreach my $ng (@{$self->{nntpd}->{grouplist}}) {
- my $c = eval { $ng->uidvalidity } // 0;
- next unless $c > $ts;
- group_line($self, $ng);
- }
- '.'
+ long_response($self, \&newgroups_i, $ts, \(my $i = 0),
+ $self->{nntpd}->{groupnames});
}
sub wildmat2re (;$) {
diff --git a/lib/PublicInbox/NNTPD.pm b/lib/PublicInbox/NNTPD.pm
index 4de1944b..33bc5fda 100644
--- a/lib/PublicInbox/NNTPD.pm
+++ b/lib/PublicInbox/NNTPD.pm
@@ -24,7 +24,6 @@ sub new {
groups => {},
err => \*STDERR,
out => \*STDOUT,
- grouplist => [],
pi_config => $pi_config,
servername => $name,
greet => \"201 $name ready - post via email\r\n",
@@ -60,9 +59,7 @@ sub refresh_groups {
delete $groups->{$ngname};
}
});
- my @names = sort(keys %$groups);
- $self->{grouplist} = [ map { $groups->{$_} } @names ];
- $self->{groupnames} = \@names;
+ $self->{groupnames} = [ sort(keys %$groups) ];
$self->{pi_config} = $pi_config;
# this will destroy old groups that got deleted
$self->{groups} = $groups;
diff --git a/t/nntp.t b/t/nntp.t
index 91a2aff7..ea2ef876 100644
--- a/t/nntp.t
+++ b/t/nntp.t
@@ -112,7 +112,6 @@ use PublicInbox::Config;
my $hdr = $mime->header_obj;
my $mock_self = {
nntpd => {
- grouplist => [],
servername => 'example.com',
pi_config => bless {}, 'PublicInbox::Config',
},
^ permalink raw reply related [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-11-28 5:09 7% [PATCH 0/5] nntp: round 2 of ->ALL extindex speedups Eric Wong
2020-11-28 5:09 6% ` [PATCH 1/5] nntp: NEWGROUPS uses long_response Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).