* [PATCH 0/12] nntp: misc updates
@ 2015-09-19 2:03 7% Eric Wong
2015-09-19 2:03 6% ` [PATCH 03/12] nntp: use long response API for LISTGROUP Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2015-09-19 2:03 UTC (permalink / raw)
To: meta
Still not changing or adding new caches or modifying our data layout,
but things seem to work well for the common case.
The most important change is the new long response API which will help
us even after any future optimizations we make. We should be able to
stream millions of messages without excessive buffering and memory
usage.
Eric Wong (12):
nntp: use write_buf_size instead write_buf
nntp: introduce long response API for streaming
nntp: use long response API for LISTGROUP
nntp: implement command argument checking
nntp: XOVER does not require range
nntp: speed up XHDR for the Message-ID case
nntp: implement XROVER, speed up XHDR for some cases
nntp: implement XPATH
nntp: fix logging of long responses
nntp: fix ARTICLE/HEAD/BODY/STAT
nntp: log to FDs given by the Nntpd module
nntp: article lookups by Message-ID may cross newsgroups
lib/PublicInbox/Msgmap.pm | 24 +--
lib/PublicInbox/NNTP.pm | 419 ++++++++++++++++++++++++++++++++-----------
lib/PublicInbox/SearchMsg.pm | 1 +
public-inbox-nntpd | 2 +
4 files changed, 330 insertions(+), 116 deletions(-)
^ permalink raw reply [relevance 7%]
* [PATCH 03/12] nntp: use long response API for LISTGROUP
2015-09-19 2:03 7% [PATCH 0/12] nntp: misc updates Eric Wong
@ 2015-09-19 2:03 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2015-09-19 2:03 UTC (permalink / raw)
To: meta
LISTGROUP can be expensive for giant groups, too. Use the
long response API to improve fairness and prevent excessive
buffering.
---
lib/PublicInbox/Msgmap.pm | 24 ++++++++----------------
lib/PublicInbox/NNTP.pm | 23 +++++++++++++----------
2 files changed, 21 insertions(+), 26 deletions(-)
diff --git a/lib/PublicInbox/Msgmap.pm b/lib/PublicInbox/Msgmap.pm
index c0fc636..2f64d90 100644
--- a/lib/PublicInbox/Msgmap.pm
+++ b/lib/PublicInbox/Msgmap.pm
@@ -155,25 +155,17 @@ sub create_tables {
'val VARCHAR(255) NOT NULL)');
}
-sub each_id_batch {
- my ($self, $cb) = @_;
+sub id_batch {
+ my ($self, $num, $cb) = @_;
my $dbh = $self->{dbh};
- my $n = 0;
- my $total = 0;
- my $nr;
my $sth = $dbh->prepare('SELECT num FROM msgmap WHERE num > ? '.
'ORDER BY num ASC LIMIT 1000');
- while (1) {
- $sth->execute($n);
- my $ary = $sth->fetchall_arrayref;
- @$ary = map { $_->[0] } @$ary;
- $nr = scalar @$ary;
- last if $nr == 0;
- $total += $nr;
- $n = $ary->[-1];
- $cb->($ary);
- }
- $total;
+ $sth->execute($num);
+ my $ary = $sth->fetchall_arrayref;
+ @$ary = map { $_->[0] } @$ary;
+ my $nr = scalar @$ary;
+ $cb->($ary) if $nr;
+ $nr;
}
1;
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index f86c633..5d770bd 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -14,6 +14,7 @@ use POSIX qw(strftime);
use Time::HiRes qw(gettimeofday tv_interval ualarm);
use constant {
r501 => '501 command syntax error',
+ long_response_limit => 0xffffffff,
};
my @OVERVIEW = qw(Subject From Date Message-ID References Bytes Lines);
@@ -142,13 +143,17 @@ sub cmd_listgroup {
more($self, $res);
}
- my $ng = $self->{ng} or return '412 no newsgroup selected';
- # Ugh this can be silly expensive for big groups
- $ng->mm->each_id_batch(sub {
- my ($ary) = @_;
- more($self, join("\r\n", @$ary));
+ $self->{ng} or return '412 no newsgroup selected';
+ $self->long_response(0, long_response_limit, sub {
+ my ($i) = @_;
+ my $nr = $self->{ng}->mm->id_batch($$i, sub {
+ my ($ary) = @_;
+ more($self, join("\r\n", @$ary));
+ });
+
+ # -1 to adjust for implicit increment in long_response
+ $$i = $nr ? $$i + $nr - 1 : long_response_limit;
});
- '.'
}
sub parse_time {
@@ -245,9 +250,7 @@ sub cmd_newnews {
$ts .= '..';
my $opts = { asc => 1, limit => 1000, offset => 0 };
-
- my $end = 0xffffffff; # would like to read 4 billion messages?
- $self->long_response(0, $end, sub {
+ $self->long_response(0, long_response_limit, sub {
my ($i) = @_;
my $srch = $srch[0];
my $res = $srch->query($ts, $opts);
@@ -262,7 +265,7 @@ sub cmd_newnews {
if (@srch) { # continue onto next newsgroup
$opts->{offset} = 0;
} else { # break out of the long response.
- $$i = $end;
+ $$i = long_response_limit;
}
}
});
--
EW
^ permalink raw reply related [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2015-09-19 2:03 7% [PATCH 0/12] nntp: misc updates Eric Wong
2015-09-19 2:03 6% ` [PATCH 03/12] nntp: use long response API for LISTGROUP Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).