* [PATCH 06/12] nntp: speed up XHDR for the Message-ID case
2015-09-19 2:03 6% [PATCH 0/12] nntp: misc updates Eric Wong
@ 2015-09-19 2:03 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2015-09-19 2:03 UTC (permalink / raw)
To: meta
We can use our msgmap database to implement "XHDR Message-ID [range]"
commands quickly. This helps immensely with slrnpull, which prefers
XHDR to LISTGROUP for some reason...
---
lib/PublicInbox/NNTP.pm | 36 ++++++++++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 8275ef0..47b03e0 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -505,9 +505,45 @@ sub long_response ($$$$) {
undef;
}
+sub xhdr_message_id ($$) { # optimize XHDR Message-ID [range] for slrnpull.
+ my ($self, $range) = @_;
+
+ my $mm = $self->{ng}->mm;
+ if (defined $range && $range =~ /\A<(.+)>\z/) { # Message-ID
+ my $n = $mm->num_for($range);
+ more($self, '221 Header follows');
+ more($self, "<$range> <$range>") if defined $n;
+ '.';
+ } else { # numeric range
+ $range = $self->{article} unless defined $range;
+ my $r = get_range($self, $range);
+ return $r unless ref $r;
+ my ($beg, $end) = @$r;
+ more($self, '221 Header follows');
+ $self->long_response($beg, $end, sub {
+ my ($i) = @_;
+ my $mid = $mm->mid_for($$i);
+ more($self, "$$i <$mid>") if defined $mid;
+ });
+ }
+}
+
sub cmd_xhdr ($$;$) {
my ($self, $header, $range) = @_;
defined $self->{ng} or return '412 no news group currently selected';
+ my $sub = $header;
+ $sub =~ tr/A-Z-/a-z_/;
+ $sub = eval {
+ no strict 'refs';
+ $sub = *{'xhdr_'.$sub}{CODE};
+ };
+ return xhdr_slow($self, $header, $range) unless defined $sub;
+ $sub->($self, $range);
+}
+
+sub xhdr_slow ($$$) {
+ my ($self, $header, $range) = @_;
+
if (defined $range && $range =~ /\A<(.+)>\z/) { # Message-ID
my $r = $self->art_lookup($range, 2);
return $r unless ref $r;
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 0/12] nntp: misc updates
@ 2015-09-19 2:03 6% Eric Wong
2015-09-19 2:03 7% ` [PATCH 06/12] nntp: speed up XHDR for the Message-ID case Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2015-09-19 2:03 UTC (permalink / raw)
To: meta
Still not changing or adding new caches or modifying our data layout,
but things seem to work well for the common case.
The most important change is the new long response API which will help
us even after any future optimizations we make. We should be able to
stream millions of messages without excessive buffering and memory
usage.
Eric Wong (12):
nntp: use write_buf_size instead write_buf
nntp: introduce long response API for streaming
nntp: use long response API for LISTGROUP
nntp: implement command argument checking
nntp: XOVER does not require range
nntp: speed up XHDR for the Message-ID case
nntp: implement XROVER, speed up XHDR for some cases
nntp: implement XPATH
nntp: fix logging of long responses
nntp: fix ARTICLE/HEAD/BODY/STAT
nntp: log to FDs given by the Nntpd module
nntp: article lookups by Message-ID may cross newsgroups
lib/PublicInbox/Msgmap.pm | 24 +--
lib/PublicInbox/NNTP.pm | 419 ++++++++++++++++++++++++++++++++-----------
lib/PublicInbox/SearchMsg.pm | 1 +
public-inbox-nntpd | 2 +
4 files changed, 330 insertions(+), 116 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2015-09-19 2:03 6% [PATCH 0/12] nntp: misc updates Eric Wong
2015-09-19 2:03 7% ` [PATCH 06/12] nntp: speed up XHDR for the Message-ID case Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).