* [PATCH 0/12] nntp: misc updates
@ 2015-09-19 2:03 7% Eric Wong
2015-09-19 2:03 6% ` [PATCH 07/12] nntp: implement XROVER, speed up XHDR for some cases Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2015-09-19 2:03 UTC (permalink / raw)
To: meta
Still not changing or adding new caches or modifying our data layout,
but things seem to work well for the common case.
The most important change is the new long response API which will help
us even after any future optimizations we make. We should be able to
stream millions of messages without excessive buffering and memory
usage.
Eric Wong (12):
nntp: use write_buf_size instead write_buf
nntp: introduce long response API for streaming
nntp: use long response API for LISTGROUP
nntp: implement command argument checking
nntp: XOVER does not require range
nntp: speed up XHDR for the Message-ID case
nntp: implement XROVER, speed up XHDR for some cases
nntp: implement XPATH
nntp: fix logging of long responses
nntp: fix ARTICLE/HEAD/BODY/STAT
nntp: log to FDs given by the Nntpd module
nntp: article lookups by Message-ID may cross newsgroups
lib/PublicInbox/Msgmap.pm | 24 +--
lib/PublicInbox/NNTP.pm | 419 ++++++++++++++++++++++++++++++++-----------
lib/PublicInbox/SearchMsg.pm | 1 +
public-inbox-nntpd | 2 +
4 files changed, 330 insertions(+), 116 deletions(-)
^ permalink raw reply [relevance 7%]
* [PATCH 07/12] nntp: implement XROVER, speed up XHDR for some cases
2015-09-19 2:03 7% [PATCH 0/12] nntp: misc updates Eric Wong
@ 2015-09-19 2:03 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2015-09-19 2:03 UTC (permalink / raw)
To: meta
Using Xapian allows us to implement XROVER without forking
new processes.
---
lib/PublicInbox/NNTP.pm | 84 +++++++++++++++++++++++++++++++++++++++-----
lib/PublicInbox/SearchMsg.pm | 1 +
2 files changed, 76 insertions(+), 9 deletions(-)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 47b03e0..939fc3a 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -528,17 +528,61 @@ sub xhdr_message_id ($$) { # optimize XHDR Message-ID [range] for slrnpull.
}
}
+sub header_obj_for {
+ my ($srch, $mid) = @_;
+ eval {
+ my $smsg = $srch->lookup_message($mid);
+ $smsg = PublicInbox::SearchMsg->load_doc($smsg->{doc});
+ $smsg->mini_mime->header_obj;
+ };
+};
+
+sub xhdr_searchmsg ($$$) {
+ my ($self, $sub, $range) = @_;
+ my $srch = $self->{ng}->search;
+ my $emit = ($sub eq 'date') ? sub {
+ my ($pfx, $m) = @_;
+ my @t = gmtime($m->header('X-PI-TS'));
+ more($self, "$pfx ". strftime('%a, %d %b %Y %T %z', @t));
+ } : sub {
+ my ($pfx, $m) = @_;
+ my $h = $m->header($sub);
+ more($self, "$pfx $h") if defined $h;
+ };
+
+ if (defined $range && $range =~ /\A<(.+)>\z/) { # Message-ID
+ more($self, '221 Header follows');
+ my $m = header_obj_for($srch, $1);
+ $emit->($range, $m) if defined $m;
+ '.';
+ } else { # numeric range
+ $range = $self->{article} unless defined $range;
+ my $mm = $self->{ng}->mm;
+ my $r = get_range($self, $range);
+ return $r unless ref $r;
+ my ($beg, $end) = @$r;
+ more($self, '221 Header follows');
+ $self->long_response($beg, $end, sub {
+ my ($i) = @_;
+ my $mid = $mm->mid_for($$i) or return;
+ my $m = header_obj_for($srch, $mid) or return;
+ $emit->($$i, $m);
+ });
+ }
+}
+
sub cmd_xhdr ($$;$) {
my ($self, $header, $range) = @_;
- defined $self->{ng} or return '412 no news group currently selected';
- my $sub = $header;
- $sub =~ tr/A-Z-/a-z_/;
- $sub = eval {
- no strict 'refs';
- $sub = *{'xhdr_'.$sub}{CODE};
- };
- return xhdr_slow($self, $header, $range) unless defined $sub;
- $sub->($self, $range);
+ my $ng = $self->{ng};
+ defined $ng or return '412 no news group currently selected';
+ my $sub = lc $header;
+ if ($sub eq 'message-id') {
+ xhdr_message_id($self, $range);
+ } elsif ($sub =~ /\A(subject|references|date)\z/ && $ng->search) {
+ xhdr_searchmsg($self, $sub, $range);
+ } else {
+ xhdr_slow($self, $header, $range);
+ }
}
sub xhdr_slow ($$$) {
@@ -568,6 +612,28 @@ sub xhdr_slow ($$$) {
}
}
+sub cmd_xrover ($;$) {
+ my ($self, $range) = @_;
+ my $ng = $self->{ng} or return '412 no newsgroup selected';
+ (defined $range && $range =~ /[<>]/) and
+ return '420 No article(s) selected'; # no message IDs
+
+ $range = $self->{article} unless defined $range;
+ my $r = get_range($self, $range);
+ return $r unless ref $r;
+ my ($beg, $end) = @$r;
+ my $mm = $ng->mm;
+ my $srch = $ng->search;
+ more($self, '224 Overview information follows');
+ $self->long_response($beg, $end, sub {
+ my ($i) = @_;
+ my $mid = $mm->mid_for($$i) or return;
+ my $m = header_obj_for($srch, $mid) or return;
+ my $h = $m->header('references');
+ more($self, "$$i $h") if defined $h;
+ });
+}
+
sub cmd_xover ($;$) {
my ($self, $range) = @_;
$range = $self->{article} unless defined $range;
diff --git a/lib/PublicInbox/SearchMsg.pm b/lib/PublicInbox/SearchMsg.pm
index e3a0460..8c55c92 100644
--- a/lib/PublicInbox/SearchMsg.pm
+++ b/lib/PublicInbox/SearchMsg.pm
@@ -141,6 +141,7 @@ sub mini_mime {
'X-PI-TS' => $self->ts,
);
if (my $refs = $self->{references_sorted}) {
+ $refs =~ s/></> </g;
push @h, References => $refs;
}
my $mime = Email::MIME->create(header_str => \@hs, header => \@h);
--
EW
^ permalink raw reply related [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2015-09-19 2:03 7% [PATCH 0/12] nntp: misc updates Eric Wong
2015-09-19 2:03 6% ` [PATCH 07/12] nntp: implement XROVER, speed up XHDR for some cases Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).