* [PATCH 0/4] www|nntp: optimize uses of Email::Simple
@ 2019-06-27 22:51 7% Eric Wong
2019-06-27 22:51 6% ` [PATCH 1/4] nntp: rework and simplify art_lookup response Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2019-06-27 22:51 UTC (permalink / raw)
To: meta
We can rely on the documented behavior of Email::Simple->new
modifying its string buffer in-place to save us the trouble
of calling ->body (or ->body_set('') to clobber the buffer).
This yields significant memory use improvements in avoiding full
message body copies in the WWW mbox code. Unfortunately, NNTP
messages with LF-only line-endings won't benefit, since the
protocol requires CRLF endings and the s/// operations get
expensive.
Memory bloat from giant emails and Email::MIME will require
more work :<
Eric Wong (4):
nntp: rework and simplify art_lookup response
mbox: use Email::Simple->new to do in-place modifications
mbox: split header and body processing
nntp: reduce syscalls for ARTICLE and BODY
lib/PublicInbox/Mbox.pm | 73 +++++++++++++++++++++--------------------
lib/PublicInbox/NNTP.pm | 59 ++++++++++++++++-----------------
2 files changed, 67 insertions(+), 65 deletions(-)
--
EW
^ permalink raw reply [relevance 7%]
* [PATCH 1/4] nntp: rework and simplify art_lookup response
2019-06-27 22:51 7% [PATCH 0/4] www|nntp: optimize uses of Email::Simple Eric Wong
@ 2019-06-27 22:51 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2019-06-27 22:51 UTC (permalink / raw)
To: meta
We don't need some of the array elements returned from
art_lookup, anymore (and haven't used them in years).
We can also shorten the lifetime of the Email::Simple object by
relying on the fact Email::Simple->new will modify it's arg if
given a SCALARREF and allow us to avoid Email::Simple::body
calls.
Unfortunately, this doesn't seem to provide any noticeable
improvement in memory usage when dealing with a 30+ MB test
message, since our previous use of ->body_set('') was saving
some memory, but forcing a LF-only body to be CRLF was making
Perl allocate extra space for s///sg.
---
lib/PublicInbox/NNTP.pm | 47 +++++++++++++++++++----------------------
1 file changed, 22 insertions(+), 25 deletions(-)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 53e18281..30d3dab6 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -510,24 +510,21 @@ find_mid:
found:
my $smsg = $ng->over->get_art($n) or return $err;
my $msg = $ng->msg_by_smsg($smsg) or return $err;
- my $s = Email::Simple->new($msg);
- if ($set_headers) {
- set_nntp_headers($self, $s->header_obj, $ng, $n, $mid);
- # must be last
- $s->body_set('') if ($set_headers == 2);
- }
- [ $n, $mid, $s, $smsg->bytes, $smsg->lines, $ng ];
+ # Email::Simple->new will modify $msg in-place as documented
+ # in its manpage, so what's left is the body and we won't need
+ # to call Email::Simple::body(), later
+ my $hdr = Email::Simple->new($msg)->header_obj;
+ set_nntp_headers($self, $hdr, $ng, $n, $mid) if $set_headers;
+ [ $n, $mid, $msg, $hdr ];
}
-sub simple_body_write ($$) {
- my ($self, $s) = @_;
- my $body = $s->body;
- $s->body_set('');
- $body =~ s/^\./../smg;
- $body =~ s/(?<!\r)\n/\r\n/sg;
- msg_more($self, $body);
- msg_more($self, "\r\n") unless $body =~ /\r\n\z/s;
+sub msg_body_write ($$) {
+ my ($self, $msg) = @_;
+ $$msg =~ s/^\./../smg;
+ $$msg =~ s/(?<!\r)\n/\r\n/sg;
+ msg_more($self, $$msg);
+ msg_more($self, "\r\n") unless $$msg =~ /\r\n\z/s;
'.'
}
@@ -537,7 +534,7 @@ sub set_art {
}
sub _header ($) {
- my $hdr = $_[0]->header_obj->as_string;
+ my $hdr = $_[0]->as_string;
utf8::encode($hdr);
$hdr =~ s/(?<!\r)\n/\r\n/sg;
@@ -554,22 +551,22 @@ sub cmd_article ($;$) {
my ($self, $art) = @_;
my $r = art_lookup($self, $art, 1);
return $r unless ref $r;
- my ($n, $mid, $s) = @$r;
+ my ($n, $mid, $msg, $hdr) = @$r;
set_art($self, $art);
more($self, "220 $n <$mid> article retrieved - head and body follow");
- msg_more($self, _header($s));
+ msg_more($self, _header($hdr));
msg_more($self, "\r\n");
- simple_body_write($self, $s);
+ msg_body_write($self, $msg);
}
sub cmd_head ($;$) {
my ($self, $art) = @_;
my $r = art_lookup($self, $art, 2);
return $r unless ref $r;
- my ($n, $mid, $s) = @$r;
+ my ($n, $mid, undef, $hdr) = @$r;
set_art($self, $art);
more($self, "221 $n <$mid> article retrieved - head follows");
- msg_more($self, _header($s));
+ msg_more($self, _header($hdr));
'.'
}
@@ -577,17 +574,17 @@ sub cmd_body ($;$) {
my ($self, $art) = @_;
my $r = art_lookup($self, $art, 0);
return $r unless ref $r;
- my ($n, $mid, $s) = @$r;
+ my ($n, $mid, $msg) = @$r;
set_art($self, $art);
more($self, "222 $n <$mid> article retrieved - body follows");
- simple_body_write($self, $s);
+ msg_body_write($self, $msg);
}
sub cmd_stat ($;$) {
my ($self, $art) = @_;
my $r = art_lookup($self, $art, 0);
return $r unless ref $r;
- my ($n, $mid, undef) = @$r;
+ my ($n, $mid) = @$r;
set_art($self, $art);
"223 $n <$mid> article retrieved - request text separately";
}
@@ -815,7 +812,7 @@ sub hdr_mid_prefix ($$$$$) {
}
sub hdr_mid_response ($$$$$$) {
- my ($self, $xhdr, $ng, $n, $mid, $v) = @_; # r: art_lookup result
+ my ($self, $xhdr, $ng, $n, $mid, $v) = @_;
my $res = '';
if ($xhdr) {
$res .= r221 . "\r\n";
--
EW
^ permalink raw reply related [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2019-06-27 22:51 7% [PATCH 0/4] www|nntp: optimize uses of Email::Simple Eric Wong
2019-06-27 22:51 6% ` [PATCH 1/4] nntp: rework and simplify art_lookup response Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).