* [PATCH 34/43] view: eml_entry: reduce parameters
2020-07-05 23:27 7% [PATCH 00/43] www: async git cat-file w/ -httpd Eric Wong
@ 2020-07-05 23:27 6% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-07-05 23:27 UTC (permalink / raw)
To: meta
We can save stack space and simplify subroutine calls, here.
---
Documentation/mknews.perl | 4 ++--
lib/PublicInbox/Feed.pm | 2 +-
lib/PublicInbox/SearchView.pm | 2 +-
lib/PublicInbox/View.pm | 17 +++++++++--------
4 files changed, 13 insertions(+), 12 deletions(-)
diff --git a/Documentation/mknews.perl b/Documentation/mknews.perl
index 51d54b716..4a5d0e563 100755
--- a/Documentation/mknews.perl
+++ b/Documentation/mknews.perl
@@ -103,9 +103,9 @@ sub mime2txt {
sub mime2html {
my ($out, $eml, $ctx) = @_;
- my $smsg = bless {}, 'PublicInbox::Smsg';
+ my $smsg = $ctx->{smsg} = bless {}, 'PublicInbox::Smsg';
$smsg->populate($eml);
- print $out PublicInbox::View::eml_entry($ctx, $smsg, $eml, 1) or die;
+ print $out PublicInbox::View::eml_entry($ctx, $eml, 1) or die;
}
sub html_start {
diff --git a/lib/PublicInbox/Feed.pm b/lib/PublicInbox/Feed.pm
index 279106d28..476d946f5 100644
--- a/lib/PublicInbox/Feed.pm
+++ b/lib/PublicInbox/Feed.pm
@@ -51,7 +51,7 @@ sub new_html_i {
my ($ctx, $eml) = @_;
$ctx->zmore($ctx->html_top) if exists $ctx->{-html_tip};
- $eml and return PublicInbox::View::eml_entry($ctx, $ctx->{smsg}, $eml,
+ $eml and return PublicInbox::View::eml_entry($ctx, $eml,
scalar @{$ctx->{msgs}});
my $smsg = shift @{$ctx->{msgs}} or
$ctx->zmore(PublicInbox::View::pagination_footer(
diff --git a/lib/PublicInbox/SearchView.pm b/lib/PublicInbox/SearchView.pm
index 921992a5d..623b16fb2 100644
--- a/lib/PublicInbox/SearchView.pm
+++ b/lib/PublicInbox/SearchView.pm
@@ -294,7 +294,7 @@ sub mset_thread {
sub mset_thread_i {
my ($ctx, $eml) = @_;
$ctx->zmore($ctx->html_top) if exists $ctx->{-html_tip};
- $eml and return PublicInbox::View::eml_entry($ctx, $ctx->{smsg}, $eml,
+ $eml and return PublicInbox::View::eml_entry($ctx, $eml,
scalar @{$ctx->{msgs}});
my $smsg = shift @{$ctx->{msgs}} or
$ctx->zmore(${delete($ctx->{skel})});
diff --git a/lib/PublicInbox/View.pm b/lib/PublicInbox/View.pm
index 16a0fcdfb..656953928 100644
--- a/lib/PublicInbox/View.pm
+++ b/lib/PublicInbox/View.pm
@@ -181,7 +181,8 @@ sub fmt_ts ($) { strftime('%Y-%m-%d %k:%M', gmtime($_[0])) }
# Displays the text of of the message for /$INBOX/$MSGID/[Tt]/ endpoint
# this is already inside a <pre>
sub eml_entry {
- my ($ctx, $smsg, $eml, $more) = @_;
+ my ($ctx, $eml, $more) = @_;
+ my $smsg = delete $ctx->{smsg};
my $subj = delete $smsg->{subject};
my $mid_raw = $smsg->{mid};
my $id = id_compress($mid_raw, 1);
@@ -370,9 +371,9 @@ sub pre_thread { # walk_thread callback
}
sub thread_eml_entry {
- my ($ctx, $level, $smsg, $eml) = @_;
- my ($beg, $end) = thread_adj_level($ctx, $level);
- $beg . '<pre>' . eml_entry($ctx, $smsg, $eml, 0) . '</pre>' . $end;
+ my ($ctx, $eml) = @_;
+ my ($beg, $end) = thread_adj_level($ctx, $ctx->{level});
+ $beg . '<pre>' . eml_entry($ctx, $eml, 0) . '</pre>' . $end;
}
sub next_in_queue ($$) {
@@ -390,12 +391,12 @@ sub stream_thread_i { # PublicInbox::WwwStream::getline callback
my ($ctx, $eml) = @_;
if ($eml) {
- my ($level, $smsg) = delete @$ctx{qw(level smsg)};
if ($ctx->{nr} == 1) {
- $ctx->{-title_html} = ascii_html($smsg->{subject});
+ $ctx->{-title_html} =
+ ascii_html($ctx->{smsg}->{subject});
$ctx->zmore($ctx->html_top);
}
- return thread_eml_entry($ctx, $level, $smsg, $eml);
+ goto &thread_eml_entry; # tail recursion
}
return unless exists($ctx->{skel});
my $ghost_ok = $ctx->{nr}++;
@@ -470,7 +471,7 @@ sub thread_html_i { # PublicInbox::WwwStream::getline callback
$ctx->{-title_html} = ascii_html($smsg->{subject});
$ctx->zmore($ctx->html_top);
}
- return eml_entry($ctx, $smsg, $eml, scalar @{$ctx->{msgs}});
+ return eml_entry($ctx, $eml, scalar @{$ctx->{msgs}});
} else {
while (my $smsg = shift @{$ctx->{msgs}}) {
return $smsg if exists($smsg->{blob});
^ permalink raw reply related [relevance 6%]
* [PATCH 00/43] www: async git cat-file w/ -httpd
@ 2020-07-05 23:27 7% Eric Wong
2020-07-05 23:27 6% ` [PATCH 34/43] view: eml_entry: reduce parameters Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-07-05 23:27 UTC (permalink / raw)
To: meta
This allows -httpd to make better use of time it spends waiting
on git-cat-file to respond. It allows us to deal with
high-latency HDD storage without a client monopolizing the event
loop. Even on a mid-range consumer-grade SSD, this seems to
give a 10+% speed improvement for HTTP responses requiring many
blobs, including all /T/, /t/, and /t.mbox.gz endpoints.
This only benefits indexed inboxes (both v1 and v2); I'm not
sure if anybody still uses unindexed v1 inboxes nowadays.
A new xt/httpd-async-stream.t maintainer test ensures checksums
for responses before and after this series match exactly as
before.
This builds off a branch I started several months ago (but never
published here) to integrate gzip responses into our codebase
and remove our optional dependency on Plack::Middleware::Deflater.
We already gzip a bunch of things independent of
Plack::Middleware::Deflater: manifest.js.gz, altid SQLite3 dumps
and all the *.mbox.gz endpoints; so being able to use gzip on
all of our responses without an extra dependency seemed logical.
Being able to consistently use our GzipFilter API to perform
buffering via ->zmore made it significantly easier to reason
about small response chunks for ghost messages interspersed with
large ones when streaming /$INBOX/$MSGID/t/ endpoints.
I'm not yet maximizing use of ->zmore for all buffering of HTTP
responses, yet; measurements need to happen, first. That may
happen in the 1.7 time frame. In particular, we would need to
ensure the Perl method dispatch and DSO overhead to Zlib.so and
libz.so of making many ->zmore calls doesn't cause performance
regressions compared to the current `.=' use and calling
->zmore/->translate fewer times.
Eric Wong (43):
gzipfilter: minor cleanups
wwwstream: oneshot: perform gzip without middleware
www*stream: gzip ->getline responses
wwwtext: gzip text/plain responses, as well
wwwtext: switch to html_oneshot
www: need: use WwwStream::html_oneshot
wwwlisting: use GzipFilter for HTML
gzipfilter: replace Compress::Raw::Deflate usages
{gzip,noop}filter: ->zmore returns undef, always
mbox: remove html_oneshot import
wwwstatic: support gzipped response directory listings
qspawn: learn to gzip streaming responses
stop auto-loading Plack::Middleware::Deflater
mboxgz: do asynchronous git blob retrievals
mboxgz: reduce object hash depth
mbox: async blob retrieval for "single message" raw mboxrd
wwwatomstream: simplify feed_update callers
wwwatomstream: use PublicInbox::Inbox->modified for feed_updated
wwwatomstream: reuse $ctx as $self
xt/httpd-async-stream: allow more options
wwwatomstream: support asynchronous blob retrievals
wwwstream: reduce object graph depth
wwwstream: reduce blob retrieval paths for ->getline
www: start making gzipfilter the parent response class
remove unused/redundant zlib-related imports
wwwstream: use parent.pm and no warnings
wwwstream: subclass off GzipFilter
view: wire up /$INBOX/$MESSAGE_ID/ permalink to async
view: /$INBOX/$MSGID/t/ reads blobs asynchronously
view: update /$INBOX/$MSGID/T/ to be async
feed: generate_i: eliminate pointless loop
feed: /$INBOX/new.html retrieves blobs asynchronously
ssearchview: /$INBOX/?q=$QUERY&x=t uses async blobs
view: eml_entry: reduce parameters
view: /$INBOX/$MSGID/t/: avoid extra hash lookup in eml case
wwwstream: eliminate ::response, use html_oneshot
www: update internal docs
view: simplify eml_entry callers further
wwwtext: simplify gzf_maybe use
wwwattach: support async blob retrievals
gzipfilter: drop HTTP connection on bugs or data corruption
daemon: warn on missing blobs
gzipfilter: check http->{forward} for client disconnects
Documentation/mknews.perl | 20 +--
Documentation/public-inbox-httpd.pod | 1 -
Documentation/technical/ds.txt | 4 +-
INSTALL | 5 -
MANIFEST | 2 +
ci/deps.perl | 1 -
examples/cgit.psgi | 8 -
examples/newswww.psgi | 8 -
examples/public-inbox.psgi | 9 --
examples/unsubscribe.psgi | 1 -
lib/PublicInbox/CompressNoop.pm | 22 +++
lib/PublicInbox/Feed.pm | 22 ++-
lib/PublicInbox/GetlineBody.pm | 4 +-
lib/PublicInbox/GzipFilter.pm | 168 +++++++++++++++++---
lib/PublicInbox/HTTP.pm | 7 +
lib/PublicInbox/HTTPD.pm | 5 +-
lib/PublicInbox/IMAP.pm | 1 +
lib/PublicInbox/Mbox.pm | 137 +++++++++--------
lib/PublicInbox/MboxGz.pm | 81 ++++------
lib/PublicInbox/NNTP.pm | 1 +
lib/PublicInbox/Qspawn.pm | 6 +-
lib/PublicInbox/SearchView.pm | 40 ++---
lib/PublicInbox/View.pm | 219 ++++++++++++++-------------
lib/PublicInbox/WWW.pm | 9 +-
lib/PublicInbox/WwwAtomStream.pm | 66 ++++----
lib/PublicInbox/WwwAttach.pm | 63 ++++++--
lib/PublicInbox/WwwListing.pm | 24 +--
lib/PublicInbox/WwwStatic.pm | 14 +-
lib/PublicInbox/WwwStream.pm | 110 ++++++++------
lib/PublicInbox/WwwText.pm | 26 ++--
script/public-inbox-httpd | 9 --
script/public-inbox.cgi | 7 -
t/httpd-corner.psgi | 7 +
t/httpd-corner.t | 9 +-
t/plack.t | 4 +
t/psgi_attach.t | 162 +++++++++++---------
t/psgi_text.t | 33 +++-
t/psgi_v2.t | 80 ++++++++--
t/www_listing.t | 8 +-
t/www_static.t | 11 +-
xt/httpd-async-stream.t | 104 +++++++++++++
41 files changed, 964 insertions(+), 554 deletions(-)
create mode 100644 lib/PublicInbox/CompressNoop.pm
create mode 100644 xt/httpd-async-stream.t
^ permalink raw reply [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-07-05 23:27 7% [PATCH 00/43] www: async git cat-file w/ -httpd Eric Wong
2020-07-05 23:27 6% ` [PATCH 34/43] view: eml_entry: reduce parameters Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).