* [PATCH 02/11] gzipfilter: lazy allocate the deflate context
2020-03-21 2:03 5% [PATCH 00/11] www: export SQLite altid dumps Eric Wong
@ 2020-03-21 2:03 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-03-21 2:03 UTC (permalink / raw)
To: meta
zlib contexts are memory-intensive, particularly when used for
compression. Since the gzip filter may be sitting in a limiter
queue for a long period, delay the allocation we actually have
data to translate, and not a moment sooner.
---
lib/PublicInbox/GzipFilter.pm | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/lib/PublicInbox/GzipFilter.pm b/lib/PublicInbox/GzipFilter.pm
index d883130f..86409586 100644
--- a/lib/PublicInbox/GzipFilter.pm
+++ b/lib/PublicInbox/GzipFilter.pm
@@ -8,11 +8,7 @@ use bytes (); # length
use Compress::Raw::Zlib qw(Z_FINISH Z_OK);
my %OPT = (-WindowBits => 15 + 16, -AppendOutput => 1);
-sub new {
- my ($gz, $err) = Compress::Raw::Zlib::Deflate->new(%OPT);
- $err == Z_OK or die "Deflate->new failed: $err";
- bless { gz => $gz }, shift;
-}
+sub new { bless {}, shift }
# for Qspawn if using $env->{'pi-httpd.async'}
sub attach {
@@ -24,6 +20,15 @@ sub attach {
# for GetlineBody (via Qspawn) when NOT using $env->{'pi-httpd.async'}
sub translate ($$) {
my $self = $_[0];
+
+ # allocate the zlib context lazily here, instead of in ->new.
+ # Deflate contexts are memory-intensive and this object may
+ # be sitting in the Qspawn limiter queue for a while.
+ my $gz = $self->{gz} ||= do {
+ my ($g, $err) = Compress::Raw::Zlib::Deflate->new(%OPT);
+ $err == Z_OK or die "Deflate->new failed: $err";
+ $g;
+ };
my $zbuf = delete($self->{zbuf});
if (defined $_[1]) { # my $buf = $_[1];
my $err = $self->{gz}->deflate($_[1], $zbuf);
^ permalink raw reply related [relevance 7%]
* [PATCH 00/11] www: export SQLite altid dumps
@ 2020-03-21 2:03 5% Eric Wong
2020-03-21 2:03 7% ` [PATCH 02/11] gzipfilter: lazy allocate the deflate context Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-03-21 2:03 UTC (permalink / raw)
To: meta
To improve reproducibility in mirrors, altid dumps can be
exported via "POST /$INBOX_URL/$prefix.sql.gz". $prefix is
something like "gmane" (though the search prefix is "gmane:"
with a colon).
Eric Wong (11):
qspawn: reinstate filter support, add gzip filter
gzipfilter: lazy allocate the deflate context
wwwstream: introduce oneshot API to avoid ->getline
extmsg: use WwwResponse::oneshot
wwwstream: oneshot sets content-length
mbox: need_gzip uses WwwStream::oneshot
qspawn: handle ENOENT (and other errors on exec)
search: clobber -user_pfx on query parser initialization
wwwtext: show thread endpoints info w/ indexlevel=basic
altid: warn about non-word prefixes
www: add endpoint to retrieve altid dumps
MANIFEST | 4 ++
lib/PublicInbox/AltId.pm | 3 +-
lib/PublicInbox/ExtMsg.pm | 4 +-
lib/PublicInbox/GetlineBody.pm | 21 ++++----
lib/PublicInbox/GzipFilter.pm | 59 +++++++++++++++++++++
lib/PublicInbox/Mbox.pm | 16 +++---
lib/PublicInbox/Qspawn.pm | 66 ++++++++++++++----------
lib/PublicInbox/Search.pm | 4 +-
lib/PublicInbox/ViewVCS.pm | 8 +--
lib/PublicInbox/WWW.pm | 14 ++++-
lib/PublicInbox/WwwAltId.pm | 94 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/WwwStream.pm | 29 +++++++++--
lib/PublicInbox/WwwText.pm | 10 +++-
t/gzip_filter.t | 37 +++++++++++++
t/httpd-corner.psgi | 16 ++++++
t/httpd-corner.t | 48 +++++++++++++++++
t/www_altid.t | 83 ++++++++++++++++++++++++++++++
17 files changed, 452 insertions(+), 64 deletions(-)
create mode 100644 lib/PublicInbox/GzipFilter.pm
create mode 100644 lib/PublicInbox/WwwAltId.pm
create mode 100644 t/gzip_filter.t
create mode 100644 t/www_altid.t
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-03-21 2:03 5% [PATCH 00/11] www: export SQLite altid dumps Eric Wong
2020-03-21 2:03 7% ` [PATCH 02/11] gzipfilter: lazy allocate the deflate context Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).