* [PATCH 1/2] wwwlisting: use first successfully loaded JSON module
@ 2020-03-21 1:10 7% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2020-03-21 1:10 UTC (permalink / raw)
To: meta
And not the last...
I only noticed this since JSON::PP::Boolean was spewing
redefinition warnings via overload.pm
Fixes: 8fb8fc52420ef669 ("wwwlisting: avoid lazy loading JSON module")
---
lib/PublicInbox/WwwListing.pm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/PublicInbox/WwwListing.pm b/lib/PublicInbox/WwwListing.pm
index 33cb0ace..42a0c0d8 100644
--- a/lib/PublicInbox/WwwListing.pm
+++ b/lib/PublicInbox/WwwListing.pm
@@ -20,7 +20,7 @@ if (eval { require IO::Compress::Gzip }) {
for my $mod (qw(JSON::MaybeXS JSON JSON::PP)) {
eval "require $mod" or next;
# ->ascii encodes non-ASCII to "\uXXXX"
- $json = $mod->new->ascii(1);
+ $json = $mod->new->ascii(1) and last;
}
}
^ permalink raw reply related [relevance 7%]
* [PATCH 0/6] daemon: reduce fragmentation via preload
@ 2020-03-19 8:32 6% Eric Wong
2020-03-19 8:32 6% ` [PATCH 3/6] wwwlisting: avoid lazy loading JSON module Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2020-03-19 8:32 UTC (permalink / raw)
To: meta
For long-lived daemons, perform immortal allocations as early as
possible to reduce the likelyhood of heap fragmentation due to
mixed-lifetime allocations happening once the process is fully
loaded and serving requests, since per-request allocations
should all be short-lived.
On a side note, I'm wondering if WWW should just preload by
default. I'm not sure if anybody uses public-inbox.cgi (or
should be using it :P). It's not like we don't ship
public-inbox-httpd; and any PSGI implementation could be used
for smaller inboxes (or powerful-enough hardware).
Eric Wong (6):
www: update ->preload for newer modules
wwwlisting: favor "use" over require
wwwlisting: avoid lazy loading JSON module
www: avoid `state' usage to perform allocations up-front
daemon: do more immortal allocations up front
viewdiff: favor `qr' to precompile regexps
lib/PublicInbox/NNTPD.pm | 4 +++
lib/PublicInbox/SolverGit.pm | 13 +++++----
lib/PublicInbox/ViewDiff.pm | 53 +++++++++++++++++++----------------
lib/PublicInbox/WWW.pm | 29 ++++++++++++++-----
lib/PublicInbox/WwwListing.pm | 30 +++++++++-----------
t/www_listing.t | 4 +--
6 files changed, 78 insertions(+), 55 deletions(-)
^ permalink raw reply [relevance 6%]
* [PATCH 3/6] wwwlisting: avoid lazy loading JSON module
2020-03-19 8:32 6% [PATCH 0/6] daemon: reduce fragmentation via preload Eric Wong
@ 2020-03-19 8:32 6% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2020-03-19 8:32 UTC (permalink / raw)
To: meta
We already lazy-load WwwListing for the CGI script, and
hiding another layer of lazy-loading makes things difficult
to do WWW->preload.
We want long-lived processes to do all long-lived allocations up
front to avoid fragmentation in the allocator, but we'll still
support short-lived processes by lazy-loading individual modules
in the PublicInbox::* namespace.
Mixing up allocation lifetimes (e.g. doing immortal allocations
while a large amount of space is taken by short-lived objects)
will cause fragmentation in any allocator which favors large
contiguous regions for performance reasons. This includes any
malloc implementation which relies on sbrk() for the primary
heap, including glibc malloc.
---
lib/PublicInbox/WwwListing.pm | 26 ++++++++++++--------------
t/www_listing.t | 4 ++--
2 files changed, 14 insertions(+), 16 deletions(-)
diff --git a/lib/PublicInbox/WwwListing.pm b/lib/PublicInbox/WwwListing.pm
index a8aecaf7..33cb0ace 100644
--- a/lib/PublicInbox/WwwListing.pm
+++ b/lib/PublicInbox/WwwListing.pm
@@ -10,11 +10,19 @@ use PublicInbox::Hval qw(ascii_html prurl);
use PublicInbox::Linkify;
use PublicInbox::View;
use PublicInbox::Inbox;
-use bytes ();
+use bytes (); # bytes::length
use HTTP::Date qw(time2str);
use Digest::SHA ();
use File::Spec ();
*try_cat = \&PublicInbox::Inbox::try_cat;
+our $json;
+if (eval { require IO::Compress::Gzip }) {
+ for my $mod (qw(JSON::MaybeXS JSON JSON::PP)) {
+ eval "require $mod" or next;
+ # ->ascii encodes non-ASCII to "\uXXXX"
+ $json = $mod->new->ascii(1);
+ }
+}
sub list_all_i {
my ($ibx, $arg) = @_;
@@ -121,16 +129,6 @@ sub html ($$) {
[ $code, $h, [ $out ] ];
}
-my $json;
-sub _json () {
- for my $mod (qw(JSON::MaybeXS JSON JSON::PP)) {
- eval "require $mod" or next;
- # ->ascii encodes non-ASCII to "\uXXXX"
- return $mod->new->ascii(1);
- }
- die;
-}
-
sub fingerprint ($) {
my ($git) = @_;
# TODO: convert to qspawn for fairness when there's
@@ -201,7 +199,8 @@ sub manifest_add ($$;$$) {
# manifest.js.gz
sub js ($$) {
my ($env, $list) = @_;
- eval { require IO::Compress::Gzip } or return [ 404, [], [] ];
+ # $json won't be defined if IO::Compress::Gzip is missing
+ $json or return [ 404, [], [] ];
my $manifest = { -abs2urlpath => {}, -mtime => 0 };
for my $ibx (@$list) {
@@ -221,8 +220,7 @@ sub js ($$) {
$repo->{reference} = $abs2urlpath->{$abs};
}
my $out;
- IO::Compress::Gzip::gzip(\(($json ||= _json())->encode($manifest)) =>
- \$out);
+ IO::Compress::Gzip::gzip(\($json->encode($manifest)) => \$out);
$manifest = undef;
[ 200, [ qw(Content-Type application/gzip),
'Last-Modified', time2str($mtime),
diff --git a/t/www_listing.t b/t/www_listing.t
index 5168e16a..39c19577 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -9,8 +9,8 @@ use PublicInbox::TestCommon;
require_mods(qw(URI::Escape Plack::Builder Digest::SHA
IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
require PublicInbox::WwwListing;
-my $json = eval { PublicInbox::WwwListing::_json() };
-plan skip_all => "JSON module missing: $@" if $@;
+my $json = $PublicInbox::WwwListing::json or
+ plan skip_all => "JSON module missing";
use_ok 'PublicInbox::Git';
^ permalink raw reply related [relevance 6%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-03-19 8:32 6% [PATCH 0/6] daemon: reduce fragmentation via preload Eric Wong
2020-03-19 8:32 6% ` [PATCH 3/6] wwwlisting: avoid lazy loading JSON module Eric Wong
2020-03-21 1:10 [PATCH 0/2] wwwlisting: fixup warnings :x Eric Wong
2020-03-21 1:10 7% ` [PATCH 1/2] wwwlisting: use first successfully loaded JSON module Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).