user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 3/6] wwwlisting: avoid lazy loading JSON module
Date: Thu, 19 Mar 2020 03:32:53 -0500
Message-ID: <20200319083256.15593-4-e@yhbt.net> (raw)
In-Reply-To: <20200319083256.15593-1-e@yhbt.net>

We already lazy-load WwwListing for the CGI script, and
hiding another layer of lazy-loading makes things difficult
to do WWW->preload.

We want long-lived processes to do all long-lived allocations up
front to avoid fragmentation in the allocator, but we'll still
support short-lived processes by lazy-loading individual modules
in the PublicInbox::* namespace.

Mixing up allocation lifetimes (e.g. doing immortal allocations
while a large amount of space is taken by short-lived objects)
will cause fragmentation in any allocator which favors large
contiguous regions for performance reasons.  This includes any
malloc implementation which relies on sbrk() for the primary
heap, including glibc malloc.
---
 lib/PublicInbox/WwwListing.pm | 26 ++++++++++++--------------
 t/www_listing.t               |  4 ++--
 2 files changed, 14 insertions(+), 16 deletions(-)

diff --git a/lib/PublicInbox/WwwListing.pm b/lib/PublicInbox/WwwListing.pm
index a8aecaf7..33cb0ace 100644
--- a/lib/PublicInbox/WwwListing.pm
+++ b/lib/PublicInbox/WwwListing.pm
@@ -10,11 +10,19 @@ use PublicInbox::Hval qw(ascii_html prurl);
 use PublicInbox::Linkify;
 use PublicInbox::View;
 use PublicInbox::Inbox;
-use bytes ();
+use bytes (); # bytes::length
 use HTTP::Date qw(time2str);
 use Digest::SHA ();
 use File::Spec ();
 *try_cat = \&PublicInbox::Inbox::try_cat;
+our $json;
+if (eval { require IO::Compress::Gzip }) {
+	for my $mod (qw(JSON::MaybeXS JSON JSON::PP)) {
+		eval "require $mod" or next;
+		# ->ascii encodes non-ASCII to "\uXXXX"
+		$json = $mod->new->ascii(1);
+	}
+}
 
 sub list_all_i {
 	my ($ibx, $arg) = @_;
@@ -121,16 +129,6 @@ sub html ($$) {
 	[ $code, $h, [ $out ] ];
 }
 
-my $json;
-sub _json () {
-	for my $mod (qw(JSON::MaybeXS JSON JSON::PP)) {
-		eval "require $mod" or next;
-		# ->ascii encodes non-ASCII to "\uXXXX"
-		return $mod->new->ascii(1);
-	}
-	die;
-}
-
 sub fingerprint ($) {
 	my ($git) = @_;
 	# TODO: convert to qspawn for fairness when there's
@@ -201,7 +199,8 @@ sub manifest_add ($$;$$) {
 # manifest.js.gz
 sub js ($$) {
 	my ($env, $list) = @_;
-	eval { require IO::Compress::Gzip } or return [ 404, [], [] ];
+	# $json won't be defined if IO::Compress::Gzip is missing
+	$json or return [ 404, [], [] ];
 
 	my $manifest = { -abs2urlpath => {}, -mtime => 0 };
 	for my $ibx (@$list) {
@@ -221,8 +220,7 @@ sub js ($$) {
 		$repo->{reference} = $abs2urlpath->{$abs};
 	}
 	my $out;
-	IO::Compress::Gzip::gzip(\(($json ||= _json())->encode($manifest)) =>
-				 \$out);
+	IO::Compress::Gzip::gzip(\($json->encode($manifest)) => \$out);
 	$manifest = undef;
 	[ 200, [ qw(Content-Type application/gzip),
 		 'Last-Modified', time2str($mtime),
diff --git a/t/www_listing.t b/t/www_listing.t
index 5168e16a..39c19577 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -9,8 +9,8 @@ use PublicInbox::TestCommon;
 require_mods(qw(URI::Escape Plack::Builder Digest::SHA
 		IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
 require PublicInbox::WwwListing;
-my $json = eval { PublicInbox::WwwListing::_json() };
-plan skip_all => "JSON module missing: $@" if $@;
+my $json = $PublicInbox::WwwListing::json or
+	plan skip_all => "JSON module missing";
 
 use_ok 'PublicInbox::Git';
 

  parent reply index

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-19  8:32 [PATCH 0/6] daemon: reduce fragmentation via preload Eric Wong
2020-03-19  8:32 ` [PATCH 1/6] www: update ->preload for newer modules Eric Wong
2020-03-19  8:32 ` [PATCH 2/6] wwwlisting: favor "use" over require Eric Wong
2020-03-19  8:32 ` Eric Wong [this message]
2020-03-21  1:10   ` [PATCH 0/2] wwwlisting: fixup warnings :x Eric Wong
2020-03-21  1:10     ` [PATCH 1/2] wwwlisting: use first successfully loaded JSON module Eric Wong
2020-03-21  1:10     ` [PATCH 2/2] t/www_listing: avoid 'once' warnings Eric Wong
2020-03-21  5:24       ` [PATCH v2] " Eric Wong
2020-03-19  8:32 ` [PATCH 4/6] www: avoid `state' usage to perform allocations up-front Eric Wong
2020-03-19  8:32 ` [PATCH 5/6] daemon: do more immortal allocations up front Eric Wong
2020-03-19  8:32 ` [PATCH 6/6] viewdiff: favor `qr' to precompile regexps Eric Wong
2020-04-21  8:52 ` Encode preloading Eric Wong
2020-05-08  1:59   ` [PATCH] www: preload: load all encodings at startup Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200319083256.15593-4-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git