From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 5/6] daemon: do more immortal allocations up front
Date: Thu, 19 Mar 2020 03:32:55 -0500 [thread overview]
Message-ID: <20200319083256.15593-6-e@yhbt.net> (raw)
In-Reply-To: <20200319083256.15593-1-e@yhbt.net>
Doing immortal allocations late can cause those allocations
to end up in places where it fragments the heap. So do more
things up front for long-lived daemons.
---
lib/PublicInbox/NNTPD.pm | 4 ++++
lib/PublicInbox/WWW.pm | 21 +++++++++++++++++----
2 files changed, 21 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/NNTPD.pm b/lib/PublicInbox/NNTPD.pm
index 7a917169..451f4d41 100644
--- a/lib/PublicInbox/NNTPD.pm
+++ b/lib/PublicInbox/NNTPD.pm
@@ -45,6 +45,10 @@ sub refresh_groups () {
# Only valid if msgmap and search works
$new->{$ngname} = $ng;
push @list, $ng;
+
+ # preload to avoid fragmentation:
+ $ng->description;
+ $ng->base_url;
}
});
@list = sort { $a->{newsgroup} cmp $b->{newsgroup} } @list;
diff --git a/lib/PublicInbox/WWW.pm b/lib/PublicInbox/WWW.pm
index 534ee028..2434f2f5 100644
--- a/lib/PublicInbox/WWW.pm
+++ b/lib/PublicInbox/WWW.pm
@@ -132,7 +132,9 @@ sub call {
}
}
-# for CoW-friendliness, MOOOOO!
+# for CoW-friendliness, MOOOOO! Even for single-process setups,
+# we want to get all immortal allocations done early to avoid heap
+# fragmentation since common allocators favor a large contiguous heap.
sub preload {
my ($self) = @_;
require PublicInbox::ExtMsg;
@@ -148,18 +150,29 @@ sub preload {
require PublicInbox::Search;
PublicInbox::Search::load_xapian();
};
- foreach (qw(PublicInbox::SearchView
- PublicInbox::MboxGz
- PublicInbox::NewsWWW)) {
+ foreach (qw(PublicInbox::SearchView PublicInbox::MboxGz)) {
eval "require $_;";
}
if (ref($self)) {
+ my $pi_config = $self->{pi_config};
+ if (defined($pi_config->{'publicinbox.cgitrc'})) {
+ $pi_config->limiter('-cgit');
+ }
$self->cgit;
$self->stylesheets_prepare($_) for ('', '../', '../../');
$self->www_listing;
+ $self->news_www;
+ $pi_config->each_inbox(\&preload_inbox);
}
}
+sub preload_inbox {
+ my $ibx = shift;
+ $ibx->cloneurl;
+ $ibx->description;
+ $ibx->base_url;
+}
+
# private functions below
sub r404 {
next prev parent reply other threads:[~2020-03-19 8:32 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-19 8:32 [PATCH 0/6] daemon: reduce fragmentation via preload Eric Wong
2020-03-19 8:32 ` [PATCH 1/6] www: update ->preload for newer modules Eric Wong
2020-03-19 8:32 ` [PATCH 2/6] wwwlisting: favor "use" over require Eric Wong
2020-03-19 8:32 ` [PATCH 3/6] wwwlisting: avoid lazy loading JSON module Eric Wong
2020-03-21 1:10 ` [PATCH 0/2] wwwlisting: fixup warnings :x Eric Wong
2020-03-21 1:10 ` [PATCH 1/2] wwwlisting: use first successfully loaded JSON module Eric Wong
2020-03-21 1:10 ` [PATCH 2/2] t/www_listing: avoid 'once' warnings Eric Wong
2020-03-21 5:24 ` [PATCH v2] " Eric Wong
2020-03-19 8:32 ` [PATCH 4/6] www: avoid `state' usage to perform allocations up-front Eric Wong
2020-03-19 8:32 ` Eric Wong [this message]
2020-03-19 8:32 ` [PATCH 6/6] viewdiff: favor `qr' to precompile regexps Eric Wong
2020-04-21 8:52 ` Encode preloading Eric Wong
2020-05-08 1:59 ` [PATCH] www: preload: load all encodings at startup Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200319083256.15593-6-e@yhbt.net \
--to=e@yhbt.net \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).