user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <>
Subject: [PATCH] www: preload: load all encodings at startup
Date: Fri, 8 May 2020 01:59:01 +0000	[thread overview]
Message-ID: <20200508015901.GA27432@dcvr> (raw)
In-Reply-To: <20200421085208.GA28087@dcvr>

Eric Wong <> wrote:
> Eric Wong <> wrote:
> > For long-lived daemons, perform immortal allocations as early as
> > possible to reduce the likelyhood of heap fragmentation due to
> > mixed-lifetime allocations happening once the process is fully
> > loaded and serving requests, since per-request allocations
> > should all be short-lived.
> Encode also loads lazily...

Subject: [PATCH] www: preload: load all encodings at startup

Encode lazy-loads encodings on an as-needed basis.  This is
great for short-lived programs, but leads to fragmentation in
long-lived daemons where immortal allocations can get
interleaved with short-lived, per-request allocations.

Since we have no idea which encodings will be needed when
there's a constant flow of incoming mail, just preload
everything available at startup.
 lib/PublicInbox/ | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/PublicInbox/ b/lib/PublicInbox/
index 275e509f..3a428218 100644
--- a/lib/PublicInbox/
+++ b/lib/PublicInbox/
@@ -141,6 +141,12 @@ sub call {
 # fragmentation since common allocators favor a large contiguous heap.
 sub preload {
 	my ($self) = @_;
+	# populate caches used by Encode internally, since emails
+	# may show up with any encoding.
+	require Encode;
+	Encode::find_encoding($_) for Encode->encodings(':all');
 	require PublicInbox::ExtMsg;
 	require PublicInbox::Feed;
 	require PublicInbox::View;

      reply	other threads:[~2020-05-08  1:59 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-19  8:32 [PATCH 0/6] daemon: reduce fragmentation via preload Eric Wong
2020-03-19  8:32 ` [PATCH 1/6] www: update ->preload for newer modules Eric Wong
2020-03-19  8:32 ` [PATCH 2/6] wwwlisting: favor "use" over require Eric Wong
2020-03-19  8:32 ` [PATCH 3/6] wwwlisting: avoid lazy loading JSON module Eric Wong
2020-03-21  1:10   ` [PATCH 0/2] wwwlisting: fixup warnings :x Eric Wong
2020-03-21  1:10     ` [PATCH 1/2] wwwlisting: use first successfully loaded JSON module Eric Wong
2020-03-21  1:10     ` [PATCH 2/2] t/www_listing: avoid 'once' warnings Eric Wong
2020-03-21  5:24       ` [PATCH v2] " Eric Wong
2020-03-19  8:32 ` [PATCH 4/6] www: avoid `state' usage to perform allocations up-front Eric Wong
2020-03-19  8:32 ` [PATCH 5/6] daemon: do more immortal allocations up front Eric Wong
2020-03-19  8:32 ` [PATCH 6/6] viewdiff: favor `qr' to precompile regexps Eric Wong
2020-04-21  8:52 ` Encode preloading Eric Wong
2020-05-08  1:59   ` Eric Wong [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200508015901.GA27432@dcvr \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).