user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: Encode preloading
Date: Tue, 21 Apr 2020 08:52:08 +0000
Message-ID: <20200421085208.GA28087@dcvr> (raw)
In-Reply-To: <20200319083256.15593-1-e@yhbt.net>

[was: Subject: [PATCH 0/6] daemon: reduce fragmentation via preload

Eric Wong <e@yhbt.net> wrote:
> For long-lived daemons, perform immortal allocations as early as
> possible to reduce the likelyhood of heap fragmentation due to
> mixed-lifetime allocations happening once the process is fully
> loaded and serving requests, since per-request allocations
> should all be short-lived.

Encode also loads lazily...

I thought about having public-inbox-{index,mda,watch} track a
list of modules for WWW to preload, but that can't predict the
future.

There's no telling what emails will show up in the future which
trigger autoloading, so I think the only way to avoid
fragmentation is to preload everything which would get
autoloaded... Ugh, that even means Encode::EBCDIC and
Encode::Symbol (for dingbats!)

> On a side note, I'm wondering if WWW should just preload by
> default.  I'm not sure if anybody uses public-inbox.cgi (or
> should be using it :P).  It's not like we don't ship
> public-inbox-httpd; and any PSGI implementation could be used
> for smaller inboxes (or powerful-enough hardware).

Yeah, likely to happen.  Trying to optimize a CGI script for
startup time while depending on git, SQLite, and Xapian just
doesn't seem worth it.

  parent reply index

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-19  8:32 [PATCH 0/6] daemon: reduce fragmentation via preload Eric Wong
2020-03-19  8:32 ` [PATCH 1/6] www: update ->preload for newer modules Eric Wong
2020-03-19  8:32 ` [PATCH 2/6] wwwlisting: favor "use" over require Eric Wong
2020-03-19  8:32 ` [PATCH 3/6] wwwlisting: avoid lazy loading JSON module Eric Wong
2020-03-21  1:10   ` [PATCH 0/2] wwwlisting: fixup warnings :x Eric Wong
2020-03-21  1:10     ` [PATCH 1/2] wwwlisting: use first successfully loaded JSON module Eric Wong
2020-03-21  1:10     ` [PATCH 2/2] t/www_listing: avoid 'once' warnings Eric Wong
2020-03-21  5:24       ` [PATCH v2] " Eric Wong
2020-03-19  8:32 ` [PATCH 4/6] www: avoid `state' usage to perform allocations up-front Eric Wong
2020-03-19  8:32 ` [PATCH 5/6] daemon: do more immortal allocations up front Eric Wong
2020-03-19  8:32 ` [PATCH 6/6] viewdiff: favor `qr' to precompile regexps Eric Wong
2020-04-21  8:52 ` Eric Wong [this message]
2020-05-08  1:59   ` [PATCH] www: preload: load all encodings at startup Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200421085208.GA28087@dcvr \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git