user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: Re: httpd memory usage?
Date: Mon, 27 Sep 2021 07:10:10 +0000	[thread overview]
Message-ID: <20210927071010.GA27389@dcvr> (raw)
In-Reply-To: <20210904235305.GA22009@dcvr>

Eric Wong <e@80x24.org> wrote:
> Is it high for anybody else?  I'm not seeing it for small
> instances (e.g. the instance that's powering
> public-inbox.org/{git,meta} is fine, but with lots of entries my
> lore mirror @ https://yhbt.net/lore/ the main sbrk heap seems to
> be growing slowly without bounds.

I suspect it's long-lived SQLite connections and associated
caches/etc (and having short-lived allocations mixed in with
long-term ones).

Anyways, I'm testing the below on yhbt.net/lore, but
there's other SQLite knobs worth investigating...
Preloading all DBs isn't realistic, either, due to the
potential number of inboxes.

diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index 1d5fc708..8fcf6e81 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -43,7 +43,7 @@ sub cleanup_task () {
 		for my $git (@{$ibx->{-repo_objs}}) {
 			$again = 1 if $git->cleanup;
 		}
-		check_inodes($ibx);
+		delete @$ibx{qw(over mm)};
 		$next->{"$ibx"} = $ibx if $again;
 	}
 	$CLEANUP = $next;

> I think I'll need to finish porting mwrap over to Perl/XS
> (from Ruby/C)...

Yeah, still on my TODO list.  But right now lei IMAP(S) on
high-latency links is a terrible...

  reply	other threads:[~2021-09-27  7:10 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-04 23:53 Eric Wong
2021-09-27  7:10 ` Eric Wong [this message]
2021-10-04  0:07 ` [PATCH 0/2] www: fix ref cycles when threading extindex Eric Wong
2021-10-04  0:07   ` [PATCH 1/2] t/thread-cycle: make Email::Simple optional Eric Wong
2021-10-04  0:07   ` [PATCH 2/2] www: fix ref cycle from threading w/ extindex Eric Wong
2021-10-04 22:51   ` [PATCH 0/2] www: fix ref cycles when threading extindex Eric Wong
2021-10-05 11:33     ` Encode.pm leak Eric Wong
2021-10-12 10:59       ` Encode.pm leak in v2.87..v3.12 Eric Wong
2021-10-13 10:16         ` [PATCH 0/7] workaround Encode leak, several test fixes Eric Wong
2021-10-13 10:16           ` [PATCH 1/7] xt/perf-msgview: drop unnecessary use_ok Eric Wong
2021-10-13 10:16           ` [PATCH 2/7] test_common: hoist out tail_f sub Eric Wong
2021-10-13 10:16           ` [PATCH 3/7] t/www_listing: require opt-in for grokmirror tests Eric Wong
2021-10-13 10:16           ` [PATCH 4/7] eml: avoid Encode 2.87..3.12 leak Eric Wong
2021-10-13 10:16           ` [PATCH 5/7] t/lei-mirror: avoid reading ~/.public-inbox/config in test Eric Wong
2021-10-13 10:16           ` [PATCH 6/7] t/git: avoid "once" warning for async_warn Eric Wong
2021-10-13 10:16           ` [PATCH 7/7] t/nntpd-tls: change diag() to like() assertion Eric Wong
2021-11-04  0:17 ` httpd memory usage? Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210927071010.GA27389@dcvr \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    --subject='Re: httpd memory usage?' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).