user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 11/15] inbox: expire resources more aggressively
Date: Thu, 30 Nov 2023 11:41:04 +0000	[thread overview]
Message-ID: <20231130114109.2577708-12-e@80x24.org> (raw)
In-Reply-To: <20231130114109.2577708-1-e@80x24.org>

We no longer trigger git cleanups from the Inbox package since
`git cat-file' users have their own cleanup to support git
coderepos not associated with any inbox.

This change means we unconditionally expire SQLite and Xapian
FDs and some internal caches regardless of git activity.  The
old logic was irrelevant to Gcf2 (libgit2) users anyways since
we couldn't determine whether or not an inbox was active based
on {inflight} git requests, and upcoming changes will make it
inaccurate for all extindex/cindex users as well.

Opening SQLite and Xapian DBs is fairly cheap; so it's a small
price to pay to reduce memory use and fragmentation.
---
 lib/PublicInbox/Inbox.pm | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)

diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index e71ef6d2..7af0ad90 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -12,28 +12,20 @@ use List::Util qw(max);
 use Carp qw(croak);
 use PublicInbox::Compat qw(uniqstr);
 
-# returns true if further checking is required
+# in case DBs get replaced (Xapcmd does it for v1)
 sub check_inodes ($) {
 	for (qw(over mm)) { $_[0]->{$_}->check_inodes if $_[0]->{$_} }
 }
 
+# search/over/mm hold onto FDs and description+cloneurl may get updated.
+# creating long-lived allocations in the same phase as short-lived
+# allocations also leads to fragmentation, so we don't want some stuff
+# living too long.
 sub do_cleanup {
 	my ($ibx) = @_;
-	my $live;
-	if (defined $ibx->{git}) {
-		$live = $ibx->isa(__PACKAGE__) ? $ibx->{git}->cleanup(1)
-					: $ibx->{git}->cleanup_if_unlinked;
-		delete($ibx->{git}) unless $live;
-	}
-	if ($live) {
-		check_inodes($ibx);
-	} else {
-		delete(@$ibx{qw(over mm description cloneurl
-				-imap_url -nntp_url -pop3_url)});
-	}
-	my $srch = $ibx->{search} // $ibx;
+	my ($srch) = delete @$ibx{qw(search over mm description cloneurl)};
+	$srch //= $ibx; # extsearch
 	delete @$srch{qw(xdb qp)};
-	PublicInbox::DS::add_uniq_timer($ibx+0, 5, \&do_cleanup, $ibx) if $live;
 }
 
 sub _cleanup_later ($) {
@@ -370,7 +362,7 @@ sub unsubscribe_unlock {
 # called by inotify
 sub on_unlock {
 	my ($self) = @_;
-	check_inodes($self);
+	check_inodes($self); # DB files may be replaced while holding lock
 	my $subs = $self->{unlock_subs} or return;
 	for my $obj (values %$subs) {
 		eval { $obj->on_inbox_unlock($self) };

  parent reply	other threads:[~2023-11-30 11:41 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-11-30 11:40 [PATCH 00/15] various cindex fixes + speedups Eric Wong
2023-11-30 11:40 ` [PATCH 01/15] cindex: fix store_repo+repo_stored on no-op Eric Wong
2023-11-30 11:40 ` [PATCH 02/15] codesearch: allow inbox count to exceed matches Eric Wong
2023-11-30 11:40 ` [PATCH 03/15] config: reject newlines consistently in dir names Eric Wong
2023-11-30 11:40 ` [PATCH 04/15] cindex: only create {-cidx_err} field on failures Eric Wong
2023-11-30 11:40 ` [PATCH 05/15] cindex: keep batch pipe for pruning SHA-256 repos Eric Wong
2023-11-30 11:40 ` [PATCH 06/15] cindex: store extensions.objectFormat with repo data Eric Wong
2023-11-30 21:36   ` Eric Wong
2023-11-30 11:41 ` [PATCH 07/15] git: share unlinked pack checking code with gcf2 Eric Wong
2023-11-30 11:41 ` [PATCH 08/15] cindex: skip getpid guard for most OnDestroy use Eric Wong
2023-11-30 11:41 ` [PATCH 09/15] spawn: drop IO layer support from redirects Eric Wong
2023-11-30 11:41 ` [PATCH 10/15] cindex: speed up initial scan setup phase Eric Wong
2023-11-30 11:41 ` Eric Wong [this message]
2023-11-30 11:41 ` [PATCH 12/15] git_async_cat: use git from "all" extindex if possible Eric Wong
2023-11-30 11:41 ` [PATCH 13/15] www_listing: support publicInbox.nameIsUrl Eric Wong
2023-12-01  1:29   ` Kyle Meyer
2023-12-01  2:01     ` [PATCH] doc: config: fix grammar for nameIsUrl Eric Wong
2023-11-30 11:41 ` [PATCH 14/15] inbox: shrink data structures for publicinbox.*.hide Eric Wong
2023-11-30 11:41 ` [PATCH 15/15] codesearch: use retry_reopen for WWW Eric Wong
2023-11-30 21:40   ` [PATCH v2] " Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20231130114109.2577708-12-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).