* [PATCH 11/15] inbox: expire resources more aggressively
2023-11-30 11:40 5% [PATCH 00/15] various cindex fixes + speedups Eric Wong
@ 2023-11-30 11:41 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2023-11-30 11:41 UTC (permalink / raw)
To: meta
We no longer trigger git cleanups from the Inbox package since
`git cat-file' users have their own cleanup to support git
coderepos not associated with any inbox.
This change means we unconditionally expire SQLite and Xapian
FDs and some internal caches regardless of git activity. The
old logic was irrelevant to Gcf2 (libgit2) users anyways since
we couldn't determine whether or not an inbox was active based
on {inflight} git requests, and upcoming changes will make it
inaccurate for all extindex/cindex users as well.
Opening SQLite and Xapian DBs is fairly cheap; so it's a small
price to pay to reduce memory use and fragmentation.
---
lib/PublicInbox/Inbox.pm | 24 ++++++++----------------
1 file changed, 8 insertions(+), 16 deletions(-)
diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index e71ef6d2..7af0ad90 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -12,28 +12,20 @@ use List::Util qw(max);
use Carp qw(croak);
use PublicInbox::Compat qw(uniqstr);
-# returns true if further checking is required
+# in case DBs get replaced (Xapcmd does it for v1)
sub check_inodes ($) {
for (qw(over mm)) { $_[0]->{$_}->check_inodes if $_[0]->{$_} }
}
+# search/over/mm hold onto FDs and description+cloneurl may get updated.
+# creating long-lived allocations in the same phase as short-lived
+# allocations also leads to fragmentation, so we don't want some stuff
+# living too long.
sub do_cleanup {
my ($ibx) = @_;
- my $live;
- if (defined $ibx->{git}) {
- $live = $ibx->isa(__PACKAGE__) ? $ibx->{git}->cleanup(1)
- : $ibx->{git}->cleanup_if_unlinked;
- delete($ibx->{git}) unless $live;
- }
- if ($live) {
- check_inodes($ibx);
- } else {
- delete(@$ibx{qw(over mm description cloneurl
- -imap_url -nntp_url -pop3_url)});
- }
- my $srch = $ibx->{search} // $ibx;
+ my ($srch) = delete @$ibx{qw(search over mm description cloneurl)};
+ $srch //= $ibx; # extsearch
delete @$srch{qw(xdb qp)};
- PublicInbox::DS::add_uniq_timer($ibx+0, 5, \&do_cleanup, $ibx) if $live;
}
sub _cleanup_later ($) {
@@ -370,7 +362,7 @@ sub unsubscribe_unlock {
# called by inotify
sub on_unlock {
my ($self) = @_;
- check_inodes($self);
+ check_inodes($self); # DB files may be replaced while holding lock
my $subs = $self->{unlock_subs} or return;
for my $obj (values %$subs) {
eval { $obj->on_inbox_unlock($self) };
^ permalink raw reply related [relevance 7%]
* [PATCH 00/15] various cindex fixes + speedups
@ 2023-11-30 11:40 5% Eric Wong
2023-11-30 11:41 7% ` [PATCH 11/15] inbox: expire resources more aggressively Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2023-11-30 11:40 UTC (permalink / raw)
To: meta
Notable changes:
10/15 provides a huge speedup which will hopefully make
future developments faster.
12/15 probably obsoletes libgit2 for extindex "all" users.
13/15 can save some memory with many inboxes while making
configuration easier.
Eric Wong (15):
cindex: fix store_repo+repo_stored on no-op
codesearch: allow inbox count to exceed matches
config: reject newlines consistently in dir names
cindex: only create {-cidx_err} field on failures
cindex: keep batch pipe for pruning SHA-256 repos
cindex: store extensions.objectFormat with repo data
git: share unlinked pack checking code with gcf2
cindex: skip getpid guard for most OnDestroy use
spawn: drop IO layer support from redirects
cindex: speed up initial scan setup phase
inbox: expire resources more aggressively
git_async_cat: use git from "all" extindex if possible
www_listing: support publicInbox.nameIsUrl
inbox: shrink data structures for publicinbox.*.hide
codesearch: use retry_reopen for WWW
Documentation/public-inbox-config.pod | 19 +-
lib/PublicInbox/CodeSearch.pm | 54 +++--
lib/PublicInbox/CodeSearchIdx.pm | 286 ++++++++++++++++----------
lib/PublicInbox/Config.pm | 32 ++-
lib/PublicInbox/Gcf2.pm | 16 +-
lib/PublicInbox/Git.pm | 27 +--
lib/PublicInbox/GitAsyncCat.pm | 8 +-
lib/PublicInbox/Inbox.pm | 32 +--
lib/PublicInbox/MailDiff.pm | 3 +-
lib/PublicInbox/SearchIdx.pm | 5 +-
lib/PublicInbox/Spawn.pm | 32 +--
lib/PublicInbox/WwwListing.pm | 21 +-
12 files changed, 303 insertions(+), 232 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2023-11-30 11:40 5% [PATCH 00/15] various cindex fixes + speedups Eric Wong
2023-11-30 11:41 7% ` [PATCH 11/15] inbox: expire resources more aggressively Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).