* [PATCH 0/5] warnings and such
@ 2021-10-13 7:00 6% Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2021-10-13 7:00 UTC (permalink / raw)
To: meta
Patches 1-4/5 are all things which will probably make
more fsck-type things nicer across different CLI tools.
5/5 was something I noticed while working on 2a3c8d7a2c40
(daemon: unconditionally close Xapian shards on cleanup, 2021-10-12)
Eric Wong (5):
extindex: show OID on bad blob failure
lei: use standard warn() in more places
treewide: use warn() or carp() instead of env->{psgi.errors}
extindex: set {current_info} in eidxq processing
www: preload: load ExtSearch via ->ALL
lib/PublicInbox/ExtSearchIdx.pm | 9 +++---
lib/PublicInbox/Fetch.pm | 2 +-
lib/PublicInbox/GitHTTPBackend.pm | 39 ++++++--------------------
lib/PublicInbox/GzipFilter.pm | 3 +-
lib/PublicInbox/HTTP.pm | 16 ++++-------
lib/PublicInbox/LEI.pm | 18 ++++++++----
lib/PublicInbox/LeiBlob.pm | 3 +-
lib/PublicInbox/LeiConvert.pm | 2 +-
lib/PublicInbox/LeiForgetExternal.pm | 5 ++--
lib/PublicInbox/LeiImport.pm | 7 ++---
lib/PublicInbox/LeiImportKw.pm | 2 +-
lib/PublicInbox/LeiInput.pm | 4 +--
lib/PublicInbox/LeiMailSync.pm | 4 +--
lib/PublicInbox/LeiMirror.pm | 8 +++---
lib/PublicInbox/LeiP2q.pm | 2 +-
lib/PublicInbox/LeiRediff.pm | 3 +-
lib/PublicInbox/LeiRemote.pm | 2 +-
lib/PublicInbox/LeiViewText.pm | 2 +-
lib/PublicInbox/LeiXSearch.pm | 42 +++++++++++++++-------------
lib/PublicInbox/Qspawn.pm | 13 +++------
lib/PublicInbox/SearchView.pm | 7 ++---
lib/PublicInbox/Unsubscribe.pm | 5 +---
lib/PublicInbox/ViewVCS.pm | 6 ++--
lib/PublicInbox/WWW.pm | 1 +
lib/PublicInbox/WwwAltId.pm | 3 +-
lib/PublicInbox/WwwHighlight.pm | 2 +-
26 files changed, 89 insertions(+), 121 deletions(-)
^ permalink raw reply [relevance 6%]
* [PATCH 7/9] daemon: unconditionally close Xapian shards on cleanup
2021-10-12 11:46 7% [PATCH 0/9] various read-only daemon and WWW things Eric Wong
@ 2021-10-12 11:47 7% ` Eric Wong
0 siblings, 0 replies; 3+ results
From: Eric Wong @ 2021-10-12 11:47 UTC (permalink / raw)
To: meta
The cost of opening a Xapian DB (even with shards) isn't high,
so save some FDs and just close it. We hit Xapian far less than
over.sqlite3 and we discard the MSet ASAP even when streaming
large responses.
This simplifies our code a bit and hopefully helps reduce
fragmentation by increasing mortality of late allocations.
---
lib/PublicInbox/Inbox.pm | 18 +++++++++---------
lib/PublicInbox/Search.pm | 14 --------------
t/extsearch.t | 8 +-------
3 files changed, 10 insertions(+), 30 deletions(-)
diff --git a/lib/PublicInbox/Inbox.pm b/lib/PublicInbox/Inbox.pm
index 724df50a..61d153bf 100644
--- a/lib/PublicInbox/Inbox.pm
+++ b/lib/PublicInbox/Inbox.pm
@@ -11,8 +11,6 @@ use List::Util qw(max);
use Carp qw(croak);
# returns true if further checking is required
-sub cleanup_shards { $_[0]->{search} ? $_[0]->{search}->cleanup_shards : undef }
-
sub check_inodes ($) {
for (qw(over mm)) { $_[0]->{$_}->check_inodes if $_[0]->{$_} }
}
@@ -31,7 +29,8 @@ sub do_cleanup {
delete(@$ibx{qw(over mm description cloneurl
-imap_url -nntp_url)});
}
- $ibx->cleanup_shards and $live = 1;
+ my $srch = $ibx->{search} // $ibx;
+ delete @$srch{qw(xdb qp)};
for my $git (@{$ibx->{-repo_objs} // []}) {
$live = 1 if $git->cleanup(1);
}
@@ -138,17 +137,18 @@ sub max_git_epoch {
$cur;
}
+sub mm_file {
+ my ($self) = @_;
+ my $d = $self->{inboxdir};
+ ($self->version >= 2 ? $d : "$d/public-inbox").'/msgmap.sqlite3';
+}
+
sub mm {
my ($self, $req) = @_;
$self->{mm} //= eval {
require PublicInbox::Msgmap;
_cleanup_later($self);
- my $dir = $self->{inboxdir};
- if ($self->version >= 2) {
- PublicInbox::Msgmap->new_file("$dir/msgmap.sqlite3");
- } else {
- PublicInbox::Msgmap->new($dir);
- }
+ PublicInbox::Msgmap->new_file(mm_file($self));
} // ($req ? croak("E: $@") : undef);
}
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index dd6d3710..f0e7ed0c 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -243,20 +243,6 @@ sub xdb ($) {
};
}
-# returns true if a future rescan is desired
-sub cleanup_shards {
- my ($self) = @_;
- return unless exists($self->{xdb});
- my $xpfx = $self->{xpfx};
- return reopen($self) if $xpfx =~ m!/xapian[0-9]+\z!; # true
- opendir(my $dh, $xpfx) or return warn("$xpfx gone: $!\n"); # true
- my $nr = grep(/\A[0-9]+\z/, readdir($dh)) or
- return warn("$xpfx has no shards\n"); # true
- return reopen($self) if $nr == ($self->{nshard} // -1);
- delete @$self{qw(xdb qp)};
- undef;
-}
-
sub new {
my ($class, $ibx) = @_;
ref $ibx or die "BUG: expected PublicInbox::Inbox object: $ibx";
diff --git a/t/extsearch.t b/t/extsearch.t
index 8190de17..dfc190e2 100644
--- a/t/extsearch.t
+++ b/t/extsearch.t
@@ -447,9 +447,6 @@ SKIP: {
ok(ref($es->{xdb}), '{xdb} created');
my $nshards1 = $es->{nshard};
is($nshards1, 1, 'correct shard count');
- my $xdb_str = "$es->{xdb}";
- ok($es->cleanup_shards, 'cleanup_shards noop');
- is("$es->{xdb}", $xdb_str, '{xdb} unchanged');
my @ei_dir = glob("$d/ei*/");
chmod 0755, $ei_dir[0] or xbail "chmod: $!";
@@ -463,11 +460,8 @@ SKIP: {
my $m = sprintf('%04o', 07777 & (stat($dirs[$i]))[2]);
is($m, $mode, "shard [$i] mode");
}
- is($es->cleanup_shards, undef, 'cleanup_shards cleaned');
- ok(!defined($es->{xdb}), 'old {xdb} gone');
- is($es->cleanup_shards, undef, 'cleanup_shards clean idempotent');
+ delete @$es{qw(xdb qp)};
is($es->mset('z:0..')->size, $nresult0, 'new shards, same results');
- ok($es->cleanup_shards, 'cleanup_shards true after open');
for my $i (4..5) {
is(grep(m!/ei[0-9]+/$i\z!, @dirs), 0, "no shard [$i]");
^ permalink raw reply related [relevance 7%]
* [PATCH 0/9] various read-only daemon and WWW things
@ 2021-10-12 11:46 7% Eric Wong
2021-10-12 11:47 7% ` [PATCH 7/9] daemon: unconditionally close Xapian shards on cleanup Eric Wong
0 siblings, 1 reply; 3+ results
From: Eric Wong @ 2021-10-12 11:46 UTC (permalink / raw)
To: meta
Read-only daemons close idle descriptors much more aggressively,
now. This should reduce userspace and kernel memory use during
non-peak times with negligible slowdown under heavy load.
Some fairly big msgmap-related cleanups and WWW now exposes
mm->created_at AKA UIDVALIDITY as the Last-Modified time
of the _/text/config/raw endpoint. (see 9/9)
Eric Wong (9):
daemon: use v5.10.1, disable local warnings
daemon: quiet down Eml-related warnings
search: delete QueryParser along with DB handle
nntp: use defined-OR from Perl 5.10 for msgid check
msgmap: use DBI->prepare_cached
msgmap: share most of check_inodes w/ over
daemon: unconditionally close Xapian shards on cleanup
msgmap: ->new_file to supports $ibx arg, drop ->new
www: _/text/config/raw Last-Modified: is mm->created_at
lib/PublicInbox/Daemon.pm | 8 ++-
lib/PublicInbox/Inbox.pm | 18 +++---
lib/PublicInbox/InboxWritable.pm | 2 +-
lib/PublicInbox/LeiMirror.pm | 13 ++--
lib/PublicInbox/Msgmap.pm | 102 +++++++++++++------------------
lib/PublicInbox/NNTP.pm | 6 +-
lib/PublicInbox/Search.pm | 14 -----
lib/PublicInbox/SearchIdx.pm | 3 +-
lib/PublicInbox/V2Writable.pm | 4 +-
lib/PublicInbox/WwwText.pm | 3 +
t/altid.t | 4 +-
t/altid_v2.t | 4 +-
t/extsearch.t | 10 +--
t/filter_rubylang.t | 2 +-
t/init.t | 3 +-
t/lei-mirror.t | 31 +++++++++-
t/msgmap.t | 5 +-
t/psgi_v2.t | 10 ++-
18 files changed, 120 insertions(+), 122 deletions(-)
^ permalink raw reply [relevance 7%]
Results 1-3 of 3 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-10-12 11:46 7% [PATCH 0/9] various read-only daemon and WWW things Eric Wong
2021-10-12 11:47 7% ` [PATCH 7/9] daemon: unconditionally close Xapian shards on cleanup Eric Wong
2021-10-13 7:00 6% [PATCH 0/5] warnings and such Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).