* [PATCH 00/11] cleanups, mostly indexing related
@ 2020-09-02 11:04 5% Eric Wong
2020-09-02 11:04 7% ` [PATCH 11/11] v2writable: reuse read-only shard counting code Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-09-02 11:04 UTC (permalink / raw)
To: meta
Some cleanups ahead of detached index support.
Found some dead code, too.
Eric Wong (11):
msgmap: note how we use ->created_at
disambiguate OverIdx and Over by field name
use more idiomatic internal API for ->over access
search: remove special case for blank query
tests: add "use strict" and declare v5.10.1 compatibility
search: replace ->query with ->mset
search: remove {over_ro} field
imap: drop old, pre-Parse::RecDescent search parser
wwwaltid: drop unused sqlite3_missing function
overidx: document column uses
v2writable: reuse read-only shard counting code
lib/PublicInbox/ExtMsg.pm | 4 +-
lib/PublicInbox/IMAP.pm | 63 +----------------
lib/PublicInbox/Inbox.pm | 11 ++-
lib/PublicInbox/Mbox.pm | 6 +-
lib/PublicInbox/Msgmap.pm | 1 +
lib/PublicInbox/OverIdx.pm | 18 ++---
lib/PublicInbox/Search.pm | 32 ++++-----
lib/PublicInbox/SearchIdx.pm | 32 ++++-----
lib/PublicInbox/SearchView.pm | 3 +-
lib/PublicInbox/SolverGit.pm | 5 +-
lib/PublicInbox/V2Writable.pm | 59 ++++++----------
lib/PublicInbox/WwwAltId.pm | 16 +----
scripts/dupe-finder | 3 +-
t/altid.t | 8 +--
t/altid_v2.t | 7 +-
t/index-git-times.t | 17 +++--
t/indexlevels-mirror.t | 8 +--
t/mda_filter_rubylang.t | 6 +-
t/replace.t | 8 +--
t/search-thr-index.t | 8 +--
t/search.t | 126 +++++++++++++++++-----------------
t/v1reindex.t | 4 +-
t/v2mda.t | 16 +++--
t/v2mirror.t | 24 +++----
t/v2reindex.t | 9 +--
t/v2writable.t | 14 ++--
t/watch_filter_rubylang.t | 12 ++--
t/watch_maildir_v2.t | 30 ++++----
t/xcpdb-reshard.t | 3 +-
xt/eml_check_limits.t | 2 +
xt/perf-threading.t | 2 +-
31 files changed, 232 insertions(+), 325 deletions(-)
^ permalink raw reply [relevance 5%]
* [PATCH 11/11] v2writable: reuse read-only shard counting code
2020-09-02 11:04 5% [PATCH 00/11] cleanups, mostly indexing related Eric Wong
@ 2020-09-02 11:04 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-09-02 11:04 UTC (permalink / raw)
To: meta
We'll also fix the read-only code to ensure we notice missing
Xapian shards, since gaps would throw off our expectation that
Xapian document IDs and NNTP article numbers are interchangeable.
---
lib/PublicInbox/Search.pm | 5 ++++-
lib/PublicInbox/V2Writable.pm | 23 +++--------------------
2 files changed, 7 insertions(+), 21 deletions(-)
diff --git a/lib/PublicInbox/Search.pm b/lib/PublicInbox/Search.pm
index b07f4ea6..fb35b747 100644
--- a/lib/PublicInbox/Search.pm
+++ b/lib/PublicInbox/Search.pm
@@ -7,6 +7,7 @@ package PublicInbox::Search;
use strict;
use parent qw(Exporter);
our @EXPORT_OK = qw(mdocid);
+use List::Util qw(max);
# values for searching, changing the numeric value breaks
# compatibility with old indices (so don't change them it)
@@ -203,7 +204,9 @@ sub _xdb ($) {
# We need numeric sorting so shard[0] is first for reading
# Xapian metadata, if needed
- for (sort { $a <=> $b } grep(/\A[0-9]+\z/, readdir($dh))) {
+ my $last = max(grep(/\A[0-9]+\z/, readdir($dh)));
+ return if !defined($last);
+ for (0..$last) {
my $shard_dir = "$dir/$_";
if (-d $shard_dir && -r _) {
push @xdb, $X{Database}->new($shard_dir);
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index c8334645..a1f6048f 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -65,28 +65,11 @@ sub nproc_shards ($) {
sub count_shards ($) {
my ($self) = @_;
- my $n = 0;
- my $xpfx = $self->{xpfx};
-
# always load existing shards in case core count changes:
# Also, shard count may change while -watch is running
- # due to "xcpdb --reshard"
- if (-d $xpfx) {
- my $XapianDatabase;
- foreach my $shard (<$xpfx/*>) {
- -d $shard && $shard =~ m!/[0-9]+\z! or next;
- $XapianDatabase //= do {
- require PublicInbox::Search;
- PublicInbox::Search::load_xapian();
- $PublicInbox::Search::X{Database};
- };
- eval {
- $XapianDatabase->new($shard)->close;
- $n++;
- };
- }
- }
- $n;
+ my $srch = $self->{ibx}->search or return 0;
+ delete $self->{ibx}->{search};
+ $srch->{nshard} // 0
}
sub new {
^ permalink raw reply related [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-09-02 11:04 5% [PATCH 00/11] cleanups, mostly indexing related Eric Wong
2020-09-02 11:04 7% ` [PATCH 11/11] v2writable: reuse read-only shard counting code Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).