user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 10/20] v2: rename SearchIdxPart => SearchIdxShard
Date: Sat, 15 Jun 2019 08:47:06 +0000
Message-ID: <20190615084716.3075-11-e@80x24.org> (raw)
In-Reply-To: <20190615084716.3075-1-e@80x24.org>

Another step towards keeping our file and package names
consistent with Xapian terminology.
---
 MANIFEST                                      |  2 +-
 .../{SearchIdxPart.pm => SearchIdxShard.pm}   | 24 +++++++++----------
 lib/PublicInbox/V2Writable.pm                 |  4 ++--
 3 files changed, 15 insertions(+), 15 deletions(-)
 rename lib/PublicInbox/{SearchIdxPart.pm => SearchIdxShard.pm} (83%)

diff --git a/MANIFEST b/MANIFEST
index 3f0a79a..c769397 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -119,7 +119,7 @@ lib/PublicInbox/Reply.pm
 lib/PublicInbox/SaPlugin/ListMirror.pm
 lib/PublicInbox/Search.pm
 lib/PublicInbox/SearchIdx.pm
-lib/PublicInbox/SearchIdxPart.pm
+lib/PublicInbox/SearchIdxShard.pm
 lib/PublicInbox/SearchMsg.pm
 lib/PublicInbox/SearchThread.pm
 lib/PublicInbox/SearchView.pm
diff --git a/lib/PublicInbox/SearchIdxPart.pm b/lib/PublicInbox/SearchIdxShard.pm
similarity index 83%
rename from lib/PublicInbox/SearchIdxPart.pm
rename to lib/PublicInbox/SearchIdxShard.pm
index 77fb7d9..15ec657 100644
--- a/lib/PublicInbox/SearchIdxPart.pm
+++ b/lib/PublicInbox/SearchIdxShard.pm
@@ -3,23 +3,23 @@
 
 # used to interface with a single Xapian shard in V2 repos.
 # See L<public-inbox-v2-format(5)> for more info on how we shard Xapian
-package PublicInbox::SearchIdxPart;
+package PublicInbox::SearchIdxShard;
 use strict;
 use warnings;
 use base qw(PublicInbox::SearchIdx);
 
 sub new {
-	my ($class, $v2writable, $part) = @_;
-	my $self = $class->SUPER::new($v2writable->{-inbox}, 1, $part);
+	my ($class, $v2writable, $shard) = @_;
+	my $self = $class->SUPER::new($v2writable->{-inbox}, 1, $shard);
 	# create the DB before forking:
 	$self->_xdb_acquire;
 	$self->_xdb_release;
-	$self->spawn_worker($v2writable, $part) if $v2writable->{parallel};
+	$self->spawn_worker($v2writable, $shard) if $v2writable->{parallel};
 	$self;
 }
 
 sub spawn_worker {
-	my ($self, $v2writable, $part) = @_;
+	my ($self, $v2writable, $shard) = @_;
 	my ($r, $w);
 	pipe($r, $w) or die "pipe failed: $!\n";
 	binmode $r, ':raw';
@@ -35,8 +35,8 @@ sub spawn_worker {
 		# speeds V2Writable batch imports across 8 cores by nearly 20%
 		fcntl($r, 1031, 1048576) if $^O eq 'linux';
 
-		eval { partition_worker_loop($self, $r, $part, $bnote) };
-		die "worker $part died: $@\n" if $@;
+		eval { shard_worker_loop($self, $r, $shard, $bnote) };
+		die "worker $shard died: $@\n" if $@;
 		die "unexpected MM $self->{mm}" if $self->{mm};
 		exit;
 	}
@@ -45,14 +45,14 @@ sub spawn_worker {
 	close $r or die "failed to close: $!";
 }
 
-sub partition_worker_loop ($$$$) {
-	my ($self, $r, $part, $bnote) = @_;
-	$0 = "pi-v2-shard[$part]";
+sub shard_worker_loop ($$$$) {
+	my ($self, $r, $shard, $bnote) = @_;
+	$0 = "pi-v2-shard[$shard]";
 	my $current_info = '';
 	my $warn_cb = $SIG{__WARN__} || sub { print STDERR @_ };
 	local $SIG{__WARN__} = sub {
 		chomp $current_info;
-		$warn_cb->("[$part] $current_info: ", @_);
+		$warn_cb->("[$shard] $current_info: ", @_);
 	};
 	$self->begin_txn_lazy;
 	while (my $line = $r->getline) {
@@ -64,7 +64,7 @@ sub partition_worker_loop ($$$$) {
 		} elsif ($line eq "barrier\n") {
 			$self->commit_txn_lazy;
 			# no need to lock < 512 bytes is atomic under POSIX
-			print $bnote "barrier $part\n" or
+			print $bnote "barrier $shard\n" or
 					die "write failed for barrier $!\n";
 		} elsif ($line =~ /\AD ([a-f0-9]{40,}) (.+)\n\z/s) {
 			my ($oid, $mid) = ($1, $2);
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index aa13aa8..cc9ebfe 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -7,7 +7,7 @@ package PublicInbox::V2Writable;
 use strict;
 use warnings;
 use base qw(PublicInbox::Lock);
-use PublicInbox::SearchIdxPart;
+use PublicInbox::SearchIdxShard;
 use PublicInbox::MIME;
 use PublicInbox::Git;
 use PublicInbox::Import;
@@ -300,7 +300,7 @@ sub idx_init {
 		# idx_parts must be visible to all forked processes
 		my $idx = $self->{idx_parts} = [];
 		for my $i (0..$max) {
-			push @$idx, PublicInbox::SearchIdxPart->new($self, $i);
+			push @$idx, PublicInbox::SearchIdxShard->new($self, $i);
 		}
 
 		# Now that all subprocesses are up, we can open the FDs
-- 
EW


  parent reply index

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-15  8:46 [PATCH 00/20] v2: use consistent terminology Eric Wong
2019-06-15  8:46 ` [PATCH 01/20] doc: rename our Xapian "partitions" to "shards" Eric Wong
2019-06-15  8:46 ` [PATCH 02/20] v2writable: update comments regarding xcpdb --reshard Eric Wong
2019-06-15  8:46 ` [PATCH 03/20] admin|xapcmd: user-facing messages say "shard" Eric Wong
2019-06-15  8:47 ` [PATCH 04/20] rename reference to git epochs as "partitions" Eric Wong
2019-06-15  8:47 ` [PATCH 05/20] searchidxpart: start using "shard" in user-visible places Eric Wong
2019-06-15  8:47 ` [PATCH 06/20] v2writable: count_partitions => count_shards Eric Wong
2019-06-15  8:47 ` [PATCH 07/20] v2writable: rename {partitions} field to {shards} Eric Wong
2019-06-15  8:47 ` [PATCH 08/20] tests: change messages to use "shard" instead of partition Eric Wong
2019-06-15  8:47 ` [PATCH 09/20] inboxwritable: s/partitions/shards/ in local var Eric Wong
2019-06-15  8:47 ` Eric Wong [this message]
2019-06-15  8:47 ` [PATCH 11/20] xapcmd: update comments referencing "partitions" Eric Wong
2019-06-15  8:47 ` [PATCH 12/20] search*: rename {partition} => {shard} Eric Wong
2019-06-15  8:47 ` [PATCH 13/20] v2writable: avoid "part" in internal subs and fields Eric Wong
2019-06-15  8:47 ` [PATCH 14/20] v2writable: rename local vars to match Xapian terminology Eric Wong
2019-06-15  8:47 ` [PATCH 15/20] adminedit: "part" => "shard" for local variables Eric Wong
2019-06-15  8:47 ` [PATCH 16/20] v2writable: use "epoch" consistently when referring to git repos Eric Wong
2019-06-15  8:47 ` [PATCH 17/20] search: use "shard" for local variable Eric Wong
2019-06-15  8:47 ` [PATCH 18/20] xapcmd: favor 'shard' over 'part' in local variables Eric Wong
2019-06-15  8:47 ` [PATCH 19/20] t/xcpdb-reshard: use 'shard' term " Eric Wong
2019-06-15  8:47 ` [PATCH 20/20] comments: replace "partition" with "shard" Eric Wong

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190615084716.3075-11-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git