* [PATCH 1/4] v2 + lei/store: always wait for fast-import checkpoint
2024-04-16 20:56 5% [PATCH 0/4] lei parallelism fixes Eric Wong
@ 2024-04-16 20:56 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2024-04-16 20:56 UTC (permalink / raw)
To: meta
Since data going to git is the most important, always ensure
data is written to git before attempting to write anything to
SQLite or Xapian.
---
lib/PublicInbox/LeiStore.pm | 4 +---
lib/PublicInbox/V2Writable.pm | 8 +-------
2 files changed, 2 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 2eb09eca..0df2352c 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -573,9 +573,7 @@ sub set_xvmd {
sub checkpoint {
my ($self, $wait) = @_;
- if (my $im = $self->{im}) {
- $wait ? $im->barrier : $im->checkpoint;
- }
+ $self->{im}->barrier if $self->{im};
delete $self->{lms};
$self->{priv_eidx}->checkpoint($wait);
}
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index fb259396..43f37f60 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -507,13 +507,7 @@ sub set_last_commits ($) { # this is NOT for ExtSearchIdx
sub checkpoint ($;$) {
my ($self, $wait) = @_;
- if (my $im = $self->{im}) {
- if ($wait) {
- $im->barrier;
- } else {
- $im->checkpoint;
- }
- }
+ $self->{im}->barrier if $self->{im};
my $shards = $self->{idx_shards};
if ($shards) {
my $dbh = $self->{mm}->{dbh} if $self->{mm};
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] lei parallelism fixes
@ 2024-04-16 20:56 5% Eric Wong
2024-04-16 20:56 7% ` [PATCH 1/4] v2 + lei/store: always wait for fast-import checkpoint Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2024-04-16 20:56 UTC (permalink / raw)
To: meta
This series allows `lei reindex' to run in parallel with other
lei commands which write to lei/store.
Eric Wong (4):
v2 + lei/store: always wait for fast-import checkpoint
lei: use ->barrier to commit to lei/store
lei/store: stop shard workers + cat-file on idle
lei: use async barrier for --import-before
lib/PublicInbox/EOFpipe.pm | 7 ++--
lib/PublicInbox/ExtSearchIdx.pm | 1 +
lib/PublicInbox/LEI.pm | 6 ++--
lib/PublicInbox/LeiInput.pm | 2 +-
lib/PublicInbox/LeiRefreshMailSync.pm | 2 +-
lib/PublicInbox/LeiRemote.pm | 4 +--
lib/PublicInbox/LeiStore.pm | 46 ++++++++++++++++-----------
lib/PublicInbox/LeiToMail.pm | 28 ++++++++++++----
lib/PublicInbox/LeiXSearch.pm | 17 ++++++----
lib/PublicInbox/V2Writable.pm | 8 +----
t/lei-store-fail.t | 2 +-
11 files changed, 74 insertions(+), 49 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2024-04-16 20:56 5% [PATCH 0/4] lei parallelism fixes Eric Wong
2024-04-16 20:56 7% ` [PATCH 1/4] v2 + lei/store: always wait for fast-import checkpoint Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).