user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 3/5] index: optimize after all SQLite DB commits
Date: Tue, 12 Oct 2021 22:44:58 +0000	[thread overview]
Message-ID: <20211012224500.2882-4-e@80x24.org> (raw)
In-Reply-To: <20211012224500.2882-1-e@80x24.org>

This covers v1 inboxes, as well.  We also guard the execution
since "PRAGMA optimize" was only introduced in SQLite 3.18.0
(2017-03-30)
---
 lib/PublicInbox/LeiMailSync.pm |  2 +-
 lib/PublicInbox/OverIdx.pm     |  1 +
 lib/PublicInbox/SearchIdx.pm   |  1 +
 lib/PublicInbox/V2Writable.pm  | 16 +++++-----------
 4 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/lib/PublicInbox/LeiMailSync.pm b/lib/PublicInbox/LeiMailSync.pm
index c6cd1bc58d0a..f7e37ad9ca80 100644
--- a/lib/PublicInbox/LeiMailSync.pm
+++ b/lib/PublicInbox/LeiMailSync.pm
@@ -48,7 +48,7 @@ sub lms_pause {
 	my ($self) = @_;
 	$self->{fmap} = {};
 	my $dbh = delete $self->{dbh};
-	$dbh->do('PRAGMA optimize') if $dbh;
+	eval { $dbh->do('PRAGMA optimize') } if $dbh;
 }
 
 sub create_tables {
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index d6d706f7fed0..9fdb26c0d5c2 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -434,6 +434,7 @@ sub commit_lazy {
 	my ($self) = @_;
 	delete $self->{txn} or return;
 	$self->{dbh}->commit;
+	eval { $self->{dbh}->do('PRAGMA optimize') };
 }
 
 sub begin_lazy {
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index a2ed94993993..928152ec4df4 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -793,6 +793,7 @@ sub v1_checkpoint ($$;$) {
 	${$sync->{max}} = $self->{batch_bytes};
 
 	$self->{mm}->{dbh}->commit;
+	eval { $self->{mm}->{dbh}->do('PRAGMA optimize') };
 	my $xdb = $self->{xdb};
 	if ($newest && $xdb) {
 		my $cur = $xdb->get_metadata('last_commit');
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index efcc1fc21a18..3914383cc9d3 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -547,11 +547,11 @@ sub checkpoint ($;$) {
 	}
 	my $shards = $self->{idx_shards};
 	if ($shards) {
-		my $mm = $self->{mm};
-		my $dbh = $mm->{dbh} if $mm;
+		my $dbh = $self->{mm}->{dbh} if $self->{mm};
 
 		# SQLite msgmap data is second in importance
 		$dbh->commit if $dbh;
+		eval { $dbh->do('PRAGMA optimize') };
 
 		# SQLite overview is third
 		$self->{oidx}->commit_lazy;
@@ -620,16 +620,10 @@ sub done {
 		my $m = $err ? 'rollback' : 'commit';
 		eval { $mm->{dbh}->$m };
 		$err .= "msgmap $m: $@\n" if $@;
-		eval { $mm->{dbh}->do('PRAGMA optimize') };
-		$err .= "msgmap optimize: $@\n" if $@;
 	}
-	if ($self->{oidx} && $self->{oidx}->{dbh}) {
-		if ($err) {
-			eval { $self->{oidx}->rollback_lazy };
-			$err .= "overview rollback: $@\n" if $@;
-		}
-		eval { $self->{oidx}->{dbh}->do('PRAGMA optimize') };
-		$err .= "overview optimize: $@\n" if $@;
+	if ($self->{oidx} && $self->{oidx}->{dbh} && $err) {
+		eval { $self->{oidx}->rollback_lazy };
+		$err .= "overview rollback: $@\n" if $@;
 	}
 
 	my $shards = delete $self->{idx_shards};

  parent reply	other threads:[~2021-10-12 22:45 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-10-12 22:44 [PATCH 0/5] fix extindex reindex harder Eric Wong
2021-10-12 22:44 ` [PATCH 1/5] extindex: flush pending reindex before unref Eric Wong
2021-10-12 22:44 ` [PATCH 2/5] lei/store: use remove_doc to save some LoC Eric Wong
2021-10-12 22:44 ` Eric Wong [this message]
2021-10-12 22:44 ` [PATCH 4/5] doc: relnotes: note some recent improvements Eric Wong
2021-10-12 22:45 ` [PATCH 5/5] lei up --all: show output for warnings Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211012224500.2882-4-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).