user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 15/15] codesearch: use retry_reopen for WWW
Date: Thu, 30 Nov 2023 11:41:08 +0000	[thread overview]
Message-ID: <20231130114109.2577708-16-e@80x24.org> (raw)
In-Reply-To: <20231130114109.2577708-1-e@80x24.org>

As with mail search, a cindex may be updated while WWW is
serving requests.  Thus we must reopen the Xapian DB when
the revision we're using becomes stale.
---
 lib/PublicInbox/CodeSearch.pm | 25 +++++++++++++++----------
 1 file changed, 15 insertions(+), 10 deletions(-)

diff --git a/lib/PublicInbox/CodeSearch.pm b/lib/PublicInbox/CodeSearch.pm
index f4694686..a2f4bae8 100644
--- a/lib/PublicInbox/CodeSearch.pm
+++ b/lib/PublicInbox/CodeSearch.pm
@@ -256,17 +256,22 @@ sub load_ct { # retry_reopen cb
 	}
 }
 
+sub load_ct { # retry_reopen cb
+	my ($self, $git_dir) = @_;
+	my @ids = docids_of_git_dir $self, $git_dir or return;
+	for (@ids) {
+		my $doc = $self->get_doc($_) // next;
+		return int_val($doc, CT);
+	}
+}
+
 sub load_commit_times { # each_cindex callback
 	my ($self, $todo) = @_; # todo = [ [ time, git ], [ time, git ] ...]
-	my (@pending, $rec, $dir, @ids, $doc);
+	my (@pending, $rec, $ct);
 	while ($rec = shift @$todo) {
-		@ids = docids_of_git_dir $self, $rec->[1]->{git_dir};
-		if (@ids) {
-			for (@ids) {
-				$doc = $self->get_doc($_) // next;
-				$rec->[0] = int_val($doc, CT);
-				last;
-			}
+		$ct = $self->retry_reopen(\&load_ct, $rec->[1]->{git_dir});
+		if (defined $ct) {
+			$rec->[0] = $ct;
 		} else { # may be in another cindex:
 			push @pending, $rec;
 		}
@@ -295,7 +300,7 @@ EOM
 			$git;
 		};
 	}
-	my $jd = join_data($self) or return warn <<EOM;
+	my $jd = $self->retry_reopen(\&join_data, $self) or return warn <<EOM;
 W: cindex.$name.topdir=$self->{topdir} has no usable join data for $cfg_f
 EOM
 	my ($ekeys, $roots, $ibx2root) = @$jd{qw(ekeys roots ibx2root)};
@@ -366,7 +371,7 @@ sub repos_sorted {
 	my @recs = map { [ 0, $_ ] } @_; # PublicInbox::Git objects
 	my @todo = @recs;
 	$pi_cfg->each_cindex(\&load_commit_times, \@todo);
-	@recs = sort { $b->[0] <=> $a->[0] } @recs;
+	@recs = sort { $b->[0] <=> $a->[0] } @recs; # sort by commit time
 }
 
 1;

  parent reply	other threads:[~2023-11-30 11:41 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-11-30 11:40 [PATCH 00/15] various cindex fixes + speedups Eric Wong
2023-11-30 11:40 ` [PATCH 01/15] cindex: fix store_repo+repo_stored on no-op Eric Wong
2023-11-30 11:40 ` [PATCH 02/15] codesearch: allow inbox count to exceed matches Eric Wong
2023-11-30 11:40 ` [PATCH 03/15] config: reject newlines consistently in dir names Eric Wong
2023-11-30 11:40 ` [PATCH 04/15] cindex: only create {-cidx_err} field on failures Eric Wong
2023-11-30 11:40 ` [PATCH 05/15] cindex: keep batch pipe for pruning SHA-256 repos Eric Wong
2023-11-30 11:40 ` [PATCH 06/15] cindex: store extensions.objectFormat with repo data Eric Wong
2023-11-30 21:36   ` Eric Wong
2023-11-30 11:41 ` [PATCH 07/15] git: share unlinked pack checking code with gcf2 Eric Wong
2023-11-30 11:41 ` [PATCH 08/15] cindex: skip getpid guard for most OnDestroy use Eric Wong
2023-11-30 11:41 ` [PATCH 09/15] spawn: drop IO layer support from redirects Eric Wong
2023-11-30 11:41 ` [PATCH 10/15] cindex: speed up initial scan setup phase Eric Wong
2023-11-30 11:41 ` [PATCH 11/15] inbox: expire resources more aggressively Eric Wong
2023-11-30 11:41 ` [PATCH 12/15] git_async_cat: use git from "all" extindex if possible Eric Wong
2023-11-30 11:41 ` [PATCH 13/15] www_listing: support publicInbox.nameIsUrl Eric Wong
2023-12-01  1:29   ` Kyle Meyer
2023-12-01  2:01     ` [PATCH] doc: config: fix grammar for nameIsUrl Eric Wong
2023-11-30 11:41 ` [PATCH 14/15] inbox: shrink data structures for publicinbox.*.hide Eric Wong
2023-11-30 11:41 ` Eric Wong [this message]
2023-11-30 21:40   ` [PATCH v2] codesearch: use retry_reopen for WWW Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20231130114109.2577708-16-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).