user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 2/3] lei <q|up>: wait on remote mboxrd imports synchronously
Date: Sat, 14 Aug 2021 00:29:43 +0000	[thread overview]
Message-ID: <20210814002944.5126-3-e@80x24.org> (raw)
In-Reply-To: <20210814002944.5126-1-e@80x24.org>

This ought to avoid /Document \d+ not found/ errors from Xapian
when seeing a message for the first time by not attempting to
read keywords for totally unseen messages.
---
 lib/PublicInbox/LeiRemote.pm  |  7 ++++---
 lib/PublicInbox/LeiStore.pm   |  1 +
 lib/PublicInbox/LeiXSearch.pm | 10 +++++++---
 3 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
index 945d9990..e7deecb8 100644
--- a/lib/PublicInbox/LeiRemote.pm
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -26,11 +26,12 @@ sub _each_mboxrd_eml { # callback for MboxReader->mboxrd
 	my ($eml, $self) = @_;
 	my $lei = $self->{lei};
 	my $xoids = $lei->{ale}->xoids_for($eml, 1);
+	my $smsg = bless {}, 'PublicInbox::Smsg';
 	if ($lei->{sto} && !$xoids) { # memoize locally
-		$lei->{sto}->ipc_do('add_eml', $eml);
+		my $res = $lei->{sto}->ipc_do('add_eml', $eml);
+		$smsg = $res if ref($res) eq ref($smsg);
 	}
-	my $smsg = bless {}, 'PublicInbox::Smsg';
-	$smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+	$smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
 				: git_sha(1, $eml)->hexdigest;
 	$smsg->populate($eml);
 	$smsg->{mid} //= '(none)';
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index e26b622d..ce66014f 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -329,6 +329,7 @@ sub add_eml {
 		}
 		\@docids;
 	} else { # totally new message
+		delete $smsg->{-oidx}; # for IPC-friendliness
 		$smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
 		$oidx->add_overview($eml, $smsg);
 		$oidx->add_xref3($smsg->{num}, -1, $smsg->{blob}, '.');
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 393f25bf..971f3a06 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -266,11 +266,15 @@ sub _smsg_fill ($$) {
 sub each_remote_eml { # callback for MboxReader->mboxrd
 	my ($eml, $self, $lei, $each_smsg) = @_;
 	my $xoids = $lei->{ale}->xoids_for($eml, 1);
+	my $smsg = bless {}, 'PublicInbox::Smsg';
 	if ($self->{import_sto} && !$xoids) {
-		$self->{import_sto}->ipc_do('add_eml', $eml);
+		my $res = $self->{import_sto}->ipc_do('add_eml', $eml);
+		if (ref($res) eq ref($smsg)) { # totally new message
+			$smsg = $res;
+			$smsg->{kw} = []; # short-circuit xsmsg_vmd
+		}
 	}
-	my $smsg = bless {}, 'PublicInbox::Smsg';
-	$smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+	$smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
 				: git_sha(1, $eml)->hexdigest;
 	_smsg_fill($smsg, $eml);
 	wait_startq($lei);

  parent reply	other threads:[~2021-08-14  0:29 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-14  0:29 [PATCH 0/3] lei: hopefully kill /Document \d+ not found/ errors Eric Wong
2021-08-14  0:29 ` [PATCH 1/3] lei: diagnostics for " Eric Wong
2021-08-14  0:29 ` Eric Wong [this message]
2021-08-14  0:29 ` [PATCH 3/3] lei: hexdigest mocks account for unwanted headers Eric Wong
2021-08-24 20:14 ` [PATCH 0/3] lei: hopefully^W kill /Document \d+ not found/ errors Eric Wong
2021-10-14  5:31   ` Eric Wong
2021-10-15  9:52     ` [PATCH] lei q: avoid kw lookup failure on remote mboxrd Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210814002944.5126-3-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).