From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 2/3] lei <q|up>: wait on remote mboxrd imports synchronously
Date: Sat, 14 Aug 2021 00:29:43 +0000 [thread overview]
Message-ID: <20210814002944.5126-3-e@80x24.org> (raw)
In-Reply-To: <20210814002944.5126-1-e@80x24.org>
This ought to avoid /Document \d+ not found/ errors from Xapian
when seeing a message for the first time by not attempting to
read keywords for totally unseen messages.
---
lib/PublicInbox/LeiRemote.pm | 7 ++++---
lib/PublicInbox/LeiStore.pm | 1 +
lib/PublicInbox/LeiXSearch.pm | 10 +++++++---
3 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/lib/PublicInbox/LeiRemote.pm b/lib/PublicInbox/LeiRemote.pm
index 945d9990..e7deecb8 100644
--- a/lib/PublicInbox/LeiRemote.pm
+++ b/lib/PublicInbox/LeiRemote.pm
@@ -26,11 +26,12 @@ sub _each_mboxrd_eml { # callback for MboxReader->mboxrd
my ($eml, $self) = @_;
my $lei = $self->{lei};
my $xoids = $lei->{ale}->xoids_for($eml, 1);
+ my $smsg = bless {}, 'PublicInbox::Smsg';
if ($lei->{sto} && !$xoids) { # memoize locally
- $lei->{sto}->ipc_do('add_eml', $eml);
+ my $res = $lei->{sto}->ipc_do('add_eml', $eml);
+ $smsg = $res if ref($res) eq ref($smsg);
}
- my $smsg = bless {}, 'PublicInbox::Smsg';
- $smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+ $smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
: git_sha(1, $eml)->hexdigest;
$smsg->populate($eml);
$smsg->{mid} //= '(none)';
diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index e26b622d..ce66014f 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -329,6 +329,7 @@ sub add_eml {
}
\@docids;
} else { # totally new message
+ delete $smsg->{-oidx}; # for IPC-friendliness
$smsg->{num} = $oidx->adj_counter('eidx_docid', '+');
$oidx->add_overview($eml, $smsg);
$oidx->add_xref3($smsg->{num}, -1, $smsg->{blob}, '.');
diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 393f25bf..971f3a06 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -266,11 +266,15 @@ sub _smsg_fill ($$) {
sub each_remote_eml { # callback for MboxReader->mboxrd
my ($eml, $self, $lei, $each_smsg) = @_;
my $xoids = $lei->{ale}->xoids_for($eml, 1);
+ my $smsg = bless {}, 'PublicInbox::Smsg';
if ($self->{import_sto} && !$xoids) {
- $self->{import_sto}->ipc_do('add_eml', $eml);
+ my $res = $self->{import_sto}->ipc_do('add_eml', $eml);
+ if (ref($res) eq ref($smsg)) { # totally new message
+ $smsg = $res;
+ $smsg->{kw} = []; # short-circuit xsmsg_vmd
+ }
}
- my $smsg = bless {}, 'PublicInbox::Smsg';
- $smsg->{blob} = $xoids ? (keys(%$xoids))[0]
+ $smsg->{blob} //= $xoids ? (keys(%$xoids))[0]
: git_sha(1, $eml)->hexdigest;
_smsg_fill($smsg, $eml);
wait_startq($lei);
next prev parent reply other threads:[~2021-08-14 0:29 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-08-14 0:29 [PATCH 0/3] lei: hopefully kill /Document \d+ not found/ errors Eric Wong
2021-08-14 0:29 ` [PATCH 1/3] lei: diagnostics for " Eric Wong
2021-08-14 0:29 ` Eric Wong [this message]
2021-08-14 0:29 ` [PATCH 3/3] lei: hexdigest mocks account for unwanted headers Eric Wong
2021-08-24 20:14 ` [PATCH 0/3] lei: hopefully^W kill /Document \d+ not found/ errors Eric Wong
2021-10-14 5:31 ` Eric Wong
2021-10-15 9:52 ` [PATCH] lei q: avoid kw lookup failure on remote mboxrd Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210814002944.5126-3-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).