user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* lei: accessing blob after import requires daemon restart
@ 2021-02-22  5:37 Kyle Meyer
  2021-02-22  6:18 ` [PATCH] lei_store: populate ALL.git/alternates with new epochs Eric Wong
  0 siblings, 1 reply; 3+ messages in thread
From: Kyle Meyer @ 2021-02-22  5:37 UTC (permalink / raw)
  To: meta

When playing around with lei-import, I was unable to display an mbox for
a just-imported message:

  # starting in an uninitialized state with no externals
  $ curl -d'' -fsS 'https://public-inbox.org/meta/?q=cgit+blob+solver&x=m' \
    | zcat >t.mbox
  $ lei import mboxrd:t.mbox
  $ lei q -f ldjson s:remove
  {"blob":"089cdf1af1d738068494a653532ed01b1844407d","docid":11,...}
  $ lei q -f mboxrd s:remove
  missing 089cdf1af1d738068494a653532ed01b1844407d

It's in the local repo though:

  $ git -C ~/.local/share/lei/store/local/0.git \
    rev-parse --verify 089cdf1af1d738068494a653532ed01b1844407d^{blob}
  089cdf1af1d738068494a653532ed01b1844407d

Killing the daemon and trying again resolves the issue:

  $ lei daemon-kill
  $ lei q -f mboxrd s:remove | head -1
  From 089cdf1af1d738068494a653532ed01b1844407d=99@mboxrd Thu Jan  1 00:00:00 1970

Sorry if I'm reporting a known to-do; with a quick search, I didn't spot
anything on the list or in the code, and it feels like enough of a
corner case to be worth mentioning.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH] lei_store: populate ALL.git/alternates with new epochs
  2021-02-22  5:37 lei: accessing blob after import requires daemon restart Kyle Meyer
@ 2021-02-22  6:18 ` Eric Wong
  2021-02-22 22:47   ` Kyle Meyer
  0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2021-02-22  6:18 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: meta

Kyle Meyer <kyle@kyleam.com> wrote:
> Sorry if I'm reporting a known to-do; with a quick search, I didn't spot
> anything on the list or in the code, and it feels like enough of a
> corner case to be worth mentioning.

Yikes :x  I definitely didn't notice since I'm constantly
restarting the daemon, and the test was insufficient.
Thanks for the bug report, I believe this fixes it:

-----8<-----
Subject: [PATCH] lei_store: populate ALL.git/alternates with new epochs

Since eidx_init updates ALL.git/objects/info/alternates, we need
to ensure new epochs we create from LeiStore->importer exist
before eidx_init writes alternates.

Reported-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/8735xou0gq.fsf@kyleam.com/
---
 lib/PublicInbox/LeiStore.pm |  4 ++--
 t/lei-import.t              | 13 +++++++++++++
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/LeiStore.pm b/lib/PublicInbox/LeiStore.pm
index 546d500b..77601828 100644
--- a/lib/PublicInbox/LeiStore.pm
+++ b/lib/PublicInbox/LeiStore.pm
@@ -192,10 +192,10 @@ sub maildir_keywords {
 
 sub add_eml {
 	my ($self, $eml, @kw) = @_;
-	my $eidx = eidx_init($self);
+	my $im = $self->importer; # may create new epoch
+	my $eidx = eidx_init($self); # writes ALL.git/objects/info/alternates
 	my $oidx = $eidx->{oidx};
 	my $smsg = bless { -oidx => $oidx }, 'PublicInbox::Smsg';
-	my $im = $self->importer;
 	$im->add($eml, undef, $smsg) or return; # duplicate returns undef
 
 	local $self->{current_info} = $smsg->{blob};
diff --git a/t/lei-import.t b/t/lei-import.t
index 46747a91..fa4fc504 100644
--- a/t/lei-import.t
+++ b/t/lei-import.t
@@ -13,6 +13,19 @@ lei_ok([qw(import -f eml -)], undef, { %$lei_opt, 0 => $fh },
 	\'import single file from stdin') or diag $lei_err;
 close $fh;
 lei_ok(qw(q s:boolean), \'search hit after import');
+lei_ok(qw(q s:boolean -f mboxrd), \'blob accessible after import');
+{
+	my $expect = [ eml_load('t/data/0001.patch') ];
+	require PublicInbox::MboxReader;
+	my @cmp;
+	open my $fh, '<', \$lei_out or BAIL_OUT "open :scalar: $!";
+	PublicInbox::MboxReader->mboxrd($fh, sub {
+		my ($eml) = @_;
+		$eml->header_set('Status');
+		push @cmp, $eml;
+	});
+	is_deeply(\@cmp, $expect, 'got expected message in mboxrd');
+}
 lei_ok(qw(import -f eml), 't/data/message_embed.eml',
 	\'import single file by path');
 

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] lei_store: populate ALL.git/alternates with new epochs
  2021-02-22  6:18 ` [PATCH] lei_store: populate ALL.git/alternates with new epochs Eric Wong
@ 2021-02-22 22:47   ` Kyle Meyer
  0 siblings, 0 replies; 3+ messages in thread
From: Kyle Meyer @ 2021-02-22 22:47 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong writes:

> Thanks for the bug report, I believe this fixes it:
>
> -----8<-----
> Subject: [PATCH] lei_store: populate ALL.git/alternates with new epochs

Confirmed.  Thanks for the very quick fix.

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2021-02-22 22:47 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-22  5:37 lei: accessing blob after import requires daemon restart Kyle Meyer
2021-02-22  6:18 ` [PATCH] lei_store: populate ALL.git/alternates with new epochs Eric Wong
2021-02-22 22:47   ` Kyle Meyer

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.mail.public-inbox.meta
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.mail.public-inbox.meta
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git