Large delays in mailing list delivery?

Large delays in mailing list delivery?

[Elijah Newren]

Are there some rather large delays in mailing list delivery these
days?  Anyone know who to contact to investigate?  [*]

Some examples:

* Stolee reviewed v2 of en/keep-cwd on Nov 29, three days after v3 was
posted, and I mentioned v3 was current, he said that v3 had not yet
appeared in his inbox[2] (though by the time of that response, half of
it had appeared).

I periodically look at lore.kernel.org/git because of the threaded
view.  While I'm accustomed to a few hours delay between messages
arriving there and in my inbox, this week I noticed a few, among them
two examples that I immediately remember:

* https://lore.kernel.org/git/20211202030458.GA48278@newk/ from Dec 1
still hasn't arrived in my inbox.  I typed up a response, but waited
for it to show up so I could respond.  (I could use git-send-email to
respond, but that's a pain so I tend not to.)

* v7 of ak/protect-any-current-branch has not arrived in my inbox yet,
despite being posted about 2 days ago.  I read through the series and
then was going to post a tiny comment or two yesterday, but only v6
and older are in my email.


Thanks,
Elijah

[1] And what are the odds that this issue will cause this email to be
delayed in delivery to everyone other than lore.kernel.org/git, and no
one sees it, until after the issue is resolved?  ;-)

[2] https://lore.kernel.org/git/068b7faf-2ade-14a7-fee3-83fec26ae856@gmail.com/

Re: Large delays in mailing list delivery?

[Konstantin Ryabitsev]

On Fri, Dec 03, 2021 at 11:52:58AM -0800, Elijah Newren wrote:
> Are there some rather large delays in mailing list delivery these
> days?  Anyone know who to contact to investigate?  [*]

The right person to contact is postmaster@vger.kernel.org, however I can
actually answer your question (despite not actually being in charge of
vger.kernel.org). Periodically, Gmail decides that there's just too much
incoming mail for some accounts and will arbitrarily delay delivery by
returning a "this account is receiving too much mail, please try later."

I am willing to bet that this is what happened to you.

Best regards,
-K

Re: Large delays in mailing list delivery?

[Ævar Arnfjörð Bjarmason]

On Fri, 3 Dec 2021 11:52:58 -0800 Elijah Newren wrote:

> Are there some rather large delays in mailing list delivery these
> days?  Anyone know who to contact to investigate?  [*]

I've got E-Mail delays so large that I manually crafted In-Reply-To
etc. to reply to this :)

I'm 99% sure it's some weird thing at GMail, and nothing to do with
kernel.org.

When I've experienced delays (sometimes of half a day or more) both
https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
updated.

Konstantin Ryabitsev notes (and would be in a position to know) that
GMail throttles delivery (with an smtp 451 error?).

But the other day when Junio sent out a What's Cooking and I saw it on
lore, but it wasn't in my mailbox for some hours. That time I looked a
bi tinto it.

I went into the gmail UI and searched for rfc822msgid:$id, nothing. It
was neither there on IMAP or in the UI.

However when the E-Mail finally arrived I looked at the raw headers, and
all the "Received" headers (including GMail's own internal routing) were
within a couple of minutes of Junio having sent the mail (and in the
meantime it went through vger etc.).

So, I'm hazy on E-Mail infrastructure details these days (but worked no
it in a past life), but that really seems to me like GMail in fact got
the E-Mail, but it was just sitting in some local queue of theirs before
it got served to me.

Right now I can't see the mail I'm replying to in my inbox[1], but I can
report the full headers once it arrives if that helps.

1. A search for:
   rfc822msgid:CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com

Re: Large delays in mailing list delivery?

[Konstantin Ryabitsev]

On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> When I've experienced delays (sometimes of half a day or more) both
> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> updated.

Btw, you can source lore.kernel.org straight into your gmail inbox. :)

    https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
    https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap

Or, you can read it via nntp://nntp.lore.kernel.org/.

Best regards,
-K

Re: Large delays in mailing list delivery?

[Ævar Arnfjörð Bjarmason]

On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:

> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> When I've experienced delays (sometimes of half a day or more) both
>> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
>> updated.
>
> Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>
>     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>
> Or, you can read it via nntp://nntp.lore.kernel.org/.

Nice. I was looking into doing that the other day but stopped at not
finding a way to do that with the public-inbox software itself.

Well, "the other day" was probably ~6 months ago, but at the time I
could only find some TODO item for a Maildir export. Looks like that
works now, great!

Re: Large delays in mailing list delivery?

[Jeff King]

On Fri, Dec 03, 2021 at 03:24:27PM -0500, Konstantin Ryabitsev wrote:

> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> > When I've experienced delays (sometimes of half a day or more) both
> > https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> > updated.
> 
> Btw, you can source lore.kernel.org straight into your gmail inbox. :)
> 
>     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> 
> Or, you can read it via nntp://nntp.lore.kernel.org/.

I've been watching the lei stuff, and it's pretty cool. I was already
indexing my local archive with notmuch, so right now I have an "in
between" solution where I pull from lore and deliver into a local
mailbox, like:

-- >8 --
ROOT=$HOME/.cache/lists

test -n "$QUIET" && exec >/dev/null
test $# = 0 && set -- $(cd "$ROOT" && echo *)

for list in "$@"; do
	cd "$ROOT/$list" || exit 1
	git fetch -v ${QUIET:+--quiet} || exit 1
	git rev-list refs/lists/delivered..HEAD |
	git diff-tree --format= --stdin --raw |
	awk '{print $4}' |
	while read blob; do
		test -n "$QUIET" || echo >&2 "Delivering $blob..."
		git cat-file blob "$blob" |
		safecat maildir/tmp maildir/new ||
		exit 1
	done || exit 1
	git update-ref refs/lists/delivered HEAD || exit 1
done
-- >8 --

Some notes:

  - ~/.cache/lists/git is a bare clone of https://lore.kernel.org/git/0;
    I know this will run into problems if we eventually get enough
    messages to start a new epoch, but that's still years away by the
    current counting.

  - maildir in the bare repo is a symlink to the actual maildir I
    deliver to (~/mail/git)

  - I use safecat here to deliver into the maildir, but notmuch-insert
    would probably make more sense.

I think this is less featureful than lei (especially some of the
advanced queries), but it was a drop-in replacement for my existing
queries and workflows. And it has low dependencies.

Of course it doesn't help much if you're using gmail or something. :)
I guess you could replace the safecat delivery with git-imap-send or
similar.

It's polling, of course, but I assume that a noop fetch against the lore
repo is pretty cheap. I think Eric's public-inbox/lei code for doing
updates has an extra HTTP-endpoint check to avoid invoking even the noop
Git (though it results in an extra HTTP request when there _is_
something to fetch).

And of course it's holding two copies of the messages (one in Git, and
then the delivered one). That's OK for my purposes, but I have noticed
that lei is generally much faster to answer queries, because maildirs
have awful cold-cache behavior because of all the inodes (and my
mailserver is still on spinning disks).

So I don't really recommend anybody going down my same path if they
could just jump to using lei. But I thought the script above might help
somebody who wants to just replace one small bit of their
infrastructure/workflow without retraining fingers, etc.

-Peff

Re: Large delays in mailing list delivery?

[Thomas Guyot-Sionnest]

On 2021-12-03 14:58, Konstantin Ryabitsev wrote:
> On Fri, Dec 03, 2021 at 11:52:58AM -0800, Elijah Newren wrote:
>> Are there some rather large delays in mailing list delivery these
>> days?  Anyone know who to contact to investigate?  [*]
> The right person to contact is postmaster@vger.kernel.org, however I can
> actually answer your question (despite not actually being in charge of
> vger.kernel.org). Periodically, Gmail decides that there's just too much
> incoming mail for some accounts and will arbitrarily delay delivery by
> returning a "this account is receiving too much mail, please try later."
>
> I am willing to bet that this is what happened to you.

I have already contacted postmaster and they are aware, however you add
very good point. Somehow I knew it but it's the way you put it that made
me realize: the emails are sent in bulk to gmail (to many or all
recipients for a single copy at once). The mail header even shows this,
listing the first recipient followed by "+ 99 others". If it's delayed
for all because of a single recipient that can only be because the 4xx
error is returned at the end of the DATA command (rather than on a
specific RCPT TO command), so no way for the sender to retry that single
faulty address alone later.

I'm not sure if their software allows it but I suggested sending to a
single recipient at a time for gmail, that would solve the issue.

In the meantime what I did is switch my subscription to another email
using a forwarder address on a domain I own - I still haven't received
all the backlog but at least I'm getting new posts timely now.

Regards,

--
Thomas

Re: Large delays in mailing list delivery?

[Ævar Arnfjörð Bjarmason]

On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:

> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> When I've experienced delays (sometimes of half a day or more) both
>> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
>> updated.
>
> Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>
>     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>
> Or, you can read it via nntp://nntp.lore.kernel.org/.

[CC'd meta@public-inbox.org, probably best to move this thread over
there sooner than later, but CC'ing git@ still in case this is
interesting to others]

I poked a bit at setting this up but couldn't find from building
public-inbox.org & trying to page through the docs how I'd get from an
existing public-inbox.org/git/ checkout to a local Maildir.

If you could share some recipe or a pointer to the right docs for that
that would be much appreciated. Thanks!

Re: Large delays in mailing list delivery?

[Eric Wong]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
> 
> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> When I've experienced delays (sometimes of half a day or more) both
> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> >> updated.
> >
> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
> >
> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> >
> > Or, you can read it via nntp://nntp.lore.kernel.org/.
> 
> [CC'd meta@public-inbox.org, probably best to move this thread over
> there sooner than later, but CC'ing git@ still in case this is
> interesting to others]
> 
> I poked a bit at setting this up but couldn't find from building
> public-inbox.org & trying to page through the docs how I'd get from an
> existing public-inbox.org/git/ checkout to a local Maildir.

Existing, public-inboxes can be set as "externals" and managed
via {add,forget,ls}-external sub-commands:

	# for locally-cloned inboxes:
	public-inbox-index /path/to/existing/inbox
	lei add-external /path/to/existing/inbox

	# relies on curl, memoizes data downloaded for each search:
	lei add-external https://yhbt.net/lore/git

Local externals will be included by every "lei q" invocation;
HTTP(S) ones require "lei q --remote"

If you only want to use an external as a one-off without adding
it, the -I/--include and -O/--only flags are useful:

  lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS

> If you could share some recipe or a pointer to the right docs for that
> that would be much appreciated. Thanks!

lei-overview(7) manpage documents some things, at least:
https://public-inbox.org/lei-overview.html  Patches welcome :>

IMHO lei still kinda sucks, and I probably won't have time to
work on it for a bit :<

Re: Large delays in mailing list delivery?

[Ævar Arnfjörð Bjarmason]

On Fri, Dec 03 2021, Ævar Arnfjörð Bjarmason wrote:

> On Fri, 3 Dec 2021 11:52:58 -0800 Elijah Newren wrote:
>
>> Are there some rather large delays in mailing list delivery these
>> days?  Anyone know who to contact to investigate?  [*]
>
> I've got E-Mail delays so large that I manually crafted In-Reply-To
> etc. to reply to this :)
>
> I'm 99% sure it's some weird thing at GMail, and nothing to do with
> kernel.org.
>
> When I've experienced delays (sometimes of half a day or more) both
> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> updated.
>
> Konstantin Ryabitsev notes (and would be in a position to know) that
> GMail throttles delivery (with an smtp 451 error?).
>
> But the other day when Junio sent out a What's Cooking and I saw it on
> lore, but it wasn't in my mailbox for some hours. That time I looked a
> bi tinto it.
>
> I went into the gmail UI and searched for rfc822msgid:$id, nothing. It
> was neither there on IMAP or in the UI.
>
> However when the E-Mail finally arrived I looked at the raw headers, and
> all the "Received" headers (including GMail's own internal routing) were
> within a couple of minutes of Junio having sent the mail (and in the
> meantime it went through vger etc.).
>
> So, I'm hazy on E-Mail infrastructure details these days (but worked no
> it in a past life), but that really seems to me like GMail in fact got
> the E-Mail, but it was just sitting in some local queue of theirs before
> it got served to me.
>
> Right now I can't see the mail I'm replying to in my inbox[1], but I can
> report the full headers once it arrives if that helps.
>
> 1. A search for:
>    rfc822msgid:CABPp-BF_xsOpQ6GSaWs9u9JcnPQT_OXP-gCsAuxPtMj-X1tgOg@mail.gmail.com

For what it's worth after it finally arrived the relevant part of the
headers is, which shows the issue Konstantin Ryabitsev
described. I.e. it was sitting for ~3 days between vger and GMail:
    
    [...]
    Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18])
            by mx.google.com with ESMTP id g10si21274138pfj.188.2021.12.06.17.50.50;
            Mon, 06 Dec 2021 17:51:02 -0800 (PST)
    Received-SPF: pass (google.com: domain of git-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18;
    Authentication-Results: mx.google.com;
           dkim=pass header.i=@gmail.com header.s=20210112 header.b=V2iU5Wyg;
           spf=pass (google.com: domain of git-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=git-owner@vger.kernel.org;
           dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com
    Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
            id S1343798AbhLCT4g (ORCPT <rfc822;bojan.kljakic.tech@gmail.com>
            + 99 others); Fri, 3 Dec 2021 14:56:36 -0500
    [...]

I.e. not at all what I alluded to above, and at this point I'm not sure
I ever really saw a mail like that (maybe I just misread the headers at
the time + confirmation bias, and I didn't go re-digging again now...).

Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 06 2021, Eric Wong wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
>> 
>> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> >> When I've experienced delays (sometimes of half a day or more) both
>> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
>> >> updated.
>> >
>> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>> >
>> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
>> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>> >
>> > Or, you can read it via nntp://nntp.lore.kernel.org/.
>> 
>> [CC'd meta@public-inbox.org, probably best to move this thread over
>> there sooner than later, but CC'ing git@ still in case this is
>> interesting to others]
>> 
>> I poked a bit at setting this up but couldn't find from building
>> public-inbox.org & trying to page through the docs how I'd get from an
>> existing public-inbox.org/git/ checkout to a local Maildir.
>
> Existing, public-inboxes can be set as "externals" and managed
> via {add,forget,ls}-external sub-commands:
>
> 	# for locally-cloned inboxes:
> 	public-inbox-index /path/to/existing/inbox
> 	lei add-external /path/to/existing/inbox
>
> 	# relies on curl, memoizes data downloaded for each search:
> 	lei add-external https://yhbt.net/lore/git
>
> Local externals will be included by every "lei q" invocation;
> HTTP(S) ones require "lei q --remote"
>
> If you only want to use an external as a one-off without adding
> it, the -I/--include and -O/--only flags are useful:
>
>   lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
>
>> If you could share some recipe or a pointer to the right docs for that
>> that would be much appreciated. Thanks!
>
> lei-overview(7) manpage documents some things, at least:
> https://public-inbox.org/lei-overview.html  Patches welcome :>
>
> IMHO lei still kinda sucks, and I probably won't have time to
> work on it for a bit :<

Thanks. I finally got around to setting this up.

The above instructions didn't quite work for me, but here's what I did
(in the form of a script lifted from a screen(1) config I've
got). Indented with un-indented comments. The "stuff" is screen's way of
"run this command" (or well, input these characters):

I had a ~/g/git-ml clone already, but this makes one:
    
    stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"

The initial index:

    ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
    ## the first time.
    stuff "time public-inbox-index -v \$PWD^M"

I fiddled with this for a bit because it refused to work, turns out it
was missing the .git at the end, i.e. it expected a bare repo[1]:

    ## When we add the lei external it *must* have the ".git" part,
    ## because it'll try to find the "public-inbox" folder at wherever we
    ## point it.
    stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"

A bit of a UX wart not to be able to specify no --limit, or maybe I'm
missing a way:

    ## The one-off massive import of the Git ML. TODO: No way to specify
    ## an infinite limit? Not --no-limit or --limit=0.
    stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"

The initial indexing:

    ## After the one-off import this will take forever *the first time*
    ## (or around 20m), but subsequent invocations will be fast:
    stuff "time lei up ~/Maildir/lei-q-git-ml^M"

Runs an ad-hoc script to keep it up-to-date, which is quoted below:

    ## Run it in a loop
    stuff "public-inbox-lei-pull-index^M"

That script (which I whipped up just now. Is there a better/more
standard way? to keep a public-inbox+lei pair up-to-date with
sleep/backoff etc?
	
	#!/bin/sh
	set -xe
	
	repo=~/g/git-ml
	while true
	do
		oid=$(git -C $repo rev-parse HEAD)
		git -C $repo pull
		noid=$(git -C $repo rev-parse HEAD)
		if test "$oid" = "$noid"
		then
			echo Nothing to update
			sleep 60
			continue
		fi
		(
			cd $repo &&
			public-inbox-index -v "$PWD"
		)
		lei up ~/Maildir/lei-q-git-ml
		sleep 1
	done

I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
files dropped in there will be added to its index. Then I just changed
my saved search to also look through that maildir (I guess the entire
first condition could be dropped, but whatever):

    "(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"

Because "mu" is generally good about de-duplicating stuff I've now got
an inbox with mixed messages I can sync from GMail (including my "Sent"
folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
delayed for 3-4 months at least).

So new messages are generally from the "lei" directory, but when I send
one it'll be dropped in the personal-gmail.

I still need to check if it's doing the wrong thing with e.g. "read"
flags if I read a mail synced via lei that later arrives in GMail. But I
mostly don't use "read" statuses anyway...

1.  Maybe this "I only tested if it complied" patch would make sense to catch that?

diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
index 2958d3f9..be49621f 100644
--- a/lib/PublicInbox/LeiXSearch.pm
+++ b/lib/PublicInbox/LeiXSearch.pm
@@ -613,7 +613,7 @@ sub add_uri {
 	}
 }
 
-sub prepare_external {
+sub _prepare_external {
 	my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
 	if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
 		return add_uri($self, $loc) if $loc->can('scheme');
@@ -638,6 +638,14 @@ sub prepare_external {
 	push @{$self->{locals}}, $loc;
 }
 
+sub prepare_external {
+	my ($self, $loc, $boost) = @_;
+	my $ret = _prepare_external($self, $loc, $boost);
+	warn "W: we got nothing from $loc, did you mean $loc/.git?"
+		if !$ret && -e "$loc/.git";
+	return $ret;
+}
+
 sub _lcat_i { # LeiMailSync->each_src iterator callback
 	my ($oidbin, $id, $each_smsg) = @_;
 	$each_smsg->({blob => unpack('H*', $oidbin), pct => 100});

Re: Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?)

[Eric Wong]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> On Mon, Dec 06 2021, Eric Wong wrote:
> > Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
> >> On Fri, Dec 03 2021, Konstantin Ryabitsev wrote:
> >> 
> >> > On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> >> >> When I've experienced delays (sometimes of half a day or more) both
> >> >> https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> >> >> updated.
> >> >
> >> > Btw, you can source lore.kernel.org straight into your gmail inbox. :)
> >> >
> >> >     https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> >> >     https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
> >> >
> >> > Or, you can read it via nntp://nntp.lore.kernel.org/.
> >> 
> >> [CC'd meta@public-inbox.org, probably best to move this thread over
> >> there sooner than later, but CC'ing git@ still in case this is
> >> interesting to others]
> >> 
> >> I poked a bit at setting this up but couldn't find from building
> >> public-inbox.org & trying to page through the docs how I'd get from an
> >> existing public-inbox.org/git/ checkout to a local Maildir.
> >
> > Existing, public-inboxes can be set as "externals" and managed
> > via {add,forget,ls}-external sub-commands:
> >
> > 	# for locally-cloned inboxes:
> > 	public-inbox-index /path/to/existing/inbox
> > 	lei add-external /path/to/existing/inbox
> >
> > 	# relies on curl, memoizes data downloaded for each search:
> > 	lei add-external https://yhbt.net/lore/git
> >
> > Local externals will be included by every "lei q" invocation;
> > HTTP(S) ones require "lei q --remote"
> >
> > If you only want to use an external as a one-off without adding
> > it, the -I/--include and -O/--only flags are useful:
> >
> >   lei q -O https://yhbt.net/lore/git -o /tmp/results SEARCH_TERMS
> >
> >> If you could share some recipe or a pointer to the right docs for that
> >> that would be much appreciated. Thanks!
> >
> > lei-overview(7) manpage documents some things, at least:
> > https://public-inbox.org/lei-overview.html  Patches welcome :>
> >
> > IMHO lei still kinda sucks, and I probably won't have time to
> > work on it for a bit :<
> 
> Thanks. I finally got around to setting this up.
> 
> The above instructions didn't quite work for me, but here's what I did
> (in the form of a script lifted from a screen(1) config I've
> got). Indented with un-indented comments. The "stuff" is screen's way of
> "run this command" (or well, input these characters):
> 
> I had a ~/g/git-ml clone already, but this makes one:
>     
>     stuff "cd ~/g/git-ml || git clone https://public-inbox.org/git ~/g/git-ml^M"

Wait, lack of --mirror or --bare for git-clone on any
public-inbox is a big mistake in terms of inode+disk use

> The initial index:
> 
>     ## This will create a .git/publici-inbox in ~/g/git-ml. Takes a while
>     ## the first time.
>     stuff "time public-inbox-index -v \$PWD^M"
> 
> I fiddled with this for a bit because it refused to work, turns out it
> was missing the .git at the end, i.e. it expected a bare repo[1]:
> 
>     ## When we add the lei external it *must* have the ".git" part,
>     ## because it'll try to find the "public-inbox" folder at wherever we
>     ## point it.
>     stuff "test -d ~/.config/lei || lei add-external ~/g/git-ml/.git^M"

Yeah, public-inbox has never supported non-bare usage.

> A bit of a UX wart not to be able to specify no --limit, or maybe I'm
> missing a way:
> 
>     ## The one-off massive import of the Git ML. TODO: No way to specify
>     ## an infinite limit? Not --no-limit or --limit=0.
>     stuff "test -d ~/Maildir/lei-q-git-ml || time lei q --limit=999999999 -v -o ~/Maildir/lei-q-git-ml l:git.vger.kernel.org^M"

Oops.  --no-limit or --limit=0 or --limit=-1 support will be
added at some point...

> The initial indexing:
> 
>     ## After the one-off import this will take forever *the first time*
>     ## (or around 20m), but subsequent invocations will be fast:
>     stuff "time lei up ~/Maildir/lei-q-git-ml^M"
> 
> Runs an ad-hoc script to keep it up-to-date, which is quoted below:
> 
>     ## Run it in a loop
>     stuff "public-inbox-lei-pull-index^M"
> 
> That script (which I whipped up just now. Is there a better/more
> standard way? to keep a public-inbox+lei pair up-to-date with
> sleep/backoff etc?

Not yet, unfortunately.  I would like to have some sort of
long-polling support (similar to IDLE in IMAP) because I hate
sleep/backoff.  But I don't think I'm physically nor mentally
capable of doing that or much of anything, anymore.

> 	#!/bin/sh
> 	set -xe
> 	
> 	repo=~/g/git-ml
> 	while true
> 	do
> 		oid=$(git -C $repo rev-parse HEAD)
> 		git -C $repo pull
> 		noid=$(git -C $repo rev-parse HEAD)
> 		if test "$oid" = "$noid"
> 		then
> 			echo Nothing to update
> 			sleep 60
> 			continue
> 		fi
> 		(
> 			cd $repo &&
> 			public-inbox-index -v "$PWD"
> 		)
> 		lei up ~/Maildir/lei-q-git-ml
> 		sleep 1
> 	done
> 
> I use Emacs+mu4e for my E-Mail. And since I index ~/Maildir having these
> files dropped in there will be added to its index. Then I just changed
> my saved search to also look through that maildir (I guess the entire
> first condition could be dropped, but whatever):
> 
>     "(maildir:/personal-gmail/* OR maildir:/lei-q-git-ml/*) AND list:git.vger.kernel.org OR recip:git@vger.kernel.org OR recip:git-packagers@googlegroups.com"
> 
> Because "mu" is generally good about de-duplicating stuff I've now got
> an inbox with mixed messages I can sync from GMail (including my "Sent"
> folder), and I get up-to-the-minute ML traffic now (it's bee 10hrs-4day
> delayed for 3-4 months at least).
> 
> So new messages are generally from the "lei" directory, but when I send
> one it'll be dropped in the personal-gmail.
> 
> I still need to check if it's doing the wrong thing with e.g. "read"
> flags if I read a mail synced via lei that later arrives in GMail. But I
> mostly don't use "read" statuses anyway...

lei can export flags from its internal DBs to IMAP via "lei export-kw"
see lei-mail-sync-overview(7) for some details, but usability is still
rough...

> 1.  Maybe this "I only tested if it complied" patch would make sense to catch that?
> 
> diff --git a/lib/PublicInbox/LeiXSearch.pm b/lib/PublicInbox/LeiXSearch.pm
> index 2958d3f9..be49621f 100644
> --- a/lib/PublicInbox/LeiXSearch.pm
> +++ b/lib/PublicInbox/LeiXSearch.pm
> @@ -613,7 +613,7 @@ sub add_uri {
>  	}
>  }
>  
> -sub prepare_external {
> +sub _prepare_external {
>  	my ($self, $loc, $boost) = @_; # n.b. already ordered by boost
>  	if (ref $loc) { # already a URI, or PublicInbox::Inbox-like object
>  		return add_uri($self, $loc) if $loc->can('scheme');
> @@ -638,6 +638,14 @@ sub prepare_external {
>  	push @{$self->{locals}}, $loc;
>  }
>  
> +sub prepare_external {
> +	my ($self, $loc, $boost) = @_;
> +	my $ret = _prepare_external($self, $loc, $boost);
> +	warn "W: we got nothing from $loc, did you mean $loc/.git?"
> +		if !$ret && -e "$loc/.git";
> +	return $ret;
> +}

Probably add a note saying non-bare repos are a terrible idea,
anyways, especially for v1 public-inboxes like public-inbox.org/git