From: Jeff King <peff@peff.net>
To: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Cc: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Elijah Newren" <newren@gmail.com>,
"Git Mailing List" <git@vger.kernel.org>,
"Derrick Stolee" <stolee@gmail.com>
Subject: Re: Large delays in mailing list delivery?
Date: Fri, 3 Dec 2021 16:52:47 -0500 [thread overview]
Message-ID: <YaqRr//Yu8cn+7s7@coredump.intra.peff.net> (raw)
In-Reply-To: <20211203202427.o575sgrx4auqkmjp@meerkat.local>
On Fri, Dec 03, 2021 at 03:24:27PM -0500, Konstantin Ryabitsev wrote:
> On Fri, Dec 03, 2021 at 09:02:48PM +0100, Ævar Arnfjörð Bjarmason wrote:
> > When I've experienced delays (sometimes of half a day or more) both
> > https://public-inbox.org/git/ and https://lore.kernel.org/git/ have been
> > updated.
>
> Btw, you can source lore.kernel.org straight into your gmail inbox. :)
>
> https://people.kernel.org/monsieuricon/lore-lei-part-1-getting-started
> https://people.kernel.org/monsieuricon/lore-lei-part-2-now-with-imap
>
> Or, you can read it via nntp://nntp.lore.kernel.org/.
I've been watching the lei stuff, and it's pretty cool. I was already
indexing my local archive with notmuch, so right now I have an "in
between" solution where I pull from lore and deliver into a local
mailbox, like:
-- >8 --
ROOT=$HOME/.cache/lists
test -n "$QUIET" && exec >/dev/null
test $# = 0 && set -- $(cd "$ROOT" && echo *)
for list in "$@"; do
cd "$ROOT/$list" || exit 1
git fetch -v ${QUIET:+--quiet} || exit 1
git rev-list refs/lists/delivered..HEAD |
git diff-tree --format= --stdin --raw |
awk '{print $4}' |
while read blob; do
test -n "$QUIET" || echo >&2 "Delivering $blob..."
git cat-file blob "$blob" |
safecat maildir/tmp maildir/new ||
exit 1
done || exit 1
git update-ref refs/lists/delivered HEAD || exit 1
done
-- >8 --
Some notes:
- ~/.cache/lists/git is a bare clone of https://lore.kernel.org/git/0;
I know this will run into problems if we eventually get enough
messages to start a new epoch, but that's still years away by the
current counting.
- maildir in the bare repo is a symlink to the actual maildir I
deliver to (~/mail/git)
- I use safecat here to deliver into the maildir, but notmuch-insert
would probably make more sense.
I think this is less featureful than lei (especially some of the
advanced queries), but it was a drop-in replacement for my existing
queries and workflows. And it has low dependencies.
Of course it doesn't help much if you're using gmail or something. :)
I guess you could replace the safecat delivery with git-imap-send or
similar.
It's polling, of course, but I assume that a noop fetch against the lore
repo is pretty cheap. I think Eric's public-inbox/lei code for doing
updates has an extra HTTP-endpoint check to avoid invoking even the noop
Git (though it results in an extra HTTP request when there _is_
something to fetch).
And of course it's holding two copies of the messages (one in Git, and
then the delivered one). That's OK for my purposes, but I have noticed
that lei is generally much faster to answer queries, because maildirs
have awful cold-cache behavior because of all the inodes (and my
mailserver is still on spinning disks).
So I don't really recommend anybody going down my same path if they
could just jump to using lei. But I thought the script above might help
somebody who wants to just replace one small bit of their
infrastructure/workflow without retraining fingers, etc.
-Peff
next prev parent reply other threads:[~2021-12-03 21:52 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-12-03 19:52 Large delays in mailing list delivery? Elijah Newren
2021-12-03 19:58 ` Konstantin Ryabitsev
2021-12-04 16:51 ` Thomas Guyot-Sionnest
2021-12-03 20:02 ` Ævar Arnfjörð Bjarmason
2021-12-03 20:24 ` Konstantin Ryabitsev
2021-12-03 20:26 ` Ævar Arnfjörð Bjarmason
2021-12-03 21:52 ` Jeff King [this message]
2021-12-06 16:12 ` Ævar Arnfjörð Bjarmason
2021-12-06 16:36 ` Eric Wong
2022-02-02 9:34 ` Using public-inbox+lei+Emacs+mu+mu4e (was: Large delays in mailing list delivery?) Ævar Arnfjörð Bjarmason
2022-02-07 21:27 ` Eric Wong
2021-12-07 4:20 ` Large delays in mailing list delivery? Ævar Arnfjörð Bjarmason
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YaqRr//Yu8cn+7s7@coredump.intra.peff.net \
--to=peff@peff.net \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=konstantin@linuxfoundation.org \
--cc=newren@gmail.com \
--cc=stolee@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).