Re: RFC: Using public-inbox v2 repos for distributed patch lifecycle tracking

[Eric Wong <e@80x24.org>]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> Hi, all:
> 
> Here's something I've been mulling over for a while, and sorta goes
> hand-in-hand with what Eric has been doing with diff highlights work.
> We have significant duplication of functionality on lore.kernel.org
> between the public-inbox repository for LKML and the patchwork
> instance for the same list (https://lore.kernel.org/patchwork). I am
> currently working on a tool that would move a lot of patchwork's
> functionality to the developers' workstation by using the public-inbox
> archive of the mailing list that receives patches. After it is done,
> the tool should be able to do all of the following:
> 
> - Keep track of patches and patch series, including series revisions
> - Allow developers to easily apply patch series directly from the
> public-inbox repository (by creating a mbox file of the series behind
> the scenes with adjusted git by-llines -- e.g. if someone replies to a
> patch with a "Acked-by: Foo Dev" or "Tested-by: Foo Bot", the original
> patch trailers are adjusted to reflect that new data)
> - Automatically recognising when patches have been applied to a repo
> and auto-"accepting" them
> - Allowing developers to see changes between series using interdiff

Cool!  I haven't looked into patchwork, much, but it looks like
a centralized system that's not easily possible to replicate
to every developer system?

And your tool could be seen as a local/replicatible version of
patchwork?

> Currently, the tool sticks various patch tracking information into a
> custom sqlite3 db, but I'm increasingly wondering if it makes more
> sense to have this machine-parseable data available as part of the
> repository itself -- say, as a git note on the commit-id of the
> message. In other words, if the patch is in a message in abcd1234:m, a
> refs/notes/patches entry for abcd1234 would have json-formatted data
> with patch tracking information similar to what patchwork has for it
> (see, for example,
> https://lore.kernel.org/patchwork/api/1.1/patches/1035986/), but
> without patchwork-specific bits and duplicate info like headers and
> patch contents. If we add these notes on lore.kernel.org itself, then
> we save a lot of redundant data processing on the client end. A
> developer who has mirror-cloned a public-inbox archive of a
> development mailing list would be able to start applying patches and
> series without having to parse tens of thousands of messages.

How much of this data is reproducible/recreatable using existing
mails + git repos?  If all of it is reproducible, I don't see
storing extra/redundant data in git as necessary...

Especially since SQLite supports online backup.  The overhead for
parsing a lot of mails could be alleviated by offering DB snapshots
for download and seeding of developer machines.  Incremental updates
would be similar to public-inbox: git fetch && $FOO-index

(To that end, public-inbox could offer a way to download the msgmap and
 overview DBs, too; but offering Xapian DB downloads would be tough,
 as it doesn't offer online backups/snapshots).

> I have limited experience with notes, however, and I'm curious if they
> are a good candidate for such task. A public-inbox repo of LKML, even
> after sharding, contains hundreds of thousands of messages. If many of
> them carry such notes, would that significantly increase the
> repository size and reduce its performance?

I'm not familiar with notes, either (or I forgot what I knew).
However, I know pain points w.r.t. git performance are:

1) too many refs
2) big trees

On a quick glance git notes seems to hit #2 by having a giant
top-level tree in "refs/notes/commit".  There's been some
work in git.git over the years to fix #1 with reftable, reftrees
or lmdb; but it's not yet an accepted solution in git.git.

> I have similar thoughts about publishing CI-related information as
> notes to public-inbox commits, as well. This would allow centralised
> archival of distributed CI efforts, which is a common problem in
> distributed projects. Currently, bots flood lists with automated email
> as patch follow-ups, but if they could publish their CI information as
> refs/notes/ci/projectname in a public repository, we could mirror that
> back to lore.kernel.org via regular pulls of that ref for each defined
> projects and developers would be able to view CI reports from multiple
> distributed projects inside the same interface.

I'm thinking this can can be done with the current public-inbox using
custom search queries (or perhaps enhancements/customizations to search).

Bots would still be flooding lists with automated mails, but their
emails could be searched by patch URL/Message-ID/Subject and made into
a CI report.

> Again, I'm not sure if git notes is the right tool for this. Another
> way to go about it would be to use something like a custom blockchain
> ledger, but I'm afraid that picking that would result in lower
> adoption rate due to general unease people have about blockchain (it's
> new, it's overhyped, and it's mired by association with cryptocoins).

Just call it "git" instead of "blockchain" :)

Philosophically, I am heavily influenced by the choices git made around
rename-detection and delta-compression.  Doing these things
after-the-fact allows freedom for improvements later on as algorithms
improve.

To that end, public-inbox stores minimal information up-front and relies
heavily on search indices for after-the-fact querying.

Right now, one could search by a patch subject + "Acked-by" in
non-quoted text:

	s:"$SUBJECT" AND ( nq:Acked-by OR nq:Tested-by )

More advanced queries are possible, of course; and the way data is
indexed is not set in stone at all (we're up to "xap15", already)

> I'd love to hear your thoughts. One of my goals is to find a way to
> keep the distributed nature of Linux Kernel development without
> locking it into a single vendor (like GitHub) or a suite of tools
> (like GitLab, CircleCI, whatnot). We need to find a way to preserve
> and archive the data generated by such tools in a way that is easy to
> replicate and verify.

Good to know our goals are aligned.  Beyond the Linux kernel, my goal
since 2004(*) was to make ALL development distributed and free from
any lock-in.  Sadly, the world's gone the opposite direction :<


(*) with svn-arch-mirror and git-svn