Re: RFC: monthly epochs for v2

[ebiederm@xmission.com (Eric W. Biederman)]

Eric Wong <e@80x24.org> writes:

> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
>> On Fri, Oct 25, 2019 at 12:22:14PM +0000, Eric Wong wrote:
>> > > I'm not sure about a libpublicinbox... I have been really
>> > > hesitant to depend on shared C/C++ libraries whenever I use Perl
>> > > or Ruby because of build and install complexity; especially for
>> > > stuff that's not-yet-available on distros.
>> > > 
>> > > Well-defined and stable protocols + data formats?
>> > > Yes. 100 times yes.
>> > > 
>> > > What would be nice is to have a local server so they could
>> > > access everything via HTTP using curl or whatever HTTP library
>> > > users want.  On shared systems, it could be HTTP over a UNIX
>> > > socket.  I don't think libcurl supports Unix domain sockets,
>> > > yet, but HTTP/1.1 parsers are pretty common.
>> > > 
>> > > JSON is a possibility, too; but I'm not sure if JSON is even
>> > > necessary if all that's exchanged are git blob OIDs and URLs for
>> > > mboxes.  Parsing MIME + RFC822(-ish) are already sunk costs.
>> > 
>> > More on that. As much as I may be in favor of "software freedom",
>> > I'm even more in favor of "freedom _from_ software".  Reusing
>> > existing data formats as much as possible to minimize the
>> > bug and attack surface is something I've been trying to do.
>> 
>> I understand the sentiment, but it's the exact problem that kernel
>> maintainers are struggling with. Almost every maintainer I've spoken to have
>> complained that without dedicated tools automating a lot of tasks for them,
>> they quickly run out of scaling capacity. In fact, most of the ones I spoke
>> with have rigged up some kind of fragile solution that sort-of works for
>> them, often involving spittle and baling wire (someone I know runs patchwork
>> in a container). After they set it up, they are hesitant to touch it, which
>> means they don't want to perform system or library upgrades for the fear
>> that something may break at the worst possible time during the merge window.
>> Which means they are potentially leaving their systems exposed by not
>> applying security updates.
>> 
>> It's little wonder why many are clamoring for a centralized forge solution
>> that would put the responsibility of maintaining things on someone else's
>> shoulders.
>
> Yeah.  I've tried to make public-inbox somewhat easy-to-install
> compared to typical web apps, at least on Debian-based systems.
> I guess the CentOS7 experience is acceptable, but maybe less than
> ideal due to the lack of Search::Xapian.
>
> Not sure what other distros and dependencies we'd have to worry
> about with less package availability than CentOS/RHEL.
>
>> If we want to avoid this, we need to provide them with convenient and robust
>> tools that they can use and adapt to their needs. Otherwise we aren't really
>> solving the problem.
>
> Right.  Much of the public-inbox search and blob-solver logic
> can be adapted to command-line tools (as they are for testing)
> and/or exposed locally via a git-instaweb-style HTTP server
> which can be curl-ed.  But ALSO hosting it on a giant server is
> an option for organizations that want such things.
>
> Maybe some tooling can be piggy-backed into git.git or its
> contrib/, section, too.  I certainly want to make git operation
> totally network transparent, at least.
>
>> (I know this really belongs on workflows more than on meta.)
>
> I posted a little bit more about local tools, here:
>   https://lore.kernel.org/workflows/20191025223915.GA22959@dcvr/
> (but I've posted to similar effect here, I think)

So not monthly epochs.  But it would be very handing to have a
public-inbox command command that refreshes git mirrors.  It would
be even more awesome if there was something like the IMAP IDLE command
in http that would let a process block until an update happened and
then fetch the updated data.

ssoma had some of that.

I have a very rough script that works, but I periodically need to find
the new epochs start a new mirror by hand.  So it would be nice if we
could get a more polished tool that I could just tell it mirror this
mailing list.  Or mirror all of the mailling lists on lore.

I don't think the case of optimizing for people who have a one time use
and want a small download in the git case makes sense.  As history tends
to be useful, and having people mirror things tends to be useful, and
the sizes we are talking about is comparatively small.

For distributed use I generally think make it cheap enough that people
don't have to optimize their current setup.  Which is roughly the policy
that git uses and it has worked.

Eric