Re: RFC: monthly epochs for v2

[Eric Wong <e@80x24.org>]

Eric Wong <e@80x24.org> wrote:
> Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> > On Thu, Oct 24, 2019 at 08:35:03PM +0000, Eric Wong wrote:
> > > > - if someone is only interested in a few months worth of archives, they
> > > > don't have to clone the entire collection
> > > > - similarly, someone using public-inbox to feed messages to their inbox
> > > > (e.g. using the l2md tool [1]) doesn't need to waste gigabytes storing
> > > > archives they aren't interested in
> > > 
> > > NNTP or d:YYYYMMDD..YYYYMMDD mboxrd downloads via HTTP search
> > > are better suited for those cases.
> > 
> > I know you really like nntp, but I'm worried that with Big Corp's love of
> > deep packet inspection and filtering, NNTP ports aren't going to be usable
> > by a large subset of developers. We already have enough problems with port
> > 9418 not being reachable (and sometimes not even port 22).  Since usenet's
> > descent into mostly illegal content, many corporate environments probably
> > have ports 119 and 563 blocked off entirely and changing that would be
> > futile.
> 
> I would consider the possibility of an HTTP API which looks like
> NNTP commands, too.  But it wouldn't work with existing NNTP
> clients...  Maybe websockets can be used *shrug*

I forget, HTTP CONNECT exists for tunneling anything off
HTTP/HTTPS, too; so generic LD_PRELOAD like proxychains should
work with most NNTP clients/libraries.

Net::NNTP is in every Perl5 install I know of, and every nearly
hacker's *nix system has Perl5, but proxychains isn't ubiquitous
(Debian packages it, at least)

> NNTP can also run off 80/443 if somebody has an extra IP.  Not
> sure if supporting HTTP and NNTP off the same port is a
> possibility since some HTTP clients pre-connect TCP and NNTP is
> server-talks-first whereas HTTP is client-talks-first.


> > > If people only want a backup via git (and not host HTTP/NNTP),
> > > it's FAR easier for them to run ubiquitous commands such as
> > > "git clone --mirror && git fetch" rather than
> > > "install $TOOL which may be out-of-date-or-missing-on-your-distro"
> > 
> > I think that anyone who is likely to use public-inbox repositories for more
> > than just a copy of archives is likely to be using some kind of tool. I
> > mean, SMTP can be used with "telnet" but nobody really does. :) If we
> > provide a convenient library that supports things like intelligent selective
> > cloning, indexing, fetching messages, etc, then that would avoid everyone
> > doing it badly. In fact, libpublicinbox and bindings to most common
> > languages is probably something that should happen early on.
> 
> I'm not sure about a libpublicinbox... I have been really
> hesitant to depend on shared C/C++ libraries whenever I use Perl
> or Ruby because of build and install complexity; especially for
> stuff that's not-yet-available on distros.
> 
> Well-defined and stable protocols + data formats?
> Yes. 100 times yes.
> 
> What would be nice is to have a local server so they could
> access everything via HTTP using curl or whatever HTTP library
> users want.  On shared systems, it could be HTTP over a UNIX
> socket.  I don't think libcurl supports Unix domain sockets,
> yet, but HTTP/1.1 parsers are pretty common.
> 
> JSON is a possibility, too; but I'm not sure if JSON is even
> necessary if all that's exchanged are git blob OIDs and URLs for
> mboxes.  Parsing MIME + RFC822(-ish) are already sunk costs.

More on that. As much as I may be in favor of "software freedom",
I'm even more in favor of "freedom _from_ software".  Reusing
existing data formats as much as possible to minimize the
bug and attack surface is something I've been trying to do.