Thank you for public-inbox!

Thank you for public-inbox!

[Johannes Schindelin]

Hi Eric,

I would like to take five minutes to thank you for public-inbox. It is
invaluable for me in the meantime. And I think I will never be able to
thank you enough for it.

Just a couple of things where it is super useful to me:

- Recently, my mail provider started dropping mails left and right. They
  might even be addressed to me, and they never arrive in my inbox (and
  not even in the spam folder: they just never arrive). I spent some ten
  hours this past weekend to script identifying the mails in public-inbox
  that I never received, and to weed through them.

  It seems I missed some 4,000 mails. Thanks to you, I now saw those mails
  (although I had to delete most of them after reading only their subject,
  in the interest of time).

- Many a times when my automated builds identified a problem with the test
  suite, it was a lot quicker to use public inbox to identify the mail to
  respond to than to use my mail program.

- Sometimes, I find myself in want of replying to a past patch, but back
  in the day when it was sent I thought it would not be interesting to me,
  so I deleted it. With public-inbox, I can easily get it in raw format,
  i.e. put it back into my inbox so I can reply.

- I cannot tell you how many times I send a link to public-inbox to my
  colleagues rather than forwarding a mail, because the former will give
  them more context (and also semi-live updates in case somebody replied
  to said mail after I sent the link).

A couple of things in the future that public-inbox will make possible for
me:

- You probably are aware of my GitGitGadget endeavor, a project similar in
  aim to SubmitGit, but a lot more integrated with the GitHub experience
  (and not requiring you to hand over your mail sending credentials to
  AWS).

  One particular feature I found myself really wanting in SubmitGit (but
  not possible due to its one-way design): I want my Pull Requests to be
  closed once the patches are integrated into git.git's `master` branch.

  While this feature is not available in GitGitGadget yet, I am well
  underway there. I already have a notes ref (`commit-to-mail`, available
  via https://github.git/gitgitgadget/git) that annotates commits in
  git.git with the Message-ID of the corresponding mail. By "I have", I
  mean: there is an automated task that uses public-inbox to keep that ref
  up to date.

  I also have an accompanying `mail-to-commit` notes ref that maps
  Message-IDs back to the corresponding commit in git.git. That notes ref
  "annotates" the (non-existing) blob you get when piping the Message-ID
  with a trailing newline to `git hash-object`.

  Again, this is information that would be absolutely unobtainable without
  public-inbox.

- Related, I want to annotate the GitHub Pull Requests handled by
  GitGitGadvget with the corresponding name of the branch in
  https://github.com/gitster/git.

  This requires that `mail-to-commit` I mentioned in the previous bullet
  point, and therefore would not be possible without public-inbox.

- A feature I plan on introducing into GitGitGadget is to attach comments
  to the GitHub Pull Request when anybody replies to the patch thread sent
  out by GitGitGadget.

  Also this feature would be impossible without public-inbox.

- Another really useful feature I plan on introducing is to attach
  comments to those PRs whenever a What's Cooking is talking about the
  corresponding branch.

  Once again, would be impossible without public-inbox.

So thank you, thank you, thank you, for public-inbox!

Ciao,
Dscho

P.S.: FWIW I added a mirror of public-inbox to
https://git-for-windows.visualstudio.com/git.public-inbox, so that my
automated tasks, as well as my playing around, does not stress your server
too much.

Re: Thank you for public-inbox!

[Eric Wong]

<snip>

You're very welcome, Johannes.  And I'm hoping to have a few
more goodies live this fall/winter for public-inbox :>

Re: Thank you for public-inbox!

[Jeff King]

On Mon, Aug 27, 2018 at 04:25:13PM +0200, Johannes Schindelin wrote:

> I would like to take five minutes to thank you for public-inbox. It is
> invaluable for me in the meantime. And I think I will never be able to
> thank you enough for it.

Let me echo that appreciation. I have always kept my own archive of the
list, but it's so valuable to have stable URLs for communicating with
other folks (and I find the general design of public-inbox _way_ more
useful than gmane ever was).

> P.S.: FWIW I added a mirror of public-inbox to
> https://git-for-windows.visualstudio.com/git.public-inbox, so that my
> automated tasks, as well as my playing around, does not stress your server
> too much.

I've thought about mirroring it to a public server as well, just for
redundancy. But without the same domain, I'm not sure it would be all
that useful as a community resource.

Eric, let me know if there's something there that would help (e.g.,
putting more servers in DNS round-robin).

-Peff

Re: Thank you for public-inbox!

[Eric Wong]

Jeff King <peff@peff.net> wrote:
> I've thought about mirroring it to a public server as well, just for
> redundancy. But without the same domain, I'm not sure it would be all
> that useful as a community resource.

I wouldn't get too attached to the domain, "public-inbox.org" is
too long for my tastes anyways.  "peff.net/git/$MESSAGE_ID"
would actually be more user-friendly :>

A generic Message-ID redirection/finding service would be good,
(maybe some DHT thing, but... has that taken off for git blobs, yet?)

Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking
strings "<XXXXXXXXXXX@XXXXXXXX>" into URLs for domain-portability,
(but it's ambiguous with email addresses).  But yeah, I don't
like things being tied to domain names.

I've also been considering setting up a parallel instance
on 80x24.org to use the more-scalable "v2" repository format
developed for https://lore.kernel.org/lkml/

Speaking of lore, Konstantin confirmed he'll be getting more
vger lists up: https://public-inbox.org/meta/20180827205811.GA1611@chatter

Re: Thank you for public-inbox!

[Andrei Rybak]

On 2018-08-29 12:02, Eric Wong wrote:
> Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking
> strings "<XXXXXXXXXXX@XXXXXXXX>" into URLs for domain-portability,
> (but it's ambiguous with email addresses).  But yeah, I don't
> like things being tied to domain names.

This would be very useful for people who use MUAs without
Message-ID lookup capabilities, even without domain-portability.

Re: Thank you for public-inbox!

[Ævar Arnfjörð Bjarmason]

On Wed, Aug 29 2018, Andrei Rybak wrote:

> On 2018-08-29 12:02, Eric Wong wrote:
>> Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking
>> strings "<XXXXXXXXXXX@XXXXXXXX>" into URLs for domain-portability,
>> (but it's ambiguous with email addresses).  But yeah, I don't
>> like things being tied to domain names.
>
> This would be very useful for people who use MUAs without
> Message-ID lookup capabilities, even without domain-portability.

FWIW Many MUAs have some hidden way to do this, for example to find the
E-Mail I'm replying to (your E-Mail) on GMail:
https://mail.google.com/mail/u/0/#search/rfc822msgid%3A3eb1c5e8-3e89-0d2e-30b1-339f38c4c703%40gmail.com

I.e. rfc822msgid:<message-id>

Confusingly Google Groups accepts the same syntax, but will barf if you
include the <>'s, but that's from memory, maybe I misrecall or they've
fixed it.

Re: Thank you for public-inbox!

[Jeff King]

On Wed, Aug 29, 2018 at 10:02:43AM +0000, Eric Wong wrote:

> Jeff King <peff@peff.net> wrote:
> > I've thought about mirroring it to a public server as well, just for
> > redundancy. But without the same domain, I'm not sure it would be all
> > that useful as a community resource.
> 
> I wouldn't get too attached to the domain, "public-inbox.org" is
> too long for my tastes anyways.  "peff.net/git/$MESSAGE_ID"
> would actually be more user-friendly :>
> 
> A generic Message-ID redirection/finding service would be good,
> (maybe some DHT thing, but... has that taken off for git blobs, yet?)

Yes, and I agree that the URL portability is one of the things I really
love about public-inbox (after all, I do have my own archive and now I
can follow people's public-inbox links into my very-fast local copy).

I guess I just wonder if I set up a mirror on another domain, would
anybody actually _use_ it? I'd think most people would just go to
public-inbox.org as the de facto URL.

> Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking
> strings "<XXXXXXXXXXX@XXXXXXXX>" into URLs for domain-portability,
> (but it's ambiguous with email addresses).  But yeah, I don't
> like things being tied to domain names.

That would be neat, but I think it actually makes references less useful
in a lot of cases. URLs are universally understood, which means:

 - people who don't know about public-inbox can just follow the link
   (and in fact, that's how they learn how useful it is!)

 - even for people who do know about it, they are likely to read mails
   in their MUA. And most MUAs have some mechanism for easily following
   a URL, but won't know how to auto-linkify a message-id.

So I too dream of a world where I can say "give me more information on
this identifier" and my tools search a peer to peer distributed hash
table for it. But I don't think we live in that world yet.

At the very least, I think if we plan to reference without an http URL
that we would use something like URI-ish, like <mid:ABC@XYZ>. That gives
tools a better chance to say "OK, I know how to find message-ids"
(though I still think that it's much less helpful out of the box
compared to an http URL).

-Peff

Re: Thank you for public-inbox!

[Jonathan Nieder]

Jeff King wrote:

> I guess I just wonder if I set up a mirror on another domain, would
> anybody actually _use_ it? I'd think most people would just go to
> public-inbox.org as the de facto URL.

If it's faster than public-inbox.org and you don't mind the traffic I
would send, then I'll use it. :)

[...]
> That would be neat, but I think it actually makes references less useful
> in a lot of cases. URLs are universally understood, which means:
>
>  - people who don't know about public-inbox can just follow the link
>    (and in fact, that's how they learn how useful it is!)

I agree: please don't stop using URLs.

Having the message-id in the URL is very useful for being able to
migrate to another server.

In an ideal world, we'd be habitually using URLs that reference a
redirector, in the same spirit as https://www.kernel.org/lore.html.
That way, there is a very restricted syntax to use (e.g. just
message-ids) that is stable and can be reconfigured to redirect to
another service as appropriate.  In principle it also allows tricks
like redirecting based on geography or making the redirector go to the
user's preferred archive interface based on a cookie.

Thanks,
Jonathan

Re: Thank you for public-inbox!

[Eric Wong]

Jonathan Nieder <jrnieder@gmail.com> wrote:
> Jeff King wrote:
> 
> > I guess I just wonder if I set up a mirror on another domain, would
> > anybody actually _use_ it? I'd think most people would just go to
> > public-inbox.org as the de facto URL.
> 
> If it's faster than public-inbox.org and you don't mind the traffic I
> would send, then I'll use it. :)

Is performance a problem on public-inbox.org for you?

It should be pretty fast, but maybe there's corner-cases
and I haven't had time to pay close attention, lately...

I've also been sorta considering downgrading to a $5/month VPS
(from a $20/month VPS) to force myself to pay more attention to
performance while saving myself a few bucks.  But I wouldn't get
to dogfood on SMP, anymore...

Re: Thank you for public-inbox!

[Eric Wong]

Jeff King <peff@peff.net> wrote:
> On Wed, Aug 29, 2018 at 10:02:43AM +0000, Eric Wong wrote:
> > Anyways I hope to teach public-inbox to auto-linkify Message-ID-looking
> > strings "<XXXXXXXXXXX@XXXXXXXX>" into URLs for domain-portability,
> > (but it's ambiguous with email addresses).  But yeah, I don't
> > like things being tied to domain names.
> 
> That would be neat, but I think it actually makes references less useful
> in a lot of cases. URLs are universally understood, which means:
> 
>  - people who don't know about public-inbox can just follow the link
>    (and in fact, that's how they learn how useful it is!)
> 
>  - even for people who do know about it, they are likely to read mails
>    in their MUA. And most MUAs have some mechanism for easily following
>    a URL, but won't know how to auto-linkify a message-id.

Heh, one of the (unstated?) goals of public-inbox is to educate
the users on how Message-IDs (and email in general) works.
And to that end...

> So I too dream of a world where I can say "give me more information on
> this identifier" and my tools search a peer to peer distributed hash
> table for it. But I don't think we live in that world yet.

....More than dreaming, our goal should be to BUILD such a world :>
After all, it was my intense dislike of centralization which
drew me to DVCS and git in the first place.

> At the very least, I think if we plan to reference without an http URL
> that we would use something like URI-ish, like <mid:ABC@XYZ>. That gives
> tools a better chance to say "OK, I know how to find message-ids"
> (though I still think that it's much less helpful out of the box
> compared to an http URL).

That would be awesome if somelike like <mid:ABC@XYZ> could be a
standard and adopted (likewise with <git:$object_id>).

I haven't checked, but are there existing/similar RFCs?
Surely somebody has tried to get <git:$object_id>
adopted by now, right?

Re: Thank you for public-inbox!

[Jeff King]

On Thu, Aug 30, 2018 at 07:20:49AM +0000, Eric Wong wrote:

> > At the very least, I think if we plan to reference without an http URL
> > that we would use something like URI-ish, like <mid:ABC@XYZ>. That gives
> > tools a better chance to say "OK, I know how to find message-ids"
> > (though I still think that it's much less helpful out of the box
> > compared to an http URL).
> 
> That would be awesome if somelike like <mid:ABC@XYZ> could be a
> standard and adopted (likewise with <git:$object_id>).
> 
> I haven't checked, but are there existing/similar RFCs?
> Surely somebody has tried to get <git:$object_id>
> adopted by now, right?

There's https://tools.ietf.org/html/rfc2392. They even call it "mid:".  :)

I don't know of any git URI scheme, though it's similar in spirit to
magnet: links. Those are a bit verbose, though, because they're really a
meta-format for a bunch of different content-addressable schemes.

-Peff

Re: Thank you for public-inbox!

[Jonathan Nieder]

Eric Wong wrote:
> Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Jeff King wrote:

>>> I guess I just wonder if I set up a mirror on another domain, would
>>> anybody actually _use_ it? I'd think most people would just go to
>>> public-inbox.org as the de facto URL.
>>
>> If it's faster than public-inbox.org and you don't mind the traffic I
>> would send, then I'll use it. :)
>
> Is performance a problem on public-inbox.org for you?

It's pretty fast.  I'm just very, very picky about latency. ;-)

It's good to know you're interested in which corner cases are bad.
The next time I have a noticeably slow page load, I'll contact meta@.

[...]
> I've also been sorta considering downgrading to a $5/month VPS
> (from a $20/month VPS) to force myself to pay more attention to
> performance while saving myself a few bucks.  But I wouldn't get
> to dogfood on SMP, anymore...

Sounds reasonable to me.  If performance gets bad, that's just a
reason for people to help out (either with patches or e.g. with
donated VMs for hosting).

Speaking of the latter: what are your current resource requirements?
E.g. which of the dimensions in [1] do you not fit into?

Thanks,
Jonathan

[1] https://cloud.google.com/free/docs/always-free-usage-limits#compute_name

Re: Thank you for public-inbox!

[Eric Wong]

Jonathan Nieder <jrnieder@gmail.com> wrote:
> Eric Wong wrote:
> > Jonathan Nieder <jrnieder@gmail.com> wrote:
> >> Jeff King wrote:
> 
> >>> I guess I just wonder if I set up a mirror on another domain, would
> >>> anybody actually _use_ it? I'd think most people would just go to
> >>> public-inbox.org as the de facto URL.
> >>
> >> If it's faster than public-inbox.org and you don't mind the traffic I
> >> would send, then I'll use it. :)
> >
> > Is performance a problem on public-inbox.org for you?
> 
> It's pretty fast.  I'm just very, very picky about latency. ;-)

Best way for good latency is to have a local mirror, but I guess
Googlers still aren't allowed to run AGPL software?

> It's good to know you're interested in which corner cases are bad.
> The next time I have a noticeably slow page load, I'll contact meta@.

Alright. It could also be a general datacenter/networking
problem so https://status.linode.com/ (my VPS provider) is worth
checking.

> [...]
> > I've also been sorta considering downgrading to a $5/month VPS
> > (from a $20/month VPS) to force myself to pay more attention to
> > performance while saving myself a few bucks.  But I wouldn't get
> > to dogfood on SMP, anymore...
> 
> Sounds reasonable to me.  If performance gets bad, that's just a
> reason for people to help out (either with patches or e.g. with
> donated VMs for hosting).

> Speaking of the latter: what are your current resource requirements?

Not too much; but could always be better on the software side.

> E.g. which of the dimensions in [1] do you not fit into?

> [1] https://cloud.google.com/free/docs/always-free-usage-limits#compute_name

Dunno, I'm not seeing RAM, there.

Depending on traffic, it's around 200MB per-public-inbox-httpd
worker (2 workers for 2 cores) when there's a traffic surge on
from popular sites.  Memory usage is the biggest disappointment
and only happens when Varnish can't read fast enough.
Everything in the PSGI code is is streamed if possible(*).
My goal is to maintain <50MB per worker process, but it could
be tough in Perl5. Anyways $20/month gets me 4GB RAM (so I have
way more than I need).

CPU usage isn't even noticeable (only bursts) and I do other
stuff on that server all the time.

HDDs wouldn't work well at all and I've noticed differences
based on SSD quality with Xapian.  Storage for Xapian+SQLite is
4-5x what's in git, so for this list, it's under 7G total
(but more will be needed for Xapian reindexing/compact and
git repacking).



(*) Individual messages for returning giant mboxes and threads are
    all lazily fleshed out from skeleton data structures as the
    client socket becomes writable (and quickly discarded after writing).
    Technically it's all compatible with any PSGI server, but all
    the streaming stuff is tailored to run on public-inbox-httpd.
    But there's also git-http-backend memory use which comes in
    bursts (bitmaps enabled, of course)