git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
* "IMAP IDLE"-like long-polling "git fetch"
       [not found] <20181229034342.11543-1-e@80x24.org>
@ 2018-12-29  3:56 ` Eric Wong
  2018-12-29  4:38   ` Konstantin Ryabitsev
  0 siblings, 1 reply; 5+ messages in thread
From: Eric Wong @ 2018-12-29  3:56 UTC (permalink / raw)
  To: git; +Cc: meta

Hey all, I just added this to the TODO file for public-inbox[1] but
obviously it's intended for git.git (meta@public-inbox cc-ed):

> +* Contribute something like IMAP IDLE for "git fetch".
> +  Inboxes (and any git repos) can be kept up-to-date without
> +  relying on polling.

I would've thought somebody had done this by now, but I guess
it's dependent on a bunch of things (TLS layer nowadays, maybe
HTTP/2), so git-daemon support alone wouldn't cut it...

Anyways, until this is implemented, feel free to continue
hammering a way on https://public-inbox.org/git/ with frequent
"git fetch".  I write C10K servers in my sleep -_-


[1] https://public-inbox.org/meta/20181229034342.11543-1-e@80x24.org/

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: "IMAP IDLE"-like long-polling "git fetch"
  2018-12-29  3:56 ` "IMAP IDLE"-like long-polling "git fetch" Eric Wong
@ 2018-12-29  4:38   ` Konstantin Ryabitsev
  2018-12-29  6:13     ` Eric Wong
  2019-01-09 22:27     ` Stefan Beller
  0 siblings, 2 replies; 5+ messages in thread
From: Konstantin Ryabitsev @ 2018-12-29  4:38 UTC (permalink / raw)
  To: Eric Wong; +Cc: git, meta

On Sat, Dec 29, 2018 at 03:56:21AM +0000, Eric Wong wrote:
> Hey all, I just added this to the TODO file for public-inbox[1] but
> obviously it's intended for git.git (meta@public-inbox cc-ed):
> 
> > +* Contribute something like IMAP IDLE for "git fetch".
> > +  Inboxes (and any git repos) can be kept up-to-date without
> > +  relying on polling.
> 
> I would've thought somebody had done this by now, but I guess
> it's dependent on a bunch of things (TLS layer nowadays, maybe
> HTTP/2), so git-daemon support alone wouldn't cut it...

Polling is not all bad, especially for large repository collections. I'm
not sure you want to "idle" individual repositories when there's
thousands of them. We ended up writing grokmirror for replicating
repo collections using manifest files.

> Anyways, until this is implemented, feel free to continue
> hammering a way on https://public-inbox.org/git/ with frequent
> "git fetch".  I write C10K servers in my sleep -_-

The archive is also mirrored at
https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/git.git, and
also on kernel.googlesource.com.

-K

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: "IMAP IDLE"-like long-polling "git fetch"
  2018-12-29  4:38   ` Konstantin Ryabitsev
@ 2018-12-29  6:13     ` Eric Wong
  2019-01-09 22:27     ` Stefan Beller
  1 sibling, 0 replies; 5+ messages in thread
From: Eric Wong @ 2018-12-29  6:13 UTC (permalink / raw)
  To: Konstantin Ryabitsev; +Cc: git, meta

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:
> On Sat, Dec 29, 2018 at 03:56:21AM +0000, Eric Wong wrote:
> > Hey all, I just added this to the TODO file for public-inbox[1] but
> > obviously it's intended for git.git (meta@public-inbox cc-ed):
> > 
> > > +* Contribute something like IMAP IDLE for "git fetch".
> > > +  Inboxes (and any git repos) can be kept up-to-date without
> > > +  relying on polling.
> > 
> > I would've thought somebody had done this by now, but I guess
> > it's dependent on a bunch of things (TLS layer nowadays, maybe
> > HTTP/2), so git-daemon support alone wouldn't cut it...
> 
> Polling is not all bad, especially for large repository collections. I'm
> not sure you want to "idle" individual repositories when there's
> thousands of them. We ended up writing grokmirror for replicating
> repo collections using manifest files.

I wasn't intending it for giant sites like korg, but for
individual hackers on their workstations tracking a handful of
projects they follow.

The cost for a hackers' machine would be the same as the current
situation where developers idle on IRC channels for the projects
they're involved in.

> > Anyways, until this is implemented, feel free to continue
> > hammering a way on https://public-inbox.org/git/ with frequent
> > "git fetch".  I write C10K servers in my sleep -_-
> 
> The archive is also mirrored at
> https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/git.git, and
> also on kernel.googlesource.com.

Now, I'm wondering if you can make a v2 public-inbox mirror of
git@vger and run it on lore.  Converting public-inbox.org/git to
v2 would break things for everybody fetching, right now :<

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: "IMAP IDLE"-like long-polling "git fetch"
  2018-12-29  4:38   ` Konstantin Ryabitsev
  2018-12-29  6:13     ` Eric Wong
@ 2019-01-09 22:27     ` Stefan Beller
  2019-01-09 22:49       ` Konstantin Ryabitsev
  1 sibling, 1 reply; 5+ messages in thread
From: Stefan Beller @ 2019-01-09 22:27 UTC (permalink / raw)
  To: Eric Wong, git, meta

On Fri, Dec 28, 2018 at 8:39 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Sat, Dec 29, 2018 at 03:56:21AM +0000, Eric Wong wrote:
> > Hey all, I just added this to the TODO file for public-inbox[1] but
> > obviously it's intended for git.git (meta@public-inbox cc-ed):
> >
> > > +* Contribute something like IMAP IDLE for "git fetch".
> > > +  Inboxes (and any git repos) can be kept up-to-date without
> > > +  relying on polling.
> >
> > I would've thought somebody had done this by now, but I guess
> > it's dependent on a bunch of things (TLS layer nowadays, maybe
> > HTTP/2), so git-daemon support alone wouldn't cut it...
>
> Polling is not all bad, especially for large repository collections.

I disagree with that statement.

IIRC, More than half the bandwidth of Googles git servers are used
for ls-remote calls (i.e. polling a lot of repos, most of them did *not*
change, by build bots which are really eager to try again after a minute).

That is why we use a superproject, with all other repositories as
a submodule for polling, as that would slash the ls-remote traffic
approximately by the number of repositories.

There was an attempt in JGit to support this type of communication
of long polling at
https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/2adc572628f9382ace5fbd791325dc64f7c968d3
but not a whole lot is left over in JGit as it was refactored at least
once again.

IIRC the issues where in the lack of protocol definition that made it
usable for a wider audience.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: "IMAP IDLE"-like long-polling "git fetch"
  2019-01-09 22:27     ` Stefan Beller
@ 2019-01-09 22:49       ` Konstantin Ryabitsev
  0 siblings, 0 replies; 5+ messages in thread
From: Konstantin Ryabitsev @ 2019-01-09 22:49 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Eric Wong, git, meta

On Wed, Jan 09, 2019 at 02:27:25PM -0800, Stefan Beller wrote:
> > > I would've thought somebody had done this by now, but I guess
> > > it's dependent on a bunch of things (TLS layer nowadays, maybe
> > > HTTP/2), so git-daemon support alone wouldn't cut it...
> >
> > Polling is not all bad, especially for large repository collections.
> 
> I disagree with that statement.
> 
> IIRC, More than half the bandwidth of Googles git servers are used
> for ls-remote calls (i.e. polling a lot of repos, most of them did *not*
> change, by build bots which are really eager to try again after a minute).

Oh, that's not the kind of polling I meant -- we monitor a single
manifest file containing the state of all repositories. It's a static
file served directly by any httpd daemon, and the only traffic is
usually the "not modified" http header.

-K

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <20181229034342.11543-1-e@80x24.org>
2018-12-29  3:56 ` "IMAP IDLE"-like long-polling "git fetch" Eric Wong
2018-12-29  4:38   ` Konstantin Ryabitsev
2018-12-29  6:13     ` Eric Wong
2019-01-09 22:27     ` Stefan Beller
2019-01-09 22:49       ` Konstantin Ryabitsev

git@vger.kernel.org list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox