git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Eric Wong <e@80x24.org>
Cc: Stefan Beller <sbeller@google.com>,
	git@vger.kernel.org, meta@public-inbox.org
Subject: Re: "IMAP IDLE"-like long-polling "git fetch"
Date: Thu, 02 May 2019 11:21:05 +0200	[thread overview]
Message-ID: <87ftpxqkji.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <20190502085055.34kkll2deowat6il@dcvr>


On Thu, May 02 2019, Eric Wong wrote:

> Stefan Beller <sbeller@google.com> wrote:
>> IIRC, More than half the bandwidth of Googles git servers are used
>> for ls-remote calls (i.e. polling a lot of repos, most of them did *not*
>> change, by build bots which are really eager to try again after a minute).
>
> Thinking back at that statement; I think polling can be
> optimized in git, at least.
>
> IIRC, your repos have lots of refs; right?
> (which is why it's a bandwidth problem)
>
> Since info/refs is a static file (hopefully updated by a
> post-update hook), the smart client can make an HTTP request
> to check If-Modified-Since: to avoid the big response.
>
> The client would need to cache the mtime of the last requested
> refs file; somewhere.
>
> IOW, do refs negotiation the "dumb" way; since it's no better
> than the smart way, really.  Keep doing object transfers the
> smart way.
>
> During the initial clone, smart servers could probably
> have a header informing clients that their info/refs
> is up-to-date and clients can do dumb refs negotiation.

Doing this with If-Modified-Since sounds like an easier drop-in
replacement (just needs a client change), but I wonder if ETag isn't a
better fit for this.

I.e. we'd document some convention where the ETag is a hash of the refs
the client expects to be advertised in some format, it then sends that
to the server.

That allows the same thing without anyone keeping more state than they
keep now in their local ref store

On the fancier side I think bloom filters are something that's been
discussed (and I believe someone (Twitter?) had such an internal patch),
i.e. the client sends a bloom filter of refs they have, and the server
advertises things they don't know about yet (and due to how bloom
filters work, some things they *do* know about already but tripped up
the bloom filter...).

  reply	other threads:[~2019-05-02  9:21 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20181229034342.11543-1-e@80x24.org>
2018-12-29  3:56 ` "IMAP IDLE"-like long-polling "git fetch" Eric Wong
2018-12-29  4:38   ` Konstantin Ryabitsev
2018-12-29  6:13     ` Eric Wong
2019-01-09 22:27     ` Stefan Beller
2019-01-09 22:49       ` Konstantin Ryabitsev
2019-05-02  8:50       ` Eric Wong
2019-05-02  9:21         ` Ævar Arnfjörð Bjarmason [this message]
2019-05-02  9:42           ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87ftpxqkji.fsf@evledraar.gmail.com \
    --to=avarab@gmail.com \
    --cc=e@80x24.org \
    --cc=git@vger.kernel.org \
    --cc=meta@public-inbox.org \
    --cc=sbeller@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).