From: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
To: Junio C Hamano <gitster@pobox.com>
Cc: "Theodore Y. Ts'o" <tytso@mit.edu>,
Caleb Gray <hey@calebgray.com>,
git@vger.kernel.org
Subject: Re: Add a "Flattened Cache" to `git --clone`?
Date: Thu, 14 May 2020 17:44:04 -0400 [thread overview]
Message-ID: <20200514214404.bcbjskgi52bwedlh@chatter.i7.local> (raw)
In-Reply-To: <xmqqr1vmp5wf.fsf@gitster.c.googlers.com>
On Thu, May 14, 2020 at 02:23:44PM -0700, Junio C Hamano wrote:
> > I think something like git-caching-proxy would be a neat project,
> > because it would significantly improve mirroring for CI deployments
> > without requiring that each individual job implements clone.bundle
> > prefetching.
>
> What are we improving with such a proxy, though?
>
> Not bandwidth to the client, apparently.
Well, if it sits in front of the CI subnet, then it *does* save
bandwidth.
Here's an example with the exact situation we have:
- the Gerrit server is on the US West Coast
- the CI builder is on the East Coast
- each CI job does a full transfer of the multi-MB repo across the
continent, even when cloning shallow
We solve this by having a local mirror of the repository, but this
requires active mirroring to be pre-setup. A caching proxy that could:
- receive a request for a repository
- stream the response back to the client
- cache objects locally
- use local cache to construct future requests, so only missing objects
are fetched from the remote repo regardless of the haves on the actual
client...
..now, that would be kinda neat, but I'm not sure how sane or fragile
that setup would be. :)
> I thought that with the
> reachability bitmap on the server side with reusing packed object,
> it was more or less a solved problem that the server end spends way
> too much time enumerating, deltifying and compressing the object
> data?
Indeed, it's not really solving anything for this case.
-K
next prev parent reply other threads:[~2020-05-14 21:44 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-05-14 14:34 Add a "Flattened Cache" to `git --clone`? Caleb Gray
2020-05-14 20:33 ` Konstantin Ryabitsev
2020-05-14 20:54 ` Bryan Turner
2020-05-14 21:05 ` Theodore Y. Ts'o
2020-05-14 21:09 ` Eric Sunshine
2020-05-14 21:10 ` Konstantin Ryabitsev
2020-05-14 21:23 ` Junio C Hamano
2020-05-14 21:44 ` Konstantin Ryabitsev [this message]
2020-05-15 21:42 ` Eric Wong
2020-05-17 22:12 ` Konstantin Ryabitsev
[not found] ` <1061511589863147@mail.yandex.ru>
2020-05-25 14:02 ` Caleb Gray
2020-05-14 21:33 ` Caleb Gray
2020-05-14 21:56 ` Junio C Hamano
2020-05-14 22:04 ` Caleb Gray
2020-05-14 22:30 ` Junio C Hamano
2020-05-14 22:44 ` Bryan Turner
2020-05-14 21:19 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200514214404.bcbjskgi52bwedlh@chatter.i7.local \
--to=konstantin@linuxfoundation.org \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=hey@calebgray.com \
--cc=tytso@mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).