From: "SZEDER Gábor" <szeder.dev@gmail.com>
To: Jeff King <peff@peff.net>
Cc: Junio C Hamano <gitster@pobox.com>,
Git mailing list <git@vger.kernel.org>,
Jonathan Nieder <jrnieder@gmail.com>
Subject: Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs
Date: Wed, 21 Jun 2017 02:21:19 +0200 [thread overview]
Message-ID: <CAM0VKjnfVuVnUfnfCceD-QfUdj=6sWTdBDXkj9jZujqMJqdAQA@mail.gmail.com> (raw)
On Sat, Jun 17, 2017 at 2:33 PM, Jeff King <peff@peff.net> wrote:
> On Sat, Jun 17, 2017 at 02:25:39PM +0200, SZEDER Gábor wrote:
>> Indeed. Even in my repos with close to 10k remotes the amount of
>> memory wasted by the duplicated refspecs is not an problem, there are
>> more pressing issues there.
>
> This is sort of a side note, but I'm curious why you need 10k remotes.
For fun! :o)
I have a couple of repos containing all branches from all forks (and
forks of forks, recursively) of a few interesting repos on GitHub,
where each fork is a remote.
(Why on earth would I want to do that?
Sometimes it's good to see what others are up to. Apparently a lot
of people fork projects, push their changes, but then never upstream
them.
Such an all-forks-in-one repo allows me to run e.g. 'git log --all
-p master.. relevant.c' and then search its output for changes in
interesting functions (thankfully function names are included in
hunk headers; alas line log doesn't work with --all). Occasionally
this unearths some treasures: other people's commits and branches
scratching the same itch that I was about to scratch, or at least
solving part of my problem and I can build on top of them.)
> We used to do this at GitHub as part of our fork storage (the shared
> repository had each fork as a remote). We fixed the quadratic issues
> like d0da003d5 (use a hashmap to make remotes faster, 2014-07-29). But
> even the linear work to read the config is noticeable (and hits you on
> every command, even ones that don't care about remotes).
Indeed, that's one of those "more pressing issues".
I worked it around by storing the remotes in a separate config file
which is not included from .git/config, because they're not needed for
local operations. And when I occasionally do want to fetch, then I
run 'git -c include.path=..../.git/config.forks fetch ...'.
> Now instead we
> pass the refspecs directly to fetch whenever move objects between the
> storage repos. They were the same for every remote anyway (and I'd guess
> that is true, too, of your 10k remotes).
I do have different fetch refspecs for every remote, i.e. the repo
'github.com/user/repo' has '+refs/heads/*:refs/forks/user/repo/*'.
Incidentally, this was what triggered
sg/clone-refspec-from-command-line-config back then, because 'git
clone' didn't grab 'refs/forks/*', and the means advertised in the
docs to do so didn't work.
Gábor
next reply other threads:[~2017-06-21 0:21 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-06-21 0:21 SZEDER Gábor [this message]
2017-06-21 16:00 ` Why 10k remotes? Was: Re: [RFC/PATCH 0/5] remote: eliminate remote->{fetch,push}_refspec and lazy parsing of refspecs Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAM0VKjnfVuVnUfnfCceD-QfUdj=6sWTdBDXkj9jZujqMJqdAQA@mail.gmail.com' \
--to=szeder.dev@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jrnieder@gmail.com \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).