Fetching commit instead of ref

Fetching commit instead of ref

[Carlsson, Magnus]

Hi

I am involved in the git-subrepo project (https://github.com/ingydotnet/git-subrepo/). It's an attempt to simplify the inclusion of repos into other repos.

In a certain situation I would really need to fetch all commits related to a specific commit (SHA). I have read the git fetch documentation and found nothing regarding this. It only seems to support fetching references.

I found some traces on stack overflow:
https://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository

Following that recommendation it feels like it almost works:
$ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit
remote: Counting objects: 2311, done.
remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311
Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1174/1174), done.
----> So far so good, but then an error message appear:
error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec
----> And nothing seems to be fetched.

Is there a way to fetch a commit and any ancestors to that commit based on a SHA?

Why do I need this?
In git-subrepo we try to recreate another repo within our main repo. Creating the necessary parent references when they appear. In some cases we need to make sure that we have access to the correct commits from the subrepo, but we don't have any references except a SHA.
 
-- Magnus
 
 
MAGNUS CARLSSON
 Staff Software Engineer
 ARRIS
 
 o: +46 13 36 75 92
 e: magnus.carlsson@arris.com
 w: www.arris.com
 
 ARRIS:  Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583
 
 This electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed.  It is confidential and may be attorney-client privileged.  In any event the Sender reserves, to the fullest extent, any "legal  advice privilege".  Any further distribution or copying of this message is strictly prohibited.  If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).
      

Re: Fetching commit instead of ref

[Jonathan Tan]

On Mon, 18 Dec 2017 12:30:23 +0000
"Carlsson, Magnus" <Magnus.Carlsson@arris.com> wrote:

> In a certain situation I would really need to fetch all commits
> related to a specific commit (SHA). I have read the git fetch
> documentation and found nothing regarding this. It only seems to
> support fetching references.

The documentation has been updated in version 2.15.0 to describe this.
But as the commit message of commit 83558a412afa ("fetch doc: src side
of refspec could be full SHA-1", 2017-10-18) says, this functionality
was available earlier.

> I found some traces on stack overflow:
> https://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository
> 
> Following that recommendation it feels like it almost works:
> $ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit
> remote: Counting objects: 2311, done.
> remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311
> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.
> Resolving deltas: 100% (1174/1174), done.
> ----> So far so good, but then an error message appear:
> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec
> ----> And nothing seems to be fetched.
> 
> Is there a way to fetch a commit and any ancestors to that commit based on a SHA?

You'll need to set uploadpack.allowTipSHA1InWant,
uploadpack.allowReachableSHA1InWant, or uploadpack.allowAnySHA1InWant on
the server. (This might need to be better documented - I see this
documented for fetch-pack, but not fetch.)

Re: Fetching commit instead of ref

[Stefan Beller]

On Mon, Dec 18, 2017 at 4:30 AM, Carlsson, Magnus
<Magnus.Carlsson@arris.com> wrote:
> Hi
>
> I am involved in the git-subrepo project (https://github.com/ingydotnet/git-subrepo/). It's an attempt to simplify the inclusion of repos into other repos.
>
> In a certain situation I would really need to fetch all commits related to a specific commit (SHA). I have read the git fetch documentation and found nothing regarding this. It only seems to support fetching references.
>
> I found some traces on stack overflow:
> https://stackoverflow.com/questions/14872486/retrieve-specific-commit-from-a-remote-git-repository
>
> Following that recommendation it feels like it almost works:
> $ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit
> remote: Counting objects: 2311, done.
> remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311
> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.
> Resolving deltas: 100% (1174/1174), done.
> ----> So far so good, but then an error message appear:
> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec
> ----> And nothing seems to be fetched.
>
> Is there a way to fetch a commit and any ancestors to that commit based on a SHA?

Ask the server operator to configure the server to allow fetching commits,
specifically "git config uploadpack.allowReachableSHA1InWant 1"


>
> Why do I need this?
> In git-subrepo we try to recreate another repo within our main repo. Creating the necessary parent references when they appear. In some cases we need to make sure that we have access to the correct commits from the subrepo, but we don't have any references except a SHA.

A very similar issue happens with submodules, which tries to fetch the
branch(es) first and then by SHA1 as a fallback, but this fallback may
fail as well due to the miss-configured server, at that point
submodules just error out.

Re: Fetching commit instead of ref

[Junio C Hamano]

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

> ----> So far so good, but then an error message appear:
> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec
> ----> And nothing seems to be fetched.

Yes, that is what the error message is telling you.  

You'd need to coordinate with the server operator so that the server
allows such an request; uploadpack.allowAnySHA1InWant may help.

Re: Fetching commit instead of ref

[Carlsson, Magnus]

First: Thanks everyone for your answers.

I understand that there is a fetch pack, problem is that I can't force every git server provider to turn it on... Tested with github and they don't seem to have it on by default.

I understand that you don't want to allow people fetching single commits all the time, but is there a reason that you don't allow SHA instead of references when you fetch an entire tree?

Is there a workaround? Can I somehow ask a remote on a valid reference that includes a SHA? So I can later fetch that reference. In my case I would like to avoid to fetch more then necessary as it pollutes the main repository.

-- Magnus

MAGNUS CARLSSON
Staff Software Engineer
ARRIS

o: +46 13 36 75 92
e: magnus.carlsson@arris.com
w: www.arris.com

ARRIS:  Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583

This electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed.  It is confidential and may be attorney-client privileged.  In any event the Sender reserves, to the fullest extent, any "legal advice privilege".  Any further distribution or copying of this message is strictly prohibited.  If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).

________________________________________
From: Junio C Hamano <gitster@pobox.com>
Sent: Monday, December 18, 2017 19:44
To: Carlsson, Magnus
Cc: git@vger.kernel.org
Subject: Re: Fetching commit instead of ref

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

> ----> So far so good, but then an error message appear:
> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec
> ----> And nothing seems to be fetched.

Yes, that is what the error message is telling you.

You'd need to coordinate with the server operator so that the server
allows such an request; uploadpack.allowAnySHA1InWant may help.

Re: Fetching commit instead of ref

[Junio C Hamano]

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

> I understand that you don't want to allow people fetching single
> commits all the time, but is there a reason that you don't allow
> SHA instead of references when you fetch an entire tree?

I do not think we don't want to allow "single commit" fetch.  We do
not allow fetching histories from an arbitrary point in the graph,
unless we can prove that what gets fetched is what the serving end
intended to expose to the fetcher---you should view it as a security
issue.

The default way to determine that a fetch request is getting only
the things the serving end wanted to publish is to see the requested
tips of the histories to be fetched are all pointed by refs.  Which
means that a client of a hosting site can rewind and repoint a ref
after pushing a wrong branch that contained undesirable matter by
accident.  Moving the ref to make the undesirable object unreachable
is all that is needed to "hide" it from the public view, even when
the client does not have a way to trigger "gc" immediately on the
hosting site.

There are variants of this security measure.  If the hosting site is
willing to spend extra cycles, we could loosen the "is the request
fetching only what is published?" check to see if the requested tips
are reachable from the tips of refs, instead of exactly at refs.  It
preserves "a mistake can be hidden with a forced push" property the
same way as above, but it is costly and is prone to abuse.

If you are confident about your pushes all the time, you can take
a stance "anything in the repository is meant to be read".  That is
what the "allowAnySHA1InWant" does.  Not everybody however wants to
do this for obvious reasons, so it is not a default.

Re: Fetching commit instead of ref

[Carlsson, Magnus]

Thanks Junio for your answer!

From my simple tests it seems that github doesn't have this on by default so that seems a little dull.

Do you know if there is a way to actually find a ref that contains the SHA from a remote?

Finally, you say that its a security feature, but from the log it feels like you are actually downloading things first and then you determine if you are allowed to fetch it or?

$ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit
remote: Counting objects: 2311, done.
remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311
-- >>> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1174/1174), done.
error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec

-- Magnus


MAGNUS CARLSSON
Staff Software Engineer
ARRIS

o: +46 13 36 75 92
e: magnus.carlsson@arris.com
w: www.arris.com

ARRIS:  Legal entity: Arris Sweden AB - Registered Office: Teknikringen 2, 583 30 Linkoping, Sweden - Reg No:556518-5831 - VAT No:SE 556518-583

This electronic transmission (and any attached document) is for the sole use of the individual or entity to whom it is addressed.  It is confidential and may be attorney-client privileged.  In any event the Sender reserves, to the fullest extent, any "legal advice privilege".  Any further distribution or copying of this message is strictly prohibited.  If you received this message in error, please notify the Sender immediately and destroy the attached message (and all attached documents).

________________________________________
From: Junio C Hamano <gitster@pobox.com>
Sent: Tuesday, December 19, 2017 17:07
To: Carlsson, Magnus
Cc: git@vger.kernel.org
Subject: Re: Fetching commit instead of ref

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

> I understand that you don't want to allow people fetching single
> commits all the time, but is there a reason that you don't allow
> SHA instead of references when you fetch an entire tree?

I do not think we don't want to allow "single commit" fetch.  We do
not allow fetching histories from an arbitrary point in the graph,
unless we can prove that what gets fetched is what the serving end
intended to expose to the fetcher---you should view it as a security
issue.

The default way to determine that a fetch request is getting only
the things the serving end wanted to publish is to see the requested
tips of the histories to be fetched are all pointed by refs.  Which
means that a client of a hosting site can rewind and repoint a ref
after pushing a wrong branch that contained undesirable matter by
accident.  Moving the ref to make the undesirable object unreachable
is all that is needed to "hide" it from the public view, even when
the client does not have a way to trigger "gc" immediately on the
hosting site.

There are variants of this security measure.  If the hosting site is
willing to spend extra cycles, we could loosen the "is the request
fetching only what is published?" check to see if the requested tips
are reachable from the tips of refs, instead of exactly at refs.  It
preserves "a mistake can be hidden with a forced push" property the
same way as above, but it is costly and is prone to abuse.

If you are confident about your pushes all the time, you can take
a stance "anything in the repository is meant to be read".  That is
what the "allowAnySHA1InWant" does.  Not everybody however wants to
do this for obvious reasons, so it is not a default.

Re: Fetching commit instead of ref

[Junio C Hamano]

"Carlsson, Magnus" <Magnus.Carlsson@arris.com> writes:

> $ git fetch subrepo 50f730db793e0733b159326c5a3e78fd48cedfec:refs/remote/subrepo/foo-commit
> remote: Counting objects: 2311, done.
> remote: Total 2311 (delta 0), reused 0 (delta 0), pack-reused 2311
> -- >>> Receiving objects: 100% (2311/2311), 703.64 KiB | 0 bytes/s, done.
> Resolving deltas: 100% (1174/1174), done.
> error: Server does not allow request for unadvertised object 50f730db793e0733b159326c5a3e78fd48cedfec

Unless that "fetch" is doing a recursive fetch from another
repository (which causes the counting to be shown) or something
silly, the above makes it look like that the server is broken.

A quick test locally does not seem to show such a behaviour, but I
do not enable pack bitmaps and I know github does at least for some
repositories, so it is possible there is a problem somewhere in that
codepath.