Re: Proposal for missing blob support in Git repos

[Jonathan Tan <jonathantanmy@google.com>]

On 05/02/2017 11:32 AM, Ævar Arnfjörð Bjarmason wrote:
> On Tue, May 2, 2017 at 7:21 PM, Jonathan Tan <jonathantanmy@google.com> wrote:
>> On Mon, May 1, 2017 at 6:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>> Jonathan Tan <jonathantanmy@google.com> writes:
>>>
>>>> On 05/01/2017 04:29 PM, Junio C Hamano wrote:
>>>>> Jonathan Tan <jonathantanmy@google.com> writes:
>>>>>
>>>>>> Thanks for your comments. If you're referring to the codepath
>>>>>> involving write_sha1_file() (for example, builtin/hash-object ->
>>>>>> index_fd or builtin/unpack-objects), that is fine because
>>>>>> write_sha1_file() invokes freshen_packed_object() and
>>>>>> freshen_loose_object() directly to check if the object already exists
>>>>>> (and thus does not invoke the new mechanism in this patch).
>>>>>
>>>>> Is that a good thing, though?  It means that you an attacker can
>>>>> feed one version to the remote object store your "grab blob" hook
>>>>> gets the blobs from, and have you add a colliding object locally,
>>>>> and the usual "are we recording the same object as existing one?"
>>>>> check is bypassed.
>>>>
>>>> If I understand this correctly, what you mean is the situation where
>>>> the hook adds an object to the local repo, overriding another object
>>>> of the same name?
>>>
>>> No.
>>>
>>> write_sha1_file() pays attention to objects already in the local
>>> object store to avoid hash collisions that can be used to replace a
>>> known-to-be-good object and that is done as a security measure.
>>> What I am reading in your response was that this new mechanism
>>> bypasses that, and I was wondering if that is a good thing.
>>
>> Oh, what I meant was that write_sha1_file() bypasses the new
>> mechanism, not that the new mechanism bypasses the checks in
>> write_sha1_file().
>>
>> To be clear, here's what happens when write_sha1_file() is invoked
>> (before and after this patch - this patch does not affect
>> write_sha1_file at all):
>> 1. (some details omitted)
>> 2. call freshen_packed_object
>> 3, call freshen_loose_object if necessary
>> 4. write object (if freshen_packed_object and freshen_loose_object do
>> not both return 0)
>>
>> Nothing changes in this patch (whether a hook is defined or not).
>
> But don't the semantics change in the sense that before
> core.missingBlobCommand you couldn't write a new blob SHA1 that was
> already part of your history,

Strictly speaking, you can already do this if you don't have the blob in 
your local repo (for example, with shallow clones - you likely wouldn't 
have blobs pointed to by historical commits outside whatever depth is set).

 > whereas with this change
> write_sha1_file() might write what it considers to be a new blob, but
> it's actually colliding with an existing blob, but write_sha1_file()
> doesn't know that because knowing would involve asking the hook to
> fetch the blob?

Yes, this might happen.

I see the semantics as "don't write what you already have", where "have" 
means what you have in local storage, but if you extend "have" to what 
upstream has, then yes, you're right that this changes (ignoring shallow 
clones).

This does remove a resistance that we have against hash collision (in 
that normally we would have the correct object for a given hash and can 
resist other servers trying to introduce a wrong object, but now that is 
no longer the case), but I think it's better than consulting the hook 
whenever you want to write anything (which is also a change in semantics 
in that you're consulting an external source whenever you're writing an 
object, besides the performance implications).

>> And here's what happens when has_sha1_file (or another function listed
>> in the commit message) is invoked:
>> 1. check for existence of packed object of the requested name
>> 2. check for existence of loose object of the requested name
>> 3. check again for existence of packed object of the requested name
>> 4. if a hook is defined, invoke the hook and repeat 1-3
>>
>> Here, in step 4, the hook could do whatever it wants to the repository.
>
> This might be a bit of early bikeshedding, but then again the lack of
> early bikeshedding tends to turn into standards.
>
> Wouldn't it be better to name this core.missingObjectCommand & have
> the hook take a list on stdin like:
>
>     <id> <TAB> <object_type> <TAB> <object_id> <TAB> <request_type> <TAB> [....]
>
> And have the hook respond:
>
>     <id> <TAB> <status> <TAB> [....]
>
> I.e. what you'd do now is send this to the hook:
>
>     1 <TAB> blob <TAB> <sha1> <TAB> missing
>
> And the hook would respond:
>
>     1 <TAB> ok
>
> But this leaves open the door addressing this potential edge case with
> writing new blobs in the future, i.e. write_sha1_file() could call it
> as:
>
>     1 <TAB> blob <TAB> <sha1> <TAB> new
>
> And the hook could either respond immediately as:
>
>     1 <TAB> ok
>
> If it's in some #YOLO mode where it's not going to check for colliding
> blobs over the network, or alternatively & ask the parent repo if it
> has those blobs, and if so print:
>
>     1 <TAB> collision
>
> Or something like that.
>
> This also enables future lazy loading of trees/commits from the same
> API, and for the hook to respond out-of-order to the input it gets as
> it can, since each request is prefixed with an incrementing request
> id.

My initial thought is that it would be better to extend hook support by 
adding configuration options for separate hooks instead of extending an 
existing protocol. For example, with this, the hook would probably need 
to declare capabilities so that Git knows what the hook supports; also, 
even if the user and the hook have no issue with Git writing any object, 
Git would have to invoke the hook since it wouldn't know.

Having said that, I'm not too attached to the existing protocol and it 
can change to something like what you suggested if necessary.