Are clone/checkout operations deterministic?

Are clone/checkout operations deterministic?

[J. Paul Reed]

Hey Git Devs,

I have a bit of an odd question: do git clone/checkout operations have a
deterministic ordering?

That is: are files guaranteed to be laid down onto disk in any specific
(and deterministic) order as a clone and/or checkout operations occurs?
(And if so, is it alphabetical order? Depth-first? Something else?)

In case the answer is different (and I'd guess that it might be?), I'm
mostly interested in the initial clone case... but it would be great to
know if, indeed, the answer is different for just-checkouts too.

I did some cursory googling, but nothing hopped out at me as an answer to
this question.

Thanks!

-preed
-- 
J. Paul Reed                                              PGP: 0xDF8708F8
=========================================================================
I've never seen an airplane yet that can read the type ratings on your
pilot's license.                                       -- Chuck Boedecker

Re: Are clone/checkout operations deterministic?

[Ævar Arnfjörð Bjarmason]

On Tue, Jul 17 2018, J. Paul Reed wrote:

> Hey Git Devs,
>
> I have a bit of an odd question: do git clone/checkout operations have a
> deterministic ordering?
>
> That is: are files guaranteed to be laid down onto disk in any specific
> (and deterministic) order as a clone and/or checkout operations occurs?
> (And if so, is it alphabetical order? Depth-first? Something else?)
>
> In case the answer is different (and I'd guess that it might be?), I'm
> mostly interested in the initial clone case... but it would be great to
> know if, indeed, the answer is different for just-checkouts too.
>
> I did some cursory googling, but nothing hopped out at me as an answer to
> this question.

In practice I think clone, checkout, reset etc. always work in the same
order you see with `git ls-tree -r --name-only HEAD`, but as far as I
know this has never been guaranteed or documented, and shouldn't be
relied on.

E.g. there's probably cases where writing files in parallel is going to
be faster than writing them sequentially. We don't have such a mode just
because nobody's written a patch for it, but having that patch would
break any assumptions of our current order.

Re: Are clone/checkout operations deterministic?

[Stefan Beller]

On Tue, Jul 17, 2018 at 2:48 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Tue, Jul 17 2018, J. Paul Reed wrote:
>
> > Hey Git Devs,
> >
> > I have a bit of an odd question: do git clone/checkout operations have a
> > deterministic ordering?
> >
> > That is: are files guaranteed to be laid down onto disk in any specific
> > (and deterministic) order as a clone and/or checkout operations occurs?
> > (And if so, is it alphabetical order? Depth-first? Something else?)
> >
> > In case the answer is different (and I'd guess that it might be?), I'm
> > mostly interested in the initial clone case... but it would be great to
> > know if, indeed, the answer is different for just-checkouts too.
> >
> > I did some cursory googling, but nothing hopped out at me as an answer to
> > this question.
>
> In practice I think clone, checkout, reset etc. always work in the same
> order you see with `git ls-tree -r --name-only HEAD`, but as far as I
> know this has never been guaranteed or documented, and shouldn't be
> relied on.

The transmission of packfiles is non-deterministic, as the packfiles
(which are packed for each clone separately when using core git as
a server) are not packed in a deterministic fashion, but in a threaded
environment which allows different packing orders.

If you clone from a server that gives you exactly the same pack at
all times (assuming the remote repo doesn't change refs), then
checkout is currently deterministic in unpacking files to the working tree.

>
> E.g. there's probably cases where writing files in parallel is going to
> be faster than writing them sequentially. We don't have such a mode just
> because nobody's written a patch for it, but having that patch would
> break any assumptions of our current order.

+cc Ben who is looking into that, but hasn't spoken up on the mailing list yet.

Re: Are clone/checkout operations deterministic?

[Duy Nguyen]

On Tue, Jul 17, 2018 at 11:50 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> In practice I think clone, checkout, reset etc. always work in the same
> order you see with `git ls-tree -r --name-only HEAD`, but as far as I
> know this has never been guaranteed or documented, and shouldn't be
> relied on.
>
> E.g. there's probably cases where writing files in parallel is going to
> be faster than writing them sequentially. We don't have such a mode just
> because nobody's written a patch for it, but having that patch would

Well I did have some patches [1] but as usually I did not follow
through. Interestingly there's actually concern about timestamp order
[2] even back then

[1] https://public-inbox.org/git/20160415095139.GA3985@lanh/
[2] https://public-inbox.org/git/CAP8UFD0WZHriY340eh3K6ygzb0tXnoT+XaY8+c2k+N2x9UBYxA@mail.gmail.com/

> break any assumptions of our current order.
-- 
Duy

Re: Are clone/checkout operations deterministic?

[Jeff King]

On Tue, Jul 17, 2018 at 11:48:45AM +0200, Ævar Arnfjörð Bjarmason wrote:

> In practice I think clone, checkout, reset etc. always work in the same
> order you see with `git ls-tree -r --name-only HEAD`, but as far as I
> know this has never been guaranteed or documented, and shouldn't be
> relied on.

I think this paragraph is correct in general (and I agree with the
sentiment that this is subject to change in future versions).

There is one concrete case I know that has non-deterministic order in
current versions: long-lived clean/smudge filters can defer their
response. The LFS filter uses this to tell Git "no, I'm still
downloading the content", at which point Git will proceed with checking
out other local files (or even other LFS files that happen to arrive
sooner).

Depending on what one wants to do with the determinism, it may be OK to
ignore that case. ;)

-Peff