Possible to unshallow or deepen based on local objects?

Possible to unshallow or deepen based on local objects?

[Mark Rushakoff]

I am building some CI tooling, and I am working with a large-ish repository, so
I am trying to start with a shallow clone of the repository and deepen it on
demand. I am finding it very difficult to correctly switch between fetch
--depth and fetch --deepen.

I am looking for a way to "recover" from a fetch --depth call that made my
history shallower than before. Ultimately, I may take a different approach to
just work around this altogether, but I thought I'd ask here first.

In particular, if there is history like this:

A --- B --- C --- D --- E --- F
       \                 \
         ---------------- G
And my shallow history includes D..F, then I believe I am limited in my options
for fetching G:

- If I do a plain fetch of G, or if I use --deepen, then I pull in A and all of
  its history, which I don't strictly need or want yet.
- If I know G's parent is B, and I fetch --depth=1, then (I think) I will see
  B, E, F, and G, effectively losing sight of D

As a more contrived example, suppose I have a shallow clone of a repository:

bash-3.2$ git clone --depth=4 -q git@github.com:git/git.git
bash-3.2$ cd git
bash-3.2$ git log --format='%h %d'
3034dab  (HEAD -> master, origin/master, origin/HEAD)
98e06de
352253a
4098130  (grafted)
7b974e3
df63c2e
75ce486  (grafted)
70b39fb  (grafted)
afc3bf6  (grafted)
bash-3.2$ cat .git/shallow
409813088ad55ae4a60f55412f6b5ba6a89d89e7
70b39fbede78313656e8a6bd9b38b238ab10db2f
75ce48674889df6a2bb493fb5d6bef0ef60ca7ae
afc3bf6eb13d9fc489b569164819cff44db8ac17

And then suppose I fetch with depth smaller than before:

bash-3.2$ git fetch --depth=1 -q
bash-3.2$ git log --format='%h %d'
3034dab  (grafted, HEAD -> master, origin/master, origin/HEAD)
bash-3.2$ cat .git/shallow
3034dab9ed6b11970a53099a7b3ca981f1461365
409813088ad55ae4a60f55412f6b5ba6a89d89e7
70b39fbede78313656e8a6bd9b38b238ab10db2f
75ce48674889df6a2bb493fb5d6bef0ef60ca7ae
afc3bf6eb13d9fc489b569164819cff44db8ac17

I can still see the commits I had before the --depth=1 fetch:

bash-3.2$ git log -1 --format=oneline 75ce486
75ce48674889df6a2bb493fb5d6bef0ef60ca7ae (grafted) Merge branch
'di/readme-markup-fix'

Short of looking through .git/shallow and removing each entry whose parents I
can resolve, is there a built-in command to "unshallow based on local objects"?
I have tried many web search terms, I have looked through the docs for many
low-level git commands, I have tried variations of "git fetch .", and I am
stumped.

Re: Possible to unshallow or deepen based on local objects?

[Mark Rushakoff]

On Fri, Jul 26, 2019 at 11:43 AM Mark Rushakoff <mark@influxdata.com> wrote:
>
> I am building some CI tooling, and I am working with a large-ish repository, so
> I am trying to start with a shallow clone of the repository and deepen it on
> demand. I am finding it very difficult to correctly switch between fetch
> --depth and fetch --deepen.
>
> I am looking for a way to "recover" from a fetch --depth call that made my
> history shallower than before. Ultimately, I may take a different approach to
> just work around this altogether, but I thought I'd ask here first.
>
> In particular, if there is history like this:
>
> A --- B --- C --- D --- E --- F
>        \                 \
>          ---------------- G
> And my shallow history includes D..F, then I believe I am limited in my options
> for fetching G:
>
> - If I do a plain fetch of G, or if I use --deepen, then I pull in A and all of
>   its history, which I don't strictly need or want yet.
> - If I know G's parent is B, and I fetch --depth=1, then (I think) I will see
>   B, E, F, and G, effectively losing sight of D
>
> As a more contrived example, suppose I have a shallow clone of a repository:
>
> bash-3.2$ git clone --depth=4 -q git@github.com:git/git.git
> bash-3.2$ cd git
> bash-3.2$ git log --format='%h %d'
> 3034dab  (HEAD -> master, origin/master, origin/HEAD)
> 98e06de
> 352253a
> 4098130  (grafted)
> 7b974e3
> df63c2e
> 75ce486  (grafted)
> 70b39fb  (grafted)
> afc3bf6  (grafted)
> bash-3.2$ cat .git/shallow
> 409813088ad55ae4a60f55412f6b5ba6a89d89e7
> 70b39fbede78313656e8a6bd9b38b238ab10db2f
> 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae
> afc3bf6eb13d9fc489b569164819cff44db8ac17
>
> And then suppose I fetch with depth smaller than before:
>
> bash-3.2$ git fetch --depth=1 -q
> bash-3.2$ git log --format='%h %d'
> 3034dab  (grafted, HEAD -> master, origin/master, origin/HEAD)
> bash-3.2$ cat .git/shallow
> 3034dab9ed6b11970a53099a7b3ca981f1461365
> 409813088ad55ae4a60f55412f6b5ba6a89d89e7
> 70b39fbede78313656e8a6bd9b38b238ab10db2f
> 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae
> afc3bf6eb13d9fc489b569164819cff44db8ac17
>
> I can still see the commits I had before the --depth=1 fetch:
>
> bash-3.2$ git log -1 --format=oneline 75ce486
> 75ce48674889df6a2bb493fb5d6bef0ef60ca7ae (grafted) Merge branch
> 'di/readme-markup-fix'
>
> Short of looking through .git/shallow and removing each entry whose parents I
> can resolve, is there a built-in command to "unshallow based on local objects"?
> I have tried many web search terms, I have looked through the docs for many
> low-level git commands, I have tried variations of "git fetch .", and I am
> stumped.

I've run into another flavor of this problem now.

I have a shallow clone of a repository. Then sometime later, I obtain a bundle
containing full history.

I thought I could use git fetch --unshallow /path/to/repo.bundle
mybranch:mybranch, but I've found that git does not support shallow fetches
from a bundle and that unshallow is actually an alias for --depth=2147483647 in
4dcb167fc3 (fetch: add --unshallow for turning shallow repo into complete one,
2013-01-11).

If I do a plain fetch from the bundle, I can manually modify .git/shallow to
"unshallow" my repository like the previous email in this thread. But what I
think I still want here is a way to tell git "here are all the objects I have;
update the shallow boundaries accordingly."

Re: Possible to unshallow or deepen based on local objects?

[Junio C Hamano]

Mark Rushakoff <mark@influxdata.com> writes:

> I have a shallow clone of a repository. Then sometime later, I obtain a bundle
> containing full history.
>
> I thought I could use git fetch --unshallow /path/to/repo.bundle
> mybranch:mybranch, but I've found that git does not support shallow fetches
> from a bundle and that unshallow is actually an alias for --depth=2147483647 in
> 4dcb167fc3 (fetch: add --unshallow for turning shallow repo into complete one,
> 2013-01-11).
>
> If I do a plain fetch from the bundle, I can manually modify .git/shallow to
> "unshallow" my repository like the previous email in this thread. But what I
> think I still want here is a way to tell git "here are all the objects I have;
> update the shallow boundaries accordingly."

It sounds like a quite sensible request to me.  

Offhand, I do not think of anything that would make it fundamentally
impossible to allow us to first add objects to the .git/objects
directory by whatever means available to you (including, but not
limited to, exploding the object data from a full bundle) and then
to tell Git to see if there (still) are missing commits and objects
that ought to exist but don't that necessitates the repository to be
marked "shallow", and adjust the boundary.

I however do not think there is such a code already written X-<.