[BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Vadim Eisenberg]

/usr/local/bin/git clone --recursive --depth 10 
https://github.com/IBM-Swift/Kitura-net.git
Cloning into 'Kitura-net'...
remote: Counting objects: 253, done.
remote: Compressing objects: 100% (142/142), done.
remote: Total 253 (delta 134), reused 188 (delta 86), pack-reused 0
Receiving objects: 100% (253/253), 63.28 KiB | 0 bytes/s, done.
Resolving deltas: 100% (134/134), done.
Checking connectivity... done.
Submodule 'Kitura-Build' (https://github.com/IBM-Swift/Kitura-Build.git) 
registered for path 'Kitura-Build'
Cloning into '/home/vadime/Kitura-net/Kitura-Build'...
error: no such remote ref d0d9d6c739a79627641e6438fe4f39bd0eba83bb
Fetched in submodule path 'Kitura-Build', but it did not contain 
d0d9d6c739a79627641e6438fe4f39bd0eba83bb. Direct fetching of that commit 
failed.

Linux version:
4.2.0-25-generic #30-Ubuntu SMP Mon Jan 18 12:31:50 UTC 2016 x86_64 x86_64 
x86_64 GNU/Linux

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Jeff King]

On Sun, Jun 19, 2016 at 10:17:36AM +0300, Vadim Eisenberg wrote:

> /usr/local/bin/git clone --recursive --depth 10 
> https://github.com/IBM-Swift/Kitura-net.git
> Cloning into 'Kitura-net'...
> remote: Counting objects: 253, done.
> remote: Compressing objects: 100% (142/142), done.
> remote: Total 253 (delta 134), reused 188 (delta 86), pack-reused 0
> Receiving objects: 100% (253/253), 63.28 KiB | 0 bytes/s, done.
> Resolving deltas: 100% (134/134), done.
> Checking connectivity... done.
> Submodule 'Kitura-Build' (https://github.com/IBM-Swift/Kitura-Build.git) 
> registered for path 'Kitura-Build'
> Cloning into '/home/vadime/Kitura-net/Kitura-Build'...
> error: no such remote ref d0d9d6c739a79627641e6438fe4f39bd0eba83bb
> Fetched in submodule path 'Kitura-Build', but it did not contain 
> d0d9d6c739a79627641e6438fe4f39bd0eba83bb. Direct fetching of that commit 
> failed.

The problem seems to be the shallow clone. The super-project points to a
commit in the submodule that is not near the tip of any branch, so
shallow-cloning the submodule means we don't get that commit. Prior to
d22eb04 (clone: add `--shallow-submodules` flag, 2016-04-25), submodules
were _always_ cloned fully.

The immediate workaround is to add "--no-shallow-submodules" to your
clone invocation.

Stefan, I think it might be worth revisiting the default set by d22eb04
to propagate shallowness from the super-project clone. In an ideal
world, we would be asking each submodule for the actual commit we are
interested in, and shallowness would not matter. But until
uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
going to be a problem.

-Peff

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Vadim Eisenberg]

The problem is with other tools that use git, like Swift Package Manager - 
https://swift.org/package-manager/.
Versions of git before 2.9.0 have no option --no-shallow-submodules. So 
the tools that use git would have to check if the version of git is 
greater than or equal 2.9.0 to know to specify that option.

Best Regards,
Vadim

Jeff King <peff@peff.net> wrote on 06/19/2016 01:00:51 PM:

> From: Jeff King <peff@peff.net>
> To: Vadim Eisenberg/Haifa/IBM@IBMIL
> Cc: Stefan Beller <sbeller@google.com>, git@vger.kernel.org
> Date: 06/19/2016 01:01 PM
> Subject: Re: [BUG REPORT] git 2.9.0 clone --recursive fails on 
> cloning a submodule
> 
> On Sun, Jun 19, 2016 at 10:17:36AM +0300, Vadim Eisenberg wrote:
> 
> > /usr/local/bin/git clone --recursive --depth 10 
> > https://github.com/IBM-Swift/Kitura-net.git
> > Cloning into 'Kitura-net'...
> > remote: Counting objects: 253, done.
> > remote: Compressing objects: 100% (142/142), done.
> > remote: Total 253 (delta 134), reused 188 (delta 86), pack-reused 0
> > Receiving objects: 100% (253/253), 63.28 KiB | 0 bytes/s, done.
> > Resolving deltas: 100% (134/134), done.
> > Checking connectivity... done.
> > Submodule 'Kitura-Build' (
https://github.com/IBM-Swift/Kitura-Build.git) 
> > registered for path 'Kitura-Build'
> > Cloning into '/home/vadime/Kitura-net/Kitura-Build'...
> > error: no such remote ref d0d9d6c739a79627641e6438fe4f39bd0eba83bb
> > Fetched in submodule path 'Kitura-Build', but it did not contain 
> > d0d9d6c739a79627641e6438fe4f39bd0eba83bb. Direct fetching of that 
commit 
> > failed.
> 
> The problem seems to be the shallow clone. The super-project points to a
> commit in the submodule that is not near the tip of any branch, so
> shallow-cloning the submodule means we don't get that commit. Prior to
> d22eb04 (clone: add `--shallow-submodules` flag, 2016-04-25), submodules
> were _always_ cloned fully.
> 
> The immediate workaround is to add "--no-shallow-submodules" to your
> clone invocation.
> 
> Stefan, I think it might be worth revisiting the default set by d22eb04
> to propagate shallowness from the super-project clone. In an ideal
> world, we would be asking each submodule for the actual commit we are
> interested in, and shallowness would not matter. But until
> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
> going to be a problem.
> 
> -Peff
> 

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Jeff King]

On Sun, Jun 19, 2016 at 04:07:41PM +0300, Vadim Eisenberg wrote:

> The problem is with other tools that use git, like Swift Package Manager - 
> https://swift.org/package-manager/.
> Versions of git before 2.9.0 have no option --no-shallow-submodules. So 
> the tools that use git would have to check if the version of git is 
> greater than or equal 2.9.0 to know to specify that option.

Oh, I agree it's something we should treat as a regression. Using
`--no-shallow-submodules` is just a workaround.

-Peff

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Stefan, I think it might be worth revisiting the default set by d22eb04
> to propagate shallowness from the super-project clone. In an ideal
> world, we would be asking each submodule for the actual commit we are
> interested in, and shallowness would not matter. But until
> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
> going to be a problem.

Yup, something like this on top of d22eb04 to be merged before
v2.9.1 for the maintenance track would be necessary.

-- >8 --
Subject: clone: do not let --depth imply --shallow-submodules

In v2.9.0, we prematurely flipped the default to force cloning
submodules shallowly, when the superproject is getting cloned
shallowly.  This is likely to fail when the upstream repositories
submodules are cloned from a repository that is not prepared to
serve histories that ends at a commit that is not at the tip of a
branch, and we know the world is not yet ready.

Use a safer default to clone the submodules fully, unless the user
tells us that she knows that the upstream repository of the
submodules are willing to cooperate with "--shallow-submodules"
option.

Noticed-by: Vadim Eisenberg <VADIME@il.ibm.com>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-clone.txt | 5 ++---
 builtin/clone.c             | 5 ++---
 t/t5614-clone-submodules.sh | 4 ++--
 3 files changed, 6 insertions(+), 8 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index e1a21b7..c5a1ce2 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -192,9 +192,8 @@ objects from the source repository into a pack in the cloned repository.
 	Create a 'shallow' clone with a history truncated to the
 	specified number of revisions. Implies `--single-branch` unless
 	`--no-single-branch` is given to fetch the histories near the
-	tips of all branches. This implies `--shallow-submodules`. If
-	you want to have a shallow superproject clone, but full submodules,
-	also pass `--no-shallow-submodules`.
+	tips of all branches. If you want to clone submodules shallowly,
+	also pass `--shallow-submodules`.
 
 --[no-]single-branch::
 	Clone only the history leading to the tip of a single branch,
diff --git a/builtin/clone.c b/builtin/clone.c
index ecdf308..f267742 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -40,7 +40,7 @@ static const char * const builtin_clone_usage[] = {
 
 static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
 static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
-static int option_shallow_submodules = -1;
+static int option_shallow_submodules;
 static char *option_template, *option_depth;
 static char *option_origin = NULL;
 static char *option_branch = NULL;
@@ -730,8 +730,7 @@ static int checkout(void)
 		struct argv_array args = ARGV_ARRAY_INIT;
 		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
 
-		if (option_shallow_submodules == 1
-		    || (option_shallow_submodules == -1 && option_depth))
+		if (option_shallow_submodules == 1)
 			argv_array_push(&args, "--depth=1");
 
 		if (max_jobs != -1)
diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh
index 62044c5..f7c630b 100755
--- a/t/t5614-clone-submodules.sh
+++ b/t/t5614-clone-submodules.sh
@@ -37,9 +37,9 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
 	)
 '
 
-test_expect_success 'shallow clone implies shallow submodule' '
+test_expect_success 'shallow clone does not imply shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
-	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
+	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
 	(
 		cd super_clone &&
 		git log --oneline >lines &&

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Jeff King]

On Sun, Jun 19, 2016 at 01:51:56PM -0700, Junio C Hamano wrote:

> Yup, something like this on top of d22eb04 to be merged before
> v2.9.1 for the maintenance track would be necessary.
> 
> -- >8 --
> Subject: clone: do not let --depth imply --shallow-submodules
> 
> In v2.9.0, we prematurely flipped the default to force cloning
> submodules shallowly, when the superproject is getting cloned
> shallowly.  This is likely to fail when the upstream repositories
> submodules are cloned from a repository that is not prepared to
> serve histories that ends at a commit that is not at the tip of a
> branch, and we know the world is not yet ready.
> 
> Use a safer default to clone the submodules fully, unless the user
> tells us that she knows that the upstream repository of the
> submodules are willing to cooperate with "--shallow-submodules"
> option.

Yeah, this looks good. To minor comments:

> @@ -730,8 +730,7 @@ static int checkout(void)
>  		struct argv_array args = ARGV_ARRAY_INIT;
>  		argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>  
> -		if (option_shallow_submodules == 1
> -		    || (option_shallow_submodules == -1 && option_depth))
> +		if (option_shallow_submodules == 1)
>  			argv_array_push(&args, "--depth=1");

I hadn't paid much attention to this topic originally, but was surprised
that "--depth 10" in the clone implies "--depth 1" in the submodule.
This is not really related to your patch (in fact, your patch makes the
logic go away). But maybe something to consider if it's ever resurrected
(or possibly if somebody runs "--shallow-submodules --depth 5" we should
pass --depth=1; I dunno).

> -test_expect_success 'shallow clone implies shallow submodule' '
> +test_expect_success 'shallow clone does not imply shallow submodule' '
>  	test_when_finished "rm -rf super_clone" &&
> -	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
> +	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
>  	(
>  		cd super_clone &&
>  		git log --oneline >lines &&

We are not really testing "does not imply" here, but "passing
--shallow-submodules works". The "does not imply" test would be cloning
without the option and checking that the resulting submodules are not
shallow.

-Peff

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Stefan Beller]

On Sun, Jun 19, 2016 at 5:13 PM, Jeff King <peff@peff.net> wrote:
> On Sun, Jun 19, 2016 at 01:51:56PM -0700, Junio C Hamano wrote:
>
>> Yup, something like this on top of d22eb04 to be merged before
>> v2.9.1 for the maintenance track would be necessary.
>>
>> -- >8 --
>> Subject: clone: do not let --depth imply --shallow-submodules
>>
>> In v2.9.0, we prematurely flipped the default to force cloning
>> submodules shallowly, when the superproject is getting cloned
>> shallowly.  This is likely to fail when the upstream repositories
>> submodules are cloned from a repository that is not prepared to
>> serve histories that ends at a commit that is not at the tip of a
>> branch, and we know the world is not yet ready.
>>
>> Use a safer default to clone the submodules fully, unless the user
>> tells us that she knows that the upstream repository of the
>> submodules are willing to cooperate with "--shallow-submodules"
>> option.
>
> Yeah, this looks good. To minor comments:

I agree, but I find the second concern a bit more than just minor.

>
>> @@ -730,8 +730,7 @@ static int checkout(void)
>>               struct argv_array args = ARGV_ARRAY_INIT;
>>               argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>>
>> -             if (option_shallow_submodules == 1
>> -                 || (option_shallow_submodules == -1 && option_depth))
>> +             if (option_shallow_submodules == 1)
>>                       argv_array_push(&args, "--depth=1");
>
> I hadn't paid much attention to this topic originally, but was surprised
> that "--depth 10" in the clone implies "--depth 1" in the submodule.
> This is not really related to your patch (in fact, your patch makes the
> logic go away). But maybe something to consider if it's ever resurrected
> (or possibly if somebody runs "--shallow-submodules --depth 5" we should
> pass --depth=1; I dunno).

How often do we see a depth != 1 in practice?
I have the impression (and no data to back up my claim) that a binary
switch for nonshallow or depth 1 would serve us just as good, which is why
I did not want to ad complexity to the submodule depth.
(What if you want submodule A with depth 2 and B with 5? In that
case get them all shallow and deepen as appropriate, would be my answer)

>
>> -test_expect_success 'shallow clone implies shallow submodule' '
>> +test_expect_success 'shallow clone does not imply shallow submodule' '
>>       test_when_finished "rm -rf super_clone" &&
>> -     git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
>> +     git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
>>       (
>>               cd super_clone &&
>>               git log --oneline >lines &&
>
> We are not really testing "does not imply" here, but "passing
> --shallow-submodules works". The "does not imply" test would be cloning
> without the option and checking that the resulting submodules are not
> shallow.

In case we want to be sure that it works for 2.9.1, i.e. we treat it
as a regression,
we need to test the "does not imply" a bit more I would think. I can send that
test on top tomorrow if you'd like to.

Thanks,
Stefan

>
> -Peff

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Vadim Eisenberg]

> How often do we see a depth != 1 in practice? 

FYI: Here is an example of depth 10 from Swift Package Manager which is 
git-based 
https://github.com/apple/swift-package-manager/blob/61bdff3e9c2b8456be31861b4cda87a31440d517/Sources/Get/Git.swift

try system(Git.tool, "clone",
        "--recursive",   // get submodules too so that developers can use 
these if they so choose
        "--depth", "10",

Stefan Beller <sbeller@google.com> wrote on 06/20/2016 04:09:28 AM:

> From: Stefan Beller <sbeller@google.com>
> To: Jeff King <peff@peff.net>
> Cc: Junio C Hamano <gitster@pobox.com>, Vadim Eisenberg/Haifa/
> IBM@IBMIL, "git@vger.kernel.org" <git@vger.kernel.org>
> Date: 06/20/2016 04:09 AM
> Subject: Re: [BUG REPORT] git 2.9.0 clone --recursive fails on 
> cloning a submodule
> 
> On Sun, Jun 19, 2016 at 5:13 PM, Jeff King <peff@peff.net> wrote:
> > On Sun, Jun 19, 2016 at 01:51:56PM -0700, Junio C Hamano wrote:
> >
> >> Yup, something like this on top of d22eb04 to be merged before
> >> v2.9.1 for the maintenance track would be necessary.
> >>
> >> -- >8 --
> >> Subject: clone: do not let --depth imply --shallow-submodules
> >>
> >> In v2.9.0, we prematurely flipped the default to force cloning
> >> submodules shallowly, when the superproject is getting cloned
> >> shallowly.  This is likely to fail when the upstream repositories
> >> submodules are cloned from a repository that is not prepared to
> >> serve histories that ends at a commit that is not at the tip of a
> >> branch, and we know the world is not yet ready.
> >>
> >> Use a safer default to clone the submodules fully, unless the user
> >> tells us that she knows that the upstream repository of the
> >> submodules are willing to cooperate with "--shallow-submodules"
> >> option.
> >
> > Yeah, this looks good. To minor comments:
> 
> I agree, but I find the second concern a bit more than just minor.
> 
> >
> >> @@ -730,8 +730,7 @@ static int checkout(void)
> >>               struct argv_array args = ARGV_ARRAY_INIT;
> >>               argv_array_pushl(&args, "submodule", "update", "--
> init", "--recursive", NULL);
> >>
> >> -             if (option_shallow_submodules == 1
> >> -                 || (option_shallow_submodules == -1 && 
option_depth))
> >> +             if (option_shallow_submodules == 1)
> >>                       argv_array_push(&args, "--depth=1");
> >
> > I hadn't paid much attention to this topic originally, but was 
surprised
> > that "--depth 10" in the clone implies "--depth 1" in the submodule.
> > This is not really related to your patch (in fact, your patch makes 
the
> > logic go away). But maybe something to consider if it's ever 
resurrected
> > (or possibly if somebody runs "--shallow-submodules --depth 5" we 
should
> > pass --depth=1; I dunno).
> 
> How often do we see a depth != 1 in practice?
> I have the impression (and no data to back up my claim) that a binary
> switch for nonshallow or depth 1 would serve us just as good, which is 
why
> I did not want to ad complexity to the submodule depth.
> (What if you want submodule A with depth 2 and B with 5? In that
> case get them all shallow and deepen as appropriate, would be my answer)
> 
> >
> >> -test_expect_success 'shallow clone implies shallow submodule' '
> >> +test_expect_success 'shallow clone does not imply shallow submodule' 
'
> >>       test_when_finished "rm -rf super_clone" &&
> >> -     git clone --recurse-submodules --depth 2 "file://$pwd/." 
> super_clone &&
> >> +     git clone --recurse-submodules --depth 2 --shallow-submodules "
> file://$pwd/." super_clone &&
> >>       (
> >>               cd super_clone &&
> >>               git log --oneline >lines &&
> >
> > We are not really testing "does not imply" here, but "passing
> > --shallow-submodules works". The "does not imply" test would be 
cloning
> > without the option and checking that the resulting submodules are not
> > shallow.
> 
> In case we want to be sure that it works for 2.9.1, i.e. we treat it
> as a regression,
> we need to test the "does not imply" a bit more I would think. I cansend 
that
> test on top tomorrow if you'd like to.
> 
> Thanks,
> Stefan
> 
> >
> > -Peff
> 

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Dennis Kaarsemaker]

On zo, 2016-06-19 at 18:09 -0700, Stefan Beller wrote:

> How often do we see a depth != 1 in practice?

Travis clones with --depth=50

D.

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Jeff King]

On Sun, Jun 19, 2016 at 06:09:28PM -0700, Stefan Beller wrote:

> > I hadn't paid much attention to this topic originally, but was surprised
> > that "--depth 10" in the clone implies "--depth 1" in the submodule.
> > This is not really related to your patch (in fact, your patch makes the
> > logic go away). But maybe something to consider if it's ever resurrected
> > (or possibly if somebody runs "--shallow-submodules --depth 5" we should
> > pass --depth=1; I dunno).
> 
> How often do we see a depth != 1 in practice?
> I have the impression (and no data to back up my claim) that a binary
> switch for nonshallow or depth 1 would serve us just as good, which is why
> I did not want to ad complexity to the submodule depth.
> (What if you want submodule A with depth 2 and B with 5? In that
> case get them all shallow and deepen as appropriate, would be my answer)

To be honest, I don't know why people use anything except --depth=1, but
it's clear from my experience that they do. This example has --depth=10,
and on the server side at GitHub I have seen similar numbers from clients,
especially CI services.

(I take special note of such cases because --shallow quite often causes
performance problems on the server side, though generally --depth=10 is
not any worse than --depth=1. The worst case is really
"--no-single-branch --depth=1", which wants a ton of objects but has to
throw away all of the on-disk deltas).

> > We are not really testing "does not imply" here, but "passing
> > --shallow-submodules works". The "does not imply" test would be cloning
> > without the option and checking that the resulting submodules are not
> > shallow.
> 
> In case we want to be sure that it works for 2.9.1, i.e. we treat it
> as a regression,
> we need to test the "does not imply" a bit more I would think. I can send that
> test on top tomorrow if you'd like to.

I think it's worth doing (and testing both: the default behavior, and
that the --shallow-submodules feature works). Thanks.

-Peff

[PATCH] shallow clone to not imply shallow submodules

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---

Hi Junio, Peff,

I thought about this patch squashed into  
"clone: do not let --depth imply --shallow-submodules" will actually test
for the regression.

Thanks,
Stefan

 t/t5614-clone-submodules.sh | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh
index f7c630b..a9aaa01 100755
--- a/t/t5614-clone-submodules.sh
+++ b/t/t5614-clone-submodules.sh
@@ -37,7 +37,7 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
 	)
 '
 
-test_expect_success 'shallow clone does not imply shallow submodule' '
+test_expect_success 'shallow clone with shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
 	(
@@ -52,6 +52,21 @@ test_expect_success 'shallow clone does not imply shallow submodule' '
 	)
 '
 
+test_expect_success 'shallow clone does not imply shallow submodule' '
+	test_when_finished "rm -rf super_clone" &&
+	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
+	(
+		cd super_clone &&
+		git log --oneline >lines &&
+		test_line_count = 2 lines
+	) &&
+	(
+		cd super_clone/sub &&
+		git log --oneline >lines &&
+		test_line_count = 3 lines
+	)
+'
+
 test_expect_success 'shallow clone with non shallow submodule' '
 	test_when_finished "rm -rf super_clone" &&
 	git clone --recurse-submodules --depth 2 --no-shallow-submodules "file://$pwd/." super_clone &&
-- 
2.7.0.rc0.40.g5328432.dirty

Re: [PATCH] shallow clone to not imply shallow submodules

[Jeff King]

On Mon, Jun 20, 2016 at 09:59:58AM -0700, Stefan Beller wrote:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
> 
> Hi Junio, Peff,
> 
> I thought about this patch squashed into  
> "clone: do not let --depth imply --shallow-submodules" will actually test
> for the regression.

Yep, it looks good to me.

> +test_expect_success 'shallow clone does not imply shallow submodule' '
> +	test_when_finished "rm -rf super_clone" &&
> +	git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
> +	(
> +		cd super_clone &&
> +		git log --oneline >lines &&
> +		test_line_count = 2 lines
> +	) &&
> +	(
> +		cd super_clone/sub &&
> +		git log --oneline >lines &&
> +		test_line_count = 3 lines
> +	)
> +'

This follows the style of the other tests, so it's the right thing here.
But as a style suggestion, I think:

  git -C super_clone/sub log --oneline >lines &&
  test_line_count = 3 lines

is nicer than the subshell. It's more succinct, and it saves a process.

-Peff

Re: [PATCH] shallow clone to not imply shallow submodules

[Stefan Beller]

On Mon, Jun 20, 2016 at 10:13 AM, Jeff King <peff@peff.net> wrote:
> On Mon, Jun 20, 2016 at 09:59:58AM -0700, Stefan Beller wrote:
>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>>
>> Hi Junio, Peff,
>>
>> I thought about this patch squashed into
>> "clone: do not let --depth imply --shallow-submodules" will actually test
>> for the regression.
>
> Yep, it looks good to me.
>
>> +test_expect_success 'shallow clone does not imply shallow submodule' '
>> +     test_when_finished "rm -rf super_clone" &&
>> +     git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
>> +     (
>> +             cd super_clone &&
>> +             git log --oneline >lines &&
>> +             test_line_count = 2 lines
>> +     ) &&
>> +     (
>> +             cd super_clone/sub &&
>> +             git log --oneline >lines &&
>> +             test_line_count = 3 lines
>> +     )
>> +'
>
> This follows the style of the other tests, so it's the right thing here.
> But as a style suggestion, I think:
>
>   git -C super_clone/sub log --oneline >lines &&
>   test_line_count = 3 lines
>
> is nicer than the subshell. It's more succinct, and it saves a process.

which we would want to refactor to in a follow up, but not merge it
through to 2.9.1.

Thanks,
Stefan

>
> -Peff

Re: [PATCH] shallow clone to not imply shallow submodules

[Jeff King]

On Mon, Jun 20, 2016 at 10:14:47AM -0700, Stefan Beller wrote:

> > This follows the style of the other tests, so it's the right thing here.
> > But as a style suggestion, I think:
> >
> >   git -C super_clone/sub log --oneline >lines &&
> >   test_line_count = 3 lines
> >
> > is nicer than the subshell. It's more succinct, and it saves a process.
> 
> which we would want to refactor to in a follow up, but not merge it
> through to 2.9.1.

Yeah, exactly. That was what I meant by "here".

-Peff

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Duy Nguyen]

On Sun, Jun 19, 2016 at 12:00 PM, Jeff King <peff@peff.net> wrote:
> Stefan, I think it might be worth revisiting the default set by d22eb04
> to propagate shallowness from the super-project clone. In an ideal
> world, we would be asking each submodule for the actual commit we are
> interested in, and shallowness would not matter. But until
> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
> going to be a problem.

Maybe we can pass an option to subsequent clones that say "if
allow-...-sha1-in-want is advertised, do a shallow clone, otherwise
fall back to full clone"?
-- 
Duy

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Ævar Arnfjörð Bjarmason]

On Sun, Jun 19, 2016 at 10:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jeff King <peff@peff.net> writes:
>
>> Stefan, I think it might be worth revisiting the default set by d22eb04
>> to propagate shallowness from the super-project clone. In an ideal
>> world, we would be asking each submodule for the actual commit we are
>> interested in, and shallowness would not matter. But until
>> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
>> going to be a problem.
>
> Yup, something like this on top of d22eb04 to be merged before
> v2.9.1 for the maintenance track would be necessary.
>
> -- >8 --
> Subject: clone: do not let --depth imply --shallow-submodules
>
> In v2.9.0, we prematurely flipped the default to force cloning
> submodules shallowly, when the superproject is getting cloned
> shallowly.  This is likely to fail when the upstream repositories
> submodules are cloned from a repository that is not prepared to
> serve histories that ends at a commit that is not at the tip of a
> branch, and we know the world is not yet ready.
>
> Use a safer default to clone the submodules fully, unless the user
> tells us that she knows that the upstream repository of the
> submodules are willing to cooperate with "--shallow-submodules"
> option.
>
> Noticed-by: Vadim Eisenberg <VADIME@il.ibm.com>
> Helped-by: Jeff King <peff@peff.net>
> Signed-off-by: Junio C Hamano <gitster@pobox.com>
> ---
>  Documentation/git-clone.txt | 5 ++---
>  builtin/clone.c             | 5 ++---
>  t/t5614-clone-submodules.sh | 4 ++--
>  3 files changed, 6 insertions(+), 8 deletions(-)
>
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index e1a21b7..c5a1ce2 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -192,9 +192,8 @@ objects from the source repository into a pack in the cloned repository.
>         Create a 'shallow' clone with a history truncated to the
>         specified number of revisions. Implies `--single-branch` unless
>         `--no-single-branch` is given to fetch the histories near the
> -       tips of all branches. This implies `--shallow-submodules`. If
> -       you want to have a shallow superproject clone, but full submodules,
> -       also pass `--no-shallow-submodules`.
> +       tips of all branches. If you want to clone submodules shallowly,
> +       also pass `--shallow-submodules`.
>
>  --[no-]single-branch::
>         Clone only the history leading to the tip of a single branch,
> diff --git a/builtin/clone.c b/builtin/clone.c
> index ecdf308..f267742 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -40,7 +40,7 @@ static const char * const builtin_clone_usage[] = {
>
>  static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
>  static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
> -static int option_shallow_submodules = -1;
> +static int option_shallow_submodules;
>  static char *option_template, *option_depth;
>  static char *option_origin = NULL;
>  static char *option_branch = NULL;
> @@ -730,8 +730,7 @@ static int checkout(void)
>                 struct argv_array args = ARGV_ARRAY_INIT;
>                 argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>
> -               if (option_shallow_submodules == 1
> -                   || (option_shallow_submodules == -1 && option_depth))
> +               if (option_shallow_submodules == 1)
>                         argv_array_push(&args, "--depth=1");

Very late reply, since I'm just looking at this now with the --no-tags
opti,n, but that == 1 makes no sense anymore, and should just be `if
(option_shallow_submodules)` shouldn't it? I.e. this used to be an int
for the depth, now is a bool, but the current == 1 check is left over
probably from an earlier version where the depth was configurable.

>                 if (max_jobs != -1)
> diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh
> index 62044c5..f7c630b 100755
> --- a/t/t5614-clone-submodules.sh
> +++ b/t/t5614-clone-submodules.sh
> @@ -37,9 +37,9 @@ test_expect_success 'nonshallow clone implies nonshallow submodule' '
>         )
>  '
>
> -test_expect_success 'shallow clone implies shallow submodule' '
> +test_expect_success 'shallow clone does not imply shallow submodule' '
>         test_when_finished "rm -rf super_clone" &&
> -       git clone --recurse-submodules --depth 2 "file://$pwd/." super_clone &&
> +       git clone --recurse-submodules --depth 2 --shallow-submodules "file://$pwd/." super_clone &&
>         (
>                 cd super_clone &&
>                 git log --oneline >lines &&
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG REPORT] git 2.9.0 clone --recursive fails on cloning a submodule

[Stefan Beller]

On Wed, Apr 19, 2017 at 4:30 AM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Sun, Jun 19, 2016 at 10:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Jeff King <peff@peff.net> writes:
>>
>>> Stefan, I think it might be worth revisiting the default set by d22eb04
>>> to propagate shallowness from the super-project clone. In an ideal
>>> world, we would be asking each submodule for the actual commit we are
>>> interested in, and shallowness would not matter. But until
>>> uploadpack.allowReachableSHA1InWant works everywhere, I suspect this is
>>> going to be a problem.
>>
>> Yup, something like this on top of d22eb04 to be merged before
>> v2.9.1 for the maintenance track would be necessary.
>>
>> -- >8 --
>> Subject: clone: do not let --depth imply --shallow-submodules
>>
>> In v2.9.0, we prematurely flipped the default to force cloning
>> submodules shallowly, when the superproject is getting cloned
>> shallowly.  This is likely to fail when the upstream repositories
>> submodules are cloned from a repository that is not prepared to
>> serve histories that ends at a commit that is not at the tip of a
>> branch, and we know the world is not yet ready.
>>
>> Use a safer default to clone the submodules fully, unless the user
>> tells us that she knows that the upstream repository of the
>> submodules are willing to cooperate with "--shallow-submodules"
>> option.
>>
>> Noticed-by: Vadim Eisenberg <VADIME@il.ibm.com>
>> Helped-by: Jeff King <peff@peff.net>
>> Signed-off-by: Junio C Hamano <gitster@pobox.com>
>> ---
>>  Documentation/git-clone.txt | 5 ++---
>>  builtin/clone.c             | 5 ++---
>>  t/t5614-clone-submodules.sh | 4 ++--
>>  3 files changed, 6 insertions(+), 8 deletions(-)
>>
>> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
>> index e1a21b7..c5a1ce2 100644
>> --- a/Documentation/git-clone.txt
>> +++ b/Documentation/git-clone.txt
>> @@ -192,9 +192,8 @@ objects from the source repository into a pack in the cloned repository.
>>         Create a 'shallow' clone with a history truncated to the
>>         specified number of revisions. Implies `--single-branch` unless
>>         `--no-single-branch` is given to fetch the histories near the
>> -       tips of all branches. This implies `--shallow-submodules`. If
>> -       you want to have a shallow superproject clone, but full submodules,
>> -       also pass `--no-shallow-submodules`.
>> +       tips of all branches. If you want to clone submodules shallowly,
>> +       also pass `--shallow-submodules`.
>>
>>  --[no-]single-branch::
>>         Clone only the history leading to the tip of a single branch,
>> diff --git a/builtin/clone.c b/builtin/clone.c
>> index ecdf308..f267742 100644
>> --- a/builtin/clone.c
>> +++ b/builtin/clone.c
>> @@ -40,7 +40,7 @@ static const char * const builtin_clone_usage[] = {
>>
>>  static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
>>  static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;
>> -static int option_shallow_submodules = -1;
>> +static int option_shallow_submodules;
>>  static char *option_template, *option_depth;
>>  static char *option_origin = NULL;
>>  static char *option_branch = NULL;
>> @@ -730,8 +730,7 @@ static int checkout(void)
>>                 struct argv_array args = ARGV_ARRAY_INIT;
>>                 argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL);
>>
>> -               if (option_shallow_submodules == 1
>> -                   || (option_shallow_submodules == -1 && option_depth))
>> +               if (option_shallow_submodules == 1)
>>                         argv_array_push(&args, "--depth=1");
>
> Very late reply, since I'm just looking at this now with the --no-tags
> opti,n, but that == 1 makes no sense anymore, and should just be `if
> (option_shallow_submodules)` shouldn't it? I.e. this used to be an int
> for the depth, now is a bool, but the current == 1 check is left over
> probably from an earlier version where the depth was configurable.

Yes we can drop the "== 1" here.

Thanks,
Stefan