* [PATCH 0/3] clone --shallow-submodules flag @ 2016-04-12 23:48 Stefan Beller 2016-04-12 23:48 ` [PATCH 1/3] submodule clone: pass along `local` option Stefan Beller ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Stefan Beller @ 2016-04-12 23:48 UTC (permalink / raw) To: git; +Cc: gitster, jacob.keller, Stefan Beller (This is a resend of a series from March 15th, titled "Towards sane shallow clones with submodules", this series applies on top of sb/submodule-parallel-update, it replaces sb/clone-shallow-passthru) When creating a shallow clone of a repository with submodules, the depth argument does not influence the submodules, i.e. the submodules are done as non-shallow clones. It is unclear what the best default is for the depth of submodules of a shallow clone, so we need to have the possibility to do all kinds of combinations: * shallow super project with shallow submodules e.g. build bots starting always from scratch. They want to transmit the least amount of network data as well as using the least amount of space on their hard drive. * shallow super project with unshallow submodules e.g. The superproject is just there to track a collection of repositories and it is not important to have the relationship between the repositories intact. However the history of the individual submodules matter. * unshallow super project with shallow submodules e.g. The superproject is the actual project and the submodule is a library which is rarely touched. The new switch to select submodules to be shallow or unshallow supports all of these three cases. Changes to the previous series: * Reordered patch 1 and 2, as it sounds like a more natural ordering. Any comments welcome! Thanks, Stefan Stefan Beller (3): clone: add `--shallow-submodules` flag submodule clone: pass along `local` option clone: Add t5614 to test cloning submodules with shallowness involved Documentation/git-clone.txt | 13 +++++-- builtin/clone.c | 21 ++++++++++++ builtin/submodule--helper.c | 22 ++++++++++-- git-submodule.sh | 7 ++++ t/t5614-clone-submodules.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 139 insertions(+), 6 deletions(-) create mode 100755 t/t5614-clone-submodules.sh -- 2.5.0.264.gc776916.dirty ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 1/3] submodule clone: pass along `local` option 2016-04-12 23:48 [PATCH 0/3] clone --shallow-submodules flag Stefan Beller @ 2016-04-12 23:48 ` Stefan Beller 2016-04-25 12:18 ` Lars Schneider 2016-04-12 23:48 ` [PATCH 2/3] clone: add `--shallow-submodules` flag Stefan Beller 2016-04-12 23:48 ` [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved Stefan Beller 2 siblings, 1 reply; 10+ messages in thread From: Stefan Beller @ 2016-04-12 23:48 UTC (permalink / raw) To: git; +Cc: gitster, jacob.keller, Stefan Beller When cloning a local repository, the user may choose to use an optimization such that the transfer uses a Git agnostic protocol. Propagate the users choice to submodules or if they don't choose, propagate nothing. A test will be added in a later patch. Signed-off-by: Stefan Beller <sbeller@google.com> --- builtin/clone.c | 14 ++++++++++++++ builtin/submodule--helper.c | 22 +++++++++++++++++++--- git-submodule.sh | 7 +++++++ 3 files changed, 40 insertions(+), 3 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index b004fb4..7453244 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -727,6 +727,20 @@ static int checkout(void) struct argv_array args = ARGV_ARRAY_INIT; argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL); + switch (option_local) { + case 1: + argv_array_push(&args, "--local"); + break; + case 0: + argv_array_push(&args, "--no-local"); + break; + case -1: + /* pass nothing */ + break; + default: + die("BUG: option_local out of range"); + } + if (max_jobs != -1) argv_array_pushf(&args, "--jobs=%d", max_jobs); diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c index a484945..822ec69 100644 --- a/builtin/submodule--helper.c +++ b/builtin/submodule--helper.c @@ -125,7 +125,8 @@ static int module_name(int argc, const char **argv, const char *prefix) return 0; } static int clone_submodule(const char *path, const char *gitdir, const char *url, - const char *depth, const char *reference, int quiet) + const char *depth, const char *reference, int quiet, + int local) { struct child_process cp; child_process_init(&cp); @@ -140,6 +141,10 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url argv_array_pushl(&cp.args, "--reference", reference, NULL); if (gitdir && *gitdir) argv_array_pushl(&cp.args, "--separate-git-dir", gitdir, NULL); + if (local == 1) + argv_array_push(&cp.args, "--local"); + else if (!local) + argv_array_push(&cp.args, "--no-local"); argv_array_push(&cp.args, url); argv_array_push(&cp.args, path); @@ -156,6 +161,7 @@ static int module_clone(int argc, const char **argv, const char *prefix) const char *path = NULL, *name = NULL, *url = NULL; const char *reference = NULL, *depth = NULL; int quiet = 0; + int local = -1; FILE *submodule_dot_git; char *sm_gitdir, *cwd, *p; struct strbuf rel_path = STRBUF_INIT; @@ -180,6 +186,8 @@ static int module_clone(int argc, const char **argv, const char *prefix) OPT_STRING(0, "depth", &depth, N_("string"), N_("depth for shallow clones")), + OPT_BOOL(0, "local", &local, + N_("to clone from a local repository")), OPT__QUIET(&quiet, "Suppress output for cloning a submodule"), OPT_END() }; @@ -200,7 +208,8 @@ static int module_clone(int argc, const char **argv, const char *prefix) if (!file_exists(sm_gitdir)) { if (safe_create_leading_directories_const(sm_gitdir) < 0) die(_("could not create directory '%s'"), sm_gitdir); - if (clone_submodule(path, sm_gitdir, url, depth, reference, quiet)) + if (clone_submodule(path, sm_gitdir, url, depth, reference, + quiet, local)) die(_("clone of '%s' into submodule path '%s' failed"), url, path); } else { @@ -266,6 +275,7 @@ struct submodule_update_clone { /* configuration parameters which are passed on to the children */ int quiet; + int local; const char *reference; const char *depth; const char *recursive_prefix; @@ -278,7 +288,7 @@ struct submodule_update_clone { unsigned quickstop : 1; }; #define SUBMODULE_UPDATE_CLONE_INIT {0, MODULE_LIST_INIT, 0, \ - SUBMODULE_UPDATE_STRATEGY_INIT, 0, NULL, NULL, NULL, NULL, \ + SUBMODULE_UPDATE_STRATEGY_INIT, 0, -1, NULL, NULL, NULL, NULL, \ STRING_LIST_INIT_DUP, 0} /** @@ -367,6 +377,10 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce, child->err = -1; argv_array_push(&child->args, "submodule--helper"); argv_array_push(&child->args, "clone"); + if (suc->local == 1) + argv_array_push(&child->args, "--local"); + else if (!suc->local) + argv_array_push(&child->args, "--no-local"); if (suc->quiet) argv_array_push(&child->args, "--quiet"); if (suc->prefix) @@ -451,6 +465,8 @@ static int update_clone(int argc, const char **argv, const char *prefix) OPT_STRING(0, "depth", &suc.depth, "<depth>", N_("Create a shallow clone truncated to the " "specified number of revisions")), + OPT_BOOL(0, "local", &suc.local, + N_("to clone from a local repository")), OPT_INTEGER('j', "jobs", &max_jobs, N_("parallel jobs")), OPT__QUIET(&suc.quiet, N_("don't print cloning progress")), diff --git a/git-submodule.sh b/git-submodule.sh index 86018ee..4d5e8c7 100755 --- a/git-submodule.sh +++ b/git-submodule.sh @@ -653,6 +653,12 @@ cmd_update() --jobs=*) jobs=$1 ;; + --local) + option_local="--local" + ;; + --no-local) + option_local="--no-local" + ;; --) shift break @@ -680,6 +686,7 @@ cmd_update() ${reference:+--reference "$reference"} \ ${depth:+--depth "$depth"} \ ${jobs:+$jobs} \ + ${option_local:+$option_local} \ "$@" || echo "#unmatched" } | { err= -- 2.5.0.264.gc776916.dirty ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] submodule clone: pass along `local` option 2016-04-12 23:48 ` [PATCH 1/3] submodule clone: pass along `local` option Stefan Beller @ 2016-04-25 12:18 ` Lars Schneider 2016-04-25 17:15 ` Stefan Beller 0 siblings, 1 reply; 10+ messages in thread From: Lars Schneider @ 2016-04-25 12:18 UTC (permalink / raw) To: Stefan Beller; +Cc: git, gitster, jacob.keller > On 13 Apr 2016, at 01:48, Stefan Beller <sbeller@google.com> wrote: > > When cloning a local repository, the user may choose to use an optimization > such that the transfer uses a Git agnostic protocol. Propagate the users > choice to submodules or if they don't choose, propagate nothing. > A test will be added in a later patch. > > Signed-off-by: Stefan Beller <sbeller@google.com> > --- > builtin/clone.c | 14 ++++++++++++++ > builtin/submodule--helper.c | 22 +++++++++++++++++++--- > git-submodule.sh | 7 +++++++ > 3 files changed, 40 insertions(+), 3 deletions(-) > > diff --git a/builtin/clone.c b/builtin/clone.c > index b004fb4..7453244 100644 > --- a/builtin/clone.c > +++ b/builtin/clone.c > @@ -727,6 +727,20 @@ static int checkout(void) > struct argv_array args = ARGV_ARRAY_INIT; > argv_array_pushl(&args, "submodule", "update", "--init", "--recursive", NULL); > > + switch (option_local) { > + case 1: > + argv_array_push(&args, "--local"); > + break; > + case 0: > + argv_array_push(&args, "--no-local"); > + break; > + case -1: > + /* pass nothing */ > + break; > + default: > + die("BUG: option_local out of range"); > + } > + > if (max_jobs != -1) > argv_array_pushf(&args, "--jobs=%d", max_jobs); > > diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c > index a484945..822ec69 100644 > --- a/builtin/submodule--helper.c > +++ b/builtin/submodule--helper.c > @@ -125,7 +125,8 @@ static int module_name(int argc, const char **argv, const char *prefix) > return 0; > } > static int clone_submodule(const char *path, const char *gitdir, const char *url, > - const char *depth, const char *reference, int quiet) > + const char *depth, const char *reference, int quiet, > + int local) > { > struct child_process cp; > child_process_init(&cp); > @@ -140,6 +141,10 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url > argv_array_pushl(&cp.args, "--reference", reference, NULL); > if (gitdir && *gitdir) > argv_array_pushl(&cp.args, "--separate-git-dir", gitdir, NULL); > + if (local == 1) bikeshedding: Why do you use == 1 here and !local below? I would either compare both against integers or none ("if (local)" should work here, too, no?). Or is this a Git coding guideline that I am not yet aware of? > + argv_array_push(&cp.args, "--local"); > + else if (!local) > + argv_array_push(&cp.args, "--no-local"); > > argv_array_push(&cp.args, url); > argv_array_push(&cp.args, path); > @@ -156,6 +161,7 @@ static int module_clone(int argc, const char **argv, const char *prefix) > const char *path = NULL, *name = NULL, *url = NULL; > const char *reference = NULL, *depth = NULL; > int quiet = 0; > + int local = -1; > FILE *submodule_dot_git; > char *sm_gitdir, *cwd, *p; > struct strbuf rel_path = STRBUF_INIT; > @@ -180,6 +186,8 @@ static int module_clone(int argc, const char **argv, const char *prefix) > OPT_STRING(0, "depth", &depth, > N_("string"), > N_("depth for shallow clones")), > + OPT_BOOL(0, "local", &local, > + N_("to clone from a local repository")), TBH I think "local" could be misleading here. How about "--pass-transfer-protocol-on-to-submodules" or something? If I understand this option correctly then this could be useful for other cases besides "local". Imagine you clone a repo via HTTPS that has references to SSH submodules on the same server. If you don't have a proper SSH setup then the submodule clone will fail. > OPT__QUIET(&quiet, "Suppress output for cloning a submodule"), > OPT_END() > }; > @@ -200,7 +208,8 @@ static int module_clone(int argc, const char **argv, const char *prefix) > if (!file_exists(sm_gitdir)) { > if (safe_create_leading_directories_const(sm_gitdir) < 0) > die(_("could not create directory '%s'"), sm_gitdir); > - if (clone_submodule(path, sm_gitdir, url, depth, reference, quiet)) > + if (clone_submodule(path, sm_gitdir, url, depth, reference, > + quiet, local)) > die(_("clone of '%s' into submodule path '%s' failed"), > url, path); > } else { > @@ -266,6 +275,7 @@ struct submodule_update_clone { > > /* configuration parameters which are passed on to the children */ > int quiet; > + int local; > const char *reference; > const char *depth; > const char *recursive_prefix; > @@ -278,7 +288,7 @@ struct submodule_update_clone { > unsigned quickstop : 1; > }; > #define SUBMODULE_UPDATE_CLONE_INIT {0, MODULE_LIST_INIT, 0, \ > - SUBMODULE_UPDATE_STRATEGY_INIT, 0, NULL, NULL, NULL, NULL, \ > + SUBMODULE_UPDATE_STRATEGY_INIT, 0, -1, NULL, NULL, NULL, NULL, \ > STRING_LIST_INIT_DUP, 0} > > /** > @@ -367,6 +377,10 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce, > child->err = -1; > argv_array_push(&child->args, "submodule--helper"); > argv_array_push(&child->args, "clone"); > + if (suc->local == 1) > + argv_array_push(&child->args, "--local"); > + else if (!suc->local) > + argv_array_push(&child->args, "--no-local"); > if (suc->quiet) > argv_array_push(&child->args, "--quiet"); > if (suc->prefix) > @@ -451,6 +465,8 @@ static int update_clone(int argc, const char **argv, const char *prefix) > OPT_STRING(0, "depth", &suc.depth, "<depth>", > N_("Create a shallow clone truncated to the " > "specified number of revisions")), > + OPT_BOOL(0, "local", &suc.local, > + N_("to clone from a local repository")), same as above > OPT_INTEGER('j', "jobs", &max_jobs, > N_("parallel jobs")), > OPT__QUIET(&suc.quiet, N_("don't print cloning progress")), > diff --git a/git-submodule.sh b/git-submodule.sh > index 86018ee..4d5e8c7 100755 > --- a/git-submodule.sh > +++ b/git-submodule.sh > @@ -653,6 +653,12 @@ cmd_update() > --jobs=*) > jobs=$1 > ;; > + --local) > + option_local="--local" > + ;; > + --no-local) > + option_local="--no-local" > + ;; > --) > shift > break > @@ -680,6 +686,7 @@ cmd_update() > ${reference:+--reference "$reference"} \ > ${depth:+--depth "$depth"} \ > ${jobs:+$jobs} \ > + ${option_local:+$option_local} \ > "$@" || echo "#unmatched" > } | { > err= > -- > 2.5.0.264.gc776916.dirty > > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] submodule clone: pass along `local` option 2016-04-25 12:18 ` Lars Schneider @ 2016-04-25 17:15 ` Stefan Beller 0 siblings, 0 replies; 10+ messages in thread From: Stefan Beller @ 2016-04-25 17:15 UTC (permalink / raw) To: Lars Schneider; +Cc: git@vger.kernel.org, Junio C Hamano, Jacob Keller On Mon, Apr 25, 2016 at 5:18 AM, Lars Schneider <larsxschneider@gmail.com> wrote: >> @@ -140,6 +141,10 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url >> argv_array_pushl(&cp.args, "--reference", reference, NULL); >> if (gitdir && *gitdir) >> argv_array_pushl(&cp.args, "--separate-git-dir", gitdir, NULL); >> + if (local == 1) > bikeshedding: Why do you use == 1 here and !local below? I would either compare both against > integers or none ("if (local)" should work here, too, no?). Or is this a Git coding guideline > that I am not yet aware of? "if (local)" doesn't work as it includes -1 as well. What I want to express here is: switch (local) { case 1: argv_array_push(&cp.args, "--local"); break; /* local == 1 did that */ case 0: argv_array_push(&cp.args, "--no-local"); break; /* !local*/ case -1: default: /* pass nothing */ } This seems to be more expressive now, so I'll replace with that? >> @@ -180,6 +186,8 @@ static int module_clone(int argc, const char **argv, const char *prefix) >> OPT_STRING(0, "depth", &depth, >> N_("string"), >> N_("depth for shallow clones")), >> + OPT_BOOL(0, "local", &local, >> + N_("to clone from a local repository")), > TBH I think "local" could be misleading here. How about "--pass-transfer-protocol-on-to-submodules" > or something? If I understand this option correctly then this could be useful for other cases besides > "local". Imagine you clone a repo via HTTPS that has references to SSH submodules on the same > server. If you don't have a proper SSH setup then the submodule clone will fail. Good point. We just converted an internal repository to use relative path for submodules instead of absolute paths and that broke the auth (they used different protocols, which we were unaware of) However "--pass-transfer-protocol-on-to-submodules" would mean more than just "--[no-]local, but also passing on the protocol prefix (which can be http://, https://, ssh://, git:// or more), so it would enlarge the scope of this series a lot. (Original purpose of this patch was to allow not using the explicit file:// protocol in the test for the next patch. But I see passing on the protocol as a new issue worth resolving :) ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 2/3] clone: add `--shallow-submodules` flag 2016-04-12 23:48 [PATCH 0/3] clone --shallow-submodules flag Stefan Beller 2016-04-12 23:48 ` [PATCH 1/3] submodule clone: pass along `local` option Stefan Beller @ 2016-04-12 23:48 ` Stefan Beller 2016-04-12 23:48 ` [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved Stefan Beller 2 siblings, 0 replies; 10+ messages in thread From: Stefan Beller @ 2016-04-12 23:48 UTC (permalink / raw) To: git; +Cc: gitster, jacob.keller, Stefan Beller When creating a shallow clone of a repository with submodules, the depth argument does not influence the submodules, i.e. the submodules are done as non-shallow clones. It is unclear what the best default is for the depth of submodules of a shallow clone, so we need to have the possibility to do all kinds of combinations: * shallow super project with shallow submodules e.g. build bots starting always from scratch. They want to transmit the least amount of network data as well as using the least amount of space on their hard drive. * shallow super project with unshallow submodules e.g. The superproject is just there to track a collection of repositories and it is not important to have the relationship between the repositories intact. However the history of the individual submodules matter. * unshallow super project with shallow submodules e.g. The superproject is the actual project and the submodule is a library which is rarely touched. The new switch to select submodules to be shallow or unshallow supports all of these three cases. It is easy to transition from the first to the second case by just unshallowing the submodules (`git submodule foreach git fetch --unshallow`), but it is not possible to transition from the second to the first case (as we wouldd have already transmitted the non shallow over the network). That is why we want to make the first case the default in case of a shallow super project. This leads to the inconvenience in the second case with the shallow super project and unshallow submodules, as you need to pass `--no-shallow-submodules`. Signed-off-by: Stefan Beller <sbeller@google.com> --- Documentation/git-clone.txt | 13 ++++++++++--- builtin/clone.c | 7 +++++++ 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt index 6db7b6d..20a4577 100644 --- a/Documentation/git-clone.txt +++ b/Documentation/git-clone.txt @@ -14,8 +14,8 @@ SYNOPSIS [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>] [--dissociate] [--separate-git-dir <git dir>] [--depth <depth>] [--[no-]single-branch] - [--recursive | --recurse-submodules] [--jobs <n>] [--] <repository> - [<directory>] + [--recursive | --recurse-submodules] [--[no-]shallow-submodules] + [--jobs <n>] [--] <repository> [<directory>] DESCRIPTION ----------- @@ -190,7 +190,11 @@ objects from the source repository into a pack in the cloned repository. --depth <depth>:: Create a 'shallow' clone with a history truncated to the - specified number of revisions. + specified number of revisions. Implies `--single-branch` unless + `--no-single-branch` is given to fetch the histories near the + tips of all branches. This implies `--shallow-submodules`. If + you want to have a shallow clone, but full submodules, also pass + `--no-shallow-submodules`. --[no-]single-branch:: Clone only the history leading to the tip of a single branch, @@ -214,6 +218,9 @@ objects from the source repository into a pack in the cloned repository. repository does not have a worktree/checkout (i.e. if any of `--no-checkout`/`-n`, `--bare`, or `--mirror` is given) +--shallow-submodules:: + All submodules which are cloned, will be shallow. + --separate-git-dir=<git dir>:: Instead of placing the cloned repository where it is supposed to be, place the cloned repository at the specified directory, diff --git a/builtin/clone.c b/builtin/clone.c index 7453244..bd1d598 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -40,6 +40,7 @@ static const char * const builtin_clone_usage[] = { static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1; static int option_local = -1, option_no_hardlinks, option_shared, option_recursive; +static int option_shallow_submodules = -1; static char *option_template, *option_depth; static char *option_origin = NULL; static char *option_branch = NULL; @@ -91,6 +92,8 @@ static struct option builtin_clone_options[] = { N_("create a shallow clone of that depth")), OPT_BOOL(0, "single-branch", &option_single_branch, N_("clone only one branch, HEAD or --branch")), + OPT_BOOL(0, "shallow-submodules", &option_shallow_submodules, + N_("any cloned submodules will be shallow")), OPT_STRING(0, "separate-git-dir", &real_git_dir, N_("gitdir"), N_("separate git dir from working tree")), OPT_STRING_LIST('c', "config", &option_config, N_("key=value"), @@ -741,6 +744,10 @@ static int checkout(void) die("BUG: option_local out of range"); } + if (option_shallow_submodules == 1 + || (option_shallow_submodules == -1 && option_depth)) + argv_array_push(&args, "--depth=1"); + if (max_jobs != -1) argv_array_pushf(&args, "--jobs=%d", max_jobs); -- 2.5.0.264.gc776916.dirty ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved 2016-04-12 23:48 [PATCH 0/3] clone --shallow-submodules flag Stefan Beller 2016-04-12 23:48 ` [PATCH 1/3] submodule clone: pass along `local` option Stefan Beller 2016-04-12 23:48 ` [PATCH 2/3] clone: add `--shallow-submodules` flag Stefan Beller @ 2016-04-12 23:48 ` Stefan Beller 2016-04-13 6:52 ` Jacob Keller 2016-04-25 12:25 ` Lars Schneider 2 siblings, 2 replies; 10+ messages in thread From: Stefan Beller @ 2016-04-12 23:48 UTC (permalink / raw) To: git; +Cc: gitster, jacob.keller, Stefan Beller There are some inherent issues with shallow clones and submodules, such as having not having a commit available the superproject may point to in the submodule due to being shallow. Use the new file t5614 to document and test expectations in this area. Signed-off-by: Stefan Beller <sbeller@google.com> --- t/t5614-clone-submodules.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100755 t/t5614-clone-submodules.sh diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh new file mode 100755 index 0000000..a66c2db --- /dev/null +++ b/t/t5614-clone-submodules.sh @@ -0,0 +1,82 @@ +#!/bin/sh + +test_description='Test shallow cloning of repos with submodules' + +. ./test-lib.sh + +test_expect_success 'setup' ' + git checkout -b master && + test_commit commit1 && + test_commit commit2 && + mkdir sub && + ( + cd sub && + git init && + test_commit subcommit1 && + test_commit subcommit2 + ) && + git submodule add ./sub && + git commit -m "add submodule" +' + +test_expect_success 'nonshallow clone implies nonshallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 3 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 2 lines + ) +' + +test_expect_success 'shallow clone implies shallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules --no-local --depth 1 . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 1 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 1 lines + ) +' + +test_expect_success 'shallow clone with non shallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules --no-local --depth 1 --no-shallow-submodules . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 1 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 2 lines + ) +' + +test_expect_success 'non shallow clone with shallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules --no-local --shallow-submodules . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 3 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 1 lines + ) +' + +test_done -- 2.5.0.264.gc776916.dirty ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved 2016-04-12 23:48 ` [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved Stefan Beller @ 2016-04-13 6:52 ` Jacob Keller 2016-04-25 17:41 ` Stefan Beller 2016-04-25 12:25 ` Lars Schneider 1 sibling, 1 reply; 10+ messages in thread From: Jacob Keller @ 2016-04-13 6:52 UTC (permalink / raw) To: Stefan Beller; +Cc: Git mailing list, Junio C Hamano On Tue, Apr 12, 2016 at 4:48 PM, Stefan Beller <sbeller@google.com> wrote: > There are some inherent issues with shallow clones and submodules, such > as having not having a commit available the superproject may point to > in the submodule due to being shallow. Use the new file t5614 to document > and test expectations in this area. > The description seems to imply that there will be "test_expect_failure" tests to indicate what needs to be improved... Maybe I am just mis-reading it? Thanks, Jake > Signed-off-by: Stefan Beller <sbeller@google.com> > --- > t/t5614-clone-submodules.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 82 insertions(+) > create mode 100755 t/t5614-clone-submodules.sh > > diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh > new file mode 100755 > index 0000000..a66c2db > --- /dev/null > +++ b/t/t5614-clone-submodules.sh > @@ -0,0 +1,82 @@ > +#!/bin/sh > + > +test_description='Test shallow cloning of repos with submodules' > + > +. ./test-lib.sh > + > +test_expect_success 'setup' ' > + git checkout -b master && > + test_commit commit1 && > + test_commit commit2 && > + mkdir sub && > + ( > + cd sub && > + git init && > + test_commit subcommit1 && > + test_commit subcommit2 > + ) && > + git submodule add ./sub && > + git commit -m "add submodule" > +' > + > +test_expect_success 'nonshallow clone implies nonshallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules . super_clone && > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 3 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 2 lines > + ) > +' > + > +test_expect_success 'shallow clone implies shallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules --no-local --depth 1 . super_clone && > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) > +' > + > +test_expect_success 'shallow clone with non shallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules --no-local --depth 1 --no-shallow-submodules . super_clone && > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 2 lines > + ) > +' > + > +test_expect_success 'non shallow clone with shallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules --no-local --shallow-submodules . super_clone && > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 3 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) > +' > + > +test_done > -- > 2.5.0.264.gc776916.dirty > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved 2016-04-13 6:52 ` Jacob Keller @ 2016-04-25 17:41 ` Stefan Beller 0 siblings, 0 replies; 10+ messages in thread From: Stefan Beller @ 2016-04-25 17:41 UTC (permalink / raw) To: Jacob Keller, Lars Schneider; +Cc: Git mailing list, Junio C Hamano On Tue, Apr 12, 2016 at 11:52 PM, Jacob Keller <jacob.keller@gmail.com> wrote: > On Tue, Apr 12, 2016 at 4:48 PM, Stefan Beller <sbeller@google.com> wrote: >> There are some inherent issues with shallow clones and submodules, such >> as having not having a commit available the superproject may point to >> in the submodule due to being shallow. Use the new file t5614 to document >> and test expectations in this area. >> > > The description seems to imply that there will be > "test_expect_failure" tests to indicate what needs to be improved... > Maybe I am just mis-reading it? That commit message certainly reads like that. I'll tone it down to just say it's testing the shallowness. Lars wrote: >> +test_expect_success 'shallow clone implies shallow submodule' ' >> + test_when_finished "rm -rf super_clone" && >> + git clone --recurse-submodules --no-local --depth 1 . super_clone && > > You could add another commit to the super repo and then clone with "--depth 2". > The super repo would then contain 2 lines and the submodule still just 1. > This would make it more obvious that shallow submodules always have a depth of 1. ok will fix. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved 2016-04-12 23:48 ` [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved Stefan Beller 2016-04-13 6:52 ` Jacob Keller @ 2016-04-25 12:25 ` Lars Schneider 1 sibling, 0 replies; 10+ messages in thread From: Lars Schneider @ 2016-04-25 12:25 UTC (permalink / raw) To: Stefan Beller; +Cc: git, gitster, jacob.keller > On 13 Apr 2016, at 01:48, Stefan Beller <sbeller@google.com> wrote: > > There are some inherent issues with shallow clones and submodules, such > as having not having a commit available the superproject may point to > in the submodule due to being shallow. Use the new file t5614 to document > and test expectations in this area. > > Signed-off-by: Stefan Beller <sbeller@google.com> > --- > t/t5614-clone-submodules.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 82 insertions(+) > create mode 100755 t/t5614-clone-submodules.sh > > diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh > new file mode 100755 > index 0000000..a66c2db > --- /dev/null > +++ b/t/t5614-clone-submodules.sh > @@ -0,0 +1,82 @@ > +#!/bin/sh > + > +test_description='Test shallow cloning of repos with submodules' > + > +. ./test-lib.sh > + > +test_expect_success 'setup' ' > + git checkout -b master && > + test_commit commit1 && > + test_commit commit2 && > + mkdir sub && > + ( > + cd sub && > + git init && > + test_commit subcommit1 && > + test_commit subcommit2 > + ) && > + git submodule add ./sub && > + git commit -m "add submodule" > +' > + > +test_expect_success 'nonshallow clone implies nonshallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules . super_clone && > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 3 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 2 lines > + ) > +' > + > +test_expect_success 'shallow clone implies shallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules --no-local --depth 1 . super_clone && You could add another commit to the super repo and then clone with "--depth 2". The super repo would then contain 2 lines and the submodule still just 1. This would make it more obvious that shallow submodules always have a depth of 1. > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) > +' > + > +test_expect_success 'shallow clone with non shallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules --no-local --depth 1 --no-shallow-submodules . super_clone && > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 2 lines > + ) > +' > + > +test_expect_success 'non shallow clone with shallow submodule' ' > + test_when_finished "rm -rf super_clone" && > + git clone --recurse-submodules --no-local --shallow-submodules . super_clone && > + ( > + cd super_clone && > + git log --oneline >lines && > + test_line_count = 3 lines > + ) && > + ( > + cd super_clone/sub && > + git log --oneline >lines && > + test_line_count = 1 lines > + ) > +' > + > +test_done > -- > 2.5.0.264.gc776916.dirty > > -- > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 0/3] Towards sane shallow clones with submodules @ 2016-03-16 1:12 Stefan Beller 2016-03-16 1:12 ` [PATCH 3/3] clone: Add t5614 to test cloning submodules with shallowness involved Stefan Beller 0 siblings, 1 reply; 10+ messages in thread From: Stefan Beller @ 2016-03-16 1:12 UTC (permalink / raw) To: git, gitster Cc: jrnieder, Jens.Lehmann, pclouds, larsxschneider, Stefan Beller This is a first real series following the RFC-y $gmane/288720. For using submodules as they are now, I would imagine this is a strict improvement. See the tests in last patch to see what a workflow would look like. Thanks, Stefan Stefan Beller (3): clone: add `--shallow-submodules` flag submodule clone: pass along `local` option clone: Add t5614 to test cloning submodules with shallowness involved Documentation/git-clone.txt | 13 +++++-- builtin/clone.c | 21 ++++++++++++ builtin/submodule--helper.c | 22 ++++++++++-- git-submodule.sh | 7 ++++ t/t5614-clone-submodules.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 139 insertions(+), 6 deletions(-) create mode 100755 t/t5614-clone-submodules.sh -- 2.7.0.rc0.42.g8e9204f.dirty ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 3/3] clone: Add t5614 to test cloning submodules with shallowness involved 2016-03-16 1:12 [PATCH 0/3] Towards sane shallow clones with submodules Stefan Beller @ 2016-03-16 1:12 ` Stefan Beller 0 siblings, 0 replies; 10+ messages in thread From: Stefan Beller @ 2016-03-16 1:12 UTC (permalink / raw) To: git, gitster Cc: jrnieder, Jens.Lehmann, pclouds, larsxschneider, Stefan Beller There are some inherent issues with shallow clones and submodules, such as having not having a commit available the superproject may point to in the submodule due to being shallow. Use the new file t5614 to document and test expectations in this area. Signed-off-by: Stefan Beller <sbeller@google.com> --- Notes: $gmane/288916 does renaming operations, "[PATCH] clone tests: rename t57* => t56*", such that the first 13 spots in t56* are taken. So we'll go with ..14 to not cause conflicts. (If not having that renaming patch in mind, you may end up aksing yourself "why ..14?") t/t5614-clone-submodules.sh | 82 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100755 t/t5614-clone-submodules.sh diff --git a/t/t5614-clone-submodules.sh b/t/t5614-clone-submodules.sh new file mode 100755 index 0000000..a66c2db --- /dev/null +++ b/t/t5614-clone-submodules.sh @@ -0,0 +1,82 @@ +#!/bin/sh + +test_description='Test shallow cloning of repos with submodules' + +. ./test-lib.sh + +test_expect_success 'setup' ' + git checkout -b master && + test_commit commit1 && + test_commit commit2 && + mkdir sub && + ( + cd sub && + git init && + test_commit subcommit1 && + test_commit subcommit2 + ) && + git submodule add ./sub && + git commit -m "add submodule" +' + +test_expect_success 'nonshallow clone implies nonshallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 3 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 2 lines + ) +' + +test_expect_success 'shallow clone implies shallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules --no-local --depth 1 . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 1 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 1 lines + ) +' + +test_expect_success 'shallow clone with non shallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules --no-local --depth 1 --no-shallow-submodules . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 1 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 2 lines + ) +' + +test_expect_success 'non shallow clone with shallow submodule' ' + test_when_finished "rm -rf super_clone" && + git clone --recurse-submodules --no-local --shallow-submodules . super_clone && + ( + cd super_clone && + git log --oneline >lines && + test_line_count = 3 lines + ) && + ( + cd super_clone/sub && + git log --oneline >lines && + test_line_count = 1 lines + ) +' + +test_done -- 2.7.0.rc0.42.g8e9204f.dirty ^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-04-25 17:41 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-04-12 23:48 [PATCH 0/3] clone --shallow-submodules flag Stefan Beller 2016-04-12 23:48 ` [PATCH 1/3] submodule clone: pass along `local` option Stefan Beller 2016-04-25 12:18 ` Lars Schneider 2016-04-25 17:15 ` Stefan Beller 2016-04-12 23:48 ` [PATCH 2/3] clone: add `--shallow-submodules` flag Stefan Beller 2016-04-12 23:48 ` [PATCH 3/3] clone: add t5614 to test cloning submodules with shallowness involved Stefan Beller 2016-04-13 6:52 ` Jacob Keller 2016-04-25 17:41 ` Stefan Beller 2016-04-25 12:25 ` Lars Schneider -- strict thread matches above, loose matches on Subject: below -- 2016-03-16 1:12 [PATCH 0/3] Towards sane shallow clones with submodules Stefan Beller 2016-03-16 1:12 ` [PATCH 3/3] clone: Add t5614 to test cloning submodules with shallowness involved Stefan Beller
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).