git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
From: Elijah Newren <newren@gmail.com>
To: Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Cc: Git Mailing List <git@vger.kernel.org>,
	Junio C Hamano <gitster@pobox.com>,
	Derrick Stolee <dstolee@microsoft.com>
Subject: Re: [PATCH 3/9] clone: add --sparse mode
Date: Fri, 23 Aug 2019 16:17:06 -0700
Message-ID: <CABPp-BGyDwgTRx3A4OPJT=5rMM-onkPtg9m58M=rDMHiJQxQ6Q@mail.gmail.com> (raw)
In-Reply-To: <4ccd36b3966b62934cfbb0ec65e2851b50dfa1e3.1566313865.git.gitgitgadget@gmail.com>

On Tue, Aug 20, 2019 at 8:12 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Derrick Stolee <dstolee@microsoft.com>
>
> When someone wants to clone a large repository, but plans to work
> using a sparse-checkout file, they either need to do a full
> checkout first and then reduce the patterns they included, or
> clone with --no-checkout, set up their patterns, and then run
> a checkout manually. This requires knowing a lot about the repo
> shape and how sparse-checkout works.
>
> Add a new '--sparse' option to 'git clone' that initializes the
> sparse-checkout file to include the following patterns:
>
>         /*
>         !/*/*
>
> These patterns include every file in the root directory, but
> no directories. This allows a repo to include files like a
> README or a bootstrapping script to grow enlistments from that
> point.

Nice.

>
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  Documentation/git-clone.txt        |  8 +++++++-
>  builtin/clone.c                    | 27 +++++++++++++++++++++++++++
>  t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++
>  3 files changed, 47 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
> index 34011c2940..0fe91d2f04 100644
> --- a/Documentation/git-clone.txt
> +++ b/Documentation/git-clone.txt
> @@ -15,7 +15,7 @@ SYNOPSIS
>           [--dissociate] [--separate-git-dir <git dir>]
>           [--depth <depth>] [--[no-]single-branch] [--no-tags]
>           [--recurse-submodules[=<pathspec>]] [--[no-]shallow-submodules]
> -         [--[no-]remote-submodules] [--jobs <n>] [--] <repository>
> +         [--[no-]remote-submodules] [--jobs <n>] [--sparse] [--] <repository>
>           [<directory>]
>
>  DESCRIPTION
> @@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository.
>         used, neither remote-tracking branches nor the related
>         configuration variables are created.
>
> +--sparse::
> +       Initialize the sparse-checkout file so the working
> +       directory starts with only the files in the root
> +       of the repository. The sparse-checkout file can be
> +       modified to grow the working directory as needed.
> +
>  --mirror::
>         Set up a mirror of the source repository.  This implies `--bare`.
>         Compared to `--bare`, `--mirror` not only maps local branches of the
> diff --git a/builtin/clone.c b/builtin/clone.c
> index f665b28ccc..d6d49a73ff 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -60,6 +60,7 @@ static const char *real_git_dir;
>  static char *option_upload_pack = "git-upload-pack";
>  static int option_verbosity;
>  static int option_progress = -1;
> +static int option_sparse_checkout;
>  static enum transport_family family;
>  static struct string_list option_config = STRING_LIST_INIT_NODUP;
>  static struct string_list option_required_reference = STRING_LIST_INIT_NODUP;
> @@ -147,6 +148,8 @@ static struct option builtin_clone_options[] = {
>         OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),
>         OPT_BOOL(0, "remote-submodules", &option_remote_submodules,
>                     N_("any cloned submodules will use their remote-tracking branch")),
> +       OPT_BOOL(0, "sparse", &option_sparse_checkout,
> +                   N_("initialize sparse-checkout file to include only files at root")),
>         OPT_END()
>  };
>
> @@ -734,6 +737,27 @@ static void update_head(const struct ref *our, const struct ref *remote,
>         }
>  }
>
> +static int git_sparse_checkout_init(const char *repo)
> +{
> +       struct argv_array argv = ARGV_ARRAY_INIT;
> +       int result = 0;
> +       argv_array_pushl(&argv, "-C", repo, "sparse-checkout", "init", NULL);
> +
> +       /*
> +        * We must apply the setting in the current process
> +        * for the later checkout to use the sparse-checkout file.
> +        */
> +       core_apply_sparse_checkout = 1;
> +
> +       if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) {
> +               error(_("failed to initialize sparse-checkout"));
> +               result = 1;
> +       }

Sigh...so much forking of additional processes.  I'd really rather
that we were reducing how much of this we are doing in the codebase
instead of adding more.  Every fork makes following stuff in a
debugger harder.

> +
> +       argv_array_clear(&argv);
> +       return result;
> +}
> +
>  static int checkout(int submodule_progress)
>  {
>         struct object_id oid;
> @@ -1107,6 +1131,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>         if (option_required_reference.nr || option_optional_reference.nr)
>                 setup_reference();
>
> +       if (option_sparse_checkout && git_sparse_checkout_init(repo))
> +               return 1;
> +
>         remote = remote_get(option_origin);
>
>         strbuf_addf(&default_refspec, "+%s*:%s*", src_ref_prefix,
> diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh
> index 35ab84aabd..b7d5f15830 100755
> --- a/t/t1091-sparse-checkout-builtin.sh
> +++ b/t/t1091-sparse-checkout-builtin.sh
> @@ -87,4 +87,17 @@ test_expect_success 'init with existing sparse-checkout' '
>         test_cmp expect dir
>  '
>
> +test_expect_success 'clone --sparse' '
> +       git clone --sparse repo clone &&
> +       git -C clone sparse-checkout list >actual &&
> +       cat >expect <<-EOF &&
> +               /*
> +               !/*/*
> +       EOF
> +       test_cmp expect actual &&
> +       ls clone >dir &&
> +       echo a >expect &&
> +       test_cmp expect dir

Checking that a toplevel entry is present, but not checking that an
entry from a subdir is missing as expected?

  reply index

Thread overview: 149+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-20 15:11 [PATCH 0/9] [RFC] New sparse-checkout builtin and "cone" mode Derrick Stolee via GitGitGadget
2019-08-20 15:11 ` [PATCH 1/9] sparse-checkout: create builtin with 'list' subcommand Derrick Stolee via GitGitGadget
2019-08-23 22:30   ` Elijah Newren
2019-08-20 15:11 ` [PATCH 2/9] sparse-checkout: create 'init' subcommand Derrick Stolee via GitGitGadget
2019-08-23 23:02   ` Elijah Newren
2019-09-11 14:27     ` Derrick Stolee
2019-09-11 20:28     ` Derrick Stolee
2019-08-20 15:11 ` [PATCH 3/9] clone: add --sparse mode Derrick Stolee via GitGitGadget
2019-08-23 23:17   ` Elijah Newren [this message]
2019-09-18 13:51     ` Derrick Stolee
2019-08-20 15:11 ` [PATCH 4/9] sparse-checkout: 'add' subcommand Derrick Stolee via GitGitGadget
2019-08-23 23:30   ` Elijah Newren
2019-09-18 13:55     ` Derrick Stolee
2019-09-18 14:56       ` Elijah Newren
2019-09-18 17:23         ` Derrick Stolee
2019-08-20 15:11 ` [PATCH 6/9] trace2:experiment: clear_ce_flags_1 Jeff Hostetler via GitGitGadget
2019-08-24  0:08   ` Elijah Newren
2019-08-20 15:11 ` [PATCH 5/9] sparse-checkout: create 'disable' subcommand Derrick Stolee via GitGitGadget
2019-08-23 23:50   ` Elijah Newren
2019-08-20 15:11 ` [PATCH 7/9] sparse-checkout: add 'cone' mode Derrick Stolee via GitGitGadget
2019-08-24  0:31   ` Elijah Newren
2019-08-20 15:11 ` [PATCH 8/9] sparse-checkout: use hashmaps for cone patterns Derrick Stolee via GitGitGadget
2019-08-24  4:56   ` Elijah Newren
2019-08-20 15:11 ` [PATCH 9/9] sparse-checkout: init and add in cone mode Derrick Stolee via GitGitGadget
2019-08-24  5:07   ` Elijah Newren
2019-08-21 21:52 ` [PATCH 0/9] [RFC] New sparse-checkout builtin and "cone" mode Elijah Newren
2019-08-22 13:10   ` Derrick Stolee
2019-08-22 14:25     ` Derrick Stolee
2019-08-24  5:40     ` Elijah Newren
2019-08-26 13:29       ` Derrick Stolee
2019-08-26 18:16         ` Elijah Newren
2019-08-26 19:16           ` Derrick Stolee
2019-09-02 17:55       ` Eric Sunshine
2019-09-19 14:43 ` [PATCH v2 00/11] " Derrick Stolee via GitGitGadget
2019-09-19 14:43   ` [PATCH v2 01/11] sparse-checkout: create builtin with 'list' subcommand Derrick Stolee via GitGitGadget
2019-10-05 19:22     ` Elijah Newren
2019-09-19 14:43   ` [PATCH v2 02/11] sparse-checkout: create 'init' subcommand Derrick Stolee via GitGitGadget
2019-10-05 19:34     ` Elijah Newren
2019-09-19 14:43   ` [PATCH v2 03/11] clone: add --sparse mode Derrick Stolee via GitGitGadget
2019-10-05 19:40     ` Elijah Newren
2019-10-07 13:56       ` Derrick Stolee
2019-09-19 14:43   ` [PATCH v2 04/11] sparse-checkout: 'set' subcommand Derrick Stolee via GitGitGadget
2019-10-05 22:44     ` Elijah Newren
2019-10-06  0:30       ` Elijah Newren
2019-10-07 18:26         ` Derrick Stolee
2019-10-11 22:24           ` Elijah Newren
2019-09-19 14:43   ` [PATCH v2 05/11] sparse-checkout: add '--stdin' option to set subcommand Derrick Stolee via GitGitGadget
2019-09-19 14:43   ` [PATCH v2 06/11] sparse-checkout: create 'disable' subcommand Derrick Stolee via GitGitGadget
2019-10-06  4:10     ` Elijah Newren
2019-10-07 19:12       ` Derrick Stolee
2019-09-19 14:43   ` [PATCH v2 07/11] trace2: add region in clear_ce_flags Jeff Hostetler via GitGitGadget
2019-10-06  4:13     ` Elijah Newren
2019-09-19 14:43   ` [PATCH v2 08/11] sparse-checkout: add 'cone' mode Derrick Stolee via GitGitGadget
2019-10-06  4:22     ` Elijah Newren
2019-10-07 19:15       ` Derrick Stolee
2019-09-19 14:43   ` [PATCH v2 09/11] sparse-checkout: use hashmaps for cone patterns Derrick Stolee via GitGitGadget
2019-09-19 20:59     ` Derrick Stolee
2019-09-20 14:37       ` Derrick Stolee
2019-09-19 14:43   ` [PATCH v2 10/11] sparse-checkout: init and set in cone mode Derrick Stolee via GitGitGadget
2019-09-19 14:43   ` [PATCH v2 11/11] unpack-trees: hash less " Derrick Stolee via GitGitGadget
2019-10-01 13:40   ` [PATCH v2 00/11] New sparse-checkout builtin and "cone" mode Derrick Stolee
2019-10-01 16:54     ` Elijah Newren
2019-10-01 18:15       ` Derrick Stolee
2019-10-03 22:28     ` Junio C Hamano
2019-10-07 20:08   ` [PATCH v3 00/17] " Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 01/17] sparse-checkout: create builtin with 'list' subcommand Derrick Stolee via GitGitGadget
2019-10-11 22:01       ` Elijah Newren
2019-10-07 20:08     ` [PATCH v3 02/17] sparse-checkout: create 'init' subcommand Derrick Stolee via GitGitGadget
2019-10-11 22:14       ` Elijah Newren
2019-10-14 20:22         ` Derrick Stolee
2019-10-07 20:08     ` [PATCH v3 03/17] clone: add --sparse mode Derrick Stolee via GitGitGadget
2019-10-11 22:20       ` Elijah Newren
2019-10-07 20:08     ` [PATCH v3 04/17] sparse-checkout: 'set' subcommand Derrick Stolee via GitGitGadget
2019-10-11 22:26       ` Elijah Newren
2019-10-11 22:30         ` Elijah Newren
2019-10-07 20:08     ` [PATCH v3 05/17] sparse-checkout: add '--stdin' option to set subcommand Derrick Stolee via GitGitGadget
2019-10-11 22:27       ` Elijah Newren
2019-10-14 20:28         ` Derrick Stolee
2019-10-07 20:08     ` [PATCH v3 06/17] sparse-checkout: create 'disable' subcommand Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 07/17] trace2: add region in clear_ce_flags Jeff Hostetler via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 08/17] sparse-checkout: add 'cone' mode Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 09/17] sparse-checkout: use hashmaps for cone patterns Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 10/17] sparse-checkout: init and set in cone mode Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 11/17] unpack-trees: hash less " Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 13/17] read-tree: show progress by default Derrick Stolee via GitGitGadget
2019-10-12 22:16       ` Elijah Newren
2019-10-14 20:31         ` Derrick Stolee
2019-10-07 20:08     ` [PATCH v3 12/17] unpack-trees: add progress to clear_ce_flags() Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 14/17] sparse-checkout: sanitize for nested folders Derrick Stolee via GitGitGadget
2019-10-07 20:08     ` [PATCH v3 15/17] sparse-checkout: update working directory in-process Derrick Stolee via GitGitGadget
2019-10-12 22:57       ` Elijah Newren
2019-10-14 20:39         ` Derrick Stolee
2019-10-07 20:08     ` [PATCH v3 16/17] sparse-checkout: write using lockfile Derrick Stolee via GitGitGadget
2019-10-12 22:59       ` Elijah Newren
2019-10-14 20:41         ` Derrick Stolee
2019-10-07 20:08     ` [PATCH v3 17/17] sparse-checkout: cone mode should not interact with .gitignore Derrick Stolee via GitGitGadget
2019-10-12 23:00       ` Elijah Newren
2019-10-12 23:22     ` [PATCH v3 00/17] New sparse-checkout builtin and "cone" mode Elijah Newren
2019-10-15 13:55     ` [PATCH v4 " Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 01/17] sparse-checkout: create builtin with 'list' subcommand Derrick Stolee via GitGitGadget
2019-10-16 19:00         ` Elijah Newren
2019-10-21 12:10           ` Derrick Stolee
2019-10-18 16:07         ` SZEDER Gábor
2019-10-21 11:42           ` Derrick Stolee
2019-10-15 13:55       ` [PATCH v4 02/17] sparse-checkout: create 'init' subcommand Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 03/17] clone: add --sparse mode Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 04/17] sparse-checkout: 'set' subcommand Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 05/17] sparse-checkout: add '--stdin' option to set subcommand Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 06/17] sparse-checkout: create 'disable' subcommand Derrick Stolee via GitGitGadget
2019-10-18 16:31         ` SZEDER Gábor
2019-10-15 13:55       ` [PATCH v4 07/17] trace2: add region in clear_ce_flags Jeff Hostetler via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 08/17] sparse-checkout: add 'cone' mode Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 09/17] sparse-checkout: use hashmaps for cone patterns Derrick Stolee via GitGitGadget
2019-10-18 15:31         ` SZEDER Gábor
2019-10-21 11:44           ` Derrick Stolee
2019-10-15 13:55       ` [PATCH v4 10/17] sparse-checkout: init and set in cone mode Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 11/17] unpack-trees: hash less " Derrick Stolee via GitGitGadget
2019-10-15 13:55       ` [PATCH v4 12/17] unpack-trees: add progress to clear_ce_flags() Derrick Stolee via GitGitGadget
2019-10-15 13:56       ` [PATCH v4 13/17] read-tree: show progress by default Derrick Stolee via GitGitGadget
2019-10-15 13:56       ` [PATCH v4 14/17] sparse-checkout: sanitize for nested folders Derrick Stolee via GitGitGadget
2019-10-15 13:56       ` [PATCH v4 15/17] sparse-checkout: update working directory in-process Derrick Stolee via GitGitGadget
2019-10-18 20:24         ` SZEDER Gábor
2019-10-18 20:40           ` SZEDER Gábor
2019-10-21 11:59             ` Derrick Stolee
2019-10-15 13:56       ` [PATCH v4 16/17] sparse-checkout: write using lockfile Derrick Stolee via GitGitGadget
2019-10-15 13:56       ` [PATCH v4 17/17] sparse-checkout: cone mode should not interact with .gitignore Derrick Stolee via GitGitGadget
2019-10-16 20:07       ` [PATCH v4 00/17] New sparse-checkout builtin and "cone" mode Elijah Newren
2019-10-17 23:53       ` Jon Simons
2019-10-21 12:08         ` Derrick Stolee
2019-10-21 13:56       ` [PATCH v5 " Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 01/17] sparse-checkout: create builtin with 'list' subcommand Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 02/17] sparse-checkout: create 'init' subcommand Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 03/17] clone: add --sparse mode Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 04/17] sparse-checkout: 'set' subcommand Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 05/17] sparse-checkout: add '--stdin' option to set subcommand Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 06/17] sparse-checkout: create 'disable' subcommand Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 07/17] trace2: add region in clear_ce_flags Jeff Hostetler via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 08/17] sparse-checkout: add 'cone' mode Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 09/17] sparse-checkout: use hashmaps for cone patterns Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 10/17] sparse-checkout: init and set in cone mode Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 11/17] unpack-trees: hash less " Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 12/17] unpack-trees: add progress to clear_ce_flags() Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 13/17] read-tree: show progress by default Derrick Stolee via GitGitGadget
2019-10-21 15:04           ` Phillip Wood
2019-10-21 15:14             ` Derrick Stolee
2019-10-21 13:56         ` [PATCH v5 14/17] sparse-checkout: sanitize for nested folders Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 15/17] sparse-checkout: update working directory in-process Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 16/17] sparse-checkout: write using lockfile Derrick Stolee via GitGitGadget
2019-10-21 13:56         ` [PATCH v5 17/17] sparse-checkout: cone mode should not interact with .gitignore Derrick Stolee via GitGitGadget

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CABPp-BGyDwgTRx3A4OPJT=5rMM-onkPtg9m58M=rDMHiJQxQ6Q@mail.gmail.com' \
    --to=newren@gmail.com \
    --cc=dstolee@microsoft.com \
    --cc=git@vger.kernel.org \
    --cc=gitgitgadget@gmail.com \
    --cc=gitster@pobox.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

git@vger.kernel.org list mirror (unofficial, one of many)

Archives are clonable:
	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.org/gmane.comp.version-control.git

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox