[PATCH v2] list-objects-filter: disable 'sparse:path' filters

[PATCH v2] list-objects-filter: disable 'sparse:path' filters

[Christian Couder]

If someone wants to use as a filter a sparse file that is in the
repository, something like "--filter=sparse:oid=<ref>:<path>"
already works.

So 'sparse:path' is only interesting if the sparse file is not in
the repository. In this case though the current implementation has
a big security issue, as it makes it possible to ask the server to
read any file, like for example /etc/password, and to explore the
filesystem, as well as individual lines of files.

If someone is interested in using a sparse file that is not in the
repository as a filter, then at the minimum a config option, such
as "uploadpack.sparsePathFilter", should be implemented first to
restrict the directory from which the files specified by
'sparse:path' can be read.

For now though, let's just disable 'sparse:path' filters.

Helped-by: Matthew DeVore <matvore@google.com>
Helped-by: Jeff Hostetler <git@jeffhostetler.com>
Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
---

Changes since the RFC version are the following:

  - improved the error message when 'sparse:path' is used,
  - updated "git-completion.bash",
  - freed "sparse_path_value" field in list_objects_filter_release(),
  - updated tests (t5317 and t6112).

Thanks to Matthew and Jeff for the suggestions.

 contrib/completion/git-completion.bash |  2 +-
 list-objects-filter-options.c          | 10 ++--
 list-objects-filter-options.h          |  2 -
 list-objects-filter.c                  | 22 --------
 t/t5317-pack-objects-filter-objects.sh | 71 +++++---------------------
 t/t6112-rev-list-filters-objects.sh    | 39 +++++---------
 6 files changed, 33 insertions(+), 113 deletions(-)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index 3eefbabdb1..9f71bcde96 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -1536,7 +1536,7 @@ _git_fetch ()
 		return
 		;;
 	--filter=*)
-		__gitcomp "blob:none blob:limit= sparse:oid= sparse:path=" "" "${cur##--filter=}"
+		__gitcomp "blob:none blob:limit= sparse:oid=" "" "${cur##--filter=}"
 		return
 		;;
 	--*)
diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c
index c0036f7378..a15d0f7829 100644
--- a/list-objects-filter-options.c
+++ b/list-objects-filter-options.c
@@ -78,9 +78,12 @@ static int gently_parse_list_objects_filter(
 		return 0;
 
 	} else if (skip_prefix(arg, "sparse:path=", &v0)) {
-		filter_options->choice = LOFC_SPARSE_PATH;
-		filter_options->sparse_path_value = strdup(v0);
-		return 0;
+		if (errbuf) {
+			strbuf_addstr(
+				errbuf,
+				_("sparse:path filters support has been dropped"));
+		}
+		return 1;
 	}
 	/*
 	 * Please update _git_fetch() in git-completion.bash when you
@@ -136,7 +139,6 @@ void list_objects_filter_release(
 {
 	free(filter_options->filter_spec);
 	free(filter_options->sparse_oid_value);
-	free(filter_options->sparse_path_value);
 	memset(filter_options, 0, sizeof(*filter_options));
 }
 
diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h
index e3adc78ebf..c54f0000fb 100644
--- a/list-objects-filter-options.h
+++ b/list-objects-filter-options.h
@@ -13,7 +13,6 @@ enum list_objects_filter_choice {
 	LOFC_BLOB_LIMIT,
 	LOFC_TREE_DEPTH,
 	LOFC_SPARSE_OID,
-	LOFC_SPARSE_PATH,
 	LOFC__COUNT /* must be last */
 };
 
@@ -44,7 +43,6 @@ struct list_objects_filter_options {
 	 * choice.
 	 */
 	struct object_id *sparse_oid_value;
-	char *sparse_path_value;
 	unsigned long blob_limit_value;
 	unsigned long tree_exclude_depth;
 };
diff --git a/list-objects-filter.c b/list-objects-filter.c
index ee449de3f7..53f90442c5 100644
--- a/list-objects-filter.c
+++ b/list-objects-filter.c
@@ -478,27 +478,6 @@ static void *filter_sparse_oid__init(
 	return d;
 }
 
-static void *filter_sparse_path__init(
-	struct oidset *omitted,
-	struct list_objects_filter_options *filter_options,
-	filter_object_fn *filter_fn,
-	filter_free_fn *filter_free_fn)
-{
-	struct filter_sparse_data *d = xcalloc(1, sizeof(*d));
-	d->omits = omitted;
-	if (add_excludes_from_file_to_list(filter_options->sparse_path_value,
-					   NULL, 0, &d->el, NULL) < 0)
-		die("could not load filter specification");
-
-	ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc);
-	d->array_frame[d->nr].defval = 0; /* default to include */
-	d->array_frame[d->nr].child_prov_omit = 0;
-
-	*filter_fn = filter_sparse;
-	*filter_free_fn = filter_sparse_free;
-	return d;
-}
-
 typedef void *(*filter_init_fn)(
 	struct oidset *omitted,
 	struct list_objects_filter_options *filter_options,
@@ -514,7 +493,6 @@ static filter_init_fn s_filters[] = {
 	filter_blobs_limit__init,
 	filter_trees_depth__init,
 	filter_sparse_oid__init,
-	filter_sparse_path__init,
 };
 
 void *list_objects_filter__init(
diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh
index 4c0201c34b..2d2f5d0229 100755
--- a/t/t5317-pack-objects-filter-objects.sh
+++ b/t/t5317-pack-objects-filter-objects.sh
@@ -277,6 +277,10 @@ test_expect_success 'verify normal and blob:limit packfiles have same commits/tr
 '
 
 # Test sparse:path=<path> filter.
+# !!!!
+# NOTE: sparse:path filter support has been dropped for security reasons,
+# so the tests have been changed to make sure that using it fails.
+# !!!!
 # Use a local file containing a sparse-checkout specification to filter
 # out blobs not required for the corresponding sparse-checkout.  We do not
 # require sparse-checkout to actually be enabled.
@@ -315,73 +319,24 @@ test_expect_success 'verify blob count in normal packfile' '
 	test_cmp expected observed
 '
 
-test_expect_success 'verify sparse:path=pattern1' '
-	git -C r3 ls-files -s dir1/sparse1 dir1/sparse2 >ls_files_result &&
-	awk -f print_2.awk ls_files_result |
-	sort >expected &&
-
-	git -C r3 pack-objects --revs --stdout --filter=sparse:path=../pattern1 >filter.pack <<-EOF &&
+test_expect_success 'verify sparse:path=pattern1 fails' '
+	test_must_fail git -C r3 pack-objects --revs --stdout \
+		--filter=sparse:path=../pattern1 <<-EOF
 	HEAD
 	EOF
-	git -C r3 index-pack ../filter.pack &&
-
-	git -C r3 verify-pack -v ../filter.pack >verify_result &&
-	grep blob verify_result |
-	awk -f print_1.awk |
-	sort >observed &&
-
-	test_cmp expected observed
-'
-
-test_expect_success 'verify normal and sparse:path=pattern1 packfiles have same commits/trees' '
-	git -C r3 verify-pack -v ../all.pack >verify_result &&
-	grep -E "commit|tree" verify_result |
-	awk -f print_1.awk |
-	sort >expected &&
-
-	git -C r3 verify-pack -v ../filter.pack >verify_result &&
-	grep -E "commit|tree" verify_result |
-	awk -f print_1.awk |
-	sort >observed &&
-
-	test_cmp expected observed
 '
 
-test_expect_success 'verify sparse:path=pattern2' '
-	git -C r3 ls-files -s sparse1 dir1/sparse1 >ls_files_result &&
-	awk -f print_2.awk ls_files_result |
-	sort >expected &&
-
-	git -C r3 pack-objects --revs --stdout --filter=sparse:path=../pattern2 >filter.pack <<-EOF &&
+test_expect_success 'verify sparse:path=pattern2 fails' '
+	test_must_fail git -C r3 pack-objects --revs --stdout \
+		--filter=sparse:path=../pattern2 <<-EOF
 	HEAD
 	EOF
-	git -C r3 index-pack ../filter.pack &&
-
-	git -C r3 verify-pack -v ../filter.pack >verify_result &&
-	grep blob verify_result |
-	awk -f print_1.awk |
-	sort >observed &&
-
-	test_cmp expected observed
-'
-
-test_expect_success 'verify normal and sparse:path=pattern2 packfiles have same commits/trees' '
-	git -C r3 verify-pack -v ../all.pack >verify_result &&
-	grep -E "commit|tree" verify_result |
-	awk -f print_1.awk |
-	sort >expected &&
-
-	git -C r3 verify-pack -v ../filter.pack >verify_result &&
-	grep -E "commit|tree" verify_result |
-	awk -f print_1.awk |
-	sort >observed &&
-
-	test_cmp expected observed
 '
 
 # Test sparse:oid=<oid-ish> filter.
-# Like sparse:path, but we get the sparse-checkout specification from
-# a blob rather than a file on disk.
+# Use a blob containing a sparse-checkout specification to filter
+# out blobs not required for the corresponding sparse-checkout.  We do not
+# require sparse-checkout to actually be enabled.
 
 test_expect_success 'setup r4' '
 	git init r4 &&
diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh
index 9c11427719..acd7f5ab80 100755
--- a/t/t6112-rev-list-filters-objects.sh
+++ b/t/t6112-rev-list-filters-objects.sh
@@ -157,6 +157,10 @@ test_expect_success 'verify blob:limit=1m' '
 '
 
 # Test sparse:path=<path> filter.
+# !!!!
+# NOTE: sparse:path filter support has been dropped for security reasons,
+# so the tests have been changed to make sure that using it fails.
+# !!!!
 # Use a local file containing a sparse-checkout specification to filter
 # out blobs not required for the corresponding sparse-checkout.  We do not
 # require sparse-checkout to actually be enabled.
@@ -176,37 +180,20 @@ test_expect_success 'setup r3' '
 	echo sparse1 >pattern2
 '
 
-test_expect_success 'verify sparse:path=pattern1 omits top-level files' '
-	git -C r3 ls-files -s sparse1 sparse2 >ls_files_result &&
-	awk -f print_2.awk ls_files_result |
-	sort >expected &&
-
-	git -C r3 rev-list --quiet --objects --filter-print-omitted \
-		--filter=sparse:path=../pattern1 HEAD >revs &&
-	awk -f print_1.awk revs |
-	sed "s/~//" |
-	sort >observed &&
-
-	test_cmp expected observed
+test_expect_success 'verify sparse:path=pattern1 fails' '
+	test_must_fail git -C r3 rev-list --quiet --objects \
+		--filter-print-omitted --filter=sparse:path=../pattern1 HEAD
 '
 
-test_expect_success 'verify sparse:path=pattern2 omits both sparse2 files' '
-	git -C r3 ls-files -s sparse2 dir1/sparse2 >ls_files_result &&
-	awk -f print_2.awk ls_files_result |
-	sort >expected &&
-
-	git -C r3 rev-list --quiet --objects --filter-print-omitted \
-		--filter=sparse:path=../pattern2 HEAD >revs &&
-	awk -f print_1.awk revs |
-	sed "s/~//" |
-	sort >observed &&
-
-	test_cmp expected observed
+test_expect_success 'verify sparse:path=pattern2 fails' '
+	test_must_fail git -C r3 rev-list --quiet --objects \
+		--filter-print-omitted --filter=sparse:path=../pattern2 HEAD
 '
 
 # Test sparse:oid=<oid-ish> filter.
-# Like sparse:path, but we get the sparse-checkout specification from
-# a blob rather than a file on disk.
+# Use a blob containing a sparse-checkout specification to filter
+# out blobs not required for the corresponding sparse-checkout.  We do not
+# require sparse-checkout to actually be enabled.
 
 test_expect_success 'setup r3 part 2' '
 	echo dir1/ >r3/pattern &&
-- 
2.22.0.rc1.1.ge905c6e19a.dirty

Re: [PATCH v2] list-objects-filter: disable 'sparse:path' filters

[Jeff King]

On Sat, May 25, 2019 at 04:28:34PM +0200, Christian Couder wrote:

> If someone wants to use as a filter a sparse file that is in the
> repository, something like "--filter=sparse:oid=<ref>:<path>"
> already works.
> 
> So 'sparse:path' is only interesting if the sparse file is not in
> the repository. In this case though the current implementation has
> a big security issue, as it makes it possible to ask the server to
> read any file, like for example /etc/password, and to explore the
> filesystem, as well as individual lines of files.
> 
> If someone is interested in using a sparse file that is not in the
> repository as a filter, then at the minimum a config option, such
> as "uploadpack.sparsePathFilter", should be implemented first to
> restrict the directory from which the files specified by
> 'sparse:path' can be read.
> 
> For now though, let's just disable 'sparse:path' filters.

Thanks for picking this up. The patch looks fine to me (versus just
disabling it for remote invocations) assuming we are OK with the
possible regression. I suppose cooking this in 'next' for a while is one
way we might find out if anybody yells loudly.

-Peff

Re: [PATCH v2] list-objects-filter: disable 'sparse:path' filters

[Junio C Hamano]

Christian Couder <christian.couder@gmail.com> writes:

> If someone wants to use as a filter a sparse file that is in the
> repository, something like "--filter=sparse:oid=<ref>:<path>"
> already works.
>
> So 'sparse:path' is only interesting if the sparse file is not in
> the repository. In this case though the current implementation has
> a big security issue, as it makes it possible to ask the server to
> read any file, like for example /etc/password, and to explore the
> filesystem, as well as individual lines of files.
>
> If someone is interested in using a sparse file that is not in the
> repository as a filter, then at the minimum a config option, such
> as "uploadpack.sparsePathFilter", should be implemented first to
> restrict the directory from which the files specified by
> 'sparse:path' can be read.
>
> For now though, let's just disable 'sparse:path' filters.
>
> Helped-by: Matthew DeVore <matvore@google.com>
> Helped-by: Jeff Hostetler <git@jeffhostetler.com>
> Signed-off-by: Christian Couder <chriscool@tuxfamily.org>
> ---
>
> Changes since the RFC version are the following:
>
>   - improved the error message when 'sparse:path' is used,
>   - updated "git-completion.bash",
>   - freed "sparse_path_value" field in list_objects_filter_release(),
>   - updated tests (t5317 and t6112).
>
> Thanks to Matthew and Jeff for the suggestions.
>
>  contrib/completion/git-completion.bash |  2 +-
>  list-objects-filter-options.c          | 10 ++--
>  list-objects-filter-options.h          |  2 -
>  list-objects-filter.c                  | 22 --------
>  t/t5317-pack-objects-filter-objects.sh | 71 +++++---------------------
>  t/t6112-rev-list-filters-objects.sh    | 39 +++++---------
>  6 files changed, 33 insertions(+), 113 deletions(-)

What is curious is that this does not touch Documentation/ hierarchy
at all---is that a sign that nobody makes any serious use of the
--filter=... thing and we can freely drop "features" around it when
we see it necessary (like in this case)?

Or do we need something like this on top (or squashed in)?  I can
live with or without "Note that..." myself.

Thanks.


 Documentation/rev-list-options.txt | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
index ddbc1de43f..73aafea8d6 100644
--- a/Documentation/rev-list-options.txt
+++ b/Documentation/rev-list-options.txt
@@ -725,9 +725,6 @@ specification contained in the blob (or blob-expression) '<blob-ish>'
 to omit blobs that would not be not required for a sparse checkout on
 the requested refs.
 +
-The form '--filter=sparse:path=<path>' similarly uses a sparse-checkout
-specification contained in <path>.
-+
 The form '--filter=tree:<depth>' omits all blobs and trees whose depth
 from the root tree is >= <depth> (minimum depth if an object is located
 at multiple depths in the commits traversed). <depth>=0 will not include
@@ -737,6 +734,9 @@ tree and blobs which are referenced directly by a commit reachable from
 <commit> or an explicitly-given object. <depth>=2 is like <depth>=1
 while also including trees and blobs one more level removed from an
 explicitly-given commit or tree.
++
+Note that the form '--filter=sparse:path=<path>' that wants to read from
+an arbitrary path on the filesystem is not supported, for security reasons.
 
 --no-filter::
 	Turn off any previous `--filter=` argument.

Re: [PATCH v2] list-objects-filter: disable 'sparse:path' filters

[Christian Couder]

On Tue, May 28, 2019 at 9:42 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> What is curious is that this does not touch Documentation/ hierarchy
> at all---is that a sign that nobody makes any serious use of the
> --filter=... thing and we can freely drop "features" around it when
> we see it necessary (like in this case)?

I just forgot about the Documentation.

> Or do we need something like this on top (or squashed in)?  I can
> live with or without "Note that..." myself.

Yeah, I will squash something like what you suggest soon.

>  Documentation/rev-list-options.txt | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
> index ddbc1de43f..73aafea8d6 100644
> --- a/Documentation/rev-list-options.txt
> +++ b/Documentation/rev-list-options.txt
> @@ -725,9 +725,6 @@ specification contained in the blob (or blob-expression) '<blob-ish>'
>  to omit blobs that would not be not required for a sparse checkout on
>  the requested refs.
>  +
> -The form '--filter=sparse:path=<path>' similarly uses a sparse-checkout
> -specification contained in <path>.
> -+
>  The form '--filter=tree:<depth>' omits all blobs and trees whose depth
>  from the root tree is >= <depth> (minimum depth if an object is located
>  at multiple depths in the commits traversed). <depth>=0 will not include
> @@ -737,6 +734,9 @@ tree and blobs which are referenced directly by a commit reachable from
>  <commit> or an explicitly-given object. <depth>=2 is like <depth>=1
>  while also including trees and blobs one more level removed from an
>  explicitly-given commit or tree.
> ++
> +Note that the form '--filter=sparse:path=<path>' that wants to read from
> +an arbitrary path on the filesystem is not supported, for security reasons.

I'd rather say:

"Note that the form '--filter=sparse:path=<path>' that wants to read from
an arbitrary path on the filesystem has been dropped for security reasons."

to be more consistent with the error message (that Matthew suggested
in a comment following my initial RFC patch).

Thanks,
Christian.