[PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check

[PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1626 bytes --]

Hi,

this patch series improves the connectivity check done by stateful
git-receive-pack(1) to only consider references as reachable that have
been advertised to the client. This has two advantages:

    - A client shouldn't assume objects to exist that have not been part
      of the reference advertisement. But if it excluded an object from
      the packfile that is reachable via any ref that is excluded from
      the reference advertisement due to `transfer.hideRefs` we'd have
      accepted the push anyway. I'd argue that this is a bug in the
      current implementation.

    - Second, by using advertised refs as inputs instead of `git
      rev-list --not --all` we avoid looking up all refs that are
      irrelevant to the current push. This can be a huge performance
      improvement in repos that have a huge amount of internal, hidden
      refs. In one of our repos with 7m refs, of which 6.8m are hidden,
      this speeds up pushes from ~30s to ~4.5s.

One downside is that we need to pass in the object IDs that were part of
the reference advertisement via the standard input, which is seemingly
slower than reading them from the refdb. I'm discussing this in the
second commit.

Patrick

Patrick Steinhardt (2):
  connected: allow supplying different view of reachable objects
  receive-pack: use advertised reference tips to inform connectivity
    check

 builtin/receive-pack.c | 31 ++++++++++++++++++++++---------
 connected.c            |  9 ++++++++-
 connected.h            |  7 +++++++
 3 files changed, 37 insertions(+), 10 deletions(-)

-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH 1/2] connected: allow supplying different view of reachable objects

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2455 bytes --]

The connectivity check is executed via git-receive-pack(1) to verify
that a client has provided all references that are required to satisfy a
set of reference updates. What the connectivity check does is to walk
the object graph with all reference tips as starting points while all
preexisting reference tips are marked as uninteresting.

Preexisting references are currently marked uninteresting by passing
`--not --all` to git-rev-list(1). Some users of the connectivity check
may have a better picture of which objects should be regarded as
uninteresting though, e.g. by reusing information from the reference
advertisement when serving a push.

Add a new field to `struct check_connected_options` that allows callers
to replace the `--not --all` logic with their own set of object IDs they
regard as uninteresting.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 connected.c | 9 ++++++++-
 connected.h | 7 +++++++
 2 files changed, 15 insertions(+), 1 deletion(-)

diff --git a/connected.c b/connected.c
index 74a20cb32e..2a4c4e0025 100644
--- a/connected.c
+++ b/connected.c
@@ -98,7 +98,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 	strvec_push(&rev_list.args, "--stdin");
 	if (has_promisor_remote())
 		strvec_push(&rev_list.args, "--exclude-promisor-objects");
-	if (!opt->is_deepening_fetch) {
+	if (!opt->is_deepening_fetch && !opt->reachable_oids_fn) {
 		strvec_push(&rev_list.args, "--not");
 		strvec_push(&rev_list.args, "--all");
 	}
@@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 
 	rev_list_in = xfdopen(rev_list.in, "w");
 
+	if (opt->reachable_oids_fn) {
+		const struct object_id *reachable_oid;
+		while ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)
+			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(reachable_oid)) < 0)
+				break;
+	}
+
 	do {
 		/*
 		 * If index-pack already checked that:
diff --git a/connected.h b/connected.h
index 6e59c92aa3..f09c7d7884 100644
--- a/connected.h
+++ b/connected.h
@@ -46,6 +46,13 @@ struct check_connected_options {
 	 * during a fetch.
 	 */
 	unsigned is_deepening_fetch : 1;
+
+	/*
+	 * If non-NULL, use this iterator to determine the set of reachable
+	 * objects instead of marking all references as unreachable.
+	 */
+	oid_iterate_fn reachable_oids_fn;
+	void *reachable_oids_data;
 };
 
 #define CHECK_CONNECTED_INIT { 0 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] connected: allow supplying different view of reachable objects

[Ævar Arnfjörð Bjarmason]

On Fri, Oct 28 2022, Patrick Steinhardt wrote:

> @@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
>  
>  	rev_list_in = xfdopen(rev_list.in, "w");
>  
> +	if (opt->reachable_oids_fn) {
> +		const struct object_id *reachable_oid;
> +		while ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)
> +			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(reachable_oid)) < 0)
> +				break;

Just a style nit, we tend to avoid != NULL, != 0 etc. comparisons. I see
connected.c has some of that already, but for new code let's just check
truthiness.

Also for such a small scope a shorter variable name helps us stay at the
usual column limits:

	if (opt->reachable_oids_fn) {
		const struct object_id *oid;
		while ((oid = opt->reachable_oids_fn(opt->reachable_oids_data)))
			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(oid)) < 0)
				break;

The fprintf() return value checking seemed a bit odd, not because we
shouldn't do it, but because we usually don't bother. For other
reviewers: We have that form already in connected.c, so at least locally
we're not being diligently careful, only to have it undone by adjacent
code...

Looks good!

Re: [PATCH 1/2] connected: allow supplying different view of reachable objects

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> diff --git a/connected.c b/connected.c
> index 74a20cb32e..2a4c4e0025 100644
> --- a/connected.c
> +++ b/connected.c
> @@ -98,7 +98,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
>  	strvec_push(&rev_list.args, "--stdin");
>  	if (has_promisor_remote())
>  		strvec_push(&rev_list.args, "--exclude-promisor-objects");
> -	if (!opt->is_deepening_fetch) {
> +	if (!opt->is_deepening_fetch && !opt->reachable_oids_fn) {
>  		strvec_push(&rev_list.args, "--not");
>  		strvec_push(&rev_list.args, "--all");
>  	}
> @@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
>  
>  	rev_list_in = xfdopen(rev_list.in, "w");
>  
> +	if (opt->reachable_oids_fn) {
> +		const struct object_id *reachable_oid;
> +		while ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)
> +			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(reachable_oid)) < 0)
> +				break;
> +	}

It is good that these individual negative references are fed from
the standard input, not on the command line, as they can be many.

In the original code without the reachable_oids_fn, we refrain from
excluding when the is_deepening_fetch bit is set, but here we do not
pay attention to the bit at all.  Is that sensible, and if so why?

Re: [PATCH 1/2] connected: allow supplying different view of reachable objects

[Taylor Blau]

On Fri, Oct 28, 2022 at 11:12:33AM -0700, Junio C Hamano wrote:
> In the original code without the reachable_oids_fn, we refrain from
> excluding when the is_deepening_fetch bit is set, but here we do not
> pay attention to the bit at all.  Is that sensible, and if so why?

I was wondering the same thing. Thanks for asking.

Thanks,
Taylor

Re: [PATCH 1/2] connected: allow supplying different view of reachable objects

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2195 bytes --]

On Fri, Oct 28, 2022 at 11:12:33AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > diff --git a/connected.c b/connected.c
> > index 74a20cb32e..2a4c4e0025 100644
> > --- a/connected.c
> > +++ b/connected.c
> > @@ -98,7 +98,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
> >  	strvec_push(&rev_list.args, "--stdin");
> >  	if (has_promisor_remote())
> >  		strvec_push(&rev_list.args, "--exclude-promisor-objects");
> > -	if (!opt->is_deepening_fetch) {
> > +	if (!opt->is_deepening_fetch && !opt->reachable_oids_fn) {
> >  		strvec_push(&rev_list.args, "--not");
> >  		strvec_push(&rev_list.args, "--all");
> >  	}
> > @@ -125,6 +125,13 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
> >  
> >  	rev_list_in = xfdopen(rev_list.in, "w");
> >  
> > +	if (opt->reachable_oids_fn) {
> > +		const struct object_id *reachable_oid;
> > +		while ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)
> > +			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(reachable_oid)) < 0)
> > +				break;
> > +	}
> 
> It is good that these individual negative references are fed from
> the standard input, not on the command line, as they can be many.
> 
> In the original code without the reachable_oids_fn, we refrain from
> excluding when the is_deepening_fetch bit is set, but here we do not
> pay attention to the bit at all.  Is that sensible, and if so why?

Hm, good point. On a deepening fetch the commits that were the previous
boundary will likely get replaced by new commits that are at a deeper
point in history, so they cannot be used as a well-defined boundary.
Instead, we do a complete graph-walk that doesn't stop at any previously
known commits at all. At least that's how I understand the code, the
explanation is likely a bit fuzzy.

I guess we should thus also pay attention to `is_deepening_fetch` here.
As this means that `is_deepening_fetch` and `reachable_oids_fn` are
mutually exclusive I'm inclined to go even further and `die()` if both
are set at the same time. We only adapt git-receive-pack(1) anyway, so
we should never run into this situation for now.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] connected: allow supplying different view of reachable objects

[Taylor Blau]

On Mon, Oct 31, 2022 at 02:10:16PM +0100, Patrick Steinhardt wrote:
> I guess we should thus also pay attention to `is_deepening_fetch` here.
> As this means that `is_deepening_fetch` and `reachable_oids_fn` are
> mutually exclusive I'm inclined to go even further and `die()` if both
> are set at the same time. We only adapt git-receive-pack(1) anyway, so
> we should never run into this situation for now.

Yes, I agree.

Thanks,
Taylor

[PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8421 bytes --]

When serving a push, git-receive-pack(1) needs to verify that the
packfile sent by the client contains all objects that are required by
the updated references. This connectivity check works by marking all
preexisting references as uninteresting and using the new reference tips
as starting point for a graph walk.

This strategy has the major downside that it will not require any object
to be sent by the client that is reachable by any of the repositories'
references. While that sounds like it would be indeed what we are after
with the connectivity check, it is arguably not. The administrator that
manages the server-side Git repository may have configured certain refs
to be hidden during the reference advertisement via `transfer.hideRefs`
or `receivepack.hideRefs`. Whatever the reason, the result is that the
client shouldn't expect that any of those hidden references exists on
the remote side, and neither should they assume any of the pointed-to
objects to exist except if referenced by any visible reference. But
because we treat _all_ local refs as uninteresting in the connectivity
check, a client is free to send a packfile that references objects that
are only reachable via a hidden reference on the server-side, and we
will gladly accept it.

Besides the correctness issue there is also a performance issue. Git
forges tend to do internal bookkeeping to keep alive sets of objects for
internal use or make them easy to find via certain references. These
references are typically hidden away from the user so that they are
neither advertised nor writeable. At GitLab, we have one particular
repository that contains a total of 7 million references, of which 6.8
million are indeed internal references. With the current connectivity
check we are forced to load all these references in order to mark them
as uninteresting, and this alone takes around 15 seconds to compute.

We can fix both of these issues by changing the logic for stateful
invocations of git-receive-pack(1) where the reference advertisement and
packfile negotiation are served by the same process. Instead of marking
all preexisting references as unreachable, we will only mark those that
we have announced to the client.

Besides the stated fix to correctness this also provides a huge boost to
performance in the repository mentioned above. Pushing a new commit into
this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
as it is configured in Gitaly leads to an almost 7.5-fold speedup:

    Benchmark 1: main
      Time (mean ± σ):     29.902 s ±  0.105 s    [User: 29.176 s, System: 1.052 s]
      Range (min … max):   29.781 s … 29.969 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):      4.033 s ±  0.088 s    [User: 4.071 s, System: 0.374 s]
      Range (min … max):    3.953 s …  4.128 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        7.42 ± 0.16 times faster than 'main'

Unfortunately, this change comes with a performance hit when refs are
not hidden. Executed in the same repository:

    Benchmark 1: main
      Time (mean ± σ):     45.780 s ±  0.507 s    [User: 46.908 s, System: 4.838 s]
      Range (min … max):   45.453 s … 46.364 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):     49.886 s ±  0.282 s    [User: 51.168 s, System: 5.015 s]
      Range (min … max):   49.589 s … 50.149 s    3 runs

    Summary
      'main' ran
        1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'

This is probably caused by the overhead of reachable tips being passed
in via git-rev-list(1)'s standard input, which seems to be slower than
reading the references from disk.

It is debatable what to do about this. If this were only about improving
performance then it would be trivial to make the new logic depend on
whether or not `transfer.hideRefs` has been configured in the repo. But
as explained this is also about correctness, even though this can be
considered an edge case. Furthermore, this slowdown is really only
noticeable in outliers like the above repository with an unreasonable
amount of refs. The same benchmark in linux-stable.git with about
4500 references shows no measurable difference:

    Benchmark 1: main
      Time (mean ± σ):     375.4 ms ±  25.4 ms    [User: 312.2 ms, System: 155.7 ms]
      Range (min … max):   324.2 ms … 492.9 ms    50 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):     374.9 ms ±  36.9 ms    [User: 311.6 ms, System: 158.2 ms]
      Range (min … max):   319.2 ms … 583.1 ms    50 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        1.00 ± 0.12 times faster than 'main'

Let's keep this as-is for the time being and accept the performance hit.
It is arguably extremely noticeable to a user if a push now performs 7.5
times faster than before, but a lot less so in case an already-slow push
becomes about 10% slower.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c | 31 ++++++++++++++++++++++---------
 1 file changed, 22 insertions(+), 9 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 44bcea3a5b..50794539c6 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -326,13 +326,10 @@ static void show_one_alternate_ref(const struct object_id *oid,
 	show_ref(".have", oid);
 }
 
-static void write_head_info(void)
+static void write_head_info(struct oidset *announced_objects)
 {
-	static struct oidset seen = OIDSET_INIT;
-
-	for_each_ref(show_ref_cb, &seen);
-	for_each_alternate_ref(show_one_alternate_ref, &seen);
-	oidset_clear(&seen);
+	for_each_ref(show_ref_cb, announced_objects);
+	for_each_alternate_ref(show_one_alternate_ref, announced_objects);
 	if (!sent_capabilities)
 		show_ref("capabilities^{}", null_oid());
 
@@ -1896,12 +1893,20 @@ static void execute_commands_atomic(struct command *commands,
 	strbuf_release(&err);
 }
 
+static const struct object_id *iterate_announced_oids(void *cb_data)
+{
+	struct oidset_iter *iter = cb_data;
+	return oidset_iter_next(iter);
+}
+
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
-			     const struct string_list *push_options)
+			     const struct string_list *push_options,
+			     struct oidset *announced_oids)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
+	struct oidset_iter announced_oids_iter;
 	struct command *cmd;
 	struct iterate_data data;
 	struct async muxer;
@@ -1928,6 +1933,12 @@ static void execute_commands(struct command *commands,
 	opt.err_fd = err_fd;
 	opt.progress = err_fd && !quiet;
 	opt.env = tmp_objdir_env(tmp_objdir);
+	if (oidset_size(announced_oids) != 0) {
+		oidset_iter_init(announced_oids, &announced_oids_iter);
+		opt.reachable_oids_fn = iterate_announced_oids;
+		opt.reachable_oids_data = &announced_oids_iter;
+	}
+
 	if (check_connected(iterate_receive_command_list, &data, &opt))
 		set_connectivity_errors(commands, si);
 
@@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 {
 	int advertise_refs = 0;
 	struct command *commands;
+	struct oidset announced_oids = OIDSET_INIT;
 	struct oid_array shallow = OID_ARRAY_INIT;
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
@@ -2524,7 +2536,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 	}
 
 	if (advertise_refs || !stateless_rpc) {
-		write_head_info();
+		write_head_info(&announced_oids);
 	}
 	if (advertise_refs)
 		return 0;
@@ -2554,7 +2566,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
 		execute_commands(commands, unpack_status, &si,
-				 &push_options);
+				 &push_options, &announced_oids);
 		if (pack_lockfile)
 			unlink_or_warn(pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2591,6 +2603,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
+	oidset_clear(&announced_oids);
 	free((void *)push_cert_nonce);
 	return 0;
 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Ævar Arnfjörð Bjarmason]

On Fri, Oct 28 2022, Patrick Steinhardt wrote:

> When serving a push, git-receive-pack(1) needs to verify that the
> packfile sent by the client contains all objects that are required by
> the updated references. This connectivity check works by marking all
> preexisting references as uninteresting and using the new reference tips
> as starting point for a graph walk.
>
> This strategy has the major downside that it will not require any object
> to be sent by the client that is reachable by any of the repositories'
> references. While that sounds like it would be indeed what we are after
> with the connectivity check, it is arguably not. The administrator that
> manages the server-side Git repository may have configured certain refs
> to be hidden during the reference advertisement via `transfer.hideRefs`
> or `receivepack.hideRefs`. Whatever the reason, the result is that the
> client shouldn't expect that any of those hidden references exists on
> the remote side, and neither should they assume any of the pointed-to
> objects to exist except if referenced by any visible reference. But
> because we treat _all_ local refs as uninteresting in the connectivity
> check, a client is free to send a packfile that references objects that
> are only reachable via a hidden reference on the server-side, and we
> will gladly accept it.
>
> Besides the correctness issue there is also a performance issue. Git
> forges tend to do internal bookkeeping to keep alive sets of objects for
> internal use or make them easy to find via certain references. These
> references are typically hidden away from the user so that they are
> neither advertised nor writeable. At GitLab, we have one particular
> repository that contains a total of 7 million references, of which 6.8
> million are indeed internal references. With the current connectivity
> check we are forced to load all these references in order to mark them
> as uninteresting, and this alone takes around 15 seconds to compute.
>
> We can fix both of these issues by changing the logic for stateful
> invocations of git-receive-pack(1) where the reference advertisement and
> packfile negotiation are served by the same process. Instead of marking
> all preexisting references as unreachable, we will only mark those that
> we have announced to the client.
>
> Besides the stated fix to correctness this also provides a huge boost to
> performance in the repository mentioned above. Pushing a new commit into
> this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
> as it is configured in Gitaly leads to an almost 7.5-fold speedup:

Really well explained.

>     Benchmark 1: main
>       Time (mean ± σ):     29.902 s ±  0.105 s    [User: 29.176 s, System: 1.052 s]
>       Range (min … max):   29.781 s … 29.969 s    3 runs
>
>     Benchmark 2: pks-connectivity-check-hide-refs
>       Time (mean ± σ):      4.033 s ±  0.088 s    [User: 4.071 s, System: 0.374 s]
>       Range (min … max):    3.953 s …  4.128 s    3 runs
>
>     Summary
>       'pks-connectivity-check-hide-refs' ran
>         7.42 ± 0.16 times faster than 'main'

And impressive, thanks!

> Unfortunately, this change comes with a performance hit when refs are
> not hidden. Executed in the same repository:
>
>     Benchmark 1: main
>       Time (mean ± σ):     45.780 s ±  0.507 s    [User: 46.908 s, System: 4.838 s]
>       Range (min … max):   45.453 s … 46.364 s    3 runs
>
>     Benchmark 2: pks-connectivity-check-hide-refs
>       Time (mean ± σ):     49.886 s ±  0.282 s    [User: 51.168 s, System: 5.015 s]
>       Range (min … max):   49.589 s … 50.149 s    3 runs
>
>     Summary
>       'main' ran
>         1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'
>
> This is probably caused by the overhead of reachable tips being passed
> in via git-rev-list(1)'s standard input, which seems to be slower than
> reading the references from disk.
>
> It is debatable what to do about this. If this were only about improving
> performance then it would be trivial to make the new logic depend on
> whether or not `transfer.hideRefs` has been configured in the repo. But
> as explained this is also about correctness, even though this can be
> considered an edge case. Furthermore, this slowdown is really only
> noticeable in outliers like the above repository with an unreasonable
> amount of refs. The same benchmark in linux-stable.git with about
> 4500 references shows no measurable difference:

Do we have a test that would start failing if we changed the behavior?
Perhaps such a test is peeking too much behind the curtain, but if it's
easy come up with one I think it would be most welcome to have it
alongside this.  to have exposes

> -static void write_head_info(void)
> +static void write_head_info(struct oidset *announced_objects)
>  {
> -	static struct oidset seen = OIDSET_INIT;
> -
> -	for_each_ref(show_ref_cb, &seen);
> -	for_each_alternate_ref(show_one_alternate_ref, &seen);
> -	oidset_clear(&seen);
> +	for_each_ref(show_ref_cb, announced_objects);
> +	for_each_alternate_ref(show_one_alternate_ref, announced_objects);
>  	if (!sent_capabilities)
>  		show_ref("capabilities^{}", null_oid());

Nit: The variable rename stands out slightly,
i.e. s/&seen/announced_objects/ not s/&seen/seen/, especially as:

>  static void execute_commands(struct command *commands,
>  			     const char *unpacker_error,
>  			     struct shallow_info *si,
> -			     const struct string_list *push_options)
> +			     const struct string_list *push_options,
> +			     struct oidset *announced_oids)

Here we have the same variable, but now it's *_oids, not *objects.

> +	if (oidset_size(announced_oids) != 0) {

Nit as before: The "!= 0" can go here.

> @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>  {
>  	int advertise_refs = 0;
>  	struct command *commands;
> +	struct oidset announced_oids = OIDSET_INIT;
>  	struct oid_array shallow = OID_ARRAY_INIT;
>  	struct oid_array ref = OID_ARRAY_INIT;
>  	struct shallow_info si;
> @@ -2524,7 +2536,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>  	}
>  
>  	if (advertise_refs || !stateless_rpc) {
> -		write_head_info();
> +		write_head_info(&announced_oids);
>  	}
>  	if (advertise_refs)
>  		return 0;

This introduces a memory leak to the function., We probably have other
ones in code it calls, but from a quick eyeballing not in the function
itself.

Squashing in / combining it with this should do it, as it never returns
non-zero (except for calling die()):
	
	diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
	index 44bcea3a5b3..8d5c2fbef1c 100644
	--- a/builtin/receive-pack.c
	+++ b/builtin/receive-pack.c
	@@ -2527,7 +2527,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
	 		write_head_info();
	 	}
	 	if (advertise_refs)
	-		return 0;
	+		goto cleanup;
	 
	 	packet_reader_init(&reader, 0, NULL, 0,
	 			   PACKET_READ_CHOMP_NEWLINE |
	@@ -2587,6 +2587,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
	 			update_server_info(0);
	 		clear_shallow_info(&si);
	 	}
	+cleanup:
	 	if (use_sideband)
	 		packet_flush(1);
	 	oid_array_clear(&shallow);

> @@ -2591,6 +2603,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>  		packet_flush(1);
>  	oid_array_clear(&shallow);
>  	oid_array_clear(&ref);
> +	oidset_clear(&announced_oids);
>  	free((void *)push_cert_nonce);
>  	return 0;
>  }

We'll then properly reach this new oidset_clear()> The oid_array_clear()
are all for variables we're populating after we're past htat "if
(advertise_refs)".

I think if you're re-rolling this sqashing 1/2 and 2/2 together would be
an improvement. The 1/2 is tiny, and it's an API that's not used until
this 1/2. I found myself going back & forth more than helped in
reviewing this.

Ggoing back a bit this:

> +static const struct object_id *iterate_announced_oids(void *cb_data)
> +{
> +	struct oidset_iter *iter = cb_data;
> +	return oidset_iter_next(iter);
> +}
> +

Is just used as (from 1/2):

> +	if (opt->reachable_oids_fn) {
> +		const struct object_id *reachable_oid;
> +		while ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)
> +			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(reachable_oid)) < 0)
> +				break;
> +	}

After doing above:

> +	if (oidset_size(announced_oids) != 0) {
> +		oidset_iter_init(announced_oids, &announced_oids_iter);
> +		opt.reachable_oids_fn = iterate_announced_oids;
> +		opt.reachable_oids_data = &announced_oids_iter;
> +	}

But I don't see the reason for the indirection, but maybe I'm missing
something obvious.

Why not just pass the oidset itself and have connected.c iterate through
it, rather than going thorugh this callback / data indirection?

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4559 bytes --]

On Fri, Oct 28, 2022 at 05:01:58PM +0200, Ævar Arnfjörð Bjarmason wrote:
> On Fri, Oct 28 2022, Patrick Steinhardt wrote:
[sinp]
> > Unfortunately, this change comes with a performance hit when refs are
> > not hidden. Executed in the same repository:
> >
> >     Benchmark 1: main
> >       Time (mean ± σ):     45.780 s ±  0.507 s    [User: 46.908 s, System: 4.838 s]
> >       Range (min … max):   45.453 s … 46.364 s    3 runs
> >
> >     Benchmark 2: pks-connectivity-check-hide-refs
> >       Time (mean ± σ):     49.886 s ±  0.282 s    [User: 51.168 s, System: 5.015 s]
> >       Range (min … max):   49.589 s … 50.149 s    3 runs
> >
> >     Summary
> >       'main' ran
> >         1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'
> >
> > This is probably caused by the overhead of reachable tips being passed
> > in via git-rev-list(1)'s standard input, which seems to be slower than
> > reading the references from disk.
> >
> > It is debatable what to do about this. If this were only about improving
> > performance then it would be trivial to make the new logic depend on
> > whether or not `transfer.hideRefs` has been configured in the repo. But
> > as explained this is also about correctness, even though this can be
> > considered an edge case. Furthermore, this slowdown is really only
> > noticeable in outliers like the above repository with an unreasonable
> > amount of refs. The same benchmark in linux-stable.git with about
> > 4500 references shows no measurable difference:
> 
> Do we have a test that would start failing if we changed the behavior?
> Perhaps such a test is peeking too much behind the curtain, but if it's
> easy come up with one I think it would be most welcome to have it
> alongside this.  to have exposes

We have tests that verify that we indeed detect missing objects in
t5504. But what we're lacking is tests that verify that we stop walking
at the boundary of preexisting objects, and I honestly wouldn't quite
know how to do that as there is no functional difference, but really
only a performance issue if we overwalked.

> > -static void write_head_info(void)
> > +static void write_head_info(struct oidset *announced_objects)
> >  {
> > -	static struct oidset seen = OIDSET_INIT;
> > -
> > -	for_each_ref(show_ref_cb, &seen);
> > -	for_each_alternate_ref(show_one_alternate_ref, &seen);
> > -	oidset_clear(&seen);
> > +	for_each_ref(show_ref_cb, announced_objects);
> > +	for_each_alternate_ref(show_one_alternate_ref, announced_objects);
> >  	if (!sent_capabilities)
> >  		show_ref("capabilities^{}", null_oid());
> 
> Nit: The variable rename stands out slightly,
> i.e. s/&seen/announced_objects/ not s/&seen/seen/, especially as:
> 
> >  static void execute_commands(struct command *commands,
> >  			     const char *unpacker_error,
> >  			     struct shallow_info *si,
> > -			     const struct string_list *push_options)
> > +			     const struct string_list *push_options,
> > +			     struct oidset *announced_oids)
> 
> Here we have the same variable, but now it's *_oids, not *objects.

Hm. I think that `announced_oids` is easier to understand compared to
`seen`, so I'd prefer to keep the rename. But I'll definitely make this
consistent so we use `announced_oids` in both places.

[snip]
> > +static const struct object_id *iterate_announced_oids(void *cb_data)
> > +{
> > +	struct oidset_iter *iter = cb_data;
> > +	return oidset_iter_next(iter);
> > +}
> > +
> 
> Is just used as (from 1/2):
> 
> > +	if (opt->reachable_oids_fn) {
> > +		const struct object_id *reachable_oid;
> > +		while ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)
> > +			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(reachable_oid)) < 0)
> > +				break;
> > +	}
> 
> After doing above:
> 
> > +	if (oidset_size(announced_oids) != 0) {
> > +		oidset_iter_init(announced_oids, &announced_oids_iter);
> > +		opt.reachable_oids_fn = iterate_announced_oids;
> > +		opt.reachable_oids_data = &announced_oids_iter;
> > +	}
> 
> But I don't see the reason for the indirection, but maybe I'm missing
> something obvious.
> 
> Why not just pass the oidset itself and have connected.c iterate through
> it, rather than going thorugh this callback / data indirection?

This is done to stay consistent with the way new tips are passed in via
the `oid_iterate_fn`. I'm happy to change callers to just directly pass
a `struct oidset *` though.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Ævar Arnfjörð Bjarmason]

On Mon, Oct 31 2022, Patrick Steinhardt wrote:

> [[PGP Signed Part:Undecided]]
> On Fri, Oct 28, 2022 at 05:01:58PM +0200, Ævar Arnfjörð Bjarmason wrote:
>> On Fri, Oct 28 2022, Patrick Steinhardt wrote:
> [sinp]
>> > Unfortunately, this change comes with a performance hit when refs are
>> > not hidden. Executed in the same repository:
>> >
>> >     Benchmark 1: main
>> >       Time (mean ± σ):     45.780 s ±  0.507 s    [User: 46.908 s, System: 4.838 s]
>> >       Range (min … max):   45.453 s … 46.364 s    3 runs
>> >
>> >     Benchmark 2: pks-connectivity-check-hide-refs
>> >       Time (mean ± σ):     49.886 s ±  0.282 s    [User: 51.168 s, System: 5.015 s]
>> >       Range (min … max):   49.589 s … 50.149 s    3 runs
>> >
>> >     Summary
>> >       'main' ran
>> >         1.09 ± 0.01 times faster than 'pks-connectivity-check-hide-refs'
>> >
>> > This is probably caused by the overhead of reachable tips being passed
>> > in via git-rev-list(1)'s standard input, which seems to be slower than
>> > reading the references from disk.
>> >
>> > It is debatable what to do about this. If this were only about improving
>> > performance then it would be trivial to make the new logic depend on
>> > whether or not `transfer.hideRefs` has been configured in the repo. But
>> > as explained this is also about correctness, even though this can be
>> > considered an edge case. Furthermore, this slowdown is really only
>> > noticeable in outliers like the above repository with an unreasonable
>> > amount of refs. The same benchmark in linux-stable.git with about
>> > 4500 references shows no measurable difference:
>> 
>> Do we have a test that would start failing if we changed the behavior?
>> Perhaps such a test is peeking too much behind the curtain, but if it's
>> easy come up with one I think it would be most welcome to have it
>> alongside this.  to have exposes
>
> We have tests that verify that we indeed detect missing objects in
> t5504. But what we're lacking is tests that verify that we stop walking
> at the boundary of preexisting objects, and I honestly wouldn't quite
> know how to do that as there is no functional difference, but really
> only a performance issue if we overwalked.
>
>> > -static void write_head_info(void)
>> > +static void write_head_info(struct oidset *announced_objects)
>> >  {
>> > -	static struct oidset seen = OIDSET_INIT;
>> > -
>> > -	for_each_ref(show_ref_cb, &seen);
>> > -	for_each_alternate_ref(show_one_alternate_ref, &seen);
>> > -	oidset_clear(&seen);
>> > +	for_each_ref(show_ref_cb, announced_objects);
>> > +	for_each_alternate_ref(show_one_alternate_ref, announced_objects);
>> >  	if (!sent_capabilities)
>> >  		show_ref("capabilities^{}", null_oid());
>> 
>> Nit: The variable rename stands out slightly,
>> i.e. s/&seen/announced_objects/ not s/&seen/seen/, especially as:
>> 
>> >  static void execute_commands(struct command *commands,
>> >  			     const char *unpacker_error,
>> >  			     struct shallow_info *si,
>> > -			     const struct string_list *push_options)
>> > +			     const struct string_list *push_options,
>> > +			     struct oidset *announced_oids)
>> 
>> Here we have the same variable, but now it's *_oids, not *objects.
>
> Hm. I think that `announced_oids` is easier to understand compared to
> `seen`, so I'd prefer to keep the rename. But I'll definitely make this
> consistent so we use `announced_oids` in both places.

Sounds good, we'll need to look at the diff lines in either case (as
we're converting it to a pointer), so changing the name while at it is
fine...

> [snip]
>> > +static const struct object_id *iterate_announced_oids(void *cb_data)
>> > +{
>> > +	struct oidset_iter *iter = cb_data;
>> > +	return oidset_iter_next(iter);
>> > +}
>> > +
>> 
>> Is just used as (from 1/2):
>> 
>> > +	if (opt->reachable_oids_fn) {
>> > +		const struct object_id *reachable_oid;
>> > +		while ((reachable_oid = opt->reachable_oids_fn(opt->reachable_oids_data)) != NULL)
>> > +			if (fprintf(rev_list_in, "^%s\n", oid_to_hex(reachable_oid)) < 0)
>> > +				break;
>> > +	}
>> 
>> After doing above:
>> 
>> > +	if (oidset_size(announced_oids) != 0) {
>> > +		oidset_iter_init(announced_oids, &announced_oids_iter);
>> > +		opt.reachable_oids_fn = iterate_announced_oids;
>> > +		opt.reachable_oids_data = &announced_oids_iter;
>> > +	}
>> 
>> But I don't see the reason for the indirection, but maybe I'm missing
>> something obvious.
>> 
>> Why not just pass the oidset itself and have connected.c iterate through
>> it, rather than going thorugh this callback / data indirection?
>
> This is done to stay consistent with the way new tips are passed in via
> the `oid_iterate_fn`. I'm happy to change callers to just directly pass
> a `struct oidset *` though.

*nod*, makes sense, no need to change it. Just wondering...

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Taylor Blau]

On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:
> This strategy has the major downside that it will not require any object
> to be sent by the client that is reachable by any of the repositories'
> references. While that sounds like it would be indeed what we are after
> with the connectivity check, it is arguably not. The administrator that
> manages the server-side Git repository may have configured certain refs
> to be hidden during the reference advertisement via `transfer.hideRefs`
> or `receivepack.hideRefs`. Whatever the reason, the result is that the
> client shouldn't expect that any of those hidden references exists on
> the remote side, and neither should they assume any of the pointed-to
> objects to exist except if referenced by any visible reference. But
> because we treat _all_ local refs as uninteresting in the connectivity
> check, a client is free to send a packfile that references objects that
> are only reachable via a hidden reference on the server-side, and we
> will gladly accept it.

You mention below that this is a correctness issue, but I am not sure
that I agree.

The existing behavior is a little strange, I agree, but your argument
relies on an assumption that the history on hidden refs is not part of
the reachable set, which is not the case. Any part of the repository
that is reachable from _any_ reference, hidden or not, is reachable by
definition.

So it's perfectly fine to consider objects on hidden refs to be in the
uninteresting set, because they are reachable. It's odd from the
client's perspective, but I do not see a path to repository corruption
with thee existing behavior.

> Besides the stated fix to correctness this also provides a huge boost to
> performance in the repository mentioned above. Pushing a new commit into
> this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
> as it is configured in Gitaly leads to an almost 7.5-fold speedup:

Nice, here we expect a pretty good speed-up, and indeed...

>     Summary
>       'pks-connectivity-check-hide-refs' ran
>         7.42 ± 0.16 times faster than 'main'

...that's exactly what we get. Good.

> @@ -1928,6 +1933,12 @@ static void execute_commands(struct command *commands,
>  	opt.err_fd = err_fd;
>  	opt.progress = err_fd && !quiet;
>  	opt.env = tmp_objdir_env(tmp_objdir);
> +	if (oidset_size(announced_oids) != 0) {

I'm nitpicking, but this would be preferable as "if (oidset_size(announced_oids))"
without the "!= 0".

> +		oidset_iter_init(announced_oids, &announced_oids_iter);
> +		opt.reachable_oids_fn = iterate_announced_oids;
> +		opt.reachable_oids_data = &announced_oids_iter;
> +	}

Why do we see a slowdown when there there aren't any hidden references?
Or am I misunderstanding your patch message which instead means "we see
a slow-down when there are no hidden references [since we still must
store and enumerate all advertised references]"?

If the latter, could we avoid invoking the new machinery altogether? In
other words, shouldn't receive-pack only set the reachable_oids_fn() to
enumerate advertised references only when the set of advertised
references differs from the behavior of `--not --all`?

>  	if (check_connected(iterate_receive_command_list, &data, &opt))
>  		set_connectivity_errors(commands, si);
>
> @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>  {
>  	int advertise_refs = 0;
>  	struct command *commands;
> +	struct oidset announced_oids = OIDSET_INIT;

This looks like trading one problem for another. In your above example,
we now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the
end of the world, but it feels like an avoidable problem.

Could we enumerate the references in a callback to for_each_ref() and
only emit ones which aren't hidden? Storing these and then recalling
them after the fact is worth avoiding.

Thanks,
Taylor

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 5518 bytes --]

On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:
> On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:
> > This strategy has the major downside that it will not require any object
> > to be sent by the client that is reachable by any of the repositories'
> > references. While that sounds like it would be indeed what we are after
> > with the connectivity check, it is arguably not. The administrator that
> > manages the server-side Git repository may have configured certain refs
> > to be hidden during the reference advertisement via `transfer.hideRefs`
> > or `receivepack.hideRefs`. Whatever the reason, the result is that the
> > client shouldn't expect that any of those hidden references exists on
> > the remote side, and neither should they assume any of the pointed-to
> > objects to exist except if referenced by any visible reference. But
> > because we treat _all_ local refs as uninteresting in the connectivity
> > check, a client is free to send a packfile that references objects that
> > are only reachable via a hidden reference on the server-side, and we
> > will gladly accept it.
> 
> You mention below that this is a correctness issue, but I am not sure
> that I agree.
> 
> The existing behavior is a little strange, I agree, but your argument
> relies on an assumption that the history on hidden refs is not part of
> the reachable set, which is not the case. Any part of the repository
> that is reachable from _any_ reference, hidden or not, is reachable by
> definition.
> 
> So it's perfectly fine to consider objects on hidden refs to be in the
> uninteresting set, because they are reachable. It's odd from the
> client's perspective, but I do not see a path to repository corruption
> with thee existing behavior.

Indeed, I'm not trying to say that this can lead to repository
corruption. If at all you can argue that this is more security-related.
Suppose an object is not reachable from any public reference and that
`allowAnySHA1InWant=false`. Then you could make these hidden objects
reachable by sending a packfile with an object that references the
hidden object. It naturally requires you to somehow know about the
object ID, so I don't think this is a critical issue.

But security-related or not, I think it is safe to say that any packfile
sent by a client that does not contain objects required for the updated
reference that the client cannot know to exist on the server-side must
be generated by buggy code.

[snip]
> Why do we see a slowdown when there there aren't any hidden references?
> Or am I misunderstanding your patch message which instead means "we see
> a slow-down when there are no hidden references [since we still must
> store and enumerate all advertised references]"?

I have tried to dig down into the code of `revision.c` but ultimately
returned empty-handed. I _think_ that this is because of the different
paths we use when reading revisions from stdin as we have to resolve the
revision to an OID first, which is more involved than taking the OIDs as
returned by the reference backend. I have tried to short-circuit this
logic in case the revision read from stdin is exactly `hash_algo->hexsz`
long so that we try to parse it as an OID directly instead of trying to
do any of the magic that is required to resolve a revision. But this
only speed things up by a small margin.

Another assumption was that this is overhead caused by using stdin
instead of reading data from a file, but flame graphs didn't support
this theory, either.

> If the latter, could we avoid invoking the new machinery altogether? In
> other words, shouldn't receive-pack only set the reachable_oids_fn() to
> enumerate advertised references only when the set of advertised
> references differs from the behavior of `--not --all`?

Yeah, I was taking a "wait for feedback and see" stance on this. We can
easily make the logic conditional on whether there are any hidden refs
at all.

> >  	if (check_connected(iterate_receive_command_list, &data, &opt))
> >  		set_connectivity_errors(commands, si);
> >
> > @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
> >  {
> >  	int advertise_refs = 0;
> >  	struct command *commands;
> > +	struct oidset announced_oids = OIDSET_INIT;
> 
> This looks like trading one problem for another. In your above example,
> we now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the
> end of the world, but it feels like an avoidable problem.

We store these references in an `oidset` before this patch set already,
but yes, the lifetime is longer now. But note that this set stores the
announced objects, not the hidden ones. So we don't store 6.8m OIDs, but
only the 250k announced ones.

> Could we enumerate the references in a callback to for_each_ref() and
> only emit ones which aren't hidden? Storing these and then recalling
> them after the fact is worth avoiding.

Sorry, I don't quite get what you're proposing. `for_each_ref()` already
does exactly that: it stores every reference that is not hidden in the
above `oidset`. This is the exact set of advertised references, which in
my example repository would be about 250k. This information is used by
git-receive-pack(1) to avoid announcing the same object twice, and now
it's also used to inform the connectivity check to use these objects as
the set of already-reachable objects.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Taylor Blau]

On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:
> On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:
> > On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:
> > > This strategy has the major downside that it will not require any object
> > > to be sent by the client that is reachable by any of the repositories'
> > > references. While that sounds like it would be indeed what we are after
> > > with the connectivity check, it is arguably not. The administrator that
> > > manages the server-side Git repository may have configured certain refs
> > > to be hidden during the reference advertisement via `transfer.hideRefs`
> > > or `receivepack.hideRefs`. Whatever the reason, the result is that the
> > > client shouldn't expect that any of those hidden references exists on
> > > the remote side, and neither should they assume any of the pointed-to
> > > objects to exist except if referenced by any visible reference. But
> > > because we treat _all_ local refs as uninteresting in the connectivity
> > > check, a client is free to send a packfile that references objects that
> > > are only reachable via a hidden reference on the server-side, and we
> > > will gladly accept it.
> >
> > You mention below that this is a correctness issue, but I am not sure
> > that I agree.
> >
> > The existing behavior is a little strange, I agree, but your argument
> > relies on an assumption that the history on hidden refs is not part of
> > the reachable set, which is not the case. Any part of the repository
> > that is reachable from _any_ reference, hidden or not, is reachable by
> > definition.
> >
> > So it's perfectly fine to consider objects on hidden refs to be in the
> > uninteresting set, because they are reachable. It's odd from the
> > client's perspective, but I do not see a path to repository corruption
> > with thee existing behavior.
>
> Indeed, I'm not trying to say that this can lead to repository
> corruption.

I definitely agree with that. I have thought about this on-and-off since
you sent the topic, and I am pretty certain that there is no path to
repository corruption with the existing behavior. It would be worth
updating the commit message to make this clearer.

> But security-related or not, I think it is safe to say that any packfile
> sent by a client that does not contain objects required for the updated
> reference that the client cannot know to exist on the server-side must
> be generated by buggy code.

Maybe, though I think it's fine to let clients send us smaller packfiles
if they have some a-priori knowledge that the server has objects that it
isn't advertising. And that can all happen without buggy code. So it's
weird, but there isn't anything wrong with letting it happen.

> [snip]
> > Why do we see a slowdown when there there aren't any hidden references?
> > Or am I misunderstanding your patch message which instead means "we see
> > a slow-down when there are no hidden references [since we still must
> > store and enumerate all advertised references]"?
>
> I have tried to dig down into the code of `revision.c` but ultimately
> returned empty-handed. I _think_ that this is because of the different
> paths we use when reading revisions from stdin as we have to resolve the
> revision to an OID first, which is more involved than taking the OIDs as
> returned by the reference backend. I have tried to short-circuit this
> logic in case the revision read from stdin is exactly `hash_algo->hexsz`
> long so that we try to parse it as an OID directly instead of trying to
> do any of the magic that is required to resolve a revision. But this
> only speed things up by a small margin.
>
> Another assumption was that this is overhead caused by using stdin
> instead of reading data from a file, but flame graphs didn't support
> this theory, either.
>
> > If the latter, could we avoid invoking the new machinery altogether? In
> > other words, shouldn't receive-pack only set the reachable_oids_fn() to
> > enumerate advertised references only when the set of advertised
> > references differs from the behavior of `--not --all`?
>
> Yeah, I was taking a "wait for feedback and see" stance on this. We can
> easily make the logic conditional on whether there are any hidden refs
> at all.

Yeah, I think that this would be preferable. I'm surprised that your
data doesn't support the idea that the slowdown is caused by reading
from stdin instead of parsing `--not --all`. I'd be curious to see what
you have tried so far.

I'm almost certain that forcing rev-list to chew through a bunch of data
on stdin that is basically equivalent to saying `--not --all` is going
to be the source of the slowdown.

> > >  	if (check_connected(iterate_receive_command_list, &data, &opt))
> > >  		set_connectivity_errors(commands, si);
> > >
> > > @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
> > >  {
> > >  	int advertise_refs = 0;
> > >  	struct command *commands;
> > > +	struct oidset announced_oids = OIDSET_INIT;
> >
> > This looks like trading one problem for another. In your above example,
> > we now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the
> > end of the world, but it feels like an avoidable problem.
>
> We store these references in an `oidset` before this patch set already,
> but yes, the lifetime is longer now. But note that this set stores the
> announced objects, not the hidden ones. So we don't store 6.8m OIDs, but
> only the 250k announced ones.

Hmm, OK. I wonder, could rev-list be given a `--refs=<namespace>`
argument which is equal to the advertised references? Or something that
implies "give me all of the references which aren't hidden?"

If we call that maybe `--visible-refs=receive` (to imply the references
*not* in receive.hideRefs), then our connectivity check would become:

    git rev-list --stdin --not --visible-refs=receive

or something like that. Not having to extend the lifetime of these OIDs
in an oidset would be worthwhile. Besides, having a new rev-list option
is cool, too ;-).

> > Could we enumerate the references in a callback to for_each_ref() and
> > only emit ones which aren't hidden? Storing these and then recalling
> > them after the fact is worth avoiding.
>
> Sorry, I don't quite get what you're proposing. `for_each_ref()` already
> does exactly that: it stores every reference that is not hidden in the
> above `oidset`. This is the exact set of advertised references, which in
> my example repository would be about 250k. This information is used by
> git-receive-pack(1) to avoid announcing the same object twice, and now
> it's also used to inform the connectivity check to use these objects as
> the set of already-reachable objects.

Yes, ignore me ;-).

Thanks,
Taylor

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 7856 bytes --]

On Mon, Oct 31, 2022 at 09:28:09PM -0400, Taylor Blau wrote:
> On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:
> > On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:
> > > On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:
> > > > This strategy has the major downside that it will not require any object
> > > > to be sent by the client that is reachable by any of the repositories'
> > > > references. While that sounds like it would be indeed what we are after
> > > > with the connectivity check, it is arguably not. The administrator that
> > > > manages the server-side Git repository may have configured certain refs
> > > > to be hidden during the reference advertisement via `transfer.hideRefs`
> > > > or `receivepack.hideRefs`. Whatever the reason, the result is that the
> > > > client shouldn't expect that any of those hidden references exists on
> > > > the remote side, and neither should they assume any of the pointed-to
> > > > objects to exist except if referenced by any visible reference. But
> > > > because we treat _all_ local refs as uninteresting in the connectivity
> > > > check, a client is free to send a packfile that references objects that
> > > > are only reachable via a hidden reference on the server-side, and we
> > > > will gladly accept it.
> > >
> > > You mention below that this is a correctness issue, but I am not sure
> > > that I agree.
> > >
> > > The existing behavior is a little strange, I agree, but your argument
> > > relies on an assumption that the history on hidden refs is not part of
> > > the reachable set, which is not the case. Any part of the repository
> > > that is reachable from _any_ reference, hidden or not, is reachable by
> > > definition.
> > >
> > > So it's perfectly fine to consider objects on hidden refs to be in the
> > > uninteresting set, because they are reachable. It's odd from the
> > > client's perspective, but I do not see a path to repository corruption
> > > with thee existing behavior.
> >
> > Indeed, I'm not trying to say that this can lead to repository
> > corruption.
> 
> I definitely agree with that. I have thought about this on-and-off since
> you sent the topic, and I am pretty certain that there is no path to
> repository corruption with the existing behavior. It would be worth
> updating the commit message to make this clearer.

Fair enough, I can try to do that.

> > But security-related or not, I think it is safe to say that any packfile
> > sent by a client that does not contain objects required for the updated
> > reference that the client cannot know to exist on the server-side must
> > be generated by buggy code.
> 
> Maybe, though I think it's fine to let clients send us smaller packfiles
> if they have some a-priori knowledge that the server has objects that it
> isn't advertising. And that can all happen without buggy code. So it's
> weird, but there isn't anything wrong with letting it happen.

Well, I don't see how to achieve both at the same time though: we can
either limit the set of uninteresting tips to what we have announced to
the client, or we allow clients to omit objects that have not been
announced. These are mutually exclusive.

So if we take the stance that it was fine to send packfiles that omit
hidden objects and that this is something we want to continue to support
then this patch series probably becomes moot. Doing the proposed
optimization means that we also tighten the rules here.

> > [snip]
> > > Why do we see a slowdown when there there aren't any hidden references?
> > > Or am I misunderstanding your patch message which instead means "we see
> > > a slow-down when there are no hidden references [since we still must
> > > store and enumerate all advertised references]"?
> >
> > I have tried to dig down into the code of `revision.c` but ultimately
> > returned empty-handed. I _think_ that this is because of the different
> > paths we use when reading revisions from stdin as we have to resolve the
> > revision to an OID first, which is more involved than taking the OIDs as
> > returned by the reference backend. I have tried to short-circuit this
> > logic in case the revision read from stdin is exactly `hash_algo->hexsz`
> > long so that we try to parse it as an OID directly instead of trying to
> > do any of the magic that is required to resolve a revision. But this
> > only speed things up by a small margin.
> >
> > Another assumption was that this is overhead caused by using stdin
> > instead of reading data from a file, but flame graphs didn't support
> > this theory, either.
> >
> > > If the latter, could we avoid invoking the new machinery altogether? In
> > > other words, shouldn't receive-pack only set the reachable_oids_fn() to
> > > enumerate advertised references only when the set of advertised
> > > references differs from the behavior of `--not --all`?
> >
> > Yeah, I was taking a "wait for feedback and see" stance on this. We can
> > easily make the logic conditional on whether there are any hidden refs
> > at all.
> 
> Yeah, I think that this would be preferable. I'm surprised that your
> data doesn't support the idea that the slowdown is caused by reading
> from stdin instead of parsing `--not --all`. I'd be curious to see what
> you have tried so far.
> 
> I'm almost certain that forcing rev-list to chew through a bunch of data
> on stdin that is basically equivalent to saying `--not --all` is going
> to be the source of the slowdown.

I was basically benchmarking these two commands in the repository with
7m refs:

    # old-style connectivity check
    $ git commit-tree HEAD^{tree} -m commit -p HEAD >newtip
    $ git rev-list --objects --quiet --stdin --not --all <newtip

    # new-style connectivity check
    $ ( git for-each-ref --format='^%(objectname)' ; $ git commit-tree HEAD^{tree} -m commit -p HEAD ) >oldandnewtips
    $ git rev-list --objects --quiet --stdin <oldandnewtips

I may have another look today and maybe share some flame graphs if I
continue to not see anything obvious.

> > > >  	if (check_connected(iterate_receive_command_list, &data, &opt))
> > > >  		set_connectivity_errors(commands, si);
> > > >
> > > > @@ -2462,6 +2473,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
> > > >  {
> > > >  	int advertise_refs = 0;
> > > >  	struct command *commands;
> > > > +	struct oidset announced_oids = OIDSET_INIT;
> > >
> > > This looks like trading one problem for another. In your above example,
> > > we now need to store 20 bytes of OIDs 6.8M times, or ~130 MiB. Not the
> > > end of the world, but it feels like an avoidable problem.
> >
> > We store these references in an `oidset` before this patch set already,
> > but yes, the lifetime is longer now. But note that this set stores the
> > announced objects, not the hidden ones. So we don't store 6.8m OIDs, but
> > only the 250k announced ones.
> 
> Hmm, OK. I wonder, could rev-list be given a `--refs=<namespace>`
> argument which is equal to the advertised references? Or something that
> implies "give me all of the references which aren't hidden?"
> 
> If we call that maybe `--visible-refs=receive` (to imply the references
> *not* in receive.hideRefs), then our connectivity check would become:
> 
>     git rev-list --stdin --not --visible-refs=receive
> 
> or something like that. Not having to extend the lifetime of these OIDs
> in an oidset would be worthwhile. Besides, having a new rev-list option
> is cool, too ;-).

That is indeed an interesting idea. It would likely both fix the issue
of extended memory lifetime of the announced OIDs as well as fixing the
slowdown caused by using `--stdin` for them.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4161 bytes --]

On Tue, Nov 01, 2022 at 08:20:48AM +0100, Patrick Steinhardt wrote:
> On Mon, Oct 31, 2022 at 09:28:09PM -0400, Taylor Blau wrote:
> > On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:
> > > On Sun, Oct 30, 2022 at 03:09:04PM -0400, Taylor Blau wrote:
> > > > On Fri, Oct 28, 2022 at 04:42:27PM +0200, Patrick Steinhardt wrote:
> > > > > This strategy has the major downside that it will not require any object
> > > > > to be sent by the client that is reachable by any of the repositories'
> > > > > references. While that sounds like it would be indeed what we are after
> > > > > with the connectivity check, it is arguably not. The administrator that
> > > > > manages the server-side Git repository may have configured certain refs
> > > > > to be hidden during the reference advertisement via `transfer.hideRefs`
> > > > > or `receivepack.hideRefs`. Whatever the reason, the result is that the
> > > > > client shouldn't expect that any of those hidden references exists on
> > > > > the remote side, and neither should they assume any of the pointed-to
> > > > > objects to exist except if referenced by any visible reference. But
> > > > > because we treat _all_ local refs as uninteresting in the connectivity
> > > > > check, a client is free to send a packfile that references objects that
> > > > > are only reachable via a hidden reference on the server-side, and we
> > > > > will gladly accept it.
> > > >
> > > > You mention below that this is a correctness issue, but I am not sure
> > > > that I agree.
> > > >
> > > > The existing behavior is a little strange, I agree, but your argument
> > > > relies on an assumption that the history on hidden refs is not part of
> > > > the reachable set, which is not the case. Any part of the repository
> > > > that is reachable from _any_ reference, hidden or not, is reachable by
> > > > definition.
> > > >
> > > > So it's perfectly fine to consider objects on hidden refs to be in the
> > > > uninteresting set, because they are reachable. It's odd from the
> > > > client's perspective, but I do not see a path to repository corruption
> > > > with thee existing behavior.
> > >
> > > Indeed, I'm not trying to say that this can lead to repository
> > > corruption.
> > 
> > I definitely agree with that. I have thought about this on-and-off since
> > you sent the topic, and I am pretty certain that there is no path to
> > repository corruption with the existing behavior. It would be worth
> > updating the commit message to make this clearer.
> 
> Fair enough, I can try to do that.
> 
> > > But security-related or not, I think it is safe to say that any packfile
> > > sent by a client that does not contain objects required for the updated
> > > reference that the client cannot know to exist on the server-side must
> > > be generated by buggy code.
> > 
> > Maybe, though I think it's fine to let clients send us smaller packfiles
> > if they have some a-priori knowledge that the server has objects that it
> > isn't advertising. And that can all happen without buggy code. So it's
> > weird, but there isn't anything wrong with letting it happen.
> 
> Well, I don't see how to achieve both at the same time though: we can
> either limit the set of uninteresting tips to what we have announced to
> the client, or we allow clients to omit objects that have not been
> announced. These are mutually exclusive.
> 
> So if we take the stance that it was fine to send packfiles that omit
> hidden objects and that this is something we want to continue to support
> then this patch series probably becomes moot. Doing the proposed
> optimization means that we also tighten the rules here.

I'm wrong and you're right: we can do the optimization to limit the refs
we use but still let clients send objects that are hidden. I didn't take
into account that this is merely an optimization that we stop walking at
reachable tips. I'll reword the commit message when having another go
and will likely do something along the lines of your proposed new
`--visible-refs` option in v2 of this series.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Taylor Blau]

On Tue, Nov 01, 2022 at 12:53:42PM +0100, Patrick Steinhardt wrote:
> > > Maybe, though I think it's fine to let clients send us smaller packfiles
> > > if they have some a-priori knowledge that the server has objects that it
> > > isn't advertising. And that can all happen without buggy code. So it's
> > > weird, but there isn't anything wrong with letting it happen.
> >
> > Well, I don't see how to achieve both at the same time though: we can
> > either limit the set of uninteresting tips to what we have announced to
> > the client, or we allow clients to omit objects that have not been
> > announced. These are mutually exclusive.
> >
> > So if we take the stance that it was fine to send packfiles that omit
> > hidden objects and that this is something we want to continue to support
> > then this patch series probably becomes moot. Doing the proposed
> > optimization means that we also tighten the rules here.
>
> I'm wrong and you're right: we can do the optimization to limit the refs
> we use but still let clients send objects that are hidden. I didn't take
> into account that this is merely an optimization that we stop walking at
> reachable tips. I'll reword the commit message when having another go
> and will likely do something along the lines of your proposed new
> `--visible-refs` option in v2 of this series.

I wasn't necessarily advocating for a behavior change in this series,
more pointing out that the situation you said can only happen with buggy
code doesn't actually require a bug in practice.

Thanks,
Taylor

Re: [PATCH 2/2] receive-pack: use advertised reference tips to inform connectivity check

[Jeff King]

On Mon, Oct 31, 2022 at 03:45:47PM +0100, Patrick Steinhardt wrote:

> Indeed, I'm not trying to say that this can lead to repository
> corruption. If at all you can argue that this is more security-related.
> Suppose an object is not reachable from any public reference and that
> `allowAnySHA1InWant=false`. Then you could make these hidden objects
> reachable by sending a packfile with an object that references the
> hidden object. It naturally requires you to somehow know about the
> object ID, so I don't think this is a critical issue.

I'd have to double check, but isn't this all moot with the v2 protocol
anyway? I didn't think it even respected allowAnySHA1InWant.

Even that aside, there are other tricks you can do. E.g., pushing up an
object which claims to be a delta of X (though you have the trick then
of figuring out how to reference it), or pushing up a test object then
fetching it back while claiming to have X, treating the server as an
oracle which may give you a delta against X.

In short, I don't think it's worth considering unreachable objects in a
repository to be considered secret.

> > Why do we see a slowdown when there there aren't any hidden references?
> > Or am I misunderstanding your patch message which instead means "we see
> > a slow-down when there are no hidden references [since we still must
> > store and enumerate all advertised references]"?
> 
> I have tried to dig down into the code of `revision.c` but ultimately
> returned empty-handed. I _think_ that this is because of the different
> paths we use when reading revisions from stdin as we have to resolve the
> revision to an OID first, which is more involved than taking the OIDs as
> returned by the reference backend. I have tried to short-circuit this
> logic in case the revision read from stdin is exactly `hash_algo->hexsz`
> long so that we try to parse it as an OID directly instead of trying to
> do any of the magic that is required to resolve a revision. But this
> only speed things up by a small margin.
> 
> Another assumption was that this is overhead caused by using stdin
> instead of reading data from a file, but flame graphs didn't support
> this theory, either.

Certainly read_revisions_from_stdin() will allocate a bunch of extra
data per item it reads (via add_rev_cmdline() and add_pending), which is
going to be way more than parsing or stdin overhead.

Much worse is that it will call get_reference(), which is very keen to
actually open and parse the object (as you know, since you added
commit-graph handling there). That might have gotten a bit better in
v2.38.0 if you have any references to blobs (as we'd now skip the extra
hash check).

Of course the original "rev-list --not --all" would suffer from the same
thing, so some of that may not explain any difference between the two.

-Peff

Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> Hi,
>
> this patch series improves the connectivity check done by stateful
> git-receive-pack(1) to only consider references as reachable that have
> been advertised to the client. This has two advantages:
>
>     - A client shouldn't assume objects to exist that have not been part
>       of the reference advertisement. But if it excluded an object from
>       the packfile that is reachable via any ref that is excluded from
>       the reference advertisement due to `transfer.hideRefs` we'd have
>       accepted the push anyway. I'd argue that this is a bug in the
>       current implementation.

I agree that it is bad to accept an incoming pack that depends on an
object that is *only* reachable by a hidden ref, but what you said
above is a bit stronger than that.  Use transfer.hideRefs does not
have to be to hide objects (e.g. I could hide the tip of the
branches that holds tips of older maintenance tracks, just to
unclutter, and giving a pack that depends on older parts of history
is just fine).

Let's let it pass, as the cover letter material won't become part of
the permanent history ;-)

>     - Second, by using advertised refs as inputs instead of `git
>       rev-list --not --all` we avoid looking up all refs that are
>       irrelevant to the current push. This can be a huge performance
>       improvement in repos that have a huge amount of internal, hidden
>       refs. In one of our repos with 7m refs, of which 6.8m are hidden,
>       this speeds up pushes from ~30s to ~4.5s.

Yes.

> One downside is that we need to pass in the object IDs that were part of
> the reference advertisement via the standard input, which is seemingly
> slower than reading them from the refdb. I'm discussing this in the
> second commit.

Interesting.

Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check

[Taylor Blau]

On Fri, Oct 28, 2022 at 04:42:19PM +0200, Patrick Steinhardt wrote:
> this patch series improves the connectivity check done by stateful
> git-receive-pack(1) to only consider references as reachable that have
> been advertised to the client. This has two advantages:

A third advantage which I didn't see mentioned here is that we have a
better chance of being helped by reachability bitmaps here. Because we
aren't likely to cover hidden references with bitmaps, we would be
paying a significant cost to build up a complete picture of what's
reachable from --all.

That will inevitably involve a lot of traversing from unbitmapped
reference tips (i.e. the hidden ones) down to either a random bitmap
sprinkled throughout history, or the root of that line of history.

But if we limited the negated side of our connectivity check to just the
non-hidden references, then we will likely end up with more full bitmap
coverage of those tips, without having to do any additional traversal.

So that is definitely something worth experimenting with.

Thanks,
Taylor

Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check

[Jeff King]

On Fri, Oct 28, 2022 at 04:42:19PM +0200, Patrick Steinhardt wrote:

>     - A client shouldn't assume objects to exist that have not been part
>       of the reference advertisement. But if it excluded an object from
>       the packfile that is reachable via any ref that is excluded from
>       the reference advertisement due to `transfer.hideRefs` we'd have
>       accepted the push anyway. I'd argue that this is a bug in the
>       current implementation.

Like others, I don't think this is a bug exactly. We'd never introduce a
corruption. We're just more lenient with clients than we need to be.

But I don't think your scheme changes that. In a sense, the tips used by
"rev-list --not --all" are really an optimization. We will walk the
history from the to-be-updated ref tips all the way down to the roots if
we have to. So imagine that I have object X which is not referenced at
all (neither hidden nor visible ref). We obviously do not advertise it
to the client, but let's further imagine that a client sends us a pack
with X..Y, and a request to update some ref to Y.

Both before and after your code, if rev-list is able to walk down from Y
until either we hit all roots or all UNINTERESTING commits, it will be
satisfied. So as long as the receiving repo actually has all of the
history leading up to X, it will allow the push, regardless of your
patch.

If we wanted to stop being lenient, we'd have to actually check that
every object we traverse is either reachable, or came from the
just-pushed pack.


There's also a subtle timing issue here. Our connectivity check happens
after we've finished receiving the pack. So not only are we including
hidden refs, but we are using the ref state at the end of the push
(after receiving and processing the incoming pack), rather than the
beginning.

From the same "leniency" lens this seems like the wrong thing. But as
above, it doesn't matter in practice, because these tips are really an
optimization to tell rev-list that it can stop traversing.

If you think of the connectivity check less as "did the client try to
cheat" and more as "is it OK to update these refs without introducing a
corruption", then it makes sense that you'd want to do read the inputs
to the check as close to the ref update as possible, because it shrinks
the race window which could introduce corruption.

Imagine a situation like this:

  0. We advertise to client that we have commit X.

  1. Client starts pushing up a pack with X..Y and asks to update some
     branch to Y.

  2. Meanwhile, the branch with X is deleted, and X is pruned.

  3. Server finishes receiving the pack. All looks good, and then we
     start a connectivity check.

In the current code, that check starts with the current ref state (with
X deleted) as a given, and makes sure that we have the objects we need
to update the refs. After your patches, it would take X as a given, and
stop traversing when we see it.

That same race exists before your patch, but it's between the time of
"rev-list --not --all" running and the ref update. After your patch,
it's between the advertisement and the ref update, which can be a long
time (hours or even days, if the client is very slow).

In practice I'm not sure how big a deal this is. If we feed the
now-pruned X to rev-list, it may notice that X went away, though we've
been reducing the number of checks there in the name of efficiency
(e.g., if it's still in the commit graph, we'd say "OK, good enough"
these days, even if we don't have it on disk anymore).

But it feels like a wrong direction to make that race longer if there's
no need to.

So all that said...

>     - Second, by using advertised refs as inputs instead of `git
>       rev-list --not --all` we avoid looking up all refs that are
>       irrelevant to the current push. This can be a huge performance
>       improvement in repos that have a huge amount of internal, hidden
>       refs. In one of our repos with 7m refs, of which 6.8m are hidden,
>       this speeds up pushes from ~30s to ~4.5s.

I like the general direction here of avoiding the hidden refs. The
client _shouldn't_ have been using them, so we can optimistically assume
they're useless (and in the case of races or other weirdness, rev-list
just ends up traversing a bit further).

But we can split the two ideas in your series:

  1. Feed the advertised tips from receive-pack to rev-list.

  2. Avoid traversing from the hidden tips.

Doing (1) gets you (2) for free. But if we don't want to do (1), and I
don't think we do, we can get (2) by just teaching rev-list to narrow
the check.

I see some discussion in the other part of the thread, and we may need a
new rev-list option to do this, as mentioned there. However, you _might_
be able to do it the existing --exclude mechanism. I.e., something like:

  rev-list --stdin --not --exclude 'refs/hidden/*' --all

The gotchas are:

  - I'm not 100% sure that --exclude globbing and transfer.hideRefs
    syntax are compatible. You'd want to check.

  - these would have to come on the command line (at least with the
    current code). Probably nobody has enough hiderefs patterns for that
    to be a problem (and remember we are passing the glob pattern here,
    not the 6.8M refs themselves). But it could bite somebody in a
    pathological case.

-Peff

Re: [PATCH 0/2] receive-pack: use advertised reference tips to inform connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 6412 bytes --]

On Tue, Nov 01, 2022 at 05:00:22AM -0400, Jeff King wrote:
> On Fri, Oct 28, 2022 at 04:42:19PM +0200, Patrick Steinhardt wrote:
> 
> >     - A client shouldn't assume objects to exist that have not been part
> >       of the reference advertisement. But if it excluded an object from
> >       the packfile that is reachable via any ref that is excluded from
> >       the reference advertisement due to `transfer.hideRefs` we'd have
> >       accepted the push anyway. I'd argue that this is a bug in the
> >       current implementation.
> 
> Like others, I don't think this is a bug exactly. We'd never introduce a
> corruption. We're just more lenient with clients than we need to be.
> 
> But I don't think your scheme changes that. In a sense, the tips used by
> "rev-list --not --all" are really an optimization. We will walk the
> history from the to-be-updated ref tips all the way down to the roots if
> we have to. So imagine that I have object X which is not referenced at
> all (neither hidden nor visible ref). We obviously do not advertise it
> to the client, but let's further imagine that a client sends us a pack
> with X..Y, and a request to update some ref to Y.
> 
> Both before and after your code, if rev-list is able to walk down from Y
> until either we hit all roots or all UNINTERESTING commits, it will be
> satisfied. So as long as the receiving repo actually has all of the
> history leading up to X, it will allow the push, regardless of your
> patch.

Oh, right! Now I see where my thinko was, which means both you and
Taylor are correct. I somehow assumed that we'd fail the connectivity
check in that case, but all it means is that we now potentially walk
more objects than we'd have done if we used `--not --all`.

> If we wanted to stop being lenient, we'd have to actually check that
> every object we traverse is either reachable, or came from the
> just-pushed pack.

Yes, indeed.

> There's also a subtle timing issue here. Our connectivity check happens
> after we've finished receiving the pack. So not only are we including
> hidden refs, but we are using the ref state at the end of the push
> (after receiving and processing the incoming pack), rather than the
> beginning.
> 
> From the same "leniency" lens this seems like the wrong thing. But as
> above, it doesn't matter in practice, because these tips are really an
> optimization to tell rev-list that it can stop traversing.
> 
> If you think of the connectivity check less as "did the client try to
> cheat" and more as "is it OK to update these refs without introducing a
> corruption", then it makes sense that you'd want to do read the inputs
> to the check as close to the ref update as possible, because it shrinks
> the race window which could introduce corruption.

Agreed.

> Imagine a situation like this:
> 
>   0. We advertise to client that we have commit X.
> 
>   1. Client starts pushing up a pack with X..Y and asks to update some
>      branch to Y.
> 
>   2. Meanwhile, the branch with X is deleted, and X is pruned.
> 
>   3. Server finishes receiving the pack. All looks good, and then we
>      start a connectivity check.
> 
> In the current code, that check starts with the current ref state (with
> X deleted) as a given, and makes sure that we have the objects we need
> to update the refs. After your patches, it would take X as a given, and
> stop traversing when we see it.
> 
> That same race exists before your patch, but it's between the time of
> "rev-list --not --all" running and the ref update. After your patch,
> it's between the advertisement and the ref update, which can be a long
> time (hours or even days, if the client is very slow).
> 
> In practice I'm not sure how big a deal this is. If we feed the
> now-pruned X to rev-list, it may notice that X went away, though we've
> been reducing the number of checks there in the name of efficiency
> (e.g., if it's still in the commit graph, we'd say "OK, good enough"
> these days, even if we don't have it on disk anymore).
> 
> But it feels like a wrong direction to make that race longer if there's
> no need to.

Good point.

> So all that said...
> 
> >     - Second, by using advertised refs as inputs instead of `git
> >       rev-list --not --all` we avoid looking up all refs that are
> >       irrelevant to the current push. This can be a huge performance
> >       improvement in repos that have a huge amount of internal, hidden
> >       refs. In one of our repos with 7m refs, of which 6.8m are hidden,
> >       this speeds up pushes from ~30s to ~4.5s.
> 
> I like the general direction here of avoiding the hidden refs. The
> client _shouldn't_ have been using them, so we can optimistically assume
> they're useless (and in the case of races or other weirdness, rev-list
> just ends up traversing a bit further).
> 
> But we can split the two ideas in your series:
> 
>   1. Feed the advertised tips from receive-pack to rev-list.
> 
>   2. Avoid traversing from the hidden tips.
> 
> Doing (1) gets you (2) for free. But if we don't want to do (1), and I
> don't think we do, we can get (2) by just teaching rev-list to narrow
> the check.
> 
> I see some discussion in the other part of the thread, and we may need a
> new rev-list option to do this, as mentioned there. However, you _might_
> be able to do it the existing --exclude mechanism. I.e., something like:
> 
>   rev-list --stdin --not --exclude 'refs/hidden/*' --all

Yeah, Taylor proposed to add a new `--visible-refs=receive` option that
lets git-rev-list(1) automatically add all references that are visible
when paying attention to `receive.hideRefs`. I actually like this idea
and will likely have a look at how easy or hard it is to implement.

> The gotchas are:
> 
>   - I'm not 100% sure that --exclude globbing and transfer.hideRefs
>     syntax are compatible. You'd want to check.
> 
>   - these would have to come on the command line (at least with the
>     current code). Probably nobody has enough hiderefs patterns for that
>     to be a problem (and remember we are passing the glob pattern here,
>     not the 6.8M refs themselves). But it could bite somebody in a
>     pathological case.
> 
> -Peff

Well, we can avoid these gotchas if we used `--visible-refs`.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 0/3] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

Hi,

this is the second version of my patch series that tries to improve
performance of the connectivity check by only considering preexisting
refs as uninteresting that could actually have been advertised to the
client.

This version uses the same idea as v1, but it's basically rewritten
based on Taylor's idea of adding a `git rev-list --visible-refs=`
switch. This fixes two major concerns:

    1. The performance regression in repositories is now gone when there
       are no hidden refs. Previously, there was a 10% regression in one
       specific benchmark that was caused by reading advertised tips
       via stdin in git-rev-list(1).

    2. It fixes Jeffs concerns around a TOCTOU-style race with very slow
       clients. It could in theory happen that a push takes multiple
       days. With the previous idea of reusing advertised refs for the
       connectivity check, the end result would be that we perform the
       check with a set of refs that has been generated when the push
       started.

The series is structured as following:

    - Patch 1 does some preliminary cleanups to de-globalize the set of
      parsed hidden refs so that we can use the existing functions
      easily for the new `--visible-refs=` option.

    - Patch 2 adds the new `--visible-refs=` option to git-rev-list(1).

    - Patch 3 converts git-receive-pack(1) to use `--visible-refs=`
      instead of `--all`.

Overall the performance improvement isn't quite as strong as before:
we're only 4.5x faster compared to 6.5x in our repo. But I guess that's
still a good-enough improvement, doubly so that there are no downsides
for repos anymore that ain't got any hidden refs.

Patrick

Patrick Steinhardt (3):
  refs: get rid of global list of hidden refs
  revision: add new parameter to specify all visible refs
  receive-pack: only use visible refs for connectivity check

 Documentation/rev-list-options.txt |  15 +++--
 builtin/receive-pack.c             |  10 ++-
 builtin/rev-list.c                 |   1 +
 builtin/rev-parse.c                |   1 +
 connected.c                        |   5 +-
 connected.h                        |   6 ++
 ls-refs.c                          |  13 ++--
 refs.c                             |  14 ++--
 refs.h                             |   5 +-
 revision.c                         |  34 +++++++++-
 t/t6021-rev-list-visible-refs.sh   | 102 +++++++++++++++++++++++++++++
 upload-pack.c                      |  30 +++++----
 12 files changed, 196 insertions(+), 40 deletions(-)
 create mode 100755 t/t6021-rev-list-visible-refs.sh

-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 1/3] refs: get rid of global list of hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 12498 bytes --]

We're about to add a new argument to git-rev-list(1) that allows it to
add all references that are visible when taking `transfer.hideRefs` et
al into account. This will require us to potentially parse multiple sets
of hidden refs, which is not easily possible right now as there is only
a single, global instance of the list of parsed hidden refs.

Refactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both
take the list of hidden references as input and adjust callers to keep a
local list, instead. This allows us to easily use multiple hidden-ref
lists. Furthermore, it allows us to properly free this list before we
exit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c |  8 +++++---
 ls-refs.c              | 13 +++++++++----
 refs.c                 | 14 ++++----------
 refs.h                 |  5 +++--
 upload-pack.c          | 30 ++++++++++++++++++------------
 5 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 44bcea3a5b..1f3efc58fb 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)
 
 static int receive_pack_config(const char *var, const char *value, void *cb)
 {
-	int status = parse_hide_refs_config(var, value, "receive");
+	int status = parse_hide_refs_config(var, value, "receive", &hidden_refs);
 
 	if (status)
 		return status;
@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,
 	struct oidset *seen = data;
 	const char *path = strip_namespace(path_full);
 
-	if (ref_is_hidden(path, path_full))
+	if (ref_is_hidden(path, path_full, &hidden_refs))
 		return 0;
 
 	/*
@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)
 		strbuf_setlen(&refname_full, prefix_len);
 		strbuf_addstr(&refname_full, cmd->ref_name);
 
-		if (!ref_is_hidden(cmd->ref_name, refname_full.buf))
+		if (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))
 			continue;
 		if (is_null_oid(&cmd->new_oid))
 			cmd->error_string = "deny deleting a hidden ref";
@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
+	string_list_clear(&hidden_refs, 1);
 	free((void *)push_cert_nonce);
 	return 0;
 }
diff --git a/ls-refs.c b/ls-refs.c
index fa0d01b47c..ae89f850e9 100644
--- a/ls-refs.c
+++ b/ls-refs.c
@@ -6,6 +6,7 @@
 #include "ls-refs.h"
 #include "pkt-line.h"
 #include "config.h"
+#include "string-list.h"
 
 static int config_read;
 static int advertise_unborn;
@@ -73,6 +74,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
+	struct string_list hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 
 	strbuf_reset(&data->buf);
 
-	if (ref_is_hidden(refname_nons, refname))
+	if (ref_is_hidden(refname_nons, refname, &data->hidden_refs))
 		return 0;
 
 	if (!ref_match(&data->prefixes, refname_nons))
@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)
 }
 
 static int ls_refs_config(const char *var, const char *value,
-			  void *data UNUSED)
+			  void *cb_data)
 {
+	struct ls_refs_data *data = cb_data;
 	/*
 	 * We only serve fetches over v2 for now, so respect only "uploadpack"
 	 * config. This may need to eventually be expanded to "receive", but we
 	 * don't yet know how that information will be passed to ls-refs.
 	 */
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 int ls_refs(struct repository *r, struct packet_reader *request)
@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
+	string_list_init_dup(&data.hidden_refs);
 
 	ensure_config_read();
-	git_config(ls_refs_config, NULL);
+	git_config(ls_refs_config, &data);
 
 	while (packet_reader_read(request) == PACKET_READ_NORMAL) {
 		const char *arg = request->line;
@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
+	string_list_clear(&data.hidden_refs, 1);
 	return 0;
 }
 
diff --git a/refs.c b/refs.c
index 1491ae937e..f1711e2e9f 100644
--- a/refs.c
+++ b/refs.c
@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 					    refname, strict);
 }
 
-static struct string_list *hide_refs;
-
-int parse_hide_refs_config(const char *var, const char *value, const char *section)
+int parse_hide_refs_config(const char *var, const char *value, const char *section,
+			   struct string_list *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		if (!hide_refs) {
-			CALLOC_ARRAY(hide_refs, 1);
-			hide_refs->strdup_strings = 1;
-		}
 		string_list_append(hide_refs, ref);
 	}
 	return 0;
 }
 
-int ref_is_hidden(const char *refname, const char *refname_full)
+int ref_is_hidden(const char *refname, const char *refname_full,
+		  const struct string_list *hide_refs)
 {
 	int i;
 
-	if (!hide_refs)
-		return 0;
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
 		const char *match = hide_refs->items[i].string;
 		const char *subject;
diff --git a/refs.h b/refs.h
index 8958717a17..3266fd8f57 100644
--- a/refs.h
+++ b/refs.h
@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *new_oid, const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr);
 
-int parse_hide_refs_config(const char *var, const char *value, const char *);
+int parse_hide_refs_config(const char *var, const char *value, const char *,
+			   struct string_list *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *);
+int ref_is_hidden(const char *, const char *, const struct string_list *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
diff --git a/upload-pack.c b/upload-pack.c
index 0b8311bd68..9db17f8787 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -62,6 +62,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
+	struct string_list hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
+	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 	memset(data, 0, sizeof(*data));
 	data->symref = symref;
 	data->wanted_refs = wanted_refs;
+	data->hidden_refs = hidden_refs;
 	data->want_obj = want_obj;
 	data->have_obj = have_obj;
 	data->haves = haves;
@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
+	string_list_clear(&data->hidden_refs, 1);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * Checking for reachable shallows requires that our refs be
 		 * marked with OUR_REF.
 		 */
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, data);
+		for_each_namespaced_ref(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid)
+			const struct object_id *oid, const struct string_list *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
-	if (ref_is_hidden(refname, refname_full)) {
+	if (ref_is_hidden(refname, refname_full, hidden_refs)) {
 		o->flags |= HIDDEN_REF;
 		return 1;
 	}
@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,
 }
 
 static int check_ref(const char *refname_full, const struct object_id *oid,
-		     int flag UNUSED, void *cb_data UNUSED)
+		     int flag UNUSED, void *cb_data)
 {
 	const char *refname = strip_namespace(refname_full);
+	struct upload_pack_data *data = cb_data;
 
-	mark_our_ref(refname, refname_full, oid);
+	mark_our_ref(refname, refname_full, oid, &data->hidden_refs);
 	return 0;
 }
 
@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 	struct object_id peeled;
 	struct upload_pack_data *data = cb_data;
 
-	if (mark_our_ref(refname_nons, refname, oid))
+	if (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))
 		return 0;
 
 	if (capabilities) {
@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)
 	if (parse_object_filter_config(var, value, data) < 0)
 		return -1;
 
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)
@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		advertise_shallow_grafts(1);
 		packet_flush(1);
 	} else {
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, &data);
+		for_each_namespaced_ref(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
+			  struct string_list *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 		struct strbuf refname = STRBUF_INIT;
 
 		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
-		if (ref_is_hidden(refname_nons, refname.buf) ||
+		if (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||
 		    read_ref(refname.buf, &oid)) {
 			packet_writer_error(writer, "unknown ref %s", refname_nons);
 			die("unknown ref %s", refname_nons);
@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,
 			continue;
 		if (data->allow_ref_in_want &&
 		    parse_want_ref(&data->writer, arg, &data->wanted_refs,
-				   &data->want_obj))
+				   &data->hidden_refs, &data->want_obj))
 			continue;
 		/* process have line */
 		if (parse_have(arg, &data->haves))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 2/3] revision: add new parameter to specify all visible refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 9576 bytes --]

Users can optionally hide refs from remote users in git-upload-pack(1),
git-receive-pack(1) and others via the `transfer.hideRefs`, but there is
not an easy way to obtain the list of all visible or hidden refs right
now. We'll require just that though for a performance improvement in our
connectivity check.

Add a new pseudo-ref `--visible-refs=` that pretends as if all refs have
been added to the command line that are not hidden. The pseudo-ref
requiers either one of "transfer", "uploadpack" or "receive" as argument
to pay attention to `transfer.hideRefs`, `uploadpack.hideRefs` or
`receive.hideRefs`, respectively.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/rev-list-options.txt |  15 +++--
 builtin/rev-list.c                 |   1 +
 builtin/rev-parse.c                |   1 +
 revision.c                         |  34 +++++++++-
 t/t6021-rev-list-visible-refs.sh   | 102 +++++++++++++++++++++++++++++
 5 files changed, 145 insertions(+), 8 deletions(-)
 create mode 100755 t/t6021-rev-list-visible-refs.sh

diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
index 1837509566..a0e34b0e2b 100644
--- a/Documentation/rev-list-options.txt
+++ b/Documentation/rev-list-options.txt
@@ -180,14 +180,19 @@ endif::git-log[]
 	is automatically prepended if missing. If pattern lacks '?', '{asterisk}',
 	or '[', '/{asterisk}' at the end is implied.
 
+--visible-refs=[transfer|receive|uploadpack]::
+	Pretend as if all the refs that have not been hidden via either one of
+	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` are
+	listed on the command line.
+
 --exclude=<glob-pattern>::
 
 	Do not include refs matching '<glob-pattern>' that the next `--all`,
-	`--branches`, `--tags`, `--remotes`, or `--glob` would otherwise
-	consider. Repetitions of this option accumulate exclusion patterns
-	up to the next `--all`, `--branches`, `--tags`, `--remotes`, or
-	`--glob` option (other options or arguments do not clear
-	accumulated patterns).
+	`--branches`, `--tags`, `--remotes`, `--glob` or `--visible-refs` would
+	otherwise consider. Repetitions of this option accumulate exclusion
+	patterns up to the next `--all`, `--branches`, `--tags`, `--remotes`,
+	`--glob` or `--visible-refs` option (other options or arguments do not
+	clear accumulated patterns).
 +
 The patterns given should not begin with `refs/heads`, `refs/tags`, or
 `refs/remotes` when applied to `--branches`, `--tags`, or `--remotes`,
diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index 3acd93f71e..f719286cf8 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -38,6 +38,7 @@ static const char rev_list_usage[] =
 "    --tags\n"
 "    --remotes\n"
 "    --stdin\n"
+"    --visible-refs=[transfer|receive|uploadpack]\n"
 "    --quiet\n"
 "  ordering output:\n"
 "    --topo-order\n"
diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 8f61050bde..31617bf3d5 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -77,6 +77,7 @@ static int is_rev_argument(const char *arg)
 		"--topo-order",
 		"--date-order",
 		"--unpacked",
+		"--visible-refs=",
 		NULL
 	};
 	const char **p = rev_args;
diff --git a/revision.c b/revision.c
index 0760e78936..ef9e2947af 100644
--- a/revision.c
+++ b/revision.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "config.h"
 #include "object-store.h"
 #include "tag.h"
 #include "blob.h"
@@ -1523,6 +1524,8 @@ struct all_refs_cb {
 	struct rev_info *all_revs;
 	const char *name_for_errormsg;
 	struct worktree *wt;
+	struct string_list hidden_refs;
+	const char *hidden_refs_section;
 };
 
 int ref_excluded(struct string_list *ref_excludes, const char *path)
@@ -1542,11 +1545,13 @@ static int handle_one_ref(const char *path, const struct object_id *oid,
 			  int flag UNUSED,
 			  void *cb_data)
 {
+	const char *stripped_path = strip_namespace(path);
 	struct all_refs_cb *cb = cb_data;
 	struct object *object;
 
-	if (ref_excluded(cb->all_revs->ref_excludes, path))
-	    return 0;
+	if (ref_excluded(cb->all_revs->ref_excludes, path) ||
+	    ref_is_hidden(stripped_path, path, &cb->hidden_refs))
+		return 0;
 
 	object = get_reference(cb->all_revs, path, oid, cb->all_flags);
 	add_rev_cmdline(cb->all_revs, object, path, REV_CMD_REF, cb->all_flags);
@@ -1561,6 +1566,7 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,
 	cb->all_flags = flags;
 	revs->rev_input_given = 1;
 	cb->wt = NULL;
+	string_list_init_dup(&cb->hidden_refs);
 }
 
 void clear_ref_exclusion(struct string_list **ref_excludes_p)
@@ -1596,6 +1602,13 @@ static void handle_refs(struct ref_store *refs,
 	for_each(refs, handle_one_ref, &cb);
 }
 
+static int hide_refs_config(const char *var, const char *value, void *cb_data)
+{
+	struct all_refs_cb *cb = cb_data;
+	return parse_hide_refs_config(var, value, cb->hidden_refs_section,
+				      &cb->hidden_refs);
+}
+
 static void handle_one_reflog_commit(struct object_id *oid, void *cb_data)
 {
 	struct all_refs_cb *cb = cb_data;
@@ -2225,7 +2238,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg
 	    !strcmp(arg, "--bisect") || starts_with(arg, "--glob=") ||
 	    !strcmp(arg, "--indexed-objects") ||
 	    !strcmp(arg, "--alternate-refs") ||
-	    starts_with(arg, "--exclude=") ||
+	    starts_with(arg, "--exclude=") || starts_with(arg, "--visible-refs=") ||
 	    starts_with(arg, "--branches=") || starts_with(arg, "--tags=") ||
 	    starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk="))
 	{
@@ -2759,6 +2772,21 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		parse_list_objects_filter(&revs->filter, arg);
 	} else if (!strcmp(arg, ("--no-filter"))) {
 		list_objects_filter_set_no_filter(&revs->filter);
+	} else if (skip_prefix(arg, "--visible-refs=", &arg)) {
+		struct all_refs_cb cb;
+
+		if (strcmp(arg, "transfer") && strcmp(arg, "receive") &&
+		    strcmp(arg, "uploadpack"))
+			die(_("unsupported section for --visible-refs: %s"), arg);
+
+		init_all_refs_cb(&cb, revs, *flags);
+		cb.hidden_refs_section = arg;
+		git_config(hide_refs_config, &cb);
+
+		refs_for_each_ref(refs, handle_one_ref, &cb);
+
+		string_list_clear(&cb.hidden_refs, 1);
+		clear_ref_exclusion(&revs->ref_excludes);
 	} else {
 		return 0;
 	}
diff --git a/t/t6021-rev-list-visible-refs.sh b/t/t6021-rev-list-visible-refs.sh
new file mode 100755
index 0000000000..9e12384dcf
--- /dev/null
+++ b/t/t6021-rev-list-visible-refs.sh
@@ -0,0 +1,102 @@
+#!/bin/sh
+
+test_description='git rev-list --visible-refs test'
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit_bulk --id=commit --ref=refs/heads/main 1 &&
+	COMMIT=$(git rev-parse refs/heads/main) &&
+	test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&
+	TAG=$(git rev-parse refs/tags/lightweight) &&
+	test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&
+	HIDDEN=$(git rev-parse refs/hidden/commit)
+'
+
+test_expect_success 'invalid section' '
+	echo "fatal: unsupported section for --visible-refs: unsupported" >expected &&
+	test_must_fail git rev-list --visible-refs=unsupported 2>err &&
+	test_cmp expected err
+'
+
+test_expect_success '--visible-refs without hiddenRefs' '
+	git rev-list --visible-refs=transfer >out &&
+	cat >expected <<-EOF &&
+	$HIDDEN
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success 'hidden via transfer.hideRefs' '
+	git -c transfer.hideRefs=refs/hidden/ rev-list --visible-refs=transfer >out &&
+	cat >expected <<-EOF &&
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--all --not --visible-refs=transfer without hidden refs' '
+	git rev-list --all --not --visible-refs=transfer >out &&
+	test_must_be_empty out
+'
+
+test_expect_success '--all --not --visible-refs=transfer with hidden ref' '
+	git -c transfer.hideRefs=refs/hidden/ rev-list --all --not --visible-refs=transfer >out &&
+	cat >expected <<-EOF &&
+	$HIDDEN
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--visible-refs with --exclude' '
+	git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --visible-refs=transfer >out &&
+	cat >expected <<-EOF &&
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+for section in receive uploadpack
+do
+	test_expect_success "hidden via $section.hideRefs" '
+		git -c receive.hideRefs=refs/hidden/ rev-list --visible-refs=receive >out &&
+		cat >expected <<-EOF &&
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "--visible-refs=$section respects transfer.hideRefs" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --visible-refs=$section >out &&
+		cat >expected <<-EOF &&
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "--visible-refs=transfer ignores $section.hideRefs" '
+		git -c $section.hideRefs=refs/hidden/ rev-list --visible-refs=transfer >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "--visible-refs=$section respects both transfer.hideRefs and $section.hideRefs" '
+		git -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --visible-refs=$section >out &&
+		cat >expected <<-EOF &&
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+done
+
+test_done
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs

[Jeff King]

On Thu, Nov 03, 2022 at 03:37:32PM +0100, Patrick Steinhardt wrote:

> Users can optionally hide refs from remote users in git-upload-pack(1),
> git-receive-pack(1) and others via the `transfer.hideRefs`, but there is
> not an easy way to obtain the list of all visible or hidden refs right
> now. We'll require just that though for a performance improvement in our
> connectivity check.
> 
> Add a new pseudo-ref `--visible-refs=` that pretends as if all refs have
> been added to the command line that are not hidden. The pseudo-ref
> requiers either one of "transfer", "uploadpack" or "receive" as argument
> to pay attention to `transfer.hideRefs`, `uploadpack.hideRefs` or
> `receive.hideRefs`, respectively.

Thanks for re-working this. I think it's a sensible path forward for the
problem you're facing.

There were two parts of the implementation that surprised me a bit.
These might just be nits, but because this is a new user-facing plumbing
option that will be hard to change later, we should make sure it fits in
with the adjacent features.

The two things I saw were:

  1. The mutual-exclusion selection between "transfer", "uploadpack",
     and "receive" is not how those options work in their respective
     programs. The "transfer.hideRefs" variable is respected for either
     program. So whichever program you are running, it will always look
     at both "transfer" and itself ("uploadpack" or "receive"). Another
     way to think of it is that the "section" argument to
     parse_hide_refs_config() is not a config section so much as a
     profile. And the profiles are:

       - uploadpack: respect transfer.hideRefs and uploadpack.hideRefs
       - receive: respect transfer.hideRefs and receive.hideRefs

     So it does not make sense to ask for "transfer" as a section; each
     of the modes is already looking at transfer.hideRefs.

     In theory if this option was "look just at $section.hideRefs", it
     could be more flexible to separate out the two. But that makes it
     more of a pain to use (for normal use, you have to specify both
     "transfer" and "receive"). And that is not what your patch
     implements anyway; because it relies on parse_hide_refs_config(),
     it is always adding in "transfer" under the hood (which is why your
     final patch is correct to just say "--visible-refs=receive" without
     specifying "transfer" as well).

  2. Your "--visible-refs" implies "--all", because it's really "all
     refs minus the hidden ones". That's convenient for the intended
     caller, but not as flexible as it could be. If it were instead
     treated the way "--exclude" is, as a modifier for the next
     iteration option, then you do a few extra things:

       a. Combine multiple exclusions in a single iteration:

            git rev-list --exclude-hidden=receive \
	                 --exclude-hidden=upload \
			 --all

          That excludes both types in a single iteration. Whereas if you
	  did:

	    git rev-list --visible-refs=receive \
	                 --visible-refs=upload

	  that will do _two_ iterations, and end up with the union of
	  the sets. Equivalent to:

	    git rev-list --exclude-hidden=receive --all \
	                 --exclude-hidden=upload  --all

       b. Do exclusions on smaller sets than --all:

            git rev-list --exclude-hidden=receive \
	                 --branches

	  which would show just the branches that we'd advertise.

     Now I don't have a particular use case for either of those things.
     But they're plausible things to want in the long run, and they fit
     in nicely with the existing ref-selection scheme of rev-list. They
     do make your call from check_connected() slightly longer, but it is
     pretty negligible. It's "--exclude-hidden=receive --all" instead of
     "--visible-refs=hidden".

So looking at the patch itself, if you wanted to take my suggestions:

> +--visible-refs=[transfer|receive|uploadpack]::
> +	Pretend as if all the refs that have not been hidden via either one of
> +	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` are
> +	listed on the command line.

This would drop "transfer" as a mode, and explain that the argument is
"hide the refs that receive-pack would use", etc.

Likewise, the name would switch and pick up explanation similar to
--exclude below about how it affects the next --all, etc.

> @@ -1542,11 +1545,13 @@ static int handle_one_ref(const char *path, const struct object_id *oid,
>  			  int flag UNUSED,
>  			  void *cb_data)
>  {
> +	const char *stripped_path = strip_namespace(path);
>  	struct all_refs_cb *cb = cb_data;
>  	struct object *object;
>  
> -	if (ref_excluded(cb->all_revs->ref_excludes, path))
> -	    return 0;
> +	if (ref_excluded(cb->all_revs->ref_excludes, path) ||
> +	    ref_is_hidden(stripped_path, path, &cb->hidden_refs))
> +		return 0;

This would stay the same. We'd still exclude hidden refs during the
iteration.

> @@ -2759,6 +2772,21 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
>  		parse_list_objects_filter(&revs->filter, arg);
>  	} else if (!strcmp(arg, ("--no-filter"))) {
>  		list_objects_filter_set_no_filter(&revs->filter);
> +	} else if (skip_prefix(arg, "--visible-refs=", &arg)) {
> +		struct all_refs_cb cb;
> +
> +		if (strcmp(arg, "transfer") && strcmp(arg, "receive") &&
> +		    strcmp(arg, "uploadpack"))
> +			die(_("unsupported section for --visible-refs: %s"), arg);
> +
> +		init_all_refs_cb(&cb, revs, *flags);
> +		cb.hidden_refs_section = arg;
> +		git_config(hide_refs_config, &cb);
> +
> +		refs_for_each_ref(refs, handle_one_ref, &cb);
> +
> +		string_list_clear(&cb.hidden_refs, 1);
> +		clear_ref_exclusion(&revs->ref_excludes);

And here we'd do the same git_config() call, but drop the
refs_for_each_ref() call. We'd clear the hidden_refs field in all the
places that call clear_ref_exclusion() now.

In fact, you could argue that all of this should just be folded into
clear_ref_exclusion() and ref_excluded(), since from the perspective of
the iterating code, they are all part of the same feature. I don't mind
leaving it separate from the perspective of rev-list, though I think
if you did so, it would all Just Work for "rev-parse", too (I doubt
anybody cares in practice, but it's probably better to keep it
consistent).

-Peff

Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 5567 bytes --]

On Sat, Nov 05, 2022 at 08:46:05AM -0400, Jeff King wrote:
> On Thu, Nov 03, 2022 at 03:37:32PM +0100, Patrick Steinhardt wrote:
> 
> > Users can optionally hide refs from remote users in git-upload-pack(1),
> > git-receive-pack(1) and others via the `transfer.hideRefs`, but there is
> > not an easy way to obtain the list of all visible or hidden refs right
> > now. We'll require just that though for a performance improvement in our
> > connectivity check.
> > 
> > Add a new pseudo-ref `--visible-refs=` that pretends as if all refs have
> > been added to the command line that are not hidden. The pseudo-ref
> > requiers either one of "transfer", "uploadpack" or "receive" as argument
> > to pay attention to `transfer.hideRefs`, `uploadpack.hideRefs` or
> > `receive.hideRefs`, respectively.
> 
> Thanks for re-working this. I think it's a sensible path forward for the
> problem you're facing.
> 
> There were two parts of the implementation that surprised me a bit.
> These might just be nits, but because this is a new user-facing plumbing
> option that will be hard to change later, we should make sure it fits in
> with the adjacent features.
> 
> The two things I saw were:
> 
>   1. The mutual-exclusion selection between "transfer", "uploadpack",
>      and "receive" is not how those options work in their respective
>      programs. The "transfer.hideRefs" variable is respected for either
>      program. So whichever program you are running, it will always look
>      at both "transfer" and itself ("uploadpack" or "receive"). Another
>      way to think of it is that the "section" argument to
>      parse_hide_refs_config() is not a config section so much as a
>      profile. And the profiles are:
> 
>        - uploadpack: respect transfer.hideRefs and uploadpack.hideRefs
>        - receive: respect transfer.hideRefs and receive.hideRefs
> 
>      So it does not make sense to ask for "transfer" as a section; each
>      of the modes is already looking at transfer.hideRefs.
> 
>      In theory if this option was "look just at $section.hideRefs", it
>      could be more flexible to separate out the two. But that makes it
>      more of a pain to use (for normal use, you have to specify both
>      "transfer" and "receive"). And that is not what your patch
>      implements anyway; because it relies on parse_hide_refs_config(),
>      it is always adding in "transfer" under the hood (which is why your
>      final patch is correct to just say "--visible-refs=receive" without
>      specifying "transfer" as well).

Yup, I'm aware of this. And as you say, the current implementation
already handles this alright for both `receive` and `uploadpack` as we
rely on `parse_hide_refs_config()`, which knows to look at both
`transfer.hideRefs` and `$section.hideRefs`. But I don't see a reason
why we shouldn't allow users to ask "What is the set of hidden refs that
are shared by `uploadpack` and `receive`?", which is exactly what
`--visible-refs=transfer` does.

The implementation is not really explicit about this as we cheat a
little bit here by passing "transfer" as a section to the parsing
function. So what it does right now is to basically check for the same
section twice, once via the hard-coded "transfer.hideRefs" and once for
the "$section.hideRefs" with `section == "transfer"`. But I didn't see
much of a point in making this more explicit.

I might update the commit message and/or documentation though to point
this out.

>   2. Your "--visible-refs" implies "--all", because it's really "all
>      refs minus the hidden ones". That's convenient for the intended
>      caller, but not as flexible as it could be. If it were instead
>      treated the way "--exclude" is, as a modifier for the next
>      iteration option, then you do a few extra things:
> 
>        a. Combine multiple exclusions in a single iteration:
> 
>             git rev-list --exclude-hidden=receive \
> 	                 --exclude-hidden=upload \
> 			 --all
> 
>           That excludes both types in a single iteration. Whereas if you
> 	  did:
> 
> 	    git rev-list --visible-refs=receive \
> 	                 --visible-refs=upload
> 
> 	  that will do _two_ iterations, and end up with the union of
> 	  the sets. Equivalent to:
> 
> 	    git rev-list --exclude-hidden=receive --all \
> 	                 --exclude-hidden=upload  --all
> 
>        b. Do exclusions on smaller sets than --all:
> 
>             git rev-list --exclude-hidden=receive \
> 	                 --branches
> 
> 	  which would show just the branches that we'd advertise.
> 
>      Now I don't have a particular use case for either of those things.
>      But they're plausible things to want in the long run, and they fit
>      in nicely with the existing ref-selection scheme of rev-list. They
>      do make your call from check_connected() slightly longer, but it is
>      pretty negligible. It's "--exclude-hidden=receive --all" instead of
>      "--visible-refs=hidden".

Fair enough. I guess that the usecase where you e.g. only hide a subset
of branches via `hideRefs` is going to be rare, so in most cases you
don't gain much by modelling this so that you can `--exclude-hidden
--branches`. But as you rightfully point out, modelling it that way fits
neatly with the existing `--exclude` switch and is overall more
flexible. So there's not much of a reason to not do so.

I'll send a v3 and include your suggestion, thanks.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs

[Jeff King]

On Mon, Nov 07, 2022 at 09:20:08AM +0100, Patrick Steinhardt wrote:

> >   1. The mutual-exclusion selection between "transfer", "uploadpack",
> >      and "receive" is not how those options work in their respective
> > [...]
> 
> Yup, I'm aware of this. And as you say, the current implementation
> already handles this alright for both `receive` and `uploadpack` as we
> rely on `parse_hide_refs_config()`, which knows to look at both
> `transfer.hideRefs` and `$section.hideRefs`. But I don't see a reason
> why we shouldn't allow users to ask "What is the set of hidden refs that
> are shared by `uploadpack` and `receive`?", which is exactly what
> `--visible-refs=transfer` does.

OK. I don't have a real problem with having that mode, as long as it is
documented accurately. I do think, though, that it's not that likely to
be useful on its own (even less so than the hypotheticals I gave! ;) ),
and it would be easy to add later in a backwards-compatible way. So my
instinct would be to leave it out for now, but I don't think it's
hurting too much as-is (again, with a correct explanation in the
documentation).

> The implementation is not really explicit about this as we cheat a
> little bit here by passing "transfer" as a section to the parsing
> function. So what it does right now is to basically check for the same
> section twice, once via the hard-coded "transfer.hideRefs" and once for
> the "$section.hideRefs" with `section == "transfer"`. But I didn't see
> much of a point in making this more explicit.

Yeah, I agree the implementation works OK here. It does a duplicate
string-comparison for the section, but the important thing is that it
doesn't add each entry to the list twice.

> >      Now I don't have a particular use case for either of those things.
> >      But they're plausible things to want in the long run, and they fit
> >      in nicely with the existing ref-selection scheme of rev-list. They
> >      do make your call from check_connected() slightly longer, but it is
> >      pretty negligible. It's "--exclude-hidden=receive --all" instead of
> >      "--visible-refs=hidden".
> 
> Fair enough. I guess that the usecase where you e.g. only hide a subset
> of branches via `hideRefs` is going to be rare, so in most cases you
> don't gain much by modelling this so that you can `--exclude-hidden
> --branches`. But as you rightfully point out, modelling it that way fits
> neatly with the existing `--exclude` switch and is overall more
> flexible. So there's not much of a reason to not do so.

Thanks. I agree these aren't incredibly likely cases to come up in
practice. But unlike the "transfer" thing, it would be very hard to
switch techniques later without adding an awkward almost-the-same
option.

-Peff

Re: [PATCH v2 2/3] revision: add new parameter to specify all visible refs

[Jeff King]

On Thu, Nov 03, 2022 at 03:37:32PM +0100, Patrick Steinhardt wrote:

> diff --git a/t/t6021-rev-list-visible-refs.sh b/t/t6021-rev-list-visible-refs.sh
> new file mode 100755
> index 0000000000..9e12384dcf

Oh, I forgot to mention in my earlier response: these tests mostly look
good to me, but it may be worth having a few that cover the namespaces
feature. I _think_ your code is doing the right thing (because it calls
strip_namespace() as appropriate), but I have to admit I am generally
confused about how the namespace feature works in the first place. So
it's probably good to figure out how it is supposed to interact here and
make sure it is covered by tests.

-Peff

[PATCH v2 3/3] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4722 bytes --]

When serving a push, git-receive-pack(1) needs to verify that the
packfile sent by the client contains all objects that are required by
the updated references. This connectivity check works by marking all
preexisting references as uninteresting and using the new reference tips
as starting point for a graph walk.

Marking all preexisting references as uninteresting can be a problem
when it comes to performance. Git forges tend to do internal bookkeeping
to keep alive sets of objects for internal use or make them easy to find
via certain references. These references are typically hidden away from
the user so that they are neither advertised nor writeable. At GitLab,
we have one particular repository that contains a total of 7 million
references, of which 6.8 million are indeed internal references. With
the current connectivity check we are forced to load all these
references in order to mark them as uninteresting, and this alone takes
around 15 seconds to compute.

We can optimize this by only taking into account the set of visible refs
when marking objects as uninteresting. This means that we may now walk
more objects until we hit any object that is marked as uninteresting.
But it is rather unlikely that clients send objects that make large
parts of objects reachable that have previously only ever been hidden,
whereas the common case is to push incremental changes that build on top
of the visible object graph.

This provides a huge boost to performance in the mentioned repository,
where the vast majority of its refs hidden. Pushing a new commit into
this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
as it is configured in Gitaly leads to an almost 4.5-fold speedup:

    Benchmark 1: main
      Time (mean ± σ):     29.475 s ±  0.248 s    [User: 28.812 s, System: 1.006 s]
      Range (min … max):   29.189 s … 29.636 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):      6.657 s ±  0.027 s    [User: 6.664 s, System: 0.355 s]
      Range (min … max):    6.629 s …  6.682 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        4.43 ± 0.04 times faster than 'main'

As we mostly go through the same codepaths even in the case where there
are no hidden refs at all compared to the code before there is no change
in performance when no refs are hidden:

    Benchmark 1: main
      Time (mean ± σ):     48.688 s ±  1.535 s    [User: 49.579 s, System: 5.058 s]
      Range (min … max):   47.518 s … 50.425 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):     48.147 s ±  0.716 s    [User: 48.800 s, System: 5.297 s]
      Range (min … max):   47.694 s … 48.973 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        1.01 ± 0.04 times faster than 'main'

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c | 2 ++
 connected.c            | 5 ++++-
 connected.h            | 6 ++++++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 1f3efc58fb..be1c1d2702 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,
 	opt.err_fd = err_fd;
 	opt.progress = err_fd && !quiet;
 	opt.env = tmp_objdir_env(tmp_objdir);
+	opt.visible_refs_section = "receive";
+
 	if (check_connected(iterate_receive_command_list, &data, &opt))
 		set_connectivity_errors(commands, si);
 
diff --git a/connected.c b/connected.c
index 74a20cb32e..c64501f755 100644
--- a/connected.c
+++ b/connected.c
@@ -100,7 +100,10 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 		strvec_push(&rev_list.args, "--exclude-promisor-objects");
 	if (!opt->is_deepening_fetch) {
 		strvec_push(&rev_list.args, "--not");
-		strvec_push(&rev_list.args, "--all");
+		if (opt->visible_refs_section)
+			strvec_pushf(&rev_list.args, "--visible-refs=%s", opt->visible_refs_section);
+		else
+			strvec_push(&rev_list.args, "--all");
 	}
 	strvec_push(&rev_list.args, "--quiet");
 	strvec_push(&rev_list.args, "--alternate-refs");
diff --git a/connected.h b/connected.h
index 6e59c92aa3..d8396e5d55 100644
--- a/connected.h
+++ b/connected.h
@@ -46,6 +46,12 @@ struct check_connected_options {
 	 * during a fetch.
 	 */
 	unsigned is_deepening_fetch : 1;
+
+	/*
+	 * If not NULL, use `--visible-refs=$section` instead of `--not --all`
+	 * as the set of already-reachable references.
+	 */
+	const char *visible_refs_section;
 };
 
 #define CHECK_CONNECTED_INIT { 0 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 0/3] receive-pack: only use visible refs for connectivity check

[Taylor Blau]

On Thu, Nov 03, 2022 at 03:37:25PM +0100, Patrick Steinhardt wrote:
> Hi,
>
> this is the second version of my patch series that tries to improve
> performance of the connectivity check by only considering preexisting
> refs as uninteresting that could actually have been advertised to the
> client.

This version was delightful to read, and I don't have any concerns with
the approach or implementation.

I would appreciate another set of reviewer eyes on the topic, but if
not, I am comfortable starting to merge this down as-is.

Thanks,
Taylor

Re: [PATCH v2 0/3] receive-pack: only use visible refs for connectivity check

[Jeff King]

On Fri, Nov 04, 2022 at 08:40:35PM -0400, Taylor Blau wrote:

> On Thu, Nov 03, 2022 at 03:37:25PM +0100, Patrick Steinhardt wrote:
> > Hi,
> >
> > this is the second version of my patch series that tries to improve
> > performance of the connectivity check by only considering preexisting
> > refs as uninteresting that could actually have been advertised to the
> > client.
> 
> This version was delightful to read, and I don't have any concerns with
> the approach or implementation.
> 
> I would appreciate another set of reviewer eyes on the topic, but if
> not, I am comfortable starting to merge this down as-is.

I like the overall direction, but I had some suggestions for the
interface in patch 2. Hopefully Patrick agrees, and we'll see a v2.

-Peff

Re: [PATCH v2 0/3] receive-pack: only use visible refs for connectivity check

[Jeff King]

On Thu, Nov 03, 2022 at 03:37:25PM +0100, Patrick Steinhardt wrote:

> Overall the performance improvement isn't quite as strong as before:
> we're only 4.5x faster compared to 6.5x in our repo. But I guess that's
> still a good-enough improvement, doubly so that there are no downsides
> for repos anymore that ain't got any hidden refs.

Just a guess, but the extra time is probably spent because "rev-list"
has to iterate over your kajillion refs saying "nope, not this one".

I wonder if there is a way to carve up the ref namespace early, in a
trie-like way, similar to how for_each_fullref_in_prefixes() skips
straight to the interesting part. It's a harder problem, because we are
excluding rather than including entries based on our patterns. But it
seems like you'd be able to notice that we are on "refs/foo/bar", that
all of "refs/foo" is excluded, and jump ahead to the end of the
"refs/foo" section by binary-searching for the end within the
packed-refs file.

And that would speed up both this rev-list, but also the initial
advertisement that receive-pack does, because it's also iterating all of
these saying "nope, it's hidden". And assuming these are all internal
refs you hide from users, upload-pack would benefit, too.

I don't think that should be part of this series, but it may be an
interesting future direction.

-Peff

[PATCH v3 0/6] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]

Hi,

this is the second version of my patch series that tries to improve
performance of the connectivity check by only considering preexisting
refs as uninteresting that could actually have been advertised to the
client.

In v2 of this series, we introduced a new option `--visible-refs=` that
mostly acted as if `--all` was given, but its output was restricted to
only visible refs. As Peff rightly points out though, this is less
flexible than it needs to be. This new version instead introduces a new
option `--exclude-hidden=` that can be combined with `--all`, `--glob`,
`--branches` and so on to provide a more flexible interface.

The patch series is structured as following:

    - Patch 1-3 refactor multiple different parts of refs.c and
      revision.c so that they are more readily reusable.

    - Patch 4-5 implement `--exclude-hidden=` for git-rev-list(1) and
      git-rev-parse(1).

    - Patch 6 starts using that option in the connectivity check.

Patrick


Patrick Steinhardt (6):
  refs: get rid of global list of hidden refs
  revision: move together exclusion-related functions
  revision: introduce struct to handle exclusions
  revision: add new parameter to exclude hidden refs
  revparse: add `--exclude-hidden=` option
  receive-pack: only use visible refs for connectivity check

 Documentation/git-rev-parse.txt    |   7 ++
 Documentation/rev-list-options.txt |   7 ++
 builtin/receive-pack.c             |  10 +-
 builtin/rev-list.c                 |   1 +
 builtin/rev-parse.c                |  12 ++-
 connected.c                        |   3 +
 connected.h                        |   7 ++
 ls-refs.c                          |  13 ++-
 refs.c                             |  14 +--
 refs.h                             |   5 +-
 revision.c                         | 118 +++++++++++++--------
 revision.h                         |  29 ++++--
 t/t6018-rev-list-glob.sh           |   8 ++
 t/t6021-rev-list-exclude-hidden.sh | 159 +++++++++++++++++++++++++++++
 upload-pack.c                      |  30 +++---
 15 files changed, 341 insertions(+), 82 deletions(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 1/6] refs: get rid of global list of hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 12498 bytes --]

We're about to add a new argument to git-rev-list(1) that allows it to
add all references that are visible when taking `transfer.hideRefs` et
al into account. This will require us to potentially parse multiple sets
of hidden refs, which is not easily possible right now as there is only
a single, global instance of the list of parsed hidden refs.

Refactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both
take the list of hidden references as input and adjust callers to keep a
local list, instead. This allows us to easily use multiple hidden-ref
lists. Furthermore, it allows us to properly free this list before we
exit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c |  8 +++++---
 ls-refs.c              | 13 +++++++++----
 refs.c                 | 14 ++++----------
 refs.h                 |  5 +++--
 upload-pack.c          | 30 ++++++++++++++++++------------
 5 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 44bcea3a5b..1f3efc58fb 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)
 
 static int receive_pack_config(const char *var, const char *value, void *cb)
 {
-	int status = parse_hide_refs_config(var, value, "receive");
+	int status = parse_hide_refs_config(var, value, "receive", &hidden_refs);
 
 	if (status)
 		return status;
@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,
 	struct oidset *seen = data;
 	const char *path = strip_namespace(path_full);
 
-	if (ref_is_hidden(path, path_full))
+	if (ref_is_hidden(path, path_full, &hidden_refs))
 		return 0;
 
 	/*
@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)
 		strbuf_setlen(&refname_full, prefix_len);
 		strbuf_addstr(&refname_full, cmd->ref_name);
 
-		if (!ref_is_hidden(cmd->ref_name, refname_full.buf))
+		if (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))
 			continue;
 		if (is_null_oid(&cmd->new_oid))
 			cmd->error_string = "deny deleting a hidden ref";
@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
+	string_list_clear(&hidden_refs, 1);
 	free((void *)push_cert_nonce);
 	return 0;
 }
diff --git a/ls-refs.c b/ls-refs.c
index fa0d01b47c..ae89f850e9 100644
--- a/ls-refs.c
+++ b/ls-refs.c
@@ -6,6 +6,7 @@
 #include "ls-refs.h"
 #include "pkt-line.h"
 #include "config.h"
+#include "string-list.h"
 
 static int config_read;
 static int advertise_unborn;
@@ -73,6 +74,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
+	struct string_list hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 
 	strbuf_reset(&data->buf);
 
-	if (ref_is_hidden(refname_nons, refname))
+	if (ref_is_hidden(refname_nons, refname, &data->hidden_refs))
 		return 0;
 
 	if (!ref_match(&data->prefixes, refname_nons))
@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)
 }
 
 static int ls_refs_config(const char *var, const char *value,
-			  void *data UNUSED)
+			  void *cb_data)
 {
+	struct ls_refs_data *data = cb_data;
 	/*
 	 * We only serve fetches over v2 for now, so respect only "uploadpack"
 	 * config. This may need to eventually be expanded to "receive", but we
 	 * don't yet know how that information will be passed to ls-refs.
 	 */
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 int ls_refs(struct repository *r, struct packet_reader *request)
@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
+	string_list_init_dup(&data.hidden_refs);
 
 	ensure_config_read();
-	git_config(ls_refs_config, NULL);
+	git_config(ls_refs_config, &data);
 
 	while (packet_reader_read(request) == PACKET_READ_NORMAL) {
 		const char *arg = request->line;
@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
+	string_list_clear(&data.hidden_refs, 1);
 	return 0;
 }
 
diff --git a/refs.c b/refs.c
index 1491ae937e..f1711e2e9f 100644
--- a/refs.c
+++ b/refs.c
@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 					    refname, strict);
 }
 
-static struct string_list *hide_refs;
-
-int parse_hide_refs_config(const char *var, const char *value, const char *section)
+int parse_hide_refs_config(const char *var, const char *value, const char *section,
+			   struct string_list *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		if (!hide_refs) {
-			CALLOC_ARRAY(hide_refs, 1);
-			hide_refs->strdup_strings = 1;
-		}
 		string_list_append(hide_refs, ref);
 	}
 	return 0;
 }
 
-int ref_is_hidden(const char *refname, const char *refname_full)
+int ref_is_hidden(const char *refname, const char *refname_full,
+		  const struct string_list *hide_refs)
 {
 	int i;
 
-	if (!hide_refs)
-		return 0;
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
 		const char *match = hide_refs->items[i].string;
 		const char *subject;
diff --git a/refs.h b/refs.h
index 8958717a17..3266fd8f57 100644
--- a/refs.h
+++ b/refs.h
@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *new_oid, const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr);
 
-int parse_hide_refs_config(const char *var, const char *value, const char *);
+int parse_hide_refs_config(const char *var, const char *value, const char *,
+			   struct string_list *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *);
+int ref_is_hidden(const char *, const char *, const struct string_list *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
diff --git a/upload-pack.c b/upload-pack.c
index 0b8311bd68..9db17f8787 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -62,6 +62,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
+	struct string_list hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
+	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 	memset(data, 0, sizeof(*data));
 	data->symref = symref;
 	data->wanted_refs = wanted_refs;
+	data->hidden_refs = hidden_refs;
 	data->want_obj = want_obj;
 	data->have_obj = have_obj;
 	data->haves = haves;
@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
+	string_list_clear(&data->hidden_refs, 1);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * Checking for reachable shallows requires that our refs be
 		 * marked with OUR_REF.
 		 */
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, data);
+		for_each_namespaced_ref(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid)
+			const struct object_id *oid, const struct string_list *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
-	if (ref_is_hidden(refname, refname_full)) {
+	if (ref_is_hidden(refname, refname_full, hidden_refs)) {
 		o->flags |= HIDDEN_REF;
 		return 1;
 	}
@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,
 }
 
 static int check_ref(const char *refname_full, const struct object_id *oid,
-		     int flag UNUSED, void *cb_data UNUSED)
+		     int flag UNUSED, void *cb_data)
 {
 	const char *refname = strip_namespace(refname_full);
+	struct upload_pack_data *data = cb_data;
 
-	mark_our_ref(refname, refname_full, oid);
+	mark_our_ref(refname, refname_full, oid, &data->hidden_refs);
 	return 0;
 }
 
@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 	struct object_id peeled;
 	struct upload_pack_data *data = cb_data;
 
-	if (mark_our_ref(refname_nons, refname, oid))
+	if (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))
 		return 0;
 
 	if (capabilities) {
@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)
 	if (parse_object_filter_config(var, value, data) < 0)
 		return -1;
 
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)
@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		advertise_shallow_grafts(1);
 		packet_flush(1);
 	} else {
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, &data);
+		for_each_namespaced_ref(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
+			  struct string_list *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 		struct strbuf refname = STRBUF_INIT;
 
 		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
-		if (ref_is_hidden(refname_nons, refname.buf) ||
+		if (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||
 		    read_ref(refname.buf, &oid)) {
 			packet_writer_error(writer, "unknown ref %s", refname_nons);
 			die("unknown ref %s", refname_nons);
@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,
 			continue;
 		if (data->allow_ref_in_want &&
 		    parse_want_ref(&data->writer, arg, &data->wanted_refs,
-				   &data->want_obj))
+				   &data->hidden_refs, &data->want_obj))
 			continue;
 		/* process have line */
 		if (parse_have(arg, &data->haves))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 2/6] revision: move together exclusion-related functions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2402 bytes --]

Move together the definitions of functions that handle exclusions of
refs so that related functionality sits in a single place, only.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 revision.c | 52 ++++++++++++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/revision.c b/revision.c
index 0760e78936..be755670e2 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-struct all_refs_cb {
-	int all_flags;
-	int warned_bad_reflog;
-	struct rev_info *all_revs;
-	const char *name_for_errormsg;
-	struct worktree *wt;
-};
-
 int ref_excluded(struct string_list *ref_excludes, const char *path)
 {
 	struct string_list_item *item;
@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)
 	return 0;
 }
 
+void clear_ref_exclusion(struct string_list **ref_excludes_p)
+{
+	if (*ref_excludes_p) {
+		string_list_clear(*ref_excludes_p, 0);
+		free(*ref_excludes_p);
+	}
+	*ref_excludes_p = NULL;
+}
+
+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+{
+	if (!*ref_excludes_p) {
+		CALLOC_ARRAY(*ref_excludes_p, 1);
+		(*ref_excludes_p)->strdup_strings = 1;
+	}
+	string_list_append(*ref_excludes_p, exclude);
+}
+
+struct all_refs_cb {
+	int all_flags;
+	int warned_bad_reflog;
+	struct rev_info *all_revs;
+	const char *name_for_errormsg;
+	struct worktree *wt;
+};
+
 static int handle_one_ref(const char *path, const struct object_id *oid,
 			  int flag UNUSED,
 			  void *cb_data)
@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,
 	cb->wt = NULL;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
-{
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
-}
-
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
-{
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
-}
-
 static void handle_refs(struct ref_store *refs,
 			struct rev_info *revs, unsigned flags,
 			int (*for_each)(struct ref_store *, each_ref_fn, void *))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 3/6] revision: introduce struct to handle exclusions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8348 bytes --]

The functions that handle exclusion of refs work on a single string
list. We're about to add a second mechanism for excluding refs though,
and it makes sense to reuse much of the same architecture for both kinds
of exclusion.

Introduce a new `struct ref_exclusions` that encapsulates all the logic
related to excluding refs.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/rev-parse.c |  8 ++++----
 revision.c          | 47 ++++++++++++++++++++-------------------------
 revision.h          | 22 +++++++++++++++------
 3 files changed, 41 insertions(+), 36 deletions(-)

diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 8f61050bde..7fa5b6991b 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -39,7 +39,7 @@ static int abbrev_ref_strict;
 static int output_sq;
 
 static int stuck_long;
-static struct string_list *ref_excludes;
+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;
 
 /*
  * Some arguments are relevant "revision" arguments,
@@ -198,7 +198,7 @@ static int show_default(void)
 static int show_reference(const char *refname, const struct object_id *oid,
 			  int flag UNUSED, void *cb_data UNUSED)
 {
-	if (ref_excluded(ref_excludes, refname))
+	if (ref_excluded(&ref_excludes, refname))
 		return 0;
 	show_rev(NORMAL, oid, refname);
 	return 0;
@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)
 		for_each_glob_ref_in(show_reference, pattern, prefix, NULL);
 	else
 		for_each_ref_in(prefix, show_reference, NULL);
-	clear_ref_exclusion(&ref_excludes);
+	clear_ref_exclusions(&ref_excludes);
 }
 
 enum format_type {
@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 			}
 			if (!strcmp(arg, "--all")) {
 				for_each_ref(show_reference, NULL);
-				clear_ref_exclusion(&ref_excludes);
+				clear_ref_exclusions(&ref_excludes);
 				continue;
 			}
 			if (skip_prefix(arg, "--disambiguate=", &arg)) {
diff --git a/revision.c b/revision.c
index be755670e2..e5eaaa24ba 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,35 +1517,29 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-int ref_excluded(struct string_list *ref_excludes, const char *path)
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
 	struct string_list_item *item;
-
-	if (!ref_excludes)
-		return 0;
-	for_each_string_list_item(item, ref_excludes) {
+	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
 	return 0;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
+void init_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
+	string_list_init_dup(&exclusions->excluded_refs);
 }
 
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
+	string_list_clear(&exclusions->excluded_refs, 0);
+}
+
+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
+{
+	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
 struct all_refs_cb {
@@ -1563,7 +1557,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,
 	struct all_refs_cb *cb = cb_data;
 	struct object *object;
 
-	if (ref_excluded(cb->all_revs->ref_excludes, path))
+	if (ref_excluded(&cb->all_revs->ref_excludes, path))
 	    return 0;
 
 	object = get_reference(cb->all_revs, path, oid, cb->all_flags);
@@ -1901,6 +1895,7 @@ void repo_init_revisions(struct repository *r,
 
 	init_display_notes(&revs->notes_opt);
 	list_objects_filter_init(&revs->filter);
+	init_ref_exclusions(&revs->ref_excludes);
 }
 
 static void add_pending_commit_list(struct rev_info *revs,
@@ -2689,10 +2684,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 			init_all_refs_cb(&cb, revs, *flags);
 			other_head_refs(handle_one_ref, &cb);
 		}
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--branches")) {
 		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--bisect")) {
 		read_bisect_terms(&term_bad, &term_good);
 		handle_refs(refs, revs, *flags, for_each_bad_bisect_ref);
@@ -2701,15 +2696,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		revs->bisect = 1;
 	} else if (!strcmp(arg, "--tags")) {
 		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--remotes")) {
 		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref(handle_one_ref, optarg, &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 		return argcount;
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
@@ -2718,17 +2713,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--tags=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--reflog")) {
 		add_reflogs_to_pending(revs, *flags);
 	} else if (!strcmp(arg, "--indexed-objects")) {
diff --git a/revision.h b/revision.h
index afe1b77985..87d6824c55 100644
--- a/revision.h
+++ b/revision.h
@@ -81,6 +81,14 @@ struct rev_cmdline_info {
 	} *rev;
 };
 
+struct ref_exclusions {
+	/*
+	 * Excluded refs is a list of wildmatch patterns. If any of the
+	 * patterns matches, the reference will be excluded.
+	 */
+	struct string_list excluded_refs;
+};
+
 struct oidset;
 struct topo_walk_info;
 
@@ -103,7 +111,7 @@ struct rev_info {
 	struct list_objects_filter_options filter;
 
 	/* excluding from --branches, --refs, etc. expansion */
-	struct string_list *ref_excludes;
+	struct ref_exclusions ref_excludes;
 
 	/* Basic information */
 	const char *prefix;
@@ -439,12 +447,14 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)
 void show_object_with_name(FILE *, struct object *, const char *);
 
 /**
- * Helpers to check if a "struct string_list" item matches with
- * wildmatch().
+ * Helpers to check if a reference should be excluded.
  */
-int ref_excluded(struct string_list *, const char *path);
-void clear_ref_exclusion(struct string_list **);
-void add_ref_exclusion(struct string_list **, const char *exclude);
+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }
+
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
+void init_ref_exclusions(struct ref_exclusions *);
+void clear_ref_exclusions(struct ref_exclusions *);
+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
 
 /**
  * This function can be used if you want to add commit objects as revision
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 3/6] revision: introduce struct to handle exclusions

[Ævar Arnfjörð Bjarmason]

On Mon, Nov 07 2022, Patrick Steinhardt wrote:

> [[PGP Signed Part:Undecided]]
> The functions that handle exclusion of refs work on a single string
> list. We're about to add a second mechanism for excluding refs though,
> and it makes sense to reuse much of the same architecture for both kinds
> of exclusion.
>
> Introduce a new `struct ref_exclusions` that encapsulates all the logic
> related to excluding refs.

I think it's a good change, but probably worth mentioning tha we're
moving the "excluded_refs" from being malloc'd to the "struct
string_list" being embedded in this new "struct ref_exclusions".

That change isn't necessary for hoisting it into a container struct, but
does make things nicer down the line.

>  	struct string_list_item *item;
> -
> -	if (!ref_excludes)
> -		return 0;
> -	for_each_string_list_item(item, ref_excludes) {
> +	for_each_string_list_item(item, &exclusions->excluded_refs) {
>  		if (!wildmatch(item->string, path, 0))
>  			return 1;

E.g. because here we don't care about the distinction between NULL and
!list->nr anymore, it *does* matter in some cases, but it's always nice
to be able to clearly distinguish the cases where we don't, such as this
one....

> -void clear_ref_exclusion(struct string_list **ref_excludes_p)
> +void init_ref_exclusions(struct ref_exclusions *exclusions)
>  {
> -	if (*ref_excludes_p) {
> -		string_list_clear(*ref_excludes_p, 0);
> -		free(*ref_excludes_p);
> -	}
> -	*ref_excludes_p = NULL;

...and this becomes much nicer.

Aside: There's some churn, and this diff is worse for the
rename-while-at-it of "clear_ref_exclusion" to "add_ref_exclusion", but
that's probably worth it to have the macro match the struct name etc.

> +	string_list_init_dup(&exclusions->excluded_refs);

Okey, so this is partly my fault for not following up on f196c1e908d
(revisions API users: use release_revisions() needing REV_INFO_INIT,
2022-04-13) :); But here:

If we keep this *_init() function don't duplicate what you're adding to
the macro, just init this in terms of the macro. See the two-line
examples in:

	git grep -W memcpy.*blank

But here (and this is the part that's mostly me) as we don't malloc this
anymore you're only needing to keep this init function for
repo_init_revisions().

So, probably too big a digression for a "while at it", but FWIW this on
top of your topic would do:
	
	 revision.c | 10 ++--------
	 revision.h | 10 +++++++---
	 2 files changed, 9 insertions(+), 11 deletions(-)
	
	diff --git a/revision.c b/revision.c
	index 45652f9b0bb..cf352d1fa43 100644
	--- a/revision.c
	+++ b/revision.c
	@@ -1534,12 +1534,6 @@ int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
	 	return 0;
	 }
	 
	-void init_ref_exclusions(struct ref_exclusions *exclusions)
	-{
	-	string_list_init_dup(&exclusions->excluded_refs);
	-	string_list_init_dup(&exclusions->hidden_refs);
	-}
	-
	 void clear_ref_exclusions(struct ref_exclusions *exclusions)
	 {
	 	string_list_clear(&exclusions->excluded_refs, 0);
	@@ -1897,7 +1891,8 @@ void repo_init_revisions(struct repository *r,
	 			 struct rev_info *revs,
	 			 const char *prefix)
	 {
	-	memset(revs, 0, sizeof(*revs));
	+	struct rev_info blank = REV_INFO_INIT;
	+	memcpy(revs, &blank, sizeof(*revs));
	 
	 	revs->repo = r;
	 	revs->abbrev = DEFAULT_ABBREV;
	@@ -1933,7 +1928,6 @@ void repo_init_revisions(struct repository *r,
	 
	 	init_display_notes(&revs->notes_opt);
	 	list_objects_filter_init(&revs->filter);
	-	init_ref_exclusions(&revs->ref_excludes);
	 }
	 
	 static void add_pending_commit_list(struct rev_info *revs,
	diff --git a/revision.h b/revision.h
	index fef5e063d16..75b8ecc307b 100644
	--- a/revision.h
	+++ b/revision.h
	@@ -94,6 +94,10 @@ struct ref_exclusions {
	 	 */
	 	struct string_list hidden_refs;
	 };
	+#define REF_EXCLUSIONS_INIT { \
	+	.excluded_refs = STRING_LIST_INIT_DUP, \
	+	.hidden_refs = STRING_LIST_INIT_DUP, \
	+}
	 
	 struct oidset;
	 struct topo_walk_info;
	@@ -371,7 +375,9 @@ struct rev_info {
	  * called before release_revisions() the "struct rev_info" can be left
	  * uninitialized.
	  */
	-#define REV_INFO_INIT { 0 }
	+#define REV_INFO_INIT { \
	+	.ref_excludes = REF_EXCLUSIONS_INIT, \
	+}
	 
	 /**
	  * Initialize a rev_info structure with default values. The third parameter may
	@@ -455,10 +461,8 @@ void show_object_with_name(FILE *, struct object *, const char *);
	 /**
	  * Helpers to check if a reference should be excluded.
	  */
	-#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }
	 
	 int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
	-void init_ref_exclusions(struct ref_exclusions *);
	 void clear_ref_exclusions(struct ref_exclusions *);
	 void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
	 void exclude_hidden_refs(struct ref_exclusions *, const char *section);

But I'll submit that cleanup seperately, but for now let's not duplicate
your REF_EXCLUSIONS_INIT macro here in init_ref_exclusions(), just have
the function do what the macro is doing, now that we don't need the
malloc.

> -void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
> +void clear_ref_exclusions(struct ref_exclusions *exclusions)
>  {
> -	if (!*ref_excludes_p) {
> -		CALLOC_ARRAY(*ref_excludes_p, 1);
> -		(*ref_excludes_p)->strdup_strings = 1;
> -	}
> -	string_list_append(*ref_excludes_p, exclude);
> +	string_list_clear(&exclusions->excluded_refs, 0);

Also nicer.

>  static void add_pending_commit_list(struct rev_info *revs,
> @@ -2689,10 +2684,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
>  			init_all_refs_cb(&cb, revs, *flags);
>  			other_head_refs(handle_one_ref, &cb);
>  		}
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);
>  	} else if (!strcmp(arg, "--branches")) {
>  		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);
>  	} else if (!strcmp(arg, "--bisect")) {
>  		read_bisect_terms(&term_bad, &term_good);
>  		handle_refs(refs, revs, *flags, for_each_bad_bisect_ref);
> @@ -2701,15 +2696,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
>  		revs->bisect = 1;
>  	} else if (!strcmp(arg, "--tags")) {
>  		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);
>  	} else if (!strcmp(arg, "--remotes")) {
>  		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);
>  	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
>  		struct all_refs_cb cb;
>  		init_all_refs_cb(&cb, revs, *flags);
>  		for_each_glob_ref(handle_one_ref, optarg, &cb);
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);
>  		return argcount;
>  	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
>  		add_ref_exclusion(&revs->ref_excludes, optarg);
> @@ -2718,17 +2713,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
>  		struct all_refs_cb cb;
>  		init_all_refs_cb(&cb, revs, *flags);
>  		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);
>  	} else if (skip_prefix(arg, "--tags=", &optarg)) {
>  		struct all_refs_cb cb;
>  		init_all_refs_cb(&cb, revs, *flags);
>  		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);
>  	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
>  		struct all_refs_cb cb;
>  		init_all_refs_cb(&cb, revs, *flags);
>  		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
> -		clear_ref_exclusion(&revs->ref_excludes);
> +		clear_ref_exclusions(&revs->ref_excludes);

The churn I mentioned with the renaming, so maybe worth doing that as a
"prep" commit?

> +struct ref_exclusions {
> +	/*
> +	 * Excluded refs is a list of wildmatch patterns. If any of the
> +	 * patterns matches, the reference will be excluded.
> +	 */
> +	struct string_list excluded_refs;
> +};

Per the above POC diff though, please move...

>  struct oidset;
>  struct topo_walk_info;
>  
> @@ -103,7 +111,7 @@ struct rev_info {
>  	struct list_objects_filter_options filter;
>  
>  	/* excluding from --branches, --refs, etc. expansion */
> -	struct string_list *ref_excludes;
> +	struct ref_exclusions ref_excludes;
>  
>  	/* Basic information */
>  	const char *prefix;
> @@ -439,12 +447,14 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)
>  void show_object_with_name(FILE *, struct object *, const char *);
>  
>  /**
> - * Helpers to check if a "struct string_list" item matches with
> - * wildmatch().
> + * Helpers to check if a reference should be excluded.
>   */
> -int ref_excluded(struct string_list *, const char *path);
> -void clear_ref_exclusion(struct string_list **);
> -void add_ref_exclusion(struct string_list **, const char *exclude);
> +#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }

...this macro to right after declaring the struct, which is what we
usually do, and will help in adding it to "REV_INFO_INIT" sooner than
later.

Also, at the end of your series this end up being overly long, so per
the diff-above (which is tot he end of the series), let's start by
line-wrapping it:

	#define ..._INIT { \
        	.member = ..._INIT, \
	}

Re: [PATCH v3 3/6] revision: introduce struct to handle exclusions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8738 bytes --]

On Mon, Nov 07, 2022 at 01:51:51PM +0100, Ævar Arnfjörð Bjarmason wrote:
> 
> On Mon, Nov 07 2022, Patrick Steinhardt wrote:
[snip]
> > +	string_list_init_dup(&exclusions->excluded_refs);
> 
> Okey, so this is partly my fault for not following up on f196c1e908d
> (revisions API users: use release_revisions() needing REV_INFO_INIT,
> 2022-04-13) :); But here:
> 
> If we keep this *_init() function don't duplicate what you're adding to
> the macro, just init this in terms of the macro. See the two-line
> examples in:
> 
> 	git grep -W memcpy.*blank

Makes sense.

> But here (and this is the part that's mostly me) as we don't malloc this
> anymore you're only needing to keep this init function for
> repo_init_revisions().
> 
> So, probably too big a digression for a "while at it", but FWIW this on
> top of your topic would do:
> 	
> 	 revision.c | 10 ++--------
> 	 revision.h | 10 +++++++---
> 	 2 files changed, 9 insertions(+), 11 deletions(-)
> 	
> 	diff --git a/revision.c b/revision.c
> 	index 45652f9b0bb..cf352d1fa43 100644
> 	--- a/revision.c
> 	+++ b/revision.c
> 	@@ -1534,12 +1534,6 @@ int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
> 	 	return 0;
> 	 }
> 	 
> 	-void init_ref_exclusions(struct ref_exclusions *exclusions)
> 	-{
> 	-	string_list_init_dup(&exclusions->excluded_refs);
> 	-	string_list_init_dup(&exclusions->hidden_refs);
> 	-}
> 	-
> 	 void clear_ref_exclusions(struct ref_exclusions *exclusions)
> 	 {
> 	 	string_list_clear(&exclusions->excluded_refs, 0);
> 	@@ -1897,7 +1891,8 @@ void repo_init_revisions(struct repository *r,
> 	 			 struct rev_info *revs,
> 	 			 const char *prefix)
> 	 {
> 	-	memset(revs, 0, sizeof(*revs));
> 	+	struct rev_info blank = REV_INFO_INIT;
> 	+	memcpy(revs, &blank, sizeof(*revs));
> 	 
> 	 	revs->repo = r;
> 	 	revs->abbrev = DEFAULT_ABBREV;
> 	@@ -1933,7 +1928,6 @@ void repo_init_revisions(struct repository *r,
> 	 
> 	 	init_display_notes(&revs->notes_opt);
> 	 	list_objects_filter_init(&revs->filter);
> 	-	init_ref_exclusions(&revs->ref_excludes);
> 	 }
> 	 
> 	 static void add_pending_commit_list(struct rev_info *revs,
> 	diff --git a/revision.h b/revision.h
> 	index fef5e063d16..75b8ecc307b 100644
> 	--- a/revision.h
> 	+++ b/revision.h
> 	@@ -94,6 +94,10 @@ struct ref_exclusions {
> 	 	 */
> 	 	struct string_list hidden_refs;
> 	 };
> 	+#define REF_EXCLUSIONS_INIT { \
> 	+	.excluded_refs = STRING_LIST_INIT_DUP, \
> 	+	.hidden_refs = STRING_LIST_INIT_DUP, \
> 	+}
> 	 
> 	 struct oidset;
> 	 struct topo_walk_info;
> 	@@ -371,7 +375,9 @@ struct rev_info {
> 	  * called before release_revisions() the "struct rev_info" can be left
> 	  * uninitialized.
> 	  */
> 	-#define REV_INFO_INIT { 0 }
> 	+#define REV_INFO_INIT { \
> 	+	.ref_excludes = REF_EXCLUSIONS_INIT, \
> 	+}
> 	 
> 	 /**
> 	  * Initialize a rev_info structure with default values. The third parameter may
> 	@@ -455,10 +461,8 @@ void show_object_with_name(FILE *, struct object *, const char *);
> 	 /**
> 	  * Helpers to check if a reference should be excluded.
> 	  */
> 	-#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }
> 	 
> 	 int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
> 	-void init_ref_exclusions(struct ref_exclusions *);
> 	 void clear_ref_exclusions(struct ref_exclusions *);
> 	 void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
> 	 void exclude_hidden_refs(struct ref_exclusions *, const char *section);
> 
> But I'll submit that cleanup seperately, but for now let's not duplicate
> your REF_EXCLUSIONS_INIT macro here in init_ref_exclusions(), just have
> the function do what the macro is doing, now that we don't need the
> malloc.

Great, thanks.

[snip]
> >  static void add_pending_commit_list(struct rev_info *revs,
> > @@ -2689,10 +2684,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
> >  			init_all_refs_cb(&cb, revs, *flags);
> >  			other_head_refs(handle_one_ref, &cb);
> >  		}
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> >  	} else if (!strcmp(arg, "--branches")) {
> >  		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> >  	} else if (!strcmp(arg, "--bisect")) {
> >  		read_bisect_terms(&term_bad, &term_good);
> >  		handle_refs(refs, revs, *flags, for_each_bad_bisect_ref);
> > @@ -2701,15 +2696,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
> >  		revs->bisect = 1;
> >  	} else if (!strcmp(arg, "--tags")) {
> >  		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> >  	} else if (!strcmp(arg, "--remotes")) {
> >  		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> >  	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
> >  		struct all_refs_cb cb;
> >  		init_all_refs_cb(&cb, revs, *flags);
> >  		for_each_glob_ref(handle_one_ref, optarg, &cb);
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> >  		return argcount;
> >  	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
> >  		add_ref_exclusion(&revs->ref_excludes, optarg);
> > @@ -2718,17 +2713,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
> >  		struct all_refs_cb cb;
> >  		init_all_refs_cb(&cb, revs, *flags);
> >  		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> >  	} else if (skip_prefix(arg, "--tags=", &optarg)) {
> >  		struct all_refs_cb cb;
> >  		init_all_refs_cb(&cb, revs, *flags);
> >  		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> >  	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
> >  		struct all_refs_cb cb;
> >  		init_all_refs_cb(&cb, revs, *flags);
> >  		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
> > -		clear_ref_exclusion(&revs->ref_excludes);
> > +		clear_ref_exclusions(&revs->ref_excludes);
> 
> The churn I mentioned with the renaming, so maybe worth doing that as a
> "prep" commit?

Hm. I don't think it's too bad, and it's weird to rename things already
without any clear justification why that's visible from the diff. Like,
there is no `struct ref_exclusions` yet, why rename it?

I'll retain this in a single commit if you don't mind, but amend the
commit message to explicitly mention the rename.

> > +struct ref_exclusions {
> > +	/*
> > +	 * Excluded refs is a list of wildmatch patterns. If any of the
> > +	 * patterns matches, the reference will be excluded.
> > +	 */
> > +	struct string_list excluded_refs;
> > +};
> 
> Per the above POC diff though, please move...
> 
> >  struct oidset;
> >  struct topo_walk_info;
> >  
> > @@ -103,7 +111,7 @@ struct rev_info {
> >  	struct list_objects_filter_options filter;
> >  
> >  	/* excluding from --branches, --refs, etc. expansion */
> > -	struct string_list *ref_excludes;
> > +	struct ref_exclusions ref_excludes;
> >  
> >  	/* Basic information */
> >  	const char *prefix;
> > @@ -439,12 +447,14 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)
> >  void show_object_with_name(FILE *, struct object *, const char *);
> >  
> >  /**
> > - * Helpers to check if a "struct string_list" item matches with
> > - * wildmatch().
> > + * Helpers to check if a reference should be excluded.
> >   */
> > -int ref_excluded(struct string_list *, const char *path);
> > -void clear_ref_exclusion(struct string_list **);
> > -void add_ref_exclusion(struct string_list **, const char *exclude);
> > +#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }
> 
> ...this macro to right after declaring the struct, which is what we
> usually do, and will help in adding it to "REV_INFO_INIT" sooner than
> later.

Fair, will change.

> Also, at the end of your series this end up being overly long, so per
> the diff-above (which is tot he end of the series), let's start by
> line-wrapping it:
> 
> 	#define ..._INIT { \
>         	.member = ..._INIT, \
> 	}

Makes sense.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 11516 bytes --]

Users can optionally hide refs from remote users in git-upload-pack(1),
git-receive-pack(1) and others via the `transfer.hideRefs`, but there is
not an easy way to obtain the list of all visible or hidden refs right
now. We'll require just that though for a performance improvement in our
connectivity check.

Add a new option `--exclude-hidden=` that excludes any hidden refs from
the next pseudo-ref like `--all` or `--branches`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/rev-list-options.txt |   7 ++
 builtin/rev-list.c                 |   1 +
 revision.c                         |  43 +++++++-
 revision.h                         |   9 +-
 t/t6021-rev-list-exclude-hidden.sh | 159 +++++++++++++++++++++++++++++
 5 files changed, 217 insertions(+), 2 deletions(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
index 1837509566..a178956613 100644
--- a/Documentation/rev-list-options.txt
+++ b/Documentation/rev-list-options.txt
@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[transfer|receive|uploadpack]::
+	Do not include refs that have been hidden via either one of
+	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that
+	the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would
+	otherwise consider.  This option is cleared when seeing one of these
+	pseudo-refs.
+
 --reflog::
 	Pretend as if all objects mentioned by reflogs are listed on the
 	command line as `<commit>`.
diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index 3acd93f71e..9eace06385 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -38,6 +38,7 @@ static const char rev_list_usage[] =
 "    --tags\n"
 "    --remotes\n"
 "    --stdin\n"
+"    --exclude-hidden=[transfer|receive|uploadpack]\n"
 "    --quiet\n"
 "  ordering output:\n"
 "    --topo-order\n"
diff --git a/revision.c b/revision.c
index e5eaaa24ba..45652f9b0b 100644
--- a/revision.c
+++ b/revision.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "config.h"
 #include "object-store.h"
 #include "tag.h"
 #include "blob.h"
@@ -1519,22 +1520,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
+	const char *stripped_path = strip_namespace(path);
 	struct string_list_item *item;
+
 	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
+
+	if (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))
+		return 1;
+
 	return 0;
 }
 
 void init_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_init_dup(&exclusions->excluded_refs);
+	string_list_init_dup(&exclusions->hidden_refs);
 }
 
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
+	string_list_clear(&exclusions->hidden_refs, 1);
 }
 
 void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
@@ -1542,6 +1551,35 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
 	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
+struct exclude_hidden_refs_cb {
+	struct ref_exclusions *exclusions;
+	const char *section;
+};
+
+static int hide_refs_config(const char *var, const char *value, void *cb_data)
+{
+	struct exclude_hidden_refs_cb *cb = cb_data;
+	return parse_hide_refs_config(var, value, cb->section,
+				      &cb->exclusions->hidden_refs);
+}
+
+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)
+{
+	struct exclude_hidden_refs_cb cb;
+
+	if (strcmp(section, "transfer") && strcmp(section, "receive") &&
+	    strcmp(section, "uploadpack"))
+		die(_("unsupported section for hidden refs: %s"), section);
+
+	if (exclusions->hidden_refs.nr)
+		die(_("--exclude-hidden= passed more than once"));
+
+	cb.exclusions = exclusions;
+	cb.section = section;
+
+	git_config(hide_refs_config, &cb);
+}
+
 struct all_refs_cb {
 	int all_flags;
 	int warned_bad_reflog;
@@ -2220,7 +2258,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg
 	    !strcmp(arg, "--bisect") || starts_with(arg, "--glob=") ||
 	    !strcmp(arg, "--indexed-objects") ||
 	    !strcmp(arg, "--alternate-refs") ||
-	    starts_with(arg, "--exclude=") ||
+	    starts_with(arg, "--exclude=") || starts_with(arg, "--exclude-hidden=") ||
 	    starts_with(arg, "--branches=") || starts_with(arg, "--tags=") ||
 	    starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk="))
 	{
@@ -2709,6 +2747,9 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
 		return argcount;
+	} else if ((argcount = parse_long_opt("exclude-hidden", argv, &optarg))) {
+		exclude_hidden_refs(&revs->ref_excludes, optarg);
+		return argcount;
 	} else if (skip_prefix(arg, "--branches=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
diff --git a/revision.h b/revision.h
index 87d6824c55..fef5e063d1 100644
--- a/revision.h
+++ b/revision.h
@@ -87,6 +87,12 @@ struct ref_exclusions {
 	 * patterns matches, the reference will be excluded.
 	 */
 	struct string_list excluded_refs;
+
+	/*
+	 * Hidden refs is a list of patterns that is to be hidden via
+	 * `ref_is_hidden()`.
+	 */
+	struct string_list hidden_refs;
 };
 
 struct oidset;
@@ -449,12 +455,13 @@ void show_object_with_name(FILE *, struct object *, const char *);
 /**
  * Helpers to check if a reference should be excluded.
  */
-#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }
+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }
 
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
 void init_ref_exclusions(struct ref_exclusions *);
 void clear_ref_exclusions(struct ref_exclusions *);
 void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
+void exclude_hidden_refs(struct ref_exclusions *, const char *section);
 
 /**
  * This function can be used if you want to add commit objects as revision
diff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh
new file mode 100755
index 0000000000..d08fc2da93
--- /dev/null
+++ b/t/t6021-rev-list-exclude-hidden.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+
+test_description='git rev-list --exclude-hidden test'
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit_bulk --id=commit --ref=refs/heads/main 1 &&
+	COMMIT=$(git rev-parse refs/heads/main) &&
+	test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&
+	TAG=$(git rev-parse refs/tags/lightweight) &&
+	test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&
+	HIDDEN=$(git rev-parse refs/hidden/commit) &&
+	test_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&
+	NAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)
+'
+
+test_expect_success 'invalid section' '
+	echo "fatal: unsupported section for hidden refs: unsupported" >expected &&
+	test_must_fail git rev-list --exclude-hidden=unsupported 2>err &&
+	test_cmp expected err
+'
+
+test_expect_success 'passed multiple times' '
+	echo "fatal: --exclude-hidden= passed more than once" >expected &&
+	test_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --exclude-hidden=transfer 2>err &&
+	test_cmp expected err
+'
+
+test_expect_success '--exclude-hidden without hiddenRefs' '
+	git rev-list --exclude-hidden=transfer --all >out &&
+	cat >expected <<-EOF &&
+	$NAMESPACE
+	$HIDDEN
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success 'hidden via transfer.hideRefs' '
+	git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&
+	cat >expected <<-EOF &&
+	$NAMESPACE
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--all --exclude-hidden=transfer --not --all without hidden refs' '
+	git rev-list --all --exclude-hidden=transfer --not --all >out &&
+	test_must_be_empty out
+'
+
+test_expect_success '--all --exclude-hidden=transfer --not --all with hidden ref' '
+	git -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=transfer --not --all >out &&
+	cat >expected <<-EOF &&
+	$HIDDEN
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--exclude-hidden with --exclude' '
+	git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=transfer --all >out &&
+	cat >expected <<-EOF &&
+	$NAMESPACE
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--exclude-hidden is reset' '
+	git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=transfer --all --all >out &&
+	cat >expected <<-EOF &&
+	$NAMESPACE
+	$HIDDEN
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--exclude-hidden operates on stripped refs by default' '
+	GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=transfer --all >out &&
+	cat >expected <<-EOF &&
+	$HIDDEN
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--exclude-hidden does not hide namespace by default' '
+	GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&
+	cat >expected <<-EOF &&
+	$NAMESPACE
+	$HIDDEN
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+test_expect_success '--exclude-hidden= may operate on unstripped refs' '
+	GIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&
+	cat >expected <<-EOF &&
+	$HIDDEN
+	$TAG
+	$COMMIT
+	EOF
+	test_cmp expected out
+'
+
+for section in receive uploadpack
+do
+	test_expect_success "hidden via $section.hideRefs" '
+		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "--exclude-hidden=$section respects transfer.hideRefs" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "--exclude-hidden=transfer ignores $section.hideRefs" '
+		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "--exclude-hidden=$section respects both transfer.hideRefs and $section.hideRefs" '
+		git -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+done
+
+test_done
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Ævar Arnfjörð Bjarmason]

On Mon, Nov 07 2022, Patrick Steinhardt wrote:

> +--exclude-hidden=[transfer|receive|uploadpack]::
> +	Do not include refs that have been hidden via either one of
> +	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that

Maybe worth adding "(see linkgit:git-config[1]) after listing the config
variables.

>  int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
>  {
> +	const char *stripped_path = strip_namespace(path);
>  	struct string_list_item *item;
> +

nit: stray whitespace in otherwise "clean" commit, but the post-image looks nicer, so...

>  void init_ref_exclusions(struct ref_exclusions *exclusions)
>  {
>  	string_list_init_dup(&exclusions->excluded_refs);
> +	string_list_init_dup(&exclusions->hidden_refs);
>  }

Per my comment on 3/6 we wouldn't need this when using the macro as a
source of truth.

>  void clear_ref_exclusions(struct ref_exclusions *exclusions)
>  {
>  	string_list_clear(&exclusions->excluded_refs, 0);
> +	string_list_clear(&exclusions->hidden_refs, 1);
>  }

Hrm, I'l read on, but I don't see any use of "util" here at a glance,
should the "1" here be "0", or maybe I've just missed how it's used...

> +	if (strcmp(section, "transfer") && strcmp(section, "receive") &&
> +	    strcmp(section, "uploadpack"))
> +		die(_("unsupported section for hidden refs: %s"), section);
> +
> +	if (exclusions->hidden_refs.nr)
> +		die(_("--exclude-hidden= passed more than once"));

We usually just ignore the first of --foo=bar --foo=baz and take "baz"
in our CLI use. Is it better to die here than just clear the previous
one & continue?


> -#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }
> +#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }

...the getting overly long line I mentioned in 3/6...

> +TEST_PASSES_SANITIZE_LEAK=true

Thanks for adding this! :)

Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Ævar Arnfjörð Bjarmason]

On Mon, Nov 07 2022, Ævar Arnfjörð Bjarmason wrote:

> On Mon, Nov 07 2022, Patrick Steinhardt wrote:

>> +TEST_PASSES_SANITIZE_LEAK=true
>
> Thanks for adding this! :)

Hrm, I spoke too soon :) This series adds new leaks, so it'll fail with
the linux-leaks job. I have the following local monkeypatch on top,
which obviously doesn't address the root cause. The t6018 leak is new
due to the new tests you added.

diff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh
index f92616de12d..54221588dd0 100755
--- a/t/t6018-rev-list-glob.sh
+++ b/t/t6018-rev-list-glob.sh
@@ -5,7 +5,6 @@ test_description='rev-list/rev-parse --glob'
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
-TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 commit () {
diff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh
index d08fc2da93d..908b9dba611 100755
--- a/t/t6021-rev-list-exclude-hidden.sh
+++ b/t/t6021-rev-list-exclude-hidden.sh
@@ -2,7 +2,6 @@
 
 test_description='git rev-list --exclude-hidden test'
 
-TEST_PASSES_SANITIZE_LEAK=true
 . ./test-lib.sh
 
 test_expect_success 'setup' '

Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

On Mon, Nov 07, 2022 at 06:07:01PM +0100, Ævar Arnfjörð Bjarmason wrote:
> 
> On Mon, Nov 07 2022, Ævar Arnfjörð Bjarmason wrote:
> 
> > On Mon, Nov 07 2022, Patrick Steinhardt wrote:
> 
> >> +TEST_PASSES_SANITIZE_LEAK=true
> >
> > Thanks for adding this! :)
> 
> Hrm, I spoke too soon :) This series adds new leaks, so it'll fail with
> the linux-leaks job. I have the following local monkeypatch on top,
> which obviously doesn't address the root cause. The t6018 leak is new
> due to the new tests you added.

Right, I didn't know it was as easy to run tests with leak checking as
just executing `make test SANITIZE=leak`. Anyway, I did that now and the
issue is in fact in how the hidden refs are parsed because we already
`xstrdup()` the config value as we need to modify it anyway.

The following patch fixes the issue:

diff --git a/refs.c b/refs.c
index f1711e2e9f..2c7e88b190 100644
--- a/refs.c
+++ b/refs.c
@@ -1430,7 +1430,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		string_list_append(hide_refs, ref);
+		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1483 bytes --]

On Mon, Nov 07, 2022 at 02:34:45PM +0100, Ævar Arnfjörð Bjarmason wrote:
> On Mon, Nov 07 2022, Patrick Steinhardt wrote:
[snip]
> > +	if (strcmp(section, "transfer") && strcmp(section, "receive") &&
> > +	    strcmp(section, "uploadpack"))
> > +		die(_("unsupported section for hidden refs: %s"), section);
> > +
> > +	if (exclusions->hidden_refs.nr)
> > +		die(_("--exclude-hidden= passed more than once"));
> 
> We usually just ignore the first of --foo=bar --foo=baz and take "baz"
> in our CLI use. Is it better to die here than just clear the previous
> one & continue?

It's something I was torn on. I ultimately chose to die though because
of the difference between `--exclude` and `--exclude-hidden`: the former
one will happily add additional patterns, all of which will ultimately
be ignored. So as a user you might rightfully expect that the latter
will work the same: if both `--exclude-hidden=uploadpack` and
`--exclude-hidden=receive` are specified, you might want to have both be
ignored.

To me it wasn't quite clear how to support multiple instances of
`transfer.hideRefs` though as there is also the concept of un-hiding
already-hidden refs. So I wanted to avoid going into this discussion to
make the patch series a little bit smaller.

By dying instead of silently overriding the previous argument we retain
the ability to iterate on this at a later point though to implement
above behaviour, if the usecase ever arises.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Taylor Blau]

On Mon, Nov 07, 2022 at 01:16:35PM +0100, Patrick Steinhardt wrote:
> @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
>  or `--all`. If a trailing '/{asterisk}' is intended, it must be given
>  explicitly.
>
> +--exclude-hidden=[transfer|receive|uploadpack]::
> +	Do not include refs that have been hidden via either one of
> +	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that
> +	the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would
> +	otherwise consider.  This option is cleared when seeing one of these
> +	pseudo-refs.
> +

Hmm. I thought that part of the motivation behind this round was to drop
the 'transfer' group, since it's implied by '--exclude-hidden=receive
--exclude-hidden-uploadpack', no?

Thanks,
Taylor

Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1280 bytes --]

On Mon, Nov 07, 2022 at 07:57:29PM -0500, Taylor Blau wrote:
> On Mon, Nov 07, 2022 at 01:16:35PM +0100, Patrick Steinhardt wrote:
> > @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
> >  or `--all`. If a trailing '/{asterisk}' is intended, it must be given
> >  explicitly.
> >
> > +--exclude-hidden=[transfer|receive|uploadpack]::
> > +	Do not include refs that have been hidden via either one of
> > +	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that
> > +	the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would
> > +	otherwise consider.  This option is cleared when seeing one of these
> > +	pseudo-refs.
> > +
> 
> Hmm. I thought that part of the motivation behind this round was to drop
> the 'transfer' group, since it's implied by '--exclude-hidden=receive
> --exclude-hidden-uploadpack', no?
> 
> Thanks,
> Taylor

I didn't quite see the point in not providing the `transfer` group so
that users can ask for only the set of refs that are hidden by both
`uploadpack` and `receive`. But given that you're the second person
asking for it to be dropped now and given that I don't really have a
plausible usecase for this I'll drop it in the next version.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 4/6] revision: add new parameter to exclude hidden refs

[Jeff King]

On Tue, Nov 08, 2022 at 09:16:47AM +0100, Patrick Steinhardt wrote:

> > > +--exclude-hidden=[transfer|receive|uploadpack]::
> > > +	Do not include refs that have been hidden via either one of
> > > +	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that
> > > +	the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would
> > > +	otherwise consider.  This option is cleared when seeing one of these
> > > +	pseudo-refs.
> > > +
> > 
> > Hmm. I thought that part of the motivation behind this round was to drop
> > the 'transfer' group, since it's implied by '--exclude-hidden=receive
> > --exclude-hidden-uploadpack', no?
> > 
> > Thanks,
> > Taylor
> 
> I didn't quite see the point in not providing the `transfer` group so
> that users can ask for only the set of refs that are hidden by both
> `uploadpack` and `receive`. But given that you're the second person
> asking for it to be dropped now and given that I don't really have a
> plausible usecase for this I'll drop it in the next version.

Sorry, I'm a little slow on the review. I just left a message in
response to v2 saying that I'm OK with it _if_ it's explained. But the
explanation above still seems misleading. Saying "either one of" implies
that they are mutually exclusive, but "receive" is really pulling from
"receive.hideRefs" and "transfer.hideRefs".

I think you'd need to lay out the rules. But if we just drop "transfer"
then that is simpler still (but the explanation probably still ought to
become "refs hidden by receive-pack" and not mention receive.hideRefs
explicitly).

-Peff

[PATCH v3 5/6] revparse: add `--exclude-hidden=` option

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2873 bytes --]

Add a new `--exclude-hidden=` option that is similar to the one we just
added to git-rev-list(1). Given a seciton name `transfer`, `uploadpack`
or `receive` as argument, it causes us to exclude all references that
would be hidden by the respective `$seciton.hideRefs` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-rev-parse.txt | 7 +++++++
 builtin/rev-parse.c             | 4 ++++
 t/t6018-rev-list-glob.sh        | 8 ++++++++
 3 files changed, 19 insertions(+)

diff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt
index 6b8ca085aa..a016cb5abe 100644
--- a/Documentation/git-rev-parse.txt
+++ b/Documentation/git-rev-parse.txt
@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[transfer|receive|uploadpack]::
+	Do not include refs that have been hidden via either one of
+	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that
+	the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would
+	otherwise consider.  This option is cleared when seeing one of these
+	pseudo-refs.
+
 --disambiguate=<prefix>::
 	Show every object whose name begins with the given prefix.
 	The <prefix> must be at least 4 hexadecimal digits long to
diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 7fa5b6991b..49730c7a23 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -895,6 +895,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				add_ref_exclusion(&ref_excludes, arg);
 				continue;
 			}
+			if (skip_prefix(arg, "--exclude-hidden=", &arg)) {
+				exclude_hidden_refs(&ref_excludes, arg);
+				continue;
+			}
 			if (!strcmp(arg, "--show-toplevel")) {
 				const char *work_tree = get_git_work_tree();
 				if (work_tree)
diff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh
index e1abc5c2b3..f92616de12 100755
--- a/t/t6018-rev-list-glob.sh
+++ b/t/t6018-rev-list-glob.sh
@@ -187,6 +187,14 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '
 	compare rev-parse "--exclude=upstream/x --remotes=upstream/*" "upstream/one upstream/two"
 '
 
+test_expect_success 'rev-parse --exclude-hidden= with --all' '
+	compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--exclude-hidden=transfer --all" "--branches --tags"
+'
+
+test_expect_success 'rev-parse --exclude-hidden= with --all' '
+	compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude-hidden=transfer --all" "--exclude=refs/heads/subspace/* --all"
+'
+
 test_expect_success 'rev-list --exclude=glob with --branches=glob' '
 	compare rev-list "--exclude=subspace-* --branches=sub*" "subspace/one subspace/two"
 '
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 5/6] revparse: add `--exclude-hidden=` option

[Jeff King]

On Mon, Nov 07, 2022 at 01:16:39PM +0100, Patrick Steinhardt wrote:

> Add a new `--exclude-hidden=` option that is similar to the one we just
> added to git-rev-list(1). Given a seciton name `transfer`, `uploadpack`
> or `receive` as argument, it causes us to exclude all references that
> would be hidden by the respective `$seciton.hideRefs` configuration.

Thanks for adding this one in. I feel like rev-parse isn't used all that
much these days, and in a sense, we could just let it fall behind what
rev-list could do and probably nobody would care. But since it's only a
few lines, keeping parity with rev-list is nice to have.

-Peff

[PATCH v3 6/6] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4675 bytes --]

When serving a push, git-receive-pack(1) needs to verify that the
packfile sent by the client contains all objects that are required by
the updated references. This connectivity check works by marking all
preexisting references as uninteresting and using the new reference tips
as starting point for a graph walk.

Marking all preexisting references as uninteresting can be a problem
when it comes to performance. Git forges tend to do internal bookkeeping
to keep alive sets of objects for internal use or make them easy to find
via certain references. These references are typically hidden away from
the user so that they are neither advertised nor writeable. At GitLab,
we have one particular repository that contains a total of 7 million
references, of which 6.8 million are indeed internal references. With
the current connectivity check we are forced to load all these
references in order to mark them as uninteresting, and this alone takes
around 15 seconds to compute.

We can optimize this by only taking into account the set of visible refs
when marking objects as uninteresting. This means that we may now walk
more objects until we hit any object that is marked as uninteresting.
But it is rather unlikely that clients send objects that make large
parts of objects reachable that have previously only ever been hidden,
whereas the common case is to push incremental changes that build on top
of the visible object graph.

This provides a huge boost to performance in the mentioned repository,
where the vast majority of its refs hidden. Pushing a new commit into
this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
as it is configured in Gitaly leads to a 4.5-fold speedup:

    Benchmark 1: main
      Time (mean ± σ):     30.977 s ±  0.157 s    [User: 30.226 s, System: 1.083 s]
      Range (min … max):   30.796 s … 31.071 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):      6.799 s ±  0.063 s    [User: 6.803 s, System: 0.354 s]
      Range (min … max):    6.729 s …  6.850 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        4.56 ± 0.05 times faster than 'main'

As we mostly go through the same codepaths even in the case where there
are no hidden refs at all compared to the code before there is no change
in performance when no refs are hidden:

    Benchmark 1: main
      Time (mean ± σ):     48.188 s ±  0.432 s    [User: 49.326 s, System: 5.009 s]
      Range (min … max):   47.706 s … 48.539 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):     48.027 s ±  0.500 s    [User: 48.934 s, System: 5.025 s]
      Range (min … max):   47.504 s … 48.500 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        1.00 ± 0.01 times faster than 'main'

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c | 2 ++
 connected.c            | 3 +++
 connected.h            | 7 +++++++
 3 files changed, 12 insertions(+)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 1f3efc58fb..77ab40f123 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,
 	opt.err_fd = err_fd;
 	opt.progress = err_fd && !quiet;
 	opt.env = tmp_objdir_env(tmp_objdir);
+	opt.exclude_hidden_refs_section = "receive";
+
 	if (check_connected(iterate_receive_command_list, &data, &opt))
 		set_connectivity_errors(commands, si);
 
diff --git a/connected.c b/connected.c
index 74a20cb32e..4f6388eed7 100644
--- a/connected.c
+++ b/connected.c
@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 		strvec_push(&rev_list.args, "--exclude-promisor-objects");
 	if (!opt->is_deepening_fetch) {
 		strvec_push(&rev_list.args, "--not");
+		if (opt->exclude_hidden_refs_section)
+			strvec_pushf(&rev_list.args, "--exclude-hidden=%s",
+				     opt->exclude_hidden_refs_section);
 		strvec_push(&rev_list.args, "--all");
 	}
 	strvec_push(&rev_list.args, "--quiet");
diff --git a/connected.h b/connected.h
index 6e59c92aa3..16b2c84f2e 100644
--- a/connected.h
+++ b/connected.h
@@ -46,6 +46,13 @@ struct check_connected_options {
 	 * during a fetch.
 	 */
 	unsigned is_deepening_fetch : 1;
+
+	/*
+	 * If not NULL, use `--exclude-hidden=$section` to exclude all refs
+	 * hidden via the `$section.hideRefs` config from the set of
+	 * already-reachable refs.
+	 */
+	const char *exclude_hidden_refs_section;
 };
 
 #define CHECK_CONNECTED_INIT { 0 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 0/6] receive-pack: only use visible refs for connectivity check

[Taylor Blau]

On Mon, Nov 07, 2022 at 01:16:19PM +0100, Patrick Steinhardt wrote:
> Patrick Steinhardt (6):
>   refs: get rid of global list of hidden refs
>   revision: move together exclusion-related functions
>   revision: introduce struct to handle exclusions
>   revision: add new parameter to exclude hidden refs
>   revparse: add `--exclude-hidden=` option
>   receive-pack: only use visible refs for connectivity check

Thanks for the updated round. This version is looking pretty good,
though it looks like Ævar and I had a few minor recommendations. I am
definitely puzzled by seeing '--exclude-hidden=transfer', though.

Thanks,
Taylor

[PATCH v4 0/6] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 19538 bytes --]

Hi,

this is the fourth version of my patch series that tries to improve
performance of the connectivity check by only considering preexisting
refs as uninteresting that could actually have been advertised to the
client.

This time around there are only incremental changes compared to v3, the
overall implementation stays the same. Changes:

    - Fixed a pre-existing memory leak in how hidden refs are parsed so
      that tests now pass with TEST_PASSES_SANITIZE_LEAK=true.

    - Improved initialization of `struct ref_exclusions` to reuse the
      `REF_EXCLUSIONS_INIT` macro so we don't have to repeat the logic.

    - Dropped the `--exclude-hidden=transfer` option. Only `receive` and
      `uploadpack` are supported now.

Patrick

Patrick Steinhardt (6):
  refs: get rid of global list of hidden refs
  revision: move together exclusion-related functions
  revision: introduce struct to handle exclusions
  revision: add new parameter to exclude hidden refs
  rev-parse: add `--exclude-hidden=` option
  receive-pack: only use visible refs for connectivity check

 Documentation/git-rev-parse.txt    |   7 ++
 Documentation/rev-list-options.txt |   7 ++
 builtin/receive-pack.c             |  10 ++-
 builtin/rev-list.c                 |   1 +
 builtin/rev-parse.c                |  12 ++-
 connected.c                        |   3 +
 connected.h                        |   7 ++
 ls-refs.c                          |  13 ++-
 refs.c                             |  16 ++--
 refs.h                             |   5 +-
 revision.c                         | 117 +++++++++++++++---------
 revision.h                         |  36 ++++++--
 t/t6018-rev-list-glob.sh           |  11 +++
 t/t6021-rev-list-exclude-hidden.sh | 137 +++++++++++++++++++++++++++++
 upload-pack.c                      |  30 ++++---
 15 files changed, 329 insertions(+), 83 deletions(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

Range-diff against v3:
1:  3741f0a389 ! 1:  34afe30d60 refs: get rid of global list of hidden refs
    @@ refs.c: int parse_hide_refs_config(const char *var, const char *value, const cha
     -			CALLOC_ARRAY(hide_refs, 1);
     -			hide_refs->strdup_strings = 1;
     -		}
    - 		string_list_append(hide_refs, ref);
    +-		string_list_append(hide_refs, ref);
    ++		string_list_append_nodup(hide_refs, ref);
      	}
      	return 0;
      }
2:  a6dcc99ca9 = 2:  b4f21d0a80 revision: move together exclusion-related functions
3:  2a6a67df1d ! 3:  265b292ed5 revision: introduce struct to handle exclusions
    @@ Commit message
         of exclusion.
     
         Introduce a new `struct ref_exclusions` that encapsulates all the logic
    -    related to excluding refs.
    +    related to excluding refs and move the `struct string_list` that holds
    +    all wildmatch patterns of excluded refs into it. Rename functions that
    +    operate on this struct to match its name.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ revision.c: static void add_rev_cmdline_list(struct rev_info *revs,
     -		free(*ref_excludes_p);
     -	}
     -	*ref_excludes_p = NULL;
    -+	string_list_init_dup(&exclusions->excluded_refs);
    ++	struct ref_exclusions blank = REF_EXCLUSIONS_INIT;
    ++	memcpy(exclusions, &blank, sizeof(*exclusions));
      }
      
     -void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
    @@ revision.h: struct rev_cmdline_info {
     +	 */
     +	struct string_list excluded_refs;
     +};
    ++
    ++/**
    ++ * Initialize a `struct ref_exclusions` with a macro.
    ++ */
    ++#define REF_EXCLUSIONS_INIT { \
    ++	.excluded_refs = STRING_LIST_INIT_DUP, \
    ++}
     +
      struct oidset;
      struct topo_walk_info;
    @@ revision.h: void mark_trees_uninteresting_sparse(struct repository *r, struct oi
     -int ref_excluded(struct string_list *, const char *path);
     -void clear_ref_exclusion(struct string_list **);
     -void add_ref_exclusion(struct string_list **, const char *exclude);
    -+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }
    -+
     +int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
     +void init_ref_exclusions(struct ref_exclusions *);
     +void clear_ref_exclusions(struct ref_exclusions *);
4:  de7c1aa210 ! 4:  c7fa6698db revision: add new parameter to exclude hidden refs
    @@ Documentation/rev-list-options.txt: respectively, and they must begin with `refs
      or `--all`. If a trailing '/{asterisk}' is intended, it must be given
      explicitly.
      
    -+--exclude-hidden=[transfer|receive|uploadpack]::
    ++--exclude-hidden=[receive|uploadpack]::
     +	Do not include refs that have been hidden via either one of
    -+	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that
    -+	the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would
    -+	otherwise consider.  This option is cleared when seeing one of these
    -+	pseudo-refs.
    ++	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
    ++	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
    ++	would otherwise consider. This option is cleared when seeing one of
    ++	these pseudo-refs.
     +
      --reflog::
      	Pretend as if all objects mentioned by reflogs are listed on the
    @@ builtin/rev-list.c: static const char rev_list_usage[] =
      "    --tags\n"
      "    --remotes\n"
      "    --stdin\n"
    -+"    --exclude-hidden=[transfer|receive|uploadpack]\n"
    ++"    --exclude-hidden=[receive|uploadpack]\n"
      "    --quiet\n"
      "  ordering output:\n"
      "    --topo-order\n"
    @@ revision.c: static void add_rev_cmdline_list(struct rev_info *revs,
      	return 0;
      }
      
    - void init_ref_exclusions(struct ref_exclusions *exclusions)
    - {
    - 	string_list_init_dup(&exclusions->excluded_refs);
    -+	string_list_init_dup(&exclusions->hidden_refs);
    - }
    - 
    +@@ revision.c: void init_ref_exclusions(struct ref_exclusions *exclusions)
      void clear_ref_exclusions(struct ref_exclusions *exclusions)
      {
      	string_list_clear(&exclusions->excluded_refs, 0);
    -+	string_list_clear(&exclusions->hidden_refs, 1);
    ++	string_list_clear(&exclusions->hidden_refs, 0);
      }
      
      void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
    @@ revision.c: void add_ref_exclusion(struct ref_exclusions *exclusions, const char
     +{
     +	struct exclude_hidden_refs_cb cb;
     +
    -+	if (strcmp(section, "transfer") && strcmp(section, "receive") &&
    -+	    strcmp(section, "uploadpack"))
    ++	if (strcmp(section, "receive") && strcmp(section, "uploadpack"))
     +		die(_("unsupported section for hidden refs: %s"), section);
     +
     +	if (exclusions->hidden_refs.nr)
    @@ revision.h: struct ref_exclusions {
     +	struct string_list hidden_refs;
      };
      
    + /**
    +@@ revision.h: struct ref_exclusions {
    +  */
    + #define REF_EXCLUSIONS_INIT { \
    + 	.excluded_refs = STRING_LIST_INIT_DUP, \
    ++	.hidden_refs = STRING_LIST_INIT_DUP, \
    + }
    + 
      struct oidset;
     @@ revision.h: void show_object_with_name(FILE *, struct object *, const char *);
      /**
       * Helpers to check if a reference should be excluded.
       */
    --#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP }
    -+#define REF_EXCLUSIONS_INIT { .excluded_refs = STRING_LIST_INIT_DUP, .hidden_refs = STRING_LIST_INIT_DUP }
    - 
    ++
      int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
      void init_ref_exclusions(struct ref_exclusions *);
      void clear_ref_exclusions(struct ref_exclusions *);
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +
     +test_description='git rev-list --exclude-hidden test'
     +
    -+TEST_PASSES_SANITIZE_LEAK=true
     +. ./test-lib.sh
     +
     +test_expect_success 'setup' '
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +	test_cmp expected err
     +'
     +
    -+test_expect_success 'passed multiple times' '
    -+	echo "fatal: --exclude-hidden= passed more than once" >expected &&
    -+	test_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --exclude-hidden=transfer 2>err &&
    -+	test_cmp expected err
    -+'
    -+
    -+test_expect_success '--exclude-hidden without hiddenRefs' '
    -+	git rev-list --exclude-hidden=transfer --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$NAMESPACE
    -+	$HIDDEN
    -+	$TAG
    -+	$COMMIT
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
    -+test_expect_success 'hidden via transfer.hideRefs' '
    -+	git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$NAMESPACE
    -+	$TAG
    -+	$COMMIT
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
    -+test_expect_success '--all --exclude-hidden=transfer --not --all without hidden refs' '
    -+	git rev-list --all --exclude-hidden=transfer --not --all >out &&
    -+	test_must_be_empty out
    -+'
    -+
    -+test_expect_success '--all --exclude-hidden=transfer --not --all with hidden ref' '
    -+	git -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=transfer --not --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$HIDDEN
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
    -+test_expect_success '--exclude-hidden with --exclude' '
    -+	git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=transfer --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$NAMESPACE
    -+	$COMMIT
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
    -+test_expect_success '--exclude-hidden is reset' '
    -+	git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=transfer --all --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$NAMESPACE
    -+	$HIDDEN
    -+	$TAG
    -+	$COMMIT
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
    -+test_expect_success '--exclude-hidden operates on stripped refs by default' '
    -+	GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=transfer --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$HIDDEN
    -+	$TAG
    -+	$COMMIT
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
    -+test_expect_success '--exclude-hidden does not hide namespace by default' '
    -+	GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$NAMESPACE
    -+	$HIDDEN
    -+	$TAG
    -+	$COMMIT
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
    -+test_expect_success '--exclude-hidden= may operate on unstripped refs' '
    -+	GIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=transfer --all >out &&
    -+	cat >expected <<-EOF &&
    -+	$HIDDEN
    -+	$TAG
    -+	$COMMIT
    -+	EOF
    -+	test_cmp expected out
    -+'
    -+
     +for section in receive uploadpack
     +do
    -+	test_expect_success "hidden via $section.hideRefs" '
    -+		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
    -+		cat >expected <<-EOF &&
    -+		$NAMESPACE
    -+		$TAG
    -+		$COMMIT
    -+		EOF
    -+		test_cmp expected out
    ++	test_expect_success "$section: passed multiple times" '
    ++		echo "fatal: --exclude-hidden= passed more than once" >expected &&
    ++		test_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&
    ++		test_cmp expected err
     +	'
     +
    -+	test_expect_success "--exclude-hidden=$section respects transfer.hideRefs" '
    -+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
    -+		cat >expected <<-EOF &&
    -+		$NAMESPACE
    -+		$TAG
    -+		$COMMIT
    -+		EOF
    -+		test_cmp expected out
    -+	'
    -+
    -+	test_expect_success "--exclude-hidden=transfer ignores $section.hideRefs" '
    -+		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=transfer --all >out &&
    ++	test_expect_success "$section: without hiddenRefs" '
    ++		git rev-list --exclude-hidden=$section --all >out &&
     +		cat >expected <<-EOF &&
     +		$NAMESPACE
     +		$HIDDEN
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +		test_cmp expected out
     +	'
     +
    -+	test_expect_success "--exclude-hidden=$section respects both transfer.hideRefs and $section.hideRefs" '
    ++	test_expect_success "$section: hidden via transfer.hideRefs" '
    ++		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$NAMESPACE
    ++		$TAG
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
    ++	test_expect_success "$section: hidden via $section.hideRefs" '
    ++		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$NAMESPACE
    ++		$TAG
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
    ++	test_expect_success "$section: respects both transfer.hideRefs and $section.hideRefs" '
     +		git -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
     +		cat >expected <<-EOF &&
     +		$NAMESPACE
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +		EOF
     +		test_cmp expected out
     +	'
    ++
    ++	test_expect_success "$section: negation without hidden refs marks everything as uninteresting" '
    ++		git rev-list --all --exclude-hidden=$section --not --all >out &&
    ++		test_must_be_empty out
    ++	'
    ++
    ++	test_expect_success "$section: negation with hidden refs marks them as interesting" '
    ++		git -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$HIDDEN
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
    ++	test_expect_success "$section: hidden refs and excludes work together" '
    ++		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$NAMESPACE
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
    ++	test_expect_success "$section: excluded hidden refs get reset" '
    ++		git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$NAMESPACE
    ++		$HIDDEN
    ++		$TAG
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
    ++	test_expect_success "$section: operates on stripped refs by default" '
    ++		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$HIDDEN
    ++		$TAG
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
    ++	test_expect_success "$section: does not hide namespace by default" '
    ++		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$NAMESPACE
    ++		$HIDDEN
    ++		$TAG
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
    ++	test_expect_success "$section: can operate on unstripped refs" '
    ++		GIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
    ++		cat >expected <<-EOF &&
    ++		$HIDDEN
    ++		$TAG
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
     +done
     +
     +test_done
5:  68a5e56304 ! 5:  79c5c64a80 revparse: add `--exclude-hidden=` option
    @@ Metadata
     Author: Patrick Steinhardt <ps@pks.im>
     
      ## Commit message ##
    -    revparse: add `--exclude-hidden=` option
    +    rev-parse: add `--exclude-hidden=` option
     
         Add a new `--exclude-hidden=` option that is similar to the one we just
    -    added to git-rev-list(1). Given a seciton name `transfer`, `uploadpack`
    -    or `receive` as argument, it causes us to exclude all references that
    -    would be hidden by the respective `$seciton.hideRefs` configuration.
    +    added to git-rev-list(1). Given a seciton name `uploadpack` or `receive`
    +    as argument, it causes us to exclude all references that would be hidden
    +    by the respective `$section.hideRefs` configuration.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    @@ Documentation/git-rev-parse.txt: respectively, and they must begin with `refs/`
      or `--all`. If a trailing '/{asterisk}' is intended, it must be given
      explicitly.
      
    -+--exclude-hidden=[transfer|receive|uploadpack]::
    ++--exclude-hidden=[receive|uploadpack]::
     +	Do not include refs that have been hidden via either one of
    -+	`transfer.hideRefs`, `receive.hideRefs` or `uploadpack.hideRefs` that
    -+	the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob` would
    -+	otherwise consider.  This option is cleared when seeing one of these
    -+	pseudo-refs.
    ++	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
    ++	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
    ++	would otherwise consider. This option is cleared when seeing one of
    ++	these pseudo-refs.
     +
      --disambiguate=<prefix>::
      	Show every object whose name begins with the given prefix.
    @@ t/t6018-rev-list-glob.sh: test_expect_success 'rev-parse --exclude=ref with --re
      	compare rev-parse "--exclude=upstream/x --remotes=upstream/*" "upstream/one upstream/two"
      '
      
    -+test_expect_success 'rev-parse --exclude-hidden= with --all' '
    -+	compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--exclude-hidden=transfer --all" "--branches --tags"
    -+'
    ++for section in receive uploadpack
    ++do
    ++	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
    ++		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--exclude-hidden=$section --all" "--branches --tags"
    ++	'
     +
    -+test_expect_success 'rev-parse --exclude-hidden= with --all' '
    -+	compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude-hidden=transfer --all" "--exclude=refs/heads/subspace/* --all"
    -+'
    ++	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
    ++		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude-hidden=$section --all" "--exclude=refs/heads/subspace/* --all"
    ++	'
    ++done
     +
      test_expect_success 'rev-list --exclude=glob with --branches=glob' '
      	compare rev-list "--exclude=subspace-* --branches=sub*" "subspace/one subspace/two"
6:  9d15449559 = 6:  39b4741734 receive-pack: only use visible refs for connectivity check
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 1/6] refs: get rid of global list of hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 12546 bytes --]

We're about to add a new argument to git-rev-list(1) that allows it to
add all references that are visible when taking `transfer.hideRefs` et
al into account. This will require us to potentially parse multiple sets
of hidden refs, which is not easily possible right now as there is only
a single, global instance of the list of parsed hidden refs.

Refactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both
take the list of hidden references as input and adjust callers to keep a
local list, instead. This allows us to easily use multiple hidden-ref
lists. Furthermore, it allows us to properly free this list before we
exit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c |  8 +++++---
 ls-refs.c              | 13 +++++++++----
 refs.c                 | 16 +++++-----------
 refs.h                 |  5 +++--
 upload-pack.c          | 30 ++++++++++++++++++------------
 5 files changed, 40 insertions(+), 32 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 44bcea3a5b..1f3efc58fb 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)
 
 static int receive_pack_config(const char *var, const char *value, void *cb)
 {
-	int status = parse_hide_refs_config(var, value, "receive");
+	int status = parse_hide_refs_config(var, value, "receive", &hidden_refs);
 
 	if (status)
 		return status;
@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,
 	struct oidset *seen = data;
 	const char *path = strip_namespace(path_full);
 
-	if (ref_is_hidden(path, path_full))
+	if (ref_is_hidden(path, path_full, &hidden_refs))
 		return 0;
 
 	/*
@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)
 		strbuf_setlen(&refname_full, prefix_len);
 		strbuf_addstr(&refname_full, cmd->ref_name);
 
-		if (!ref_is_hidden(cmd->ref_name, refname_full.buf))
+		if (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))
 			continue;
 		if (is_null_oid(&cmd->new_oid))
 			cmd->error_string = "deny deleting a hidden ref";
@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
+	string_list_clear(&hidden_refs, 1);
 	free((void *)push_cert_nonce);
 	return 0;
 }
diff --git a/ls-refs.c b/ls-refs.c
index fa0d01b47c..ae89f850e9 100644
--- a/ls-refs.c
+++ b/ls-refs.c
@@ -6,6 +6,7 @@
 #include "ls-refs.h"
 #include "pkt-line.h"
 #include "config.h"
+#include "string-list.h"
 
 static int config_read;
 static int advertise_unborn;
@@ -73,6 +74,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
+	struct string_list hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 
 	strbuf_reset(&data->buf);
 
-	if (ref_is_hidden(refname_nons, refname))
+	if (ref_is_hidden(refname_nons, refname, &data->hidden_refs))
 		return 0;
 
 	if (!ref_match(&data->prefixes, refname_nons))
@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)
 }
 
 static int ls_refs_config(const char *var, const char *value,
-			  void *data UNUSED)
+			  void *cb_data)
 {
+	struct ls_refs_data *data = cb_data;
 	/*
 	 * We only serve fetches over v2 for now, so respect only "uploadpack"
 	 * config. This may need to eventually be expanded to "receive", but we
 	 * don't yet know how that information will be passed to ls-refs.
 	 */
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 int ls_refs(struct repository *r, struct packet_reader *request)
@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
+	string_list_init_dup(&data.hidden_refs);
 
 	ensure_config_read();
-	git_config(ls_refs_config, NULL);
+	git_config(ls_refs_config, &data);
 
 	while (packet_reader_read(request) == PACKET_READ_NORMAL) {
 		const char *arg = request->line;
@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
+	string_list_clear(&data.hidden_refs, 1);
 	return 0;
 }
 
diff --git a/refs.c b/refs.c
index 1491ae937e..2c7e88b190 100644
--- a/refs.c
+++ b/refs.c
@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 					    refname, strict);
 }
 
-static struct string_list *hide_refs;
-
-int parse_hide_refs_config(const char *var, const char *value, const char *section)
+int parse_hide_refs_config(const char *var, const char *value, const char *section,
+			   struct string_list *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		if (!hide_refs) {
-			CALLOC_ARRAY(hide_refs, 1);
-			hide_refs->strdup_strings = 1;
-		}
-		string_list_append(hide_refs, ref);
+		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
 
-int ref_is_hidden(const char *refname, const char *refname_full)
+int ref_is_hidden(const char *refname, const char *refname_full,
+		  const struct string_list *hide_refs)
 {
 	int i;
 
-	if (!hide_refs)
-		return 0;
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
 		const char *match = hide_refs->items[i].string;
 		const char *subject;
diff --git a/refs.h b/refs.h
index 8958717a17..3266fd8f57 100644
--- a/refs.h
+++ b/refs.h
@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *new_oid, const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr);
 
-int parse_hide_refs_config(const char *var, const char *value, const char *);
+int parse_hide_refs_config(const char *var, const char *value, const char *,
+			   struct string_list *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *);
+int ref_is_hidden(const char *, const char *, const struct string_list *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
diff --git a/upload-pack.c b/upload-pack.c
index 0b8311bd68..9db17f8787 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -62,6 +62,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
+	struct string_list hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
+	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 	memset(data, 0, sizeof(*data));
 	data->symref = symref;
 	data->wanted_refs = wanted_refs;
+	data->hidden_refs = hidden_refs;
 	data->want_obj = want_obj;
 	data->have_obj = have_obj;
 	data->haves = haves;
@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
+	string_list_clear(&data->hidden_refs, 1);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * Checking for reachable shallows requires that our refs be
 		 * marked with OUR_REF.
 		 */
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, data);
+		for_each_namespaced_ref(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid)
+			const struct object_id *oid, const struct string_list *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
-	if (ref_is_hidden(refname, refname_full)) {
+	if (ref_is_hidden(refname, refname_full, hidden_refs)) {
 		o->flags |= HIDDEN_REF;
 		return 1;
 	}
@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,
 }
 
 static int check_ref(const char *refname_full, const struct object_id *oid,
-		     int flag UNUSED, void *cb_data UNUSED)
+		     int flag UNUSED, void *cb_data)
 {
 	const char *refname = strip_namespace(refname_full);
+	struct upload_pack_data *data = cb_data;
 
-	mark_our_ref(refname, refname_full, oid);
+	mark_our_ref(refname, refname_full, oid, &data->hidden_refs);
 	return 0;
 }
 
@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 	struct object_id peeled;
 	struct upload_pack_data *data = cb_data;
 
-	if (mark_our_ref(refname_nons, refname, oid))
+	if (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))
 		return 0;
 
 	if (capabilities) {
@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)
 	if (parse_object_filter_config(var, value, data) < 0)
 		return -1;
 
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)
@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		advertise_shallow_grafts(1);
 		packet_flush(1);
 	} else {
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, &data);
+		for_each_namespaced_ref(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
+			  struct string_list *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 		struct strbuf refname = STRBUF_INIT;
 
 		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
-		if (ref_is_hidden(refname_nons, refname.buf) ||
+		if (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||
 		    read_ref(refname.buf, &oid)) {
 			packet_writer_error(writer, "unknown ref %s", refname_nons);
 			die("unknown ref %s", refname_nons);
@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,
 			continue;
 		if (data->allow_ref_in_want &&
 		    parse_want_ref(&data->writer, arg, &data->wanted_refs,
-				   &data->want_obj))
+				   &data->hidden_refs, &data->want_obj))
 			continue;
 		/* process have line */
 		if (parse_have(arg, &data->haves))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 1/6] refs: get rid of global list of hidden refs

[Ævar Arnfjörð Bjarmason]

On Tue, Nov 08 2022, Patrick Steinhardt wrote:

> @@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>  		packet_flush(1);
>  	oid_array_clear(&shallow);
>  	oid_array_clear(&ref);
> +	string_list_clear(&hidden_refs, 1);

In the v4 re-roll you got rid of the "1" for some other string_lists,
but is this one still needed, i.e. does it use "util"? At a glance it
doesn't seem so. There's another "hidden_refs" (maybe just semi-related)
in 4/6 that doesn't use it when clearing.

> diff --git a/refs.c b/refs.c
> index 1491ae937e..2c7e88b190 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
>  					    refname, strict);
>  }
>  
> -static struct string_list *hide_refs;
> -
> -int parse_hide_refs_config(const char *var, const char *value, const char *section)
> +int parse_hide_refs_config(const char *var, const char *value, const char *section,
> +			   struct string_list *hide_refs)
>  {
>  	const char *key;
>  	if (!strcmp("transfer.hiderefs", var) ||
> @@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
>  		len = strlen(ref);
>  		while (len && ref[len - 1] == '/')
>  			ref[--len] = '\0';
> -		if (!hide_refs) {
> -			CALLOC_ARRAY(hide_refs, 1);
> -			hide_refs->strdup_strings = 1;
> -		}
> -		string_list_append(hide_refs, ref);
> +		string_list_append_nodup(hide_refs, ref);
>  	}
>  	return 0;
>  }
>  

As before, this is all much nicer, thanks.

Re: [PATCH v4 1/6] refs: get rid of global list of hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 782 bytes --]

On Tue, Nov 08, 2022 at 02:36:04PM +0100, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Nov 08 2022, Patrick Steinhardt wrote:
> 
> > @@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
> >  		packet_flush(1);
> >  	oid_array_clear(&shallow);
> >  	oid_array_clear(&ref);
> > +	string_list_clear(&hidden_refs, 1);
> 
> In the v4 re-roll you got rid of the "1" for some other string_lists,
> but is this one still needed, i.e. does it use "util"? At a glance it
> doesn't seem so. There's another "hidden_refs" (maybe just semi-related)
> in 4/6 that doesn't use it when clearing.

Oh, right, I missed this one. Will wait a bit though for other feedback
to come in before sending a v5 only with this one-line change.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 1/6] refs: get rid of global list of hidden refs

[Jeff King]

On Tue, Nov 08, 2022 at 11:03:39AM +0100, Patrick Steinhardt wrote:

> -static struct string_list *hide_refs;
> -
> -int parse_hide_refs_config(const char *var, const char *value, const char *section)
> +int parse_hide_refs_config(const char *var, const char *value, const char *section,
> +			   struct string_list *hide_refs)
>  {
>  	const char *key;
>  	if (!strcmp("transfer.hiderefs", var) ||
> @@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
>  		len = strlen(ref);
>  		while (len && ref[len - 1] == '/')
>  			ref[--len] = '\0';
> -		if (!hide_refs) {
> -			CALLOC_ARRAY(hide_refs, 1);
> -			hide_refs->strdup_strings = 1;
> -		}
> -		string_list_append(hide_refs, ref);
> +		string_list_append_nodup(hide_refs, ref);
>  	}
>  	return 0;
>  }

This nodup is definitely the right thing to be doing, but it's kind of
hidden in here. AFAICT it is fixing an existing leak, because the
previous code always set strdup_strings, and we always made our own copy
of "ref".

Probably not worth a re-roll on its own, but I'd probably have pulled
that into its own commit.

The rest of the commit looks OK to me. Like Ævar, I'm confused by the
"free_util" arguments to string_list_clear().

-Peff

[PATCH v4 2/6] revision: move together exclusion-related functions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2402 bytes --]

Move together the definitions of functions that handle exclusions of
refs so that related functionality sits in a single place, only.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 revision.c | 52 ++++++++++++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/revision.c b/revision.c
index 0760e78936..be755670e2 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-struct all_refs_cb {
-	int all_flags;
-	int warned_bad_reflog;
-	struct rev_info *all_revs;
-	const char *name_for_errormsg;
-	struct worktree *wt;
-};
-
 int ref_excluded(struct string_list *ref_excludes, const char *path)
 {
 	struct string_list_item *item;
@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)
 	return 0;
 }
 
+void clear_ref_exclusion(struct string_list **ref_excludes_p)
+{
+	if (*ref_excludes_p) {
+		string_list_clear(*ref_excludes_p, 0);
+		free(*ref_excludes_p);
+	}
+	*ref_excludes_p = NULL;
+}
+
+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+{
+	if (!*ref_excludes_p) {
+		CALLOC_ARRAY(*ref_excludes_p, 1);
+		(*ref_excludes_p)->strdup_strings = 1;
+	}
+	string_list_append(*ref_excludes_p, exclude);
+}
+
+struct all_refs_cb {
+	int all_flags;
+	int warned_bad_reflog;
+	struct rev_info *all_revs;
+	const char *name_for_errormsg;
+	struct worktree *wt;
+};
+
 static int handle_one_ref(const char *path, const struct object_id *oid,
 			  int flag UNUSED,
 			  void *cb_data)
@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,
 	cb->wt = NULL;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
-{
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
-}
-
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
-{
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
-}
-
 static void handle_refs(struct ref_store *refs,
 			struct rev_info *revs, unsigned flags,
 			int (*for_each)(struct ref_store *, each_ref_fn, void *))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 3/6] revision: introduce struct to handle exclusions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8642 bytes --]

The functions that handle exclusion of refs work on a single string
list. We're about to add a second mechanism for excluding refs though,
and it makes sense to reuse much of the same architecture for both kinds
of exclusion.

Introduce a new `struct ref_exclusions` that encapsulates all the logic
related to excluding refs and move the `struct string_list` that holds
all wildmatch patterns of excluded refs into it. Rename functions that
operate on this struct to match its name.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/rev-parse.c |  8 ++++----
 revision.c          | 48 +++++++++++++++++++++------------------------
 revision.h          | 27 +++++++++++++++++++------
 3 files changed, 47 insertions(+), 36 deletions(-)

diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 8f61050bde..7fa5b6991b 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -39,7 +39,7 @@ static int abbrev_ref_strict;
 static int output_sq;
 
 static int stuck_long;
-static struct string_list *ref_excludes;
+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;
 
 /*
  * Some arguments are relevant "revision" arguments,
@@ -198,7 +198,7 @@ static int show_default(void)
 static int show_reference(const char *refname, const struct object_id *oid,
 			  int flag UNUSED, void *cb_data UNUSED)
 {
-	if (ref_excluded(ref_excludes, refname))
+	if (ref_excluded(&ref_excludes, refname))
 		return 0;
 	show_rev(NORMAL, oid, refname);
 	return 0;
@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)
 		for_each_glob_ref_in(show_reference, pattern, prefix, NULL);
 	else
 		for_each_ref_in(prefix, show_reference, NULL);
-	clear_ref_exclusion(&ref_excludes);
+	clear_ref_exclusions(&ref_excludes);
 }
 
 enum format_type {
@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 			}
 			if (!strcmp(arg, "--all")) {
 				for_each_ref(show_reference, NULL);
-				clear_ref_exclusion(&ref_excludes);
+				clear_ref_exclusions(&ref_excludes);
 				continue;
 			}
 			if (skip_prefix(arg, "--disambiguate=", &arg)) {
diff --git a/revision.c b/revision.c
index be755670e2..fe3ec98f46 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,35 +1517,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-int ref_excluded(struct string_list *ref_excludes, const char *path)
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
 	struct string_list_item *item;
-
-	if (!ref_excludes)
-		return 0;
-	for_each_string_list_item(item, ref_excludes) {
+	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
 	return 0;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
+void init_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
+	struct ref_exclusions blank = REF_EXCLUSIONS_INIT;
+	memcpy(exclusions, &blank, sizeof(*exclusions));
 }
 
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
+	string_list_clear(&exclusions->excluded_refs, 0);
+}
+
+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
+{
+	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
 struct all_refs_cb {
@@ -1563,7 +1558,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,
 	struct all_refs_cb *cb = cb_data;
 	struct object *object;
 
-	if (ref_excluded(cb->all_revs->ref_excludes, path))
+	if (ref_excluded(&cb->all_revs->ref_excludes, path))
 	    return 0;
 
 	object = get_reference(cb->all_revs, path, oid, cb->all_flags);
@@ -1901,6 +1896,7 @@ void repo_init_revisions(struct repository *r,
 
 	init_display_notes(&revs->notes_opt);
 	list_objects_filter_init(&revs->filter);
+	init_ref_exclusions(&revs->ref_excludes);
 }
 
 static void add_pending_commit_list(struct rev_info *revs,
@@ -2689,10 +2685,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 			init_all_refs_cb(&cb, revs, *flags);
 			other_head_refs(handle_one_ref, &cb);
 		}
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--branches")) {
 		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--bisect")) {
 		read_bisect_terms(&term_bad, &term_good);
 		handle_refs(refs, revs, *flags, for_each_bad_bisect_ref);
@@ -2701,15 +2697,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		revs->bisect = 1;
 	} else if (!strcmp(arg, "--tags")) {
 		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--remotes")) {
 		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref(handle_one_ref, optarg, &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 		return argcount;
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
@@ -2718,17 +2714,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--tags=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--reflog")) {
 		add_reflogs_to_pending(revs, *flags);
 	} else if (!strcmp(arg, "--indexed-objects")) {
diff --git a/revision.h b/revision.h
index afe1b77985..5c8ab16047 100644
--- a/revision.h
+++ b/revision.h
@@ -81,6 +81,21 @@ struct rev_cmdline_info {
 	} *rev;
 };
 
+struct ref_exclusions {
+	/*
+	 * Excluded refs is a list of wildmatch patterns. If any of the
+	 * patterns matches, the reference will be excluded.
+	 */
+	struct string_list excluded_refs;
+};
+
+/**
+ * Initialize a `struct ref_exclusions` with a macro.
+ */
+#define REF_EXCLUSIONS_INIT { \
+	.excluded_refs = STRING_LIST_INIT_DUP, \
+}
+
 struct oidset;
 struct topo_walk_info;
 
@@ -103,7 +118,7 @@ struct rev_info {
 	struct list_objects_filter_options filter;
 
 	/* excluding from --branches, --refs, etc. expansion */
-	struct string_list *ref_excludes;
+	struct ref_exclusions ref_excludes;
 
 	/* Basic information */
 	const char *prefix;
@@ -439,12 +454,12 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)
 void show_object_with_name(FILE *, struct object *, const char *);
 
 /**
- * Helpers to check if a "struct string_list" item matches with
- * wildmatch().
+ * Helpers to check if a reference should be excluded.
  */
-int ref_excluded(struct string_list *, const char *path);
-void clear_ref_exclusion(struct string_list **);
-void add_ref_exclusion(struct string_list **, const char *exclude);
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
+void init_ref_exclusions(struct ref_exclusions *);
+void clear_ref_exclusions(struct ref_exclusions *);
+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
 
 /**
  * This function can be used if you want to add commit objects as revision
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 4/6] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 10885 bytes --]

Users can optionally hide refs from remote users in git-upload-pack(1),
git-receive-pack(1) and others via the `transfer.hideRefs`, but there is
not an easy way to obtain the list of all visible or hidden refs right
now. We'll require just that though for a performance improvement in our
connectivity check.

Add a new option `--exclude-hidden=` that excludes any hidden refs from
the next pseudo-ref like `--all` or `--branches`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/rev-list-options.txt |   7 ++
 builtin/rev-list.c                 |   1 +
 revision.c                         |  41 ++++++++-
 revision.h                         |   9 ++
 t/t6021-rev-list-exclude-hidden.sh | 137 +++++++++++++++++++++++++++++
 5 files changed, 194 insertions(+), 1 deletion(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
index 1837509566..5b46781b35 100644
--- a/Documentation/rev-list-options.txt
+++ b/Documentation/rev-list-options.txt
@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[receive|uploadpack]::
+	Do not include refs that have been hidden via either one of
+	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
+	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
+	would otherwise consider. This option is cleared when seeing one of
+	these pseudo-refs.
+
 --reflog::
 	Pretend as if all objects mentioned by reflogs are listed on the
 	command line as `<commit>`.
diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index 3acd93f71e..d42db0b0cc 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -38,6 +38,7 @@ static const char rev_list_usage[] =
 "    --tags\n"
 "    --remotes\n"
 "    --stdin\n"
+"    --exclude-hidden=[receive|uploadpack]\n"
 "    --quiet\n"
 "  ordering output:\n"
 "    --topo-order\n"
diff --git a/revision.c b/revision.c
index fe3ec98f46..b726cfd255 100644
--- a/revision.c
+++ b/revision.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "config.h"
 #include "object-store.h"
 #include "tag.h"
 #include "blob.h"
@@ -1519,11 +1520,17 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
+	const char *stripped_path = strip_namespace(path);
 	struct string_list_item *item;
+
 	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
+
+	if (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))
+		return 1;
+
 	return 0;
 }
 
@@ -1536,6 +1543,7 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
+	string_list_clear(&exclusions->hidden_refs, 0);
 }
 
 void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
@@ -1543,6 +1551,34 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
 	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
+struct exclude_hidden_refs_cb {
+	struct ref_exclusions *exclusions;
+	const char *section;
+};
+
+static int hide_refs_config(const char *var, const char *value, void *cb_data)
+{
+	struct exclude_hidden_refs_cb *cb = cb_data;
+	return parse_hide_refs_config(var, value, cb->section,
+				      &cb->exclusions->hidden_refs);
+}
+
+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)
+{
+	struct exclude_hidden_refs_cb cb;
+
+	if (strcmp(section, "receive") && strcmp(section, "uploadpack"))
+		die(_("unsupported section for hidden refs: %s"), section);
+
+	if (exclusions->hidden_refs.nr)
+		die(_("--exclude-hidden= passed more than once"));
+
+	cb.exclusions = exclusions;
+	cb.section = section;
+
+	git_config(hide_refs_config, &cb);
+}
+
 struct all_refs_cb {
 	int all_flags;
 	int warned_bad_reflog;
@@ -2221,7 +2257,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg
 	    !strcmp(arg, "--bisect") || starts_with(arg, "--glob=") ||
 	    !strcmp(arg, "--indexed-objects") ||
 	    !strcmp(arg, "--alternate-refs") ||
-	    starts_with(arg, "--exclude=") ||
+	    starts_with(arg, "--exclude=") || starts_with(arg, "--exclude-hidden=") ||
 	    starts_with(arg, "--branches=") || starts_with(arg, "--tags=") ||
 	    starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk="))
 	{
@@ -2710,6 +2746,9 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
 		return argcount;
+	} else if ((argcount = parse_long_opt("exclude-hidden", argv, &optarg))) {
+		exclude_hidden_refs(&revs->ref_excludes, optarg);
+		return argcount;
 	} else if (skip_prefix(arg, "--branches=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
diff --git a/revision.h b/revision.h
index 5c8ab16047..a96fefebf1 100644
--- a/revision.h
+++ b/revision.h
@@ -87,6 +87,12 @@ struct ref_exclusions {
 	 * patterns matches, the reference will be excluded.
 	 */
 	struct string_list excluded_refs;
+
+	/*
+	 * Hidden refs is a list of patterns that is to be hidden via
+	 * `ref_is_hidden()`.
+	 */
+	struct string_list hidden_refs;
 };
 
 /**
@@ -94,6 +100,7 @@ struct ref_exclusions {
  */
 #define REF_EXCLUSIONS_INIT { \
 	.excluded_refs = STRING_LIST_INIT_DUP, \
+	.hidden_refs = STRING_LIST_INIT_DUP, \
 }
 
 struct oidset;
@@ -456,10 +463,12 @@ void show_object_with_name(FILE *, struct object *, const char *);
 /**
  * Helpers to check if a reference should be excluded.
  */
+
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
 void init_ref_exclusions(struct ref_exclusions *);
 void clear_ref_exclusions(struct ref_exclusions *);
 void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
+void exclude_hidden_refs(struct ref_exclusions *, const char *section);
 
 /**
  * This function can be used if you want to add commit objects as revision
diff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh
new file mode 100755
index 0000000000..4ab50e7f4f
--- /dev/null
+++ b/t/t6021-rev-list-exclude-hidden.sh
@@ -0,0 +1,137 @@
+#!/bin/sh
+
+test_description='git rev-list --exclude-hidden test'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit_bulk --id=commit --ref=refs/heads/main 1 &&
+	COMMIT=$(git rev-parse refs/heads/main) &&
+	test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&
+	TAG=$(git rev-parse refs/tags/lightweight) &&
+	test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&
+	HIDDEN=$(git rev-parse refs/hidden/commit) &&
+	test_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&
+	NAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)
+'
+
+test_expect_success 'invalid section' '
+	echo "fatal: unsupported section for hidden refs: unsupported" >expected &&
+	test_must_fail git rev-list --exclude-hidden=unsupported 2>err &&
+	test_cmp expected err
+'
+
+for section in receive uploadpack
+do
+	test_expect_success "$section: passed multiple times" '
+		echo "fatal: --exclude-hidden= passed more than once" >expected &&
+		test_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&
+		test_cmp expected err
+	'
+
+	test_expect_success "$section: without hiddenRefs" '
+		git rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden via transfer.hideRefs" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden via $section.hideRefs" '
+		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: respects both transfer.hideRefs and $section.hideRefs" '
+		git -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: negation without hidden refs marks everything as uninteresting" '
+		git rev-list --all --exclude-hidden=$section --not --all >out &&
+		test_must_be_empty out
+	'
+
+	test_expect_success "$section: negation with hidden refs marks them as interesting" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden refs and excludes work together" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: excluded hidden refs get reset" '
+		git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: operates on stripped refs by default" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: does not hide namespace by default" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: can operate on unstripped refs" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+done
+
+test_done
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 4/6] revision: add new parameter to exclude hidden refs

[Jeff King]

On Tue, Nov 08, 2022 at 11:03:51AM +0100, Patrick Steinhardt wrote:

> --- a/Documentation/rev-list-options.txt
> +++ b/Documentation/rev-list-options.txt
> @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
>  or `--all`. If a trailing '/{asterisk}' is intended, it must be given
>  explicitly.
>  
> +--exclude-hidden=[receive|uploadpack]::
> +	Do not include refs that have been hidden via either one of
> +	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
> +	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
> +	would otherwise consider. This option is cleared when seeing one of
> +	these pseudo-refs.

OK, so this one drops "transfer", which I think is good. But again, I
think the explanation here is subtly misleading, because it's not just
"hidden via those sections", but "hidden as receive-pack or upload-pack
would, respecting both those sections _and_ transfer".

I also wondered if we could simplify the explanation a bit by tying it
into the existing --exclude, and then we don't need that mouthful of
pseudo-ref options. So together something like:

  Do not include refs that would be hidden by `receive-pack` or
  `upload-pack`, by consulting the appropriate `receive.hideRefs` or
  `uploadpack.hideRefs`, along with `transfer.hideRefs`. Like
  `--exclude`, this option affects the next pseudo-ref option (`--all`,
  `--glob`, etc), and is cleared after processing them.

But then I read the next section in the --exclude docs, which says:

  The patterns given should not begin with refs/heads, refs/tags, or
  refs/remotes when applied to --branches, --tags, or --remotes,
  respectively, and they must begin with refs/ when applied to --glob or
  --all. If a trailing /* is intended, it must be given explicitly.

Yikes. So --all is going to process "refs/heads/foo", but --branches
will see just "foo". And that means that:

  git rev-list --exclude-hidden=receive --branches

will not work! Because receive.hideRefs is naming fully qualified refs,
but we'll pass unqualified ones to ref_is_hidden(). Even though that
function takes a "full" refname argument, I think that has to do with
namespaces.

I'm sure this _could_ be made to work, but I wonder if it is worth the
trouble. If it's not going to work, though, I think we'd want to detect
the situation and complain, at least for now. And likewise the
documentation needs to make clear it only works with --all and --glob.

Sorry to have misled in my initial suggestion to turn --visible-refs
into --exclude-hidden. However, I do still stand by that suggestion.
Even if we don't make it work with "--branches" now, the user-visible
framework is still there, so it becomes a matter of extending the
implementation later, rather than re-designing the options.

-Peff

Re: [PATCH v4 4/6] revision: add new parameter to exclude hidden refs

[Taylor Blau]

On Tue, Nov 08, 2022 at 10:07:35AM -0500, Jeff King wrote:
> I'm sure this _could_ be made to work, but I wonder if it is worth the
> trouble. If it's not going to work, though, I think we'd want to detect
> the situation and complain, at least for now. And likewise the
> documentation needs to make clear it only works with --all and --glob.
>
> Sorry to have misled in my initial suggestion to turn --visible-refs
> into --exclude-hidden. However, I do still stand by that suggestion.
> Even if we don't make it work with "--branches" now, the user-visible
> framework is still there, so it becomes a matter of extending the
> implementation later, rather than re-designing the options.

Good catch. I agree completely, so hopefully we will see something to
this effect in the forthcoming v5.

I'll hold off on merging this down until we see that version.


Thanks,
Taylor

Re: [PATCH v4 4/6] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3531 bytes --]

On Tue, Nov 08, 2022 at 10:07:35AM -0500, Jeff King wrote:
> On Tue, Nov 08, 2022 at 11:03:51AM +0100, Patrick Steinhardt wrote:
> 
> > --- a/Documentation/rev-list-options.txt
> > +++ b/Documentation/rev-list-options.txt
> > @@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
> >  or `--all`. If a trailing '/{asterisk}' is intended, it must be given
> >  explicitly.
> >  
> > +--exclude-hidden=[receive|uploadpack]::
> > +	Do not include refs that have been hidden via either one of
> > +	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
> > +	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
> > +	would otherwise consider. This option is cleared when seeing one of
> > +	these pseudo-refs.
> 
> OK, so this one drops "transfer", which I think is good. But again, I
> think the explanation here is subtly misleading, because it's not just
> "hidden via those sections", but "hidden as receive-pack or upload-pack
> would, respecting both those sections _and_ transfer".
> 
> I also wondered if we could simplify the explanation a bit by tying it
> into the existing --exclude, and then we don't need that mouthful of
> pseudo-ref options. So together something like:
> 
>   Do not include refs that would be hidden by `receive-pack` or
>   `upload-pack`, by consulting the appropriate `receive.hideRefs` or
>   `uploadpack.hideRefs`, along with `transfer.hideRefs`. Like
>   `--exclude`, this option affects the next pseudo-ref option (`--all`,
>   `--glob`, etc), and is cleared after processing them.

I'll try to come up with something, thanks.

> But then I read the next section in the --exclude docs, which says:
> 
>   The patterns given should not begin with refs/heads, refs/tags, or
>   refs/remotes when applied to --branches, --tags, or --remotes,
>   respectively, and they must begin with refs/ when applied to --glob or
>   --all. If a trailing /* is intended, it must be given explicitly.
> 
> Yikes. So --all is going to process "refs/heads/foo", but --branches
> will see just "foo". And that means that:
> 
>   git rev-list --exclude-hidden=receive --branches
> 
> will not work! Because receive.hideRefs is naming fully qualified refs,
> but we'll pass unqualified ones to ref_is_hidden(). Even though that
> function takes a "full" refname argument, I think that has to do with
> namespaces.

Oh, good catch. I noticed that paragraph before, but somehow it didn't
click.

> I'm sure this _could_ be made to work, but I wonder if it is worth the
> trouble. If it's not going to work, though, I think we'd want to detect
> the situation and complain, at least for now. And likewise the
> documentation needs to make clear it only works with --all and --glob.
> 
> Sorry to have misled in my initial suggestion to turn --visible-refs
> into --exclude-hidden. However, I do still stand by that suggestion.
> Even if we don't make it work with "--branches" now, the user-visible
> framework is still there, so it becomes a matter of extending the
> implementation later, rather than re-designing the options.

No worries, and I agree that it's ultimately still the better route to
use `--exclude-hidden=`. The things that don't work right now just go to
show that there is still some potential here and that it's overall the
more flexible design. And as you say, this is something we can build on
at a later point if any such usecases come up.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 5/6] rev-parse: add `--exclude-hidden=` option

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2934 bytes --]

Add a new `--exclude-hidden=` option that is similar to the one we just
added to git-rev-list(1). Given a seciton name `uploadpack` or `receive`
as argument, it causes us to exclude all references that would be hidden
by the respective `$section.hideRefs` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-rev-parse.txt |  7 +++++++
 builtin/rev-parse.c             |  4 ++++
 t/t6018-rev-list-glob.sh        | 11 +++++++++++
 3 files changed, 22 insertions(+)

diff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt
index 6b8ca085aa..393aa6e982 100644
--- a/Documentation/git-rev-parse.txt
+++ b/Documentation/git-rev-parse.txt
@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[receive|uploadpack]::
+	Do not include refs that have been hidden via either one of
+	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
+	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
+	would otherwise consider. This option is cleared when seeing one of
+	these pseudo-refs.
+
 --disambiguate=<prefix>::
 	Show every object whose name begins with the given prefix.
 	The <prefix> must be at least 4 hexadecimal digits long to
diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 7fa5b6991b..49730c7a23 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -895,6 +895,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				add_ref_exclusion(&ref_excludes, arg);
 				continue;
 			}
+			if (skip_prefix(arg, "--exclude-hidden=", &arg)) {
+				exclude_hidden_refs(&ref_excludes, arg);
+				continue;
+			}
 			if (!strcmp(arg, "--show-toplevel")) {
 				const char *work_tree = get_git_work_tree();
 				if (work_tree)
diff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh
index e1abc5c2b3..af0c55cbe7 100755
--- a/t/t6018-rev-list-glob.sh
+++ b/t/t6018-rev-list-glob.sh
@@ -187,6 +187,17 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '
 	compare rev-parse "--exclude=upstream/x --remotes=upstream/*" "upstream/one upstream/two"
 '
 
+for section in receive uploadpack
+do
+	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
+		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--exclude-hidden=$section --all" "--branches --tags"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
+		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude-hidden=$section --all" "--exclude=refs/heads/subspace/* --all"
+	'
+done
+
 test_expect_success 'rev-list --exclude=glob with --branches=glob' '
 	compare rev-list "--exclude=subspace-* --branches=sub*" "subspace/one subspace/two"
 '
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 6/6] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4675 bytes --]

When serving a push, git-receive-pack(1) needs to verify that the
packfile sent by the client contains all objects that are required by
the updated references. This connectivity check works by marking all
preexisting references as uninteresting and using the new reference tips
as starting point for a graph walk.

Marking all preexisting references as uninteresting can be a problem
when it comes to performance. Git forges tend to do internal bookkeeping
to keep alive sets of objects for internal use or make them easy to find
via certain references. These references are typically hidden away from
the user so that they are neither advertised nor writeable. At GitLab,
we have one particular repository that contains a total of 7 million
references, of which 6.8 million are indeed internal references. With
the current connectivity check we are forced to load all these
references in order to mark them as uninteresting, and this alone takes
around 15 seconds to compute.

We can optimize this by only taking into account the set of visible refs
when marking objects as uninteresting. This means that we may now walk
more objects until we hit any object that is marked as uninteresting.
But it is rather unlikely that clients send objects that make large
parts of objects reachable that have previously only ever been hidden,
whereas the common case is to push incremental changes that build on top
of the visible object graph.

This provides a huge boost to performance in the mentioned repository,
where the vast majority of its refs hidden. Pushing a new commit into
this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
as it is configured in Gitaly leads to a 4.5-fold speedup:

    Benchmark 1: main
      Time (mean ± σ):     30.977 s ±  0.157 s    [User: 30.226 s, System: 1.083 s]
      Range (min … max):   30.796 s … 31.071 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):      6.799 s ±  0.063 s    [User: 6.803 s, System: 0.354 s]
      Range (min … max):    6.729 s …  6.850 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        4.56 ± 0.05 times faster than 'main'

As we mostly go through the same codepaths even in the case where there
are no hidden refs at all compared to the code before there is no change
in performance when no refs are hidden:

    Benchmark 1: main
      Time (mean ± σ):     48.188 s ±  0.432 s    [User: 49.326 s, System: 5.009 s]
      Range (min … max):   47.706 s … 48.539 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):     48.027 s ±  0.500 s    [User: 48.934 s, System: 5.025 s]
      Range (min … max):   47.504 s … 48.500 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        1.00 ± 0.01 times faster than 'main'

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c | 2 ++
 connected.c            | 3 +++
 connected.h            | 7 +++++++
 3 files changed, 12 insertions(+)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 1f3efc58fb..77ab40f123 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,
 	opt.err_fd = err_fd;
 	opt.progress = err_fd && !quiet;
 	opt.env = tmp_objdir_env(tmp_objdir);
+	opt.exclude_hidden_refs_section = "receive";
+
 	if (check_connected(iterate_receive_command_list, &data, &opt))
 		set_connectivity_errors(commands, si);
 
diff --git a/connected.c b/connected.c
index 74a20cb32e..4f6388eed7 100644
--- a/connected.c
+++ b/connected.c
@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 		strvec_push(&rev_list.args, "--exclude-promisor-objects");
 	if (!opt->is_deepening_fetch) {
 		strvec_push(&rev_list.args, "--not");
+		if (opt->exclude_hidden_refs_section)
+			strvec_pushf(&rev_list.args, "--exclude-hidden=%s",
+				     opt->exclude_hidden_refs_section);
 		strvec_push(&rev_list.args, "--all");
 	}
 	strvec_push(&rev_list.args, "--quiet");
diff --git a/connected.h b/connected.h
index 6e59c92aa3..16b2c84f2e 100644
--- a/connected.h
+++ b/connected.h
@@ -46,6 +46,13 @@ struct check_connected_options {
 	 * during a fetch.
 	 */
 	unsigned is_deepening_fetch : 1;
+
+	/*
+	 * If not NULL, use `--exclude-hidden=$section` to exclude all refs
+	 * hidden via the `$section.hideRefs` config from the set of
+	 * already-reachable refs.
+	 */
+	const char *exclude_hidden_refs_section;
 };
 
 #define CHECK_CONNECTED_INIT { 0 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 0/7] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 17012 bytes --]

Hi,

this is the fifth version of my patch series that tries to improve
performance of the connectivity check by only considering preexisting
refs as uninteresting that could actually have been advertised to the
client.

Changes compared to v4:

    - Split out the memory leak fix when parsing hidden refs into a
      separate commit 1/7.

    - Fixed calls to `string_list_clear()` to not free the `util` field
      of `hidden_refs`.

    - Updated the documentation of the new `--exclude-hidden=` option to
      hopefully be easier to understand.

    - We now return an error when `--exclude-hidden=` is used with
      either one of `--branches`, `--tags` or `--remotes`.

    - Fixed a bug where we didn't bail when `--exclude-hidden=` was
      passed multiple times when there are no hidden refs.

    - Extended test coverage.

Patrick

Patrick Steinhardt (7):
  refs: fix memory leak when parsing hideRefs config
  refs: get rid of global list of hidden refs
  revision: move together exclusion-related functions
  revision: introduce struct to handle exclusions
  revision: add new parameter to exclude hidden refs
  rev-parse: add `--exclude-hidden=` option
  receive-pack: only use visible refs for connectivity check

 Documentation/git-rev-parse.txt    |   7 ++
 Documentation/rev-list-options.txt |   7 ++
 builtin/receive-pack.c             |  10 +-
 builtin/rev-list.c                 |   1 +
 builtin/rev-parse.c                |  18 +++-
 connected.c                        |   3 +
 connected.h                        |   7 ++
 ls-refs.c                          |  13 ++-
 refs.c                             |  16 +--
 refs.h                             |   5 +-
 revision.c                         | 131 +++++++++++++++--------
 revision.h                         |  43 ++++++--
 t/t6018-rev-list-glob.sh           |  40 +++++++
 t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++
 upload-pack.c                      |  30 +++---
 15 files changed, 411 insertions(+), 83 deletions(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

Range-diff against v4:
-:  ---------- > 1:  cfab8ba1a2 refs: fix memory leak when parsing hideRefs config
1:  34afe30d60 ! 2:  d8118c6dd8 refs: get rid of global list of hidden refs
    @@ builtin/receive-pack.c: int cmd_receive_pack(int argc, const char **argv, const
      		packet_flush(1);
      	oid_array_clear(&shallow);
      	oid_array_clear(&ref);
    -+	string_list_clear(&hidden_refs, 1);
    ++	string_list_clear(&hidden_refs, 0);
      	free((void *)push_cert_nonce);
      	return 0;
      }
    @@ ls-refs.c: int ls_refs(struct repository *r, struct packet_reader *request)
      	packet_fflush(stdout);
      	strvec_clear(&data.prefixes);
      	strbuf_release(&data.buf);
    -+	string_list_clear(&data.hidden_refs, 1);
    ++	string_list_clear(&data.hidden_refs, 0);
      	return 0;
      }
      
    @@ refs.c: int parse_hide_refs_config(const char *var, const char *value, const cha
     -			CALLOC_ARRAY(hide_refs, 1);
     -			hide_refs->strdup_strings = 1;
     -		}
    --		string_list_append(hide_refs, ref);
    -+		string_list_append_nodup(hide_refs, ref);
    + 		string_list_append_nodup(hide_refs, ref);
      	}
      	return 0;
      }
    @@ upload-pack.c: static void upload_pack_data_clear(struct upload_pack_data *data)
      {
      	string_list_clear(&data->symref, 1);
      	string_list_clear(&data->wanted_refs, 1);
    -+	string_list_clear(&data->hidden_refs, 1);
    ++	string_list_clear(&data->hidden_refs, 0);
      	object_array_clear(&data->want_obj);
      	object_array_clear(&data->have_obj);
      	oid_array_clear(&data->haves);
2:  b4f21d0a80 = 3:  93a627fb7f revision: move together exclusion-related functions
3:  265b292ed5 = 4:  ad41ade332 revision: introduce struct to handle exclusions
4:  c7fa6698db ! 5:  b5a4ce432a revision: add new parameter to exclude hidden refs
    @@ Documentation/rev-list-options.txt: respectively, and they must begin with `refs
      explicitly.
      
     +--exclude-hidden=[receive|uploadpack]::
    -+	Do not include refs that have been hidden via either one of
    -+	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
    -+	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
    -+	would otherwise consider. This option is cleared when seeing one of
    -+	these pseudo-refs.
    ++	Do not include refs that would be hidden by `git-receive-pack` or
    ++	`git-upload-pack` by consulting the appropriate `receive.hideRefs` or
    ++	`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see
    ++	linkgit:git-config[1]). This option affects the next pseudo-ref option
    ++	`--all` or `--glob` and is cleared after processing them.
     +
      --reflog::
      	Pretend as if all objects mentioned by reflogs are listed on the
    @@ revision.c: void init_ref_exclusions(struct ref_exclusions *exclusions)
      {
      	string_list_clear(&exclusions->excluded_refs, 0);
     +	string_list_clear(&exclusions->hidden_refs, 0);
    ++	exclusions->hidden_refs_configured = 0;
      }
      
      void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
    @@ revision.c: void add_ref_exclusion(struct ref_exclusions *exclusions, const char
     +static int hide_refs_config(const char *var, const char *value, void *cb_data)
     +{
     +	struct exclude_hidden_refs_cb *cb = cb_data;
    ++	cb->exclusions->hidden_refs_configured = 1;
     +	return parse_hide_refs_config(var, value, cb->section,
     +				      &cb->exclusions->hidden_refs);
     +}
    @@ revision.c: void add_ref_exclusion(struct ref_exclusions *exclusions, const char
     +	if (strcmp(section, "receive") && strcmp(section, "uploadpack"))
     +		die(_("unsupported section for hidden refs: %s"), section);
     +
    -+	if (exclusions->hidden_refs.nr)
    ++	if (exclusions->hidden_refs_configured)
     +		die(_("--exclude-hidden= passed more than once"));
     +
     +	cb.exclusions = exclusions;
    @@ revision.c: static int handle_revision_opt(struct rev_info *revs, int argc, cons
      	    starts_with(arg, "--branches=") || starts_with(arg, "--tags=") ||
      	    starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk="))
      	{
    +@@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,
    + 		}
    + 		clear_ref_exclusions(&revs->ref_excludes);
    + 	} else if (!strcmp(arg, "--branches")) {
    ++		if (revs->ref_excludes.hidden_refs_configured)
    ++			return error(_("--exclude-hidden cannot be used together with --branches"));
    + 		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
    + 		clear_ref_exclusions(&revs->ref_excludes);
    + 	} else if (!strcmp(arg, "--bisect")) {
    +@@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,
    + 			    for_each_good_bisect_ref);
    + 		revs->bisect = 1;
    + 	} else if (!strcmp(arg, "--tags")) {
    ++		if (revs->ref_excludes.hidden_refs_configured)
    ++			return error(_("--exclude-hidden cannot be used together with --tags"));
    + 		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
    + 		clear_ref_exclusions(&revs->ref_excludes);
    + 	} else if (!strcmp(arg, "--remotes")) {
    ++		if (revs->ref_excludes.hidden_refs_configured)
    ++			return error(_("--exclude-hidden cannot be used together with --remotes"));
    + 		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
    + 		clear_ref_exclusions(&revs->ref_excludes);
    + 	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
     @@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,
      	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
      		add_ref_exclusion(&revs->ref_excludes, optarg);
    @@ revision.c: static int handle_revision_pseudo_opt(struct rev_info *revs,
     +		return argcount;
      	} else if (skip_prefix(arg, "--branches=", &optarg)) {
      		struct all_refs_cb cb;
    ++		if (revs->ref_excludes.hidden_refs_configured)
    ++			return error(_("--exclude-hidden cannot be used together with --branches"));
      		init_all_refs_cb(&cb, revs, *flags);
    + 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
    + 		clear_ref_exclusions(&revs->ref_excludes);
    + 	} else if (skip_prefix(arg, "--tags=", &optarg)) {
    + 		struct all_refs_cb cb;
    ++		if (revs->ref_excludes.hidden_refs_configured)
    ++			return error(_("--exclude-hidden cannot be used together with --tags"));
    + 		init_all_refs_cb(&cb, revs, *flags);
    + 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
    + 		clear_ref_exclusions(&revs->ref_excludes);
    + 	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
    + 		struct all_refs_cb cb;
    ++		if (revs->ref_excludes.hidden_refs_configured)
    ++			return error(_("--exclude-hidden cannot be used together with --remotes"));
    + 		init_all_refs_cb(&cb, revs, *flags);
    + 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
    + 		clear_ref_exclusions(&revs->ref_excludes);
     
      ## revision.h ##
     @@ revision.h: struct ref_exclusions {
    @@ revision.h: struct ref_exclusions {
     +	 * `ref_is_hidden()`.
     +	 */
     +	struct string_list hidden_refs;
    ++
    ++	/*
    ++	 * Indicates whether hidden refs have been configured. This is to
    ++	 * distinguish between no hidden refs existing and hidden refs not
    ++	 * being parsed.
    ++	 */
    ++	char hidden_refs_configured;
      };
      
      /**
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +do
     +	test_expect_success "$section: passed multiple times" '
     +		echo "fatal: --exclude-hidden= passed more than once" >expected &&
    -+		test_must_fail git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&
    ++		test_must_fail git rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&
     +		test_cmp expected err
     +	'
     +
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +		test_cmp expected out
     +	'
     +
    ++	test_expect_success "$section: excluded hidden refs can be used with multiple pseudo-refs" '
    ++		git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&
    ++		test_must_be_empty out
    ++	'
    ++
    ++	test_expect_success "$section: works with --glob" '
    ++		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --glob=refs/h* >out &&
    ++		cat >expected <<-EOF &&
    ++		$COMMIT
    ++		EOF
    ++		test_cmp expected out
    ++	'
    ++
     +	test_expect_success "$section: operates on stripped refs by default" '
     +		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&
     +		cat >expected <<-EOF &&
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +		EOF
     +		test_cmp expected out
     +	'
    ++
    ++	for pseudoopt in remotes branches tags
    ++	do
    ++		test_expect_success "$section: fails with --$pseudoopt" '
    ++			test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt 2>err &&
    ++			test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err
    ++		'
    ++
    ++		test_expect_success "$section: fails with --$pseudoopt=pattern" '
    ++			test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt=pattern 2>err &&
    ++			test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err
    ++		'
    ++	done
     +done
     +
     +test_done
5:  79c5c64a80 ! 6:  2eeb25eef0 rev-parse: add `--exclude-hidden=` option
    @@ Documentation/git-rev-parse.txt: respectively, and they must begin with `refs/`
      explicitly.
      
     +--exclude-hidden=[receive|uploadpack]::
    -+	Do not include refs that have been hidden via either one of
    -+	`receive.hideRefs` or `uploadpack.hideRefs` (see linkgit:git-config[1])
    -+	that the next `--all`, `--branches`, `--tags`, `--remotes` or `--glob`
    -+	would otherwise consider. This option is cleared when seeing one of
    -+	these pseudo-refs.
    ++	Do not include refs that would be hidden by `git-receive-pack` or
    ++	`git-upload-pack` by consulting the appropriate `receive.hideRefs` or
    ++	`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see
    ++	linkgit:git-config[1]). This option affects the next pseudo-ref option
    ++	`--all` or `--glob` and is cleared after processing them.
     +
      --disambiguate=<prefix>::
      	Show every object whose name begins with the given prefix.
      	The <prefix> must be at least 4 hexadecimal digits long to
     
      ## builtin/rev-parse.c ##
    +@@ builtin/rev-parse.c: int cmd_rev_parse(int argc, const char **argv, const char *prefix)
    + 				continue;
    + 			}
    + 			if (opt_with_value(arg, "--branches", &arg)) {
    ++				if (ref_excludes.hidden_refs_configured)
    ++					return error(_("--exclude-hidden cannot be used together with --branches"));
    + 				handle_ref_opt(arg, "refs/heads/");
    + 				continue;
    + 			}
    + 			if (opt_with_value(arg, "--tags", &arg)) {
    ++				if (ref_excludes.hidden_refs_configured)
    ++					return error(_("--exclude-hidden cannot be used together with --tags"));
    + 				handle_ref_opt(arg, "refs/tags/");
    + 				continue;
    + 			}
    +@@ builtin/rev-parse.c: int cmd_rev_parse(int argc, const char **argv, const char *prefix)
    + 				continue;
    + 			}
    + 			if (opt_with_value(arg, "--remotes", &arg)) {
    ++				if (ref_excludes.hidden_refs_configured)
    ++					return error(_("--exclude-hidden cannot be used together with --remotes"));
    + 				handle_ref_opt(arg, "refs/remotes/");
    + 				continue;
    + 			}
     @@ builtin/rev-parse.c: int cmd_rev_parse(int argc, const char **argv, const char *prefix)
      				add_ref_exclusion(&ref_excludes, arg);
      				continue;
    @@ t/t6018-rev-list-glob.sh: test_expect_success 'rev-parse --exclude=ref with --re
     +for section in receive uploadpack
     +do
     +	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
    -+		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--exclude-hidden=$section --all" "--branches --tags"
    ++		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags" "--exclude-hidden=$section --all"
     +	'
     +
     +	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
    -+		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude-hidden=$section --all" "--exclude=refs/heads/subspace/* --all"
    ++		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --all" "--exclude-hidden=$section --all"
     +	'
    ++
    ++	test_expect_success "rev-parse --exclude-hidden=$section with --glob" '
    ++		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --glob=refs/heads/*" "--exclude-hidden=$section --glob=refs/heads/*"
    ++	'
    ++
    ++	test_expect_success "rev-parse --exclude-hidden=$section can be passed once per pseudo-ref" '
    ++		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags --branches --tags" "--exclude-hidden=$section --all --exclude-hidden=$section --all"
    ++	'
    ++
    ++	test_expect_success "rev-parse --exclude-hidden=$section can only be passed once per pseudo-ref" '
    ++		echo "fatal: --exclude-hidden= passed more than once" >expected &&
    ++		test_must_fail git rev-parse --exclude-hidden=$section --exclude-hidden=$section 2>err &&
    ++		test_cmp expected err
    ++	'
    ++
    ++	for pseudoopt in branches tags remotes
    ++	do
    ++		test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt" '
    ++			echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected &&
    ++			test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt 2>err &&
    ++			test_cmp expected err
    ++		'
    ++
    ++		test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt=pattern" '
    ++			echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected &&
    ++			test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt=pattern 2>err &&
    ++			test_cmp expected err
    ++		'
    ++	done
     +done
     +
      test_expect_success 'rev-list --exclude=glob with --branches=glob' '
6:  39b4741734 = 7:  f5f18f3939 receive-pack: only use visible refs for connectivity check
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 1/7] refs: fix memory leak when parsing hideRefs config

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

When parsing the hideRefs configuration, we first duplicate the config
value so that we can modify it. We then subsequently append it to the
`hide_refs` string list, which is initialized with `strdup_strings`
enabled. As a consequence we again reallocate the string, but never
free the first duplicate and thus have a memory leak.

While we never clean up the static `hide_refs` variable anyway, this is
no excuse to make the leak worse by leaking every value twice. We are
also about to change the way this variable will be handled so that we do
indeed start to clean it up. So let's fix the memory leak by using the
`string_list_append_nodup()` so that we pass ownership of the allocated
string to `hide_refs`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 refs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/refs.c b/refs.c
index 1491ae937e..a4ab264d74 100644
--- a/refs.c
+++ b/refs.c
@@ -1435,7 +1435,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 			CALLOC_ARRAY(hide_refs, 1);
 			hide_refs->strdup_strings = 1;
 		}
-		string_list_append(hide_refs, ref);
+		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 2/7] refs: get rid of global list of hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 12504 bytes --]

We're about to add a new argument to git-rev-list(1) that allows it to
add all references that are visible when taking `transfer.hideRefs` et
al into account. This will require us to potentially parse multiple sets
of hidden refs, which is not easily possible right now as there is only
a single, global instance of the list of parsed hidden refs.

Refactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both
take the list of hidden references as input and adjust callers to keep a
local list, instead. This allows us to easily use multiple hidden-ref
lists. Furthermore, it allows us to properly free this list before we
exit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c |  8 +++++---
 ls-refs.c              | 13 +++++++++----
 refs.c                 | 14 ++++----------
 refs.h                 |  5 +++--
 upload-pack.c          | 30 ++++++++++++++++++------------
 5 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 44bcea3a5b..1e24b31a0a 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)
 
 static int receive_pack_config(const char *var, const char *value, void *cb)
 {
-	int status = parse_hide_refs_config(var, value, "receive");
+	int status = parse_hide_refs_config(var, value, "receive", &hidden_refs);
 
 	if (status)
 		return status;
@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,
 	struct oidset *seen = data;
 	const char *path = strip_namespace(path_full);
 
-	if (ref_is_hidden(path, path_full))
+	if (ref_is_hidden(path, path_full, &hidden_refs))
 		return 0;
 
 	/*
@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)
 		strbuf_setlen(&refname_full, prefix_len);
 		strbuf_addstr(&refname_full, cmd->ref_name);
 
-		if (!ref_is_hidden(cmd->ref_name, refname_full.buf))
+		if (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))
 			continue;
 		if (is_null_oid(&cmd->new_oid))
 			cmd->error_string = "deny deleting a hidden ref";
@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
+	string_list_clear(&hidden_refs, 0);
 	free((void *)push_cert_nonce);
 	return 0;
 }
diff --git a/ls-refs.c b/ls-refs.c
index fa0d01b47c..fb6769742c 100644
--- a/ls-refs.c
+++ b/ls-refs.c
@@ -6,6 +6,7 @@
 #include "ls-refs.h"
 #include "pkt-line.h"
 #include "config.h"
+#include "string-list.h"
 
 static int config_read;
 static int advertise_unborn;
@@ -73,6 +74,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
+	struct string_list hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 
 	strbuf_reset(&data->buf);
 
-	if (ref_is_hidden(refname_nons, refname))
+	if (ref_is_hidden(refname_nons, refname, &data->hidden_refs))
 		return 0;
 
 	if (!ref_match(&data->prefixes, refname_nons))
@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)
 }
 
 static int ls_refs_config(const char *var, const char *value,
-			  void *data UNUSED)
+			  void *cb_data)
 {
+	struct ls_refs_data *data = cb_data;
 	/*
 	 * We only serve fetches over v2 for now, so respect only "uploadpack"
 	 * config. This may need to eventually be expanded to "receive", but we
 	 * don't yet know how that information will be passed to ls-refs.
 	 */
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 int ls_refs(struct repository *r, struct packet_reader *request)
@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
+	string_list_init_dup(&data.hidden_refs);
 
 	ensure_config_read();
-	git_config(ls_refs_config, NULL);
+	git_config(ls_refs_config, &data);
 
 	while (packet_reader_read(request) == PACKET_READ_NORMAL) {
 		const char *arg = request->line;
@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
+	string_list_clear(&data.hidden_refs, 0);
 	return 0;
 }
 
diff --git a/refs.c b/refs.c
index a4ab264d74..2c7e88b190 100644
--- a/refs.c
+++ b/refs.c
@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 					    refname, strict);
 }
 
-static struct string_list *hide_refs;
-
-int parse_hide_refs_config(const char *var, const char *value, const char *section)
+int parse_hide_refs_config(const char *var, const char *value, const char *section,
+			   struct string_list *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		if (!hide_refs) {
-			CALLOC_ARRAY(hide_refs, 1);
-			hide_refs->strdup_strings = 1;
-		}
 		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
 
-int ref_is_hidden(const char *refname, const char *refname_full)
+int ref_is_hidden(const char *refname, const char *refname_full,
+		  const struct string_list *hide_refs)
 {
 	int i;
 
-	if (!hide_refs)
-		return 0;
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
 		const char *match = hide_refs->items[i].string;
 		const char *subject;
diff --git a/refs.h b/refs.h
index 8958717a17..3266fd8f57 100644
--- a/refs.h
+++ b/refs.h
@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *new_oid, const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr);
 
-int parse_hide_refs_config(const char *var, const char *value, const char *);
+int parse_hide_refs_config(const char *var, const char *value, const char *,
+			   struct string_list *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *);
+int ref_is_hidden(const char *, const char *, const struct string_list *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
diff --git a/upload-pack.c b/upload-pack.c
index 0b8311bd68..551f22ffa5 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -62,6 +62,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
+	struct string_list hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
+	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 	memset(data, 0, sizeof(*data));
 	data->symref = symref;
 	data->wanted_refs = wanted_refs;
+	data->hidden_refs = hidden_refs;
 	data->want_obj = want_obj;
 	data->have_obj = have_obj;
 	data->haves = haves;
@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
+	string_list_clear(&data->hidden_refs, 0);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * Checking for reachable shallows requires that our refs be
 		 * marked with OUR_REF.
 		 */
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, data);
+		for_each_namespaced_ref(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid)
+			const struct object_id *oid, const struct string_list *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
-	if (ref_is_hidden(refname, refname_full)) {
+	if (ref_is_hidden(refname, refname_full, hidden_refs)) {
 		o->flags |= HIDDEN_REF;
 		return 1;
 	}
@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,
 }
 
 static int check_ref(const char *refname_full, const struct object_id *oid,
-		     int flag UNUSED, void *cb_data UNUSED)
+		     int flag UNUSED, void *cb_data)
 {
 	const char *refname = strip_namespace(refname_full);
+	struct upload_pack_data *data = cb_data;
 
-	mark_our_ref(refname, refname_full, oid);
+	mark_our_ref(refname, refname_full, oid, &data->hidden_refs);
 	return 0;
 }
 
@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 	struct object_id peeled;
 	struct upload_pack_data *data = cb_data;
 
-	if (mark_our_ref(refname_nons, refname, oid))
+	if (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))
 		return 0;
 
 	if (capabilities) {
@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)
 	if (parse_object_filter_config(var, value, data) < 0)
 		return -1;
 
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)
@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		advertise_shallow_grafts(1);
 		packet_flush(1);
 	} else {
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, &data);
+		for_each_namespaced_ref(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
+			  struct string_list *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 		struct strbuf refname = STRBUF_INIT;
 
 		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
-		if (ref_is_hidden(refname_nons, refname.buf) ||
+		if (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||
 		    read_ref(refname.buf, &oid)) {
 			packet_writer_error(writer, "unknown ref %s", refname_nons);
 			die("unknown ref %s", refname_nons);
@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,
 			continue;
 		if (data->allow_ref_in_want &&
 		    parse_want_ref(&data->writer, arg, &data->wanted_refs,
-				   &data->want_obj))
+				   &data->hidden_refs, &data->want_obj))
 			continue;
 		/* process have line */
 		if (parse_have(arg, &data->haves))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 3/7] revision: move together exclusion-related functions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2402 bytes --]

Move together the definitions of functions that handle exclusions of
refs so that related functionality sits in a single place, only.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 revision.c | 52 ++++++++++++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/revision.c b/revision.c
index 0760e78936..be755670e2 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-struct all_refs_cb {
-	int all_flags;
-	int warned_bad_reflog;
-	struct rev_info *all_revs;
-	const char *name_for_errormsg;
-	struct worktree *wt;
-};
-
 int ref_excluded(struct string_list *ref_excludes, const char *path)
 {
 	struct string_list_item *item;
@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)
 	return 0;
 }
 
+void clear_ref_exclusion(struct string_list **ref_excludes_p)
+{
+	if (*ref_excludes_p) {
+		string_list_clear(*ref_excludes_p, 0);
+		free(*ref_excludes_p);
+	}
+	*ref_excludes_p = NULL;
+}
+
+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+{
+	if (!*ref_excludes_p) {
+		CALLOC_ARRAY(*ref_excludes_p, 1);
+		(*ref_excludes_p)->strdup_strings = 1;
+	}
+	string_list_append(*ref_excludes_p, exclude);
+}
+
+struct all_refs_cb {
+	int all_flags;
+	int warned_bad_reflog;
+	struct rev_info *all_revs;
+	const char *name_for_errormsg;
+	struct worktree *wt;
+};
+
 static int handle_one_ref(const char *path, const struct object_id *oid,
 			  int flag UNUSED,
 			  void *cb_data)
@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,
 	cb->wt = NULL;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
-{
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
-}
-
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
-{
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
-}
-
 static void handle_refs(struct ref_store *refs,
 			struct rev_info *revs, unsigned flags,
 			int (*for_each)(struct ref_store *, each_ref_fn, void *))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 4/7] revision: introduce struct to handle exclusions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8642 bytes --]

The functions that handle exclusion of refs work on a single string
list. We're about to add a second mechanism for excluding refs though,
and it makes sense to reuse much of the same architecture for both kinds
of exclusion.

Introduce a new `struct ref_exclusions` that encapsulates all the logic
related to excluding refs and move the `struct string_list` that holds
all wildmatch patterns of excluded refs into it. Rename functions that
operate on this struct to match its name.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/rev-parse.c |  8 ++++----
 revision.c          | 48 +++++++++++++++++++++------------------------
 revision.h          | 27 +++++++++++++++++++------
 3 files changed, 47 insertions(+), 36 deletions(-)

diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 8f61050bde..7fa5b6991b 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -39,7 +39,7 @@ static int abbrev_ref_strict;
 static int output_sq;
 
 static int stuck_long;
-static struct string_list *ref_excludes;
+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;
 
 /*
  * Some arguments are relevant "revision" arguments,
@@ -198,7 +198,7 @@ static int show_default(void)
 static int show_reference(const char *refname, const struct object_id *oid,
 			  int flag UNUSED, void *cb_data UNUSED)
 {
-	if (ref_excluded(ref_excludes, refname))
+	if (ref_excluded(&ref_excludes, refname))
 		return 0;
 	show_rev(NORMAL, oid, refname);
 	return 0;
@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)
 		for_each_glob_ref_in(show_reference, pattern, prefix, NULL);
 	else
 		for_each_ref_in(prefix, show_reference, NULL);
-	clear_ref_exclusion(&ref_excludes);
+	clear_ref_exclusions(&ref_excludes);
 }
 
 enum format_type {
@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 			}
 			if (!strcmp(arg, "--all")) {
 				for_each_ref(show_reference, NULL);
-				clear_ref_exclusion(&ref_excludes);
+				clear_ref_exclusions(&ref_excludes);
 				continue;
 			}
 			if (skip_prefix(arg, "--disambiguate=", &arg)) {
diff --git a/revision.c b/revision.c
index be755670e2..fe3ec98f46 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,35 +1517,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-int ref_excluded(struct string_list *ref_excludes, const char *path)
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
 	struct string_list_item *item;
-
-	if (!ref_excludes)
-		return 0;
-	for_each_string_list_item(item, ref_excludes) {
+	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
 	return 0;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
+void init_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
+	struct ref_exclusions blank = REF_EXCLUSIONS_INIT;
+	memcpy(exclusions, &blank, sizeof(*exclusions));
 }
 
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
+	string_list_clear(&exclusions->excluded_refs, 0);
+}
+
+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
+{
+	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
 struct all_refs_cb {
@@ -1563,7 +1558,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,
 	struct all_refs_cb *cb = cb_data;
 	struct object *object;
 
-	if (ref_excluded(cb->all_revs->ref_excludes, path))
+	if (ref_excluded(&cb->all_revs->ref_excludes, path))
 	    return 0;
 
 	object = get_reference(cb->all_revs, path, oid, cb->all_flags);
@@ -1901,6 +1896,7 @@ void repo_init_revisions(struct repository *r,
 
 	init_display_notes(&revs->notes_opt);
 	list_objects_filter_init(&revs->filter);
+	init_ref_exclusions(&revs->ref_excludes);
 }
 
 static void add_pending_commit_list(struct rev_info *revs,
@@ -2689,10 +2685,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 			init_all_refs_cb(&cb, revs, *flags);
 			other_head_refs(handle_one_ref, &cb);
 		}
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--branches")) {
 		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--bisect")) {
 		read_bisect_terms(&term_bad, &term_good);
 		handle_refs(refs, revs, *flags, for_each_bad_bisect_ref);
@@ -2701,15 +2697,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		revs->bisect = 1;
 	} else if (!strcmp(arg, "--tags")) {
 		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--remotes")) {
 		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref(handle_one_ref, optarg, &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 		return argcount;
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
@@ -2718,17 +2714,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--tags=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--reflog")) {
 		add_reflogs_to_pending(revs, *flags);
 	} else if (!strcmp(arg, "--indexed-objects")) {
diff --git a/revision.h b/revision.h
index afe1b77985..5c8ab16047 100644
--- a/revision.h
+++ b/revision.h
@@ -81,6 +81,21 @@ struct rev_cmdline_info {
 	} *rev;
 };
 
+struct ref_exclusions {
+	/*
+	 * Excluded refs is a list of wildmatch patterns. If any of the
+	 * patterns matches, the reference will be excluded.
+	 */
+	struct string_list excluded_refs;
+};
+
+/**
+ * Initialize a `struct ref_exclusions` with a macro.
+ */
+#define REF_EXCLUSIONS_INIT { \
+	.excluded_refs = STRING_LIST_INIT_DUP, \
+}
+
 struct oidset;
 struct topo_walk_info;
 
@@ -103,7 +118,7 @@ struct rev_info {
 	struct list_objects_filter_options filter;
 
 	/* excluding from --branches, --refs, etc. expansion */
-	struct string_list *ref_excludes;
+	struct ref_exclusions ref_excludes;
 
 	/* Basic information */
 	const char *prefix;
@@ -439,12 +454,12 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)
 void show_object_with_name(FILE *, struct object *, const char *);
 
 /**
- * Helpers to check if a "struct string_list" item matches with
- * wildmatch().
+ * Helpers to check if a reference should be excluded.
  */
-int ref_excluded(struct string_list *, const char *path);
-void clear_ref_exclusion(struct string_list **);
-void add_ref_exclusion(struct string_list **, const char *exclude);
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
+void init_ref_exclusions(struct ref_exclusions *);
+void clear_ref_exclusions(struct ref_exclusions *);
+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
 
 /**
  * This function can be used if you want to add commit objects as revision
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 5/7] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 14480 bytes --]

Users can optionally hide refs from remote users in git-upload-pack(1),
git-receive-pack(1) and others via the `transfer.hideRefs`, but there is
not an easy way to obtain the list of all visible or hidden refs right
now. We'll require just that though for a performance improvement in our
connectivity check.

Add a new option `--exclude-hidden=` that excludes any hidden refs from
the next pseudo-ref like `--all` or `--branches`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/rev-list-options.txt |   7 ++
 builtin/rev-list.c                 |   1 +
 revision.c                         |  55 +++++++++-
 revision.h                         |  16 +++
 t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++
 5 files changed, 241 insertions(+), 1 deletion(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
index 1837509566..ff68e48406 100644
--- a/Documentation/rev-list-options.txt
+++ b/Documentation/rev-list-options.txt
@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[receive|uploadpack]::
+	Do not include refs that would be hidden by `git-receive-pack` or
+	`git-upload-pack` by consulting the appropriate `receive.hideRefs` or
+	`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see
+	linkgit:git-config[1]). This option affects the next pseudo-ref option
+	`--all` or `--glob` and is cleared after processing them.
+
 --reflog::
 	Pretend as if all objects mentioned by reflogs are listed on the
 	command line as `<commit>`.
diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index 3acd93f71e..d42db0b0cc 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -38,6 +38,7 @@ static const char rev_list_usage[] =
 "    --tags\n"
 "    --remotes\n"
 "    --stdin\n"
+"    --exclude-hidden=[receive|uploadpack]\n"
 "    --quiet\n"
 "  ordering output:\n"
 "    --topo-order\n"
diff --git a/revision.c b/revision.c
index fe3ec98f46..bc32fb819a 100644
--- a/revision.c
+++ b/revision.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "config.h"
 #include "object-store.h"
 #include "tag.h"
 #include "blob.h"
@@ -1519,11 +1520,17 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
+	const char *stripped_path = strip_namespace(path);
 	struct string_list_item *item;
+
 	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
+
+	if (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))
+		return 1;
+
 	return 0;
 }
 
@@ -1536,6 +1543,8 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
+	string_list_clear(&exclusions->hidden_refs, 0);
+	exclusions->hidden_refs_configured = 0;
 }
 
 void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
@@ -1543,6 +1552,35 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
 	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
+struct exclude_hidden_refs_cb {
+	struct ref_exclusions *exclusions;
+	const char *section;
+};
+
+static int hide_refs_config(const char *var, const char *value, void *cb_data)
+{
+	struct exclude_hidden_refs_cb *cb = cb_data;
+	cb->exclusions->hidden_refs_configured = 1;
+	return parse_hide_refs_config(var, value, cb->section,
+				      &cb->exclusions->hidden_refs);
+}
+
+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)
+{
+	struct exclude_hidden_refs_cb cb;
+
+	if (strcmp(section, "receive") && strcmp(section, "uploadpack"))
+		die(_("unsupported section for hidden refs: %s"), section);
+
+	if (exclusions->hidden_refs_configured)
+		die(_("--exclude-hidden= passed more than once"));
+
+	cb.exclusions = exclusions;
+	cb.section = section;
+
+	git_config(hide_refs_config, &cb);
+}
+
 struct all_refs_cb {
 	int all_flags;
 	int warned_bad_reflog;
@@ -2221,7 +2259,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg
 	    !strcmp(arg, "--bisect") || starts_with(arg, "--glob=") ||
 	    !strcmp(arg, "--indexed-objects") ||
 	    !strcmp(arg, "--alternate-refs") ||
-	    starts_with(arg, "--exclude=") ||
+	    starts_with(arg, "--exclude=") || starts_with(arg, "--exclude-hidden=") ||
 	    starts_with(arg, "--branches=") || starts_with(arg, "--tags=") ||
 	    starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk="))
 	{
@@ -2687,6 +2725,8 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		}
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--branches")) {
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --branches"));
 		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--bisect")) {
@@ -2696,9 +2736,13 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 			    for_each_good_bisect_ref);
 		revs->bisect = 1;
 	} else if (!strcmp(arg, "--tags")) {
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --tags"));
 		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--remotes")) {
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --remotes"));
 		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
@@ -2710,18 +2754,27 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
 		return argcount;
+	} else if ((argcount = parse_long_opt("exclude-hidden", argv, &optarg))) {
+		exclude_hidden_refs(&revs->ref_excludes, optarg);
+		return argcount;
 	} else if (skip_prefix(arg, "--branches=", &optarg)) {
 		struct all_refs_cb cb;
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --branches"));
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--tags=", &optarg)) {
 		struct all_refs_cb cb;
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --tags"));
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
 		struct all_refs_cb cb;
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --remotes"));
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
 		clear_ref_exclusions(&revs->ref_excludes);
diff --git a/revision.h b/revision.h
index 5c8ab16047..adb810c2f8 100644
--- a/revision.h
+++ b/revision.h
@@ -87,6 +87,19 @@ struct ref_exclusions {
 	 * patterns matches, the reference will be excluded.
 	 */
 	struct string_list excluded_refs;
+
+	/*
+	 * Hidden refs is a list of patterns that is to be hidden via
+	 * `ref_is_hidden()`.
+	 */
+	struct string_list hidden_refs;
+
+	/*
+	 * Indicates whether hidden refs have been configured. This is to
+	 * distinguish between no hidden refs existing and hidden refs not
+	 * being parsed.
+	 */
+	char hidden_refs_configured;
 };
 
 /**
@@ -94,6 +107,7 @@ struct ref_exclusions {
  */
 #define REF_EXCLUSIONS_INIT { \
 	.excluded_refs = STRING_LIST_INIT_DUP, \
+	.hidden_refs = STRING_LIST_INIT_DUP, \
 }
 
 struct oidset;
@@ -456,10 +470,12 @@ void show_object_with_name(FILE *, struct object *, const char *);
 /**
  * Helpers to check if a reference should be excluded.
  */
+
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
 void init_ref_exclusions(struct ref_exclusions *);
 void clear_ref_exclusions(struct ref_exclusions *);
 void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
+void exclude_hidden_refs(struct ref_exclusions *, const char *section);
 
 /**
  * This function can be used if you want to add commit objects as revision
diff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh
new file mode 100755
index 0000000000..018796d41c
--- /dev/null
+++ b/t/t6021-rev-list-exclude-hidden.sh
@@ -0,0 +1,163 @@
+#!/bin/sh
+
+test_description='git rev-list --exclude-hidden test'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit_bulk --id=commit --ref=refs/heads/main 1 &&
+	COMMIT=$(git rev-parse refs/heads/main) &&
+	test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&
+	TAG=$(git rev-parse refs/tags/lightweight) &&
+	test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&
+	HIDDEN=$(git rev-parse refs/hidden/commit) &&
+	test_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&
+	NAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)
+'
+
+test_expect_success 'invalid section' '
+	echo "fatal: unsupported section for hidden refs: unsupported" >expected &&
+	test_must_fail git rev-list --exclude-hidden=unsupported 2>err &&
+	test_cmp expected err
+'
+
+for section in receive uploadpack
+do
+	test_expect_success "$section: passed multiple times" '
+		echo "fatal: --exclude-hidden= passed more than once" >expected &&
+		test_must_fail git rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&
+		test_cmp expected err
+	'
+
+	test_expect_success "$section: without hiddenRefs" '
+		git rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden via transfer.hideRefs" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden via $section.hideRefs" '
+		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: respects both transfer.hideRefs and $section.hideRefs" '
+		git -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: negation without hidden refs marks everything as uninteresting" '
+		git rev-list --all --exclude-hidden=$section --not --all >out &&
+		test_must_be_empty out
+	'
+
+	test_expect_success "$section: negation with hidden refs marks them as interesting" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden refs and excludes work together" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: excluded hidden refs get reset" '
+		git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: excluded hidden refs can be used with multiple pseudo-refs" '
+		git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&
+		test_must_be_empty out
+	'
+
+	test_expect_success "$section: works with --glob" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --glob=refs/h* >out &&
+		cat >expected <<-EOF &&
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: operates on stripped refs by default" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: does not hide namespace by default" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: can operate on unstripped refs" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	for pseudoopt in remotes branches tags
+	do
+		test_expect_success "$section: fails with --$pseudoopt" '
+			test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt 2>err &&
+			test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err
+		'
+
+		test_expect_success "$section: fails with --$pseudoopt=pattern" '
+			test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt=pattern 2>err &&
+			test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err
+		'
+	done
+done
+
+test_done
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 6/7] rev-parse: add `--exclude-hidden=` option

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 5452 bytes --]

Add a new `--exclude-hidden=` option that is similar to the one we just
added to git-rev-list(1). Given a seciton name `uploadpack` or `receive`
as argument, it causes us to exclude all references that would be hidden
by the respective `$section.hideRefs` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-rev-parse.txt |  7 ++++++
 builtin/rev-parse.c             | 10 +++++++++
 t/t6018-rev-list-glob.sh        | 40 +++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+)

diff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt
index 6b8ca085aa..bcd8069287 100644
--- a/Documentation/git-rev-parse.txt
+++ b/Documentation/git-rev-parse.txt
@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[receive|uploadpack]::
+	Do not include refs that would be hidden by `git-receive-pack` or
+	`git-upload-pack` by consulting the appropriate `receive.hideRefs` or
+	`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see
+	linkgit:git-config[1]). This option affects the next pseudo-ref option
+	`--all` or `--glob` and is cleared after processing them.
+
 --disambiguate=<prefix>::
 	Show every object whose name begins with the given prefix.
 	The <prefix> must be at least 4 hexadecimal digits long to
diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 7fa5b6991b..b5666a03bd 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -876,10 +876,14 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (opt_with_value(arg, "--branches", &arg)) {
+				if (ref_excludes.hidden_refs_configured)
+					return error(_("--exclude-hidden cannot be used together with --branches"));
 				handle_ref_opt(arg, "refs/heads/");
 				continue;
 			}
 			if (opt_with_value(arg, "--tags", &arg)) {
+				if (ref_excludes.hidden_refs_configured)
+					return error(_("--exclude-hidden cannot be used together with --tags"));
 				handle_ref_opt(arg, "refs/tags/");
 				continue;
 			}
@@ -888,6 +892,8 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (opt_with_value(arg, "--remotes", &arg)) {
+				if (ref_excludes.hidden_refs_configured)
+					return error(_("--exclude-hidden cannot be used together with --remotes"));
 				handle_ref_opt(arg, "refs/remotes/");
 				continue;
 			}
@@ -895,6 +901,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				add_ref_exclusion(&ref_excludes, arg);
 				continue;
 			}
+			if (skip_prefix(arg, "--exclude-hidden=", &arg)) {
+				exclude_hidden_refs(&ref_excludes, arg);
+				continue;
+			}
 			if (!strcmp(arg, "--show-toplevel")) {
 				const char *work_tree = get_git_work_tree();
 				if (work_tree)
diff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh
index e1abc5c2b3..aabf590dda 100755
--- a/t/t6018-rev-list-glob.sh
+++ b/t/t6018-rev-list-glob.sh
@@ -187,6 +187,46 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '
 	compare rev-parse "--exclude=upstream/x --remotes=upstream/*" "upstream/one upstream/two"
 '
 
+for section in receive uploadpack
+do
+	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
+		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags" "--exclude-hidden=$section --all"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
+		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --all" "--exclude-hidden=$section --all"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section with --glob" '
+		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --glob=refs/heads/*" "--exclude-hidden=$section --glob=refs/heads/*"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section can be passed once per pseudo-ref" '
+		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags --branches --tags" "--exclude-hidden=$section --all --exclude-hidden=$section --all"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section can only be passed once per pseudo-ref" '
+		echo "fatal: --exclude-hidden= passed more than once" >expected &&
+		test_must_fail git rev-parse --exclude-hidden=$section --exclude-hidden=$section 2>err &&
+		test_cmp expected err
+	'
+
+	for pseudoopt in branches tags remotes
+	do
+		test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt" '
+			echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected &&
+			test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt 2>err &&
+			test_cmp expected err
+		'
+
+		test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt=pattern" '
+			echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected &&
+			test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt=pattern 2>err &&
+			test_cmp expected err
+		'
+	done
+done
+
 test_expect_success 'rev-list --exclude=glob with --branches=glob' '
 	compare rev-list "--exclude=subspace-* --branches=sub*" "subspace/one subspace/two"
 '
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v5 7/7] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4675 bytes --]

When serving a push, git-receive-pack(1) needs to verify that the
packfile sent by the client contains all objects that are required by
the updated references. This connectivity check works by marking all
preexisting references as uninteresting and using the new reference tips
as starting point for a graph walk.

Marking all preexisting references as uninteresting can be a problem
when it comes to performance. Git forges tend to do internal bookkeeping
to keep alive sets of objects for internal use or make them easy to find
via certain references. These references are typically hidden away from
the user so that they are neither advertised nor writeable. At GitLab,
we have one particular repository that contains a total of 7 million
references, of which 6.8 million are indeed internal references. With
the current connectivity check we are forced to load all these
references in order to mark them as uninteresting, and this alone takes
around 15 seconds to compute.

We can optimize this by only taking into account the set of visible refs
when marking objects as uninteresting. This means that we may now walk
more objects until we hit any object that is marked as uninteresting.
But it is rather unlikely that clients send objects that make large
parts of objects reachable that have previously only ever been hidden,
whereas the common case is to push incremental changes that build on top
of the visible object graph.

This provides a huge boost to performance in the mentioned repository,
where the vast majority of its refs hidden. Pushing a new commit into
this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
as it is configured in Gitaly leads to a 4.5-fold speedup:

    Benchmark 1: main
      Time (mean ± σ):     30.977 s ±  0.157 s    [User: 30.226 s, System: 1.083 s]
      Range (min … max):   30.796 s … 31.071 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):      6.799 s ±  0.063 s    [User: 6.803 s, System: 0.354 s]
      Range (min … max):    6.729 s …  6.850 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        4.56 ± 0.05 times faster than 'main'

As we mostly go through the same codepaths even in the case where there
are no hidden refs at all compared to the code before there is no change
in performance when no refs are hidden:

    Benchmark 1: main
      Time (mean ± σ):     48.188 s ±  0.432 s    [User: 49.326 s, System: 5.009 s]
      Range (min … max):   47.706 s … 48.539 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):     48.027 s ±  0.500 s    [User: 48.934 s, System: 5.025 s]
      Range (min … max):   47.504 s … 48.500 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        1.00 ± 0.01 times faster than 'main'

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c | 2 ++
 connected.c            | 3 +++
 connected.h            | 7 +++++++
 3 files changed, 12 insertions(+)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 1e24b31a0a..a90af30363 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,
 	opt.err_fd = err_fd;
 	opt.progress = err_fd && !quiet;
 	opt.env = tmp_objdir_env(tmp_objdir);
+	opt.exclude_hidden_refs_section = "receive";
+
 	if (check_connected(iterate_receive_command_list, &data, &opt))
 		set_connectivity_errors(commands, si);
 
diff --git a/connected.c b/connected.c
index 74a20cb32e..4f6388eed7 100644
--- a/connected.c
+++ b/connected.c
@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 		strvec_push(&rev_list.args, "--exclude-promisor-objects");
 	if (!opt->is_deepening_fetch) {
 		strvec_push(&rev_list.args, "--not");
+		if (opt->exclude_hidden_refs_section)
+			strvec_pushf(&rev_list.args, "--exclude-hidden=%s",
+				     opt->exclude_hidden_refs_section);
 		strvec_push(&rev_list.args, "--all");
 	}
 	strvec_push(&rev_list.args, "--quiet");
diff --git a/connected.h b/connected.h
index 6e59c92aa3..16b2c84f2e 100644
--- a/connected.h
+++ b/connected.h
@@ -46,6 +46,13 @@ struct check_connected_options {
 	 * during a fetch.
 	 */
 	unsigned is_deepening_fetch : 1;
+
+	/*
+	 * If not NULL, use `--exclude-hidden=$section` to exclude all refs
+	 * hidden via the `$section.hideRefs` config from the set of
+	 * already-reachable refs.
+	 */
+	const char *exclude_hidden_refs_section;
 };
 
 #define CHECK_CONNECTED_INIT { 0 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check

[Taylor Blau]

On Fri, Nov 11, 2022 at 07:49:49AM +0100, Patrick Steinhardt wrote:
> Patrick Steinhardt (7):
>   refs: fix memory leak when parsing hideRefs config
>   refs: get rid of global list of hidden refs
>   revision: move together exclusion-related functions
>   revision: introduce struct to handle exclusions
>   revision: add new parameter to exclude hidden refs
>   rev-parse: add `--exclude-hidden=` option
>   receive-pack: only use visible refs for connectivity check

The new version is looking pretty good to my eyes, though I would like
to hear from Peff, too, since he caught a few things that I missed in
the previous rounds.

I have to say, the final patch is *really* nice ;-).

Thanks,
Taylor

Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check

[Jeff King]

On Fri, Nov 11, 2022 at 05:18:23PM -0500, Taylor Blau wrote:

> On Fri, Nov 11, 2022 at 07:49:49AM +0100, Patrick Steinhardt wrote:
> > Patrick Steinhardt (7):
> >   refs: fix memory leak when parsing hideRefs config
> >   refs: get rid of global list of hidden refs
> >   revision: move together exclusion-related functions
> >   revision: introduce struct to handle exclusions
> >   revision: add new parameter to exclude hidden refs
> >   rev-parse: add `--exclude-hidden=` option
> >   receive-pack: only use visible refs for connectivity check
> 
> The new version is looking pretty good to my eyes, though I would like
> to hear from Peff, too, since he caught a few things that I missed in
> the previous rounds.

This looks good to me, too. There's a typo (s/seciton/section/) in the
commit message of patch 6, but definitely not worth a re-roll. :)

I admit I didn't think _too_ deeply about the interaction with
namespaces, and whether there are any corner cases. I was happy to see
the tests there, and I assume in writing them that Patrick matched how
receive-pack, etc, behave.

-Peff

Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check

[Taylor Blau]

On Tue, Nov 15, 2022 at 12:26:25PM -0500, Jeff King wrote:
> On Fri, Nov 11, 2022 at 05:18:23PM -0500, Taylor Blau wrote:
>
> > On Fri, Nov 11, 2022 at 07:49:49AM +0100, Patrick Steinhardt wrote:
> > > Patrick Steinhardt (7):
> > >   refs: fix memory leak when parsing hideRefs config
> > >   refs: get rid of global list of hidden refs
> > >   revision: move together exclusion-related functions
> > >   revision: introduce struct to handle exclusions
> > >   revision: add new parameter to exclude hidden refs
> > >   rev-parse: add `--exclude-hidden=` option
> > >   receive-pack: only use visible refs for connectivity check
> >
> > The new version is looking pretty good to my eyes, though I would like
> > to hear from Peff, too, since he caught a few things that I missed in
> > the previous rounds.
>
> This looks good to me, too. There's a typo (s/seciton/section/) in the
> commit message of patch 6, but definitely not worth a re-roll. :)

Hmm. It looks like this is broken in CI when the default initial branch
name is something other than "master":

    $ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main ./t6021-rev-list-exclude-hidden.sh -i --verbose-only=12 -x
    [...]
    expecting success of 6021.12 'receive: excluded hidden refs can be used with multiple pseudo-refs':
        git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&
        test_must_be_empty out

    + git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=receive --all --exclude-hidden=receive --all
    + test_must_be_empty out
    + test 1 -ne 1
    + test_path_is_file out
    + test 1 -ne 1
    + test -f out
    + test -s out
    + echo 'out' is not empty, it contains:
    'out' is not empty, it contains:
    + cat out
    d2e88f5a45c63e4ec7e1fd303542944487abe89a
    + return 1
    error: last command exited with $?=1
    not ok 12 - receive: excluded hidden refs can be used with multiple pseudo-refs
    #
    #			git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&
    #			test_must_be_empty out
    #
    1..12

I haven't looked too deeply at what is going on here, but let's make
sure to resolve this before graduating the topic down (which I would
otherwise like to do in the next push-out, probably tomorrow or the next
day).

Thanks,
Taylor

Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check

[Jeff King]

On Wed, Nov 16, 2022 at 04:22:24PM -0500, Taylor Blau wrote:

> > This looks good to me, too. There's a typo (s/seciton/section/) in the
> > commit message of patch 6, but definitely not worth a re-roll. :)
> 
> Hmm. It looks like this is broken in CI when the default initial branch
> name is something other than "master":
> 
>     $ GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main ./t6021-rev-list-exclude-hidden.sh -i --verbose-only=12 -x
>     [...]
>     expecting success of 6021.12 'receive: excluded hidden refs can be used with multiple pseudo-refs':
>         git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&
>         test_must_be_empty out
> 
>     + git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=receive --all --exclude-hidden=receive --all
>     + test_must_be_empty out
>     + test 1 -ne 1
>     + test_path_is_file out
>     + test 1 -ne 1
>     + test -f out
>     + test -s out
>     + echo 'out' is not empty, it contains:
>     'out' is not empty, it contains:
>     + cat out
>     d2e88f5a45c63e4ec7e1fd303542944487abe89a
>     + return 1
>     error: last command exited with $?=1
>     not ok 12 - receive: excluded hidden refs can be used with multiple pseudo-refs
>     #
>     #			git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&
>     #			test_must_be_empty out
>     #
>     1..12
> 
> I haven't looked too deeply at what is going on here, but let's make
> sure to resolve this before graduating the topic down (which I would
> otherwise like to do in the next push-out, probably tomorrow or the next
> day).

The issue is that some of the tests assume that hiding "refs/" should
produce no output from "--exclude-hidden=receive --all". But it will
also show HEAD, even if it points to a hidden ref (which I think is OK,
and matches what receive-pack would do).

But because the setup uses "main" as one of the sample refs, HEAD may or
may not be valid, depending on what it points to (without setting
GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME it points to master, which is
unborn).

So the fix is just:

diff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh
index 018796d41c..1543a93fe0 100755
--- a/t/t6021-rev-list-exclude-hidden.sh
+++ b/t/t6021-rev-list-exclude-hidden.sh
@@ -5,8 +5,8 @@ test_description='git rev-list --exclude-hidden test'
 . ./test-lib.sh
 
 test_expect_success 'setup' '
-	test_commit_bulk --id=commit --ref=refs/heads/main 1 &&
-	COMMIT=$(git rev-parse refs/heads/main) &&
+	test_commit_bulk --id=commit --ref=refs/heads/foo 1 &&
+	COMMIT=$(git rev-parse refs/heads/foo) &&
 	test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&
 	TAG=$(git rev-parse refs/tags/lightweight) &&
 	test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&

but Patrick may want to select a more meaningful name. :)

Notably I don't think we want to do the usual

  GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
  export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME

at the top of the script. We really don't mean to depend on having a
specific branch that HEAD points to here.

-Peff

Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check

[Taylor Blau]

On Wed, Nov 16, 2022 at 05:04:10PM -0500, Jeff King wrote:
> > I haven't looked too deeply at what is going on here, but let's make
> > sure to resolve this before graduating the topic down (which I would
> > otherwise like to do in the next push-out, probably tomorrow or the next
> > day).
>
> The issue is that some of the tests assume that hiding "refs/" should
> produce no output from "--exclude-hidden=receive --all". But it will
> also show HEAD, even if it points to a hidden ref (which I think is OK,
> and matches what receive-pack would do).
>
> But because the setup uses "main" as one of the sample refs, HEAD may or
> may not be valid, depending on what it points to (without setting
> GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME it points to master, which is
> unborn).
>
> So the fix is just:
>
> [...]

Makes perfect sense, and thanks for looking into it.

Patrick: it sounds like there was one typo in the earlier round which
you may want to pick up also, assuming you reroll this.

Thanks,
Taylor

Re: [PATCH v5 0/7] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

On Wed, Nov 16, 2022 at 05:33:12PM -0500, Taylor Blau wrote:
> On Wed, Nov 16, 2022 at 05:04:10PM -0500, Jeff King wrote:
> > > I haven't looked too deeply at what is going on here, but let's make
> > > sure to resolve this before graduating the topic down (which I would
> > > otherwise like to do in the next push-out, probably tomorrow or the next
> > > day).
> >
> > The issue is that some of the tests assume that hiding "refs/" should
> > produce no output from "--exclude-hidden=receive --all". But it will
> > also show HEAD, even if it points to a hidden ref (which I think is OK,
> > and matches what receive-pack would do).
> >
> > But because the setup uses "main" as one of the sample refs, HEAD may or
> > may not be valid, depending on what it points to (without setting
> > GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME it points to master, which is
> > unborn).
> >
> > So the fix is just:
> >
> > [...]
> 
> Makes perfect sense, and thanks for looking into it.
> 
> Patrick: it sounds like there was one typo in the earlier round which
> you may want to pick up also, assuming you reroll this.
> 
> Thanks,
> Taylor

Thanks to both of you, I'll send out v6 in a bit.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 0/7] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3380 bytes --]

Hi,

this is the sixth version of my patch series that tries to improve
performance of the connectivity check by only considering preexisting
refs as uninteresting that could actually have been advertised to the
client.

There are only two changes in this version compared to v5:

    - A fix to the test setup in commit 5/7 so that tests pass when
      GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main.

    - A typo fix in the commit message of patch 6/7.

Patrick

Patrick Steinhardt (7):
  refs: fix memory leak when parsing hideRefs config
  refs: get rid of global list of hidden refs
  revision: move together exclusion-related functions
  revision: introduce struct to handle exclusions
  revision: add new parameter to exclude hidden refs
  rev-parse: add `--exclude-hidden=` option
  receive-pack: only use visible refs for connectivity check

 Documentation/git-rev-parse.txt    |   7 ++
 Documentation/rev-list-options.txt |   7 ++
 builtin/receive-pack.c             |  10 +-
 builtin/rev-list.c                 |   1 +
 builtin/rev-parse.c                |  18 +++-
 connected.c                        |   3 +
 connected.h                        |   7 ++
 ls-refs.c                          |  13 ++-
 refs.c                             |  16 +--
 refs.h                             |   5 +-
 revision.c                         | 131 +++++++++++++++--------
 revision.h                         |  43 ++++++--
 t/t6018-rev-list-glob.sh           |  40 +++++++
 t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++
 upload-pack.c                      |  30 +++---
 15 files changed, 411 insertions(+), 83 deletions(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

Range-diff against v5:
1:  cfab8ba1a2 = 1:  ef182e4330 refs: fix memory leak when parsing hideRefs config
2:  d8118c6dd8 = 2:  48913c1493 refs: get rid of global list of hidden refs
3:  93a627fb7f = 3:  3827d6a2fc revision: move together exclusion-related functions
4:  ad41ade332 = 4:  805de80e64 revision: introduce struct to handle exclusions
5:  b5a4ce432a ! 5:  d86a3342f6 revision: add new parameter to exclude hidden refs
    @@ t/t6021-rev-list-exclude-hidden.sh (new)
     +. ./test-lib.sh
     +
     +test_expect_success 'setup' '
    -+	test_commit_bulk --id=commit --ref=refs/heads/main 1 &&
    -+	COMMIT=$(git rev-parse refs/heads/main) &&
    ++	test_commit_bulk --id=commit --ref=refs/heads/branch 1 &&
    ++	COMMIT=$(git rev-parse refs/heads/branch) &&
     +	test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&
     +	TAG=$(git rev-parse refs/tags/lightweight) &&
     +	test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&
6:  2eeb25eef0 ! 6:  f8b5eb5a7e rev-parse: add `--exclude-hidden=` option
    @@ Commit message
         rev-parse: add `--exclude-hidden=` option
     
         Add a new `--exclude-hidden=` option that is similar to the one we just
    -    added to git-rev-list(1). Given a seciton name `uploadpack` or `receive`
    +    added to git-rev-list(1). Given a section name `uploadpack` or `receive`
         as argument, it causes us to exclude all references that would be hidden
         by the respective `$section.hideRefs` configuration.
     
7:  f5f18f3939 = 7:  a7eae80ff3 receive-pack: only use visible refs for connectivity check
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 1/7] refs: fix memory leak when parsing hideRefs config

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1242 bytes --]

When parsing the hideRefs configuration, we first duplicate the config
value so that we can modify it. We then subsequently append it to the
`hide_refs` string list, which is initialized with `strdup_strings`
enabled. As a consequence we again reallocate the string, but never
free the first duplicate and thus have a memory leak.

While we never clean up the static `hide_refs` variable anyway, this is
no excuse to make the leak worse by leaking every value twice. We are
also about to change the way this variable will be handled so that we do
indeed start to clean it up. So let's fix the memory leak by using the
`string_list_append_nodup()` so that we pass ownership of the allocated
string to `hide_refs`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 refs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/refs.c b/refs.c
index 1491ae937e..a4ab264d74 100644
--- a/refs.c
+++ b/refs.c
@@ -1435,7 +1435,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 			CALLOC_ARRAY(hide_refs, 1);
 			hide_refs->strdup_strings = 1;
 		}
-		string_list_append(hide_refs, ref);
+		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 2/7] refs: get rid of global list of hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 12504 bytes --]

We're about to add a new argument to git-rev-list(1) that allows it to
add all references that are visible when taking `transfer.hideRefs` et
al into account. This will require us to potentially parse multiple sets
of hidden refs, which is not easily possible right now as there is only
a single, global instance of the list of parsed hidden refs.

Refactor `parse_hide_refs_config()` and `ref_is_hidden()` so that both
take the list of hidden references as input and adjust callers to keep a
local list, instead. This allows us to easily use multiple hidden-ref
lists. Furthermore, it allows us to properly free this list before we
exit.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c |  8 +++++---
 ls-refs.c              | 13 +++++++++----
 refs.c                 | 14 ++++----------
 refs.h                 |  5 +++--
 upload-pack.c          | 30 ++++++++++++++++++------------
 5 files changed, 39 insertions(+), 31 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 44bcea3a5b..1e24b31a0a 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -80,6 +80,7 @@ static struct object_id push_cert_oid;
 static struct signature_check sigcheck;
 static const char *push_cert_nonce;
 static const char *cert_nonce_seed;
+static struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 
 static const char *NONCE_UNSOLICITED = "UNSOLICITED";
 static const char *NONCE_BAD = "BAD";
@@ -130,7 +131,7 @@ static enum deny_action parse_deny_action(const char *var, const char *value)
 
 static int receive_pack_config(const char *var, const char *value, void *cb)
 {
-	int status = parse_hide_refs_config(var, value, "receive");
+	int status = parse_hide_refs_config(var, value, "receive", &hidden_refs);
 
 	if (status)
 		return status;
@@ -296,7 +297,7 @@ static int show_ref_cb(const char *path_full, const struct object_id *oid,
 	struct oidset *seen = data;
 	const char *path = strip_namespace(path_full);
 
-	if (ref_is_hidden(path, path_full))
+	if (ref_is_hidden(path, path_full, &hidden_refs))
 		return 0;
 
 	/*
@@ -1794,7 +1795,7 @@ static void reject_updates_to_hidden(struct command *commands)
 		strbuf_setlen(&refname_full, prefix_len);
 		strbuf_addstr(&refname_full, cmd->ref_name);
 
-		if (!ref_is_hidden(cmd->ref_name, refname_full.buf))
+		if (!ref_is_hidden(cmd->ref_name, refname_full.buf, &hidden_refs))
 			continue;
 		if (is_null_oid(&cmd->new_oid))
 			cmd->error_string = "deny deleting a hidden ref";
@@ -2591,6 +2592,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 		packet_flush(1);
 	oid_array_clear(&shallow);
 	oid_array_clear(&ref);
+	string_list_clear(&hidden_refs, 0);
 	free((void *)push_cert_nonce);
 	return 0;
 }
diff --git a/ls-refs.c b/ls-refs.c
index fa0d01b47c..fb6769742c 100644
--- a/ls-refs.c
+++ b/ls-refs.c
@@ -6,6 +6,7 @@
 #include "ls-refs.h"
 #include "pkt-line.h"
 #include "config.h"
+#include "string-list.h"
 
 static int config_read;
 static int advertise_unborn;
@@ -73,6 +74,7 @@ struct ls_refs_data {
 	unsigned symrefs;
 	struct strvec prefixes;
 	struct strbuf buf;
+	struct string_list hidden_refs;
 	unsigned unborn : 1;
 };
 
@@ -84,7 +86,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 
 	strbuf_reset(&data->buf);
 
-	if (ref_is_hidden(refname_nons, refname))
+	if (ref_is_hidden(refname_nons, refname, &data->hidden_refs))
 		return 0;
 
 	if (!ref_match(&data->prefixes, refname_nons))
@@ -137,14 +139,15 @@ static void send_possibly_unborn_head(struct ls_refs_data *data)
 }
 
 static int ls_refs_config(const char *var, const char *value,
-			  void *data UNUSED)
+			  void *cb_data)
 {
+	struct ls_refs_data *data = cb_data;
 	/*
 	 * We only serve fetches over v2 for now, so respect only "uploadpack"
 	 * config. This may need to eventually be expanded to "receive", but we
 	 * don't yet know how that information will be passed to ls-refs.
 	 */
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 int ls_refs(struct repository *r, struct packet_reader *request)
@@ -154,9 +157,10 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	memset(&data, 0, sizeof(data));
 	strvec_init(&data.prefixes);
 	strbuf_init(&data.buf, 0);
+	string_list_init_dup(&data.hidden_refs);
 
 	ensure_config_read();
-	git_config(ls_refs_config, NULL);
+	git_config(ls_refs_config, &data);
 
 	while (packet_reader_read(request) == PACKET_READ_NORMAL) {
 		const char *arg = request->line;
@@ -195,6 +199,7 @@ int ls_refs(struct repository *r, struct packet_reader *request)
 	packet_fflush(stdout);
 	strvec_clear(&data.prefixes);
 	strbuf_release(&data.buf);
+	string_list_clear(&data.hidden_refs, 0);
 	return 0;
 }
 
diff --git a/refs.c b/refs.c
index a4ab264d74..2c7e88b190 100644
--- a/refs.c
+++ b/refs.c
@@ -1414,9 +1414,8 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 					    refname, strict);
 }
 
-static struct string_list *hide_refs;
-
-int parse_hide_refs_config(const char *var, const char *value, const char *section)
+int parse_hide_refs_config(const char *var, const char *value, const char *section,
+			   struct string_list *hide_refs)
 {
 	const char *key;
 	if (!strcmp("transfer.hiderefs", var) ||
@@ -1431,21 +1430,16 @@ int parse_hide_refs_config(const char *var, const char *value, const char *secti
 		len = strlen(ref);
 		while (len && ref[len - 1] == '/')
 			ref[--len] = '\0';
-		if (!hide_refs) {
-			CALLOC_ARRAY(hide_refs, 1);
-			hide_refs->strdup_strings = 1;
-		}
 		string_list_append_nodup(hide_refs, ref);
 	}
 	return 0;
 }
 
-int ref_is_hidden(const char *refname, const char *refname_full)
+int ref_is_hidden(const char *refname, const char *refname_full,
+		  const struct string_list *hide_refs)
 {
 	int i;
 
-	if (!hide_refs)
-		return 0;
 	for (i = hide_refs->nr - 1; i >= 0; i--) {
 		const char *match = hide_refs->items[i].string;
 		const char *subject;
diff --git a/refs.h b/refs.h
index 8958717a17..3266fd8f57 100644
--- a/refs.h
+++ b/refs.h
@@ -808,7 +808,8 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *new_oid, const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr);
 
-int parse_hide_refs_config(const char *var, const char *value, const char *);
+int parse_hide_refs_config(const char *var, const char *value, const char *,
+			   struct string_list *);
 
 /*
  * Check whether a ref is hidden. If no namespace is set, both the first and
@@ -818,7 +819,7 @@ int parse_hide_refs_config(const char *var, const char *value, const char *);
  * the ref is outside that namespace, the first parameter is NULL. The second
  * parameter always points to the full ref name.
  */
-int ref_is_hidden(const char *, const char *);
+int ref_is_hidden(const char *, const char *, const struct string_list *);
 
 /* Is this a per-worktree ref living in the refs/ namespace? */
 int is_per_worktree_ref(const char *refname);
diff --git a/upload-pack.c b/upload-pack.c
index 0b8311bd68..551f22ffa5 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -62,6 +62,7 @@ struct upload_pack_data {
 	struct object_array have_obj;
 	struct oid_array haves;					/* v2 only */
 	struct string_list wanted_refs;				/* v2 only */
+	struct string_list hidden_refs;
 
 	struct object_array shallows;
 	struct string_list deepen_not;
@@ -118,6 +119,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 {
 	struct string_list symref = STRING_LIST_INIT_DUP;
 	struct string_list wanted_refs = STRING_LIST_INIT_DUP;
+	struct string_list hidden_refs = STRING_LIST_INIT_DUP;
 	struct object_array want_obj = OBJECT_ARRAY_INIT;
 	struct object_array have_obj = OBJECT_ARRAY_INIT;
 	struct oid_array haves = OID_ARRAY_INIT;
@@ -130,6 +132,7 @@ static void upload_pack_data_init(struct upload_pack_data *data)
 	memset(data, 0, sizeof(*data));
 	data->symref = symref;
 	data->wanted_refs = wanted_refs;
+	data->hidden_refs = hidden_refs;
 	data->want_obj = want_obj;
 	data->have_obj = have_obj;
 	data->haves = haves;
@@ -151,6 +154,7 @@ static void upload_pack_data_clear(struct upload_pack_data *data)
 {
 	string_list_clear(&data->symref, 1);
 	string_list_clear(&data->wanted_refs, 1);
+	string_list_clear(&data->hidden_refs, 0);
 	object_array_clear(&data->want_obj);
 	object_array_clear(&data->have_obj);
 	oid_array_clear(&data->haves);
@@ -842,8 +846,8 @@ static void deepen(struct upload_pack_data *data, int depth)
 		 * Checking for reachable shallows requires that our refs be
 		 * marked with OUR_REF.
 		 */
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, data);
+		for_each_namespaced_ref(check_ref, data);
 
 		get_reachable_list(data, &reachable_shallows);
 		result = get_shallow_commits(&reachable_shallows,
@@ -1158,11 +1162,11 @@ static void receive_needs(struct upload_pack_data *data,
 
 /* return non-zero if the ref is hidden, otherwise 0 */
 static int mark_our_ref(const char *refname, const char *refname_full,
-			const struct object_id *oid)
+			const struct object_id *oid, const struct string_list *hidden_refs)
 {
 	struct object *o = lookup_unknown_object(the_repository, oid);
 
-	if (ref_is_hidden(refname, refname_full)) {
+	if (ref_is_hidden(refname, refname_full, hidden_refs)) {
 		o->flags |= HIDDEN_REF;
 		return 1;
 	}
@@ -1171,11 +1175,12 @@ static int mark_our_ref(const char *refname, const char *refname_full,
 }
 
 static int check_ref(const char *refname_full, const struct object_id *oid,
-		     int flag UNUSED, void *cb_data UNUSED)
+		     int flag UNUSED, void *cb_data)
 {
 	const char *refname = strip_namespace(refname_full);
+	struct upload_pack_data *data = cb_data;
 
-	mark_our_ref(refname, refname_full, oid);
+	mark_our_ref(refname, refname_full, oid, &data->hidden_refs);
 	return 0;
 }
 
@@ -1204,7 +1209,7 @@ static int send_ref(const char *refname, const struct object_id *oid,
 	struct object_id peeled;
 	struct upload_pack_data *data = cb_data;
 
-	if (mark_our_ref(refname_nons, refname, oid))
+	if (mark_our_ref(refname_nons, refname, oid, &data->hidden_refs))
 		return 0;
 
 	if (capabilities) {
@@ -1327,7 +1332,7 @@ static int upload_pack_config(const char *var, const char *value, void *cb_data)
 	if (parse_object_filter_config(var, value, data) < 0)
 		return -1;
 
-	return parse_hide_refs_config(var, value, "uploadpack");
+	return parse_hide_refs_config(var, value, "uploadpack", &data->hidden_refs);
 }
 
 static int upload_pack_protected_config(const char *var, const char *value, void *cb_data)
@@ -1375,8 +1380,8 @@ void upload_pack(const int advertise_refs, const int stateless_rpc,
 		advertise_shallow_grafts(1);
 		packet_flush(1);
 	} else {
-		head_ref_namespaced(check_ref, NULL);
-		for_each_namespaced_ref(check_ref, NULL);
+		head_ref_namespaced(check_ref, &data);
+		for_each_namespaced_ref(check_ref, &data);
 	}
 
 	if (!advertise_refs) {
@@ -1441,6 +1446,7 @@ static int parse_want(struct packet_writer *writer, const char *line,
 
 static int parse_want_ref(struct packet_writer *writer, const char *line,
 			  struct string_list *wanted_refs,
+			  struct string_list *hidden_refs,
 			  struct object_array *want_obj)
 {
 	const char *refname_nons;
@@ -1451,7 +1457,7 @@ static int parse_want_ref(struct packet_writer *writer, const char *line,
 		struct strbuf refname = STRBUF_INIT;
 
 		strbuf_addf(&refname, "%s%s", get_git_namespace(), refname_nons);
-		if (ref_is_hidden(refname_nons, refname.buf) ||
+		if (ref_is_hidden(refname_nons, refname.buf, hidden_refs) ||
 		    read_ref(refname.buf, &oid)) {
 			packet_writer_error(writer, "unknown ref %s", refname_nons);
 			die("unknown ref %s", refname_nons);
@@ -1508,7 +1514,7 @@ static void process_args(struct packet_reader *request,
 			continue;
 		if (data->allow_ref_in_want &&
 		    parse_want_ref(&data->writer, arg, &data->wanted_refs,
-				   &data->want_obj))
+				   &data->hidden_refs, &data->want_obj))
 			continue;
 		/* process have line */
 		if (parse_have(arg, &data->haves))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 3/7] revision: move together exclusion-related functions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2402 bytes --]

Move together the definitions of functions that handle exclusions of
refs so that related functionality sits in a single place, only.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 revision.c | 52 ++++++++++++++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 26 deletions(-)

diff --git a/revision.c b/revision.c
index 0760e78936..be755670e2 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,14 +1517,6 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-struct all_refs_cb {
-	int all_flags;
-	int warned_bad_reflog;
-	struct rev_info *all_revs;
-	const char *name_for_errormsg;
-	struct worktree *wt;
-};
-
 int ref_excluded(struct string_list *ref_excludes, const char *path)
 {
 	struct string_list_item *item;
@@ -1538,6 +1530,32 @@ int ref_excluded(struct string_list *ref_excludes, const char *path)
 	return 0;
 }
 
+void clear_ref_exclusion(struct string_list **ref_excludes_p)
+{
+	if (*ref_excludes_p) {
+		string_list_clear(*ref_excludes_p, 0);
+		free(*ref_excludes_p);
+	}
+	*ref_excludes_p = NULL;
+}
+
+void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+{
+	if (!*ref_excludes_p) {
+		CALLOC_ARRAY(*ref_excludes_p, 1);
+		(*ref_excludes_p)->strdup_strings = 1;
+	}
+	string_list_append(*ref_excludes_p, exclude);
+}
+
+struct all_refs_cb {
+	int all_flags;
+	int warned_bad_reflog;
+	struct rev_info *all_revs;
+	const char *name_for_errormsg;
+	struct worktree *wt;
+};
+
 static int handle_one_ref(const char *path, const struct object_id *oid,
 			  int flag UNUSED,
 			  void *cb_data)
@@ -1563,24 +1581,6 @@ static void init_all_refs_cb(struct all_refs_cb *cb, struct rev_info *revs,
 	cb->wt = NULL;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
-{
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
-}
-
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
-{
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
-}
-
 static void handle_refs(struct ref_store *refs,
 			struct rev_info *revs, unsigned flags,
 			int (*for_each)(struct ref_store *, each_ref_fn, void *))
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 4/7] revision: introduce struct to handle exclusions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8642 bytes --]

The functions that handle exclusion of refs work on a single string
list. We're about to add a second mechanism for excluding refs though,
and it makes sense to reuse much of the same architecture for both kinds
of exclusion.

Introduce a new `struct ref_exclusions` that encapsulates all the logic
related to excluding refs and move the `struct string_list` that holds
all wildmatch patterns of excluded refs into it. Rename functions that
operate on this struct to match its name.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/rev-parse.c |  8 ++++----
 revision.c          | 48 +++++++++++++++++++++------------------------
 revision.h          | 27 +++++++++++++++++++------
 3 files changed, 47 insertions(+), 36 deletions(-)

diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 8f61050bde..7fa5b6991b 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -39,7 +39,7 @@ static int abbrev_ref_strict;
 static int output_sq;
 
 static int stuck_long;
-static struct string_list *ref_excludes;
+static struct ref_exclusions ref_excludes = REF_EXCLUSIONS_INIT;
 
 /*
  * Some arguments are relevant "revision" arguments,
@@ -198,7 +198,7 @@ static int show_default(void)
 static int show_reference(const char *refname, const struct object_id *oid,
 			  int flag UNUSED, void *cb_data UNUSED)
 {
-	if (ref_excluded(ref_excludes, refname))
+	if (ref_excluded(&ref_excludes, refname))
 		return 0;
 	show_rev(NORMAL, oid, refname);
 	return 0;
@@ -585,7 +585,7 @@ static void handle_ref_opt(const char *pattern, const char *prefix)
 		for_each_glob_ref_in(show_reference, pattern, prefix, NULL);
 	else
 		for_each_ref_in(prefix, show_reference, NULL);
-	clear_ref_exclusion(&ref_excludes);
+	clear_ref_exclusions(&ref_excludes);
 }
 
 enum format_type {
@@ -863,7 +863,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 			}
 			if (!strcmp(arg, "--all")) {
 				for_each_ref(show_reference, NULL);
-				clear_ref_exclusion(&ref_excludes);
+				clear_ref_exclusions(&ref_excludes);
 				continue;
 			}
 			if (skip_prefix(arg, "--disambiguate=", &arg)) {
diff --git a/revision.c b/revision.c
index be755670e2..fe3ec98f46 100644
--- a/revision.c
+++ b/revision.c
@@ -1517,35 +1517,30 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 	}
 }
 
-int ref_excluded(struct string_list *ref_excludes, const char *path)
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
 	struct string_list_item *item;
-
-	if (!ref_excludes)
-		return 0;
-	for_each_string_list_item(item, ref_excludes) {
+	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
 	return 0;
 }
 
-void clear_ref_exclusion(struct string_list **ref_excludes_p)
+void init_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (*ref_excludes_p) {
-		string_list_clear(*ref_excludes_p, 0);
-		free(*ref_excludes_p);
-	}
-	*ref_excludes_p = NULL;
+	struct ref_exclusions blank = REF_EXCLUSIONS_INIT;
+	memcpy(exclusions, &blank, sizeof(*exclusions));
 }
 
-void add_ref_exclusion(struct string_list **ref_excludes_p, const char *exclude)
+void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
-	if (!*ref_excludes_p) {
-		CALLOC_ARRAY(*ref_excludes_p, 1);
-		(*ref_excludes_p)->strdup_strings = 1;
-	}
-	string_list_append(*ref_excludes_p, exclude);
+	string_list_clear(&exclusions->excluded_refs, 0);
+}
+
+void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
+{
+	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
 struct all_refs_cb {
@@ -1563,7 +1558,7 @@ static int handle_one_ref(const char *path, const struct object_id *oid,
 	struct all_refs_cb *cb = cb_data;
 	struct object *object;
 
-	if (ref_excluded(cb->all_revs->ref_excludes, path))
+	if (ref_excluded(&cb->all_revs->ref_excludes, path))
 	    return 0;
 
 	object = get_reference(cb->all_revs, path, oid, cb->all_flags);
@@ -1901,6 +1896,7 @@ void repo_init_revisions(struct repository *r,
 
 	init_display_notes(&revs->notes_opt);
 	list_objects_filter_init(&revs->filter);
+	init_ref_exclusions(&revs->ref_excludes);
 }
 
 static void add_pending_commit_list(struct rev_info *revs,
@@ -2689,10 +2685,10 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 			init_all_refs_cb(&cb, revs, *flags);
 			other_head_refs(handle_one_ref, &cb);
 		}
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--branches")) {
 		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--bisect")) {
 		read_bisect_terms(&term_bad, &term_good);
 		handle_refs(refs, revs, *flags, for_each_bad_bisect_ref);
@@ -2701,15 +2697,15 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		revs->bisect = 1;
 	} else if (!strcmp(arg, "--tags")) {
 		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--remotes")) {
 		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref(handle_one_ref, optarg, &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 		return argcount;
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
@@ -2718,17 +2714,17 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--tags=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
 		struct all_refs_cb cb;
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
-		clear_ref_exclusion(&revs->ref_excludes);
+		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--reflog")) {
 		add_reflogs_to_pending(revs, *flags);
 	} else if (!strcmp(arg, "--indexed-objects")) {
diff --git a/revision.h b/revision.h
index afe1b77985..5c8ab16047 100644
--- a/revision.h
+++ b/revision.h
@@ -81,6 +81,21 @@ struct rev_cmdline_info {
 	} *rev;
 };
 
+struct ref_exclusions {
+	/*
+	 * Excluded refs is a list of wildmatch patterns. If any of the
+	 * patterns matches, the reference will be excluded.
+	 */
+	struct string_list excluded_refs;
+};
+
+/**
+ * Initialize a `struct ref_exclusions` with a macro.
+ */
+#define REF_EXCLUSIONS_INIT { \
+	.excluded_refs = STRING_LIST_INIT_DUP, \
+}
+
 struct oidset;
 struct topo_walk_info;
 
@@ -103,7 +118,7 @@ struct rev_info {
 	struct list_objects_filter_options filter;
 
 	/* excluding from --branches, --refs, etc. expansion */
-	struct string_list *ref_excludes;
+	struct ref_exclusions ref_excludes;
 
 	/* Basic information */
 	const char *prefix;
@@ -439,12 +454,12 @@ void mark_trees_uninteresting_sparse(struct repository *r, struct oidset *trees)
 void show_object_with_name(FILE *, struct object *, const char *);
 
 /**
- * Helpers to check if a "struct string_list" item matches with
- * wildmatch().
+ * Helpers to check if a reference should be excluded.
  */
-int ref_excluded(struct string_list *, const char *path);
-void clear_ref_exclusion(struct string_list **);
-void add_ref_exclusion(struct string_list **, const char *exclude);
+int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
+void init_ref_exclusions(struct ref_exclusions *);
+void clear_ref_exclusions(struct ref_exclusions *);
+void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
 
 /**
  * This function can be used if you want to add commit objects as revision
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 5/7] revision: add new parameter to exclude hidden refs

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 14484 bytes --]

Users can optionally hide refs from remote users in git-upload-pack(1),
git-receive-pack(1) and others via the `transfer.hideRefs`, but there is
not an easy way to obtain the list of all visible or hidden refs right
now. We'll require just that though for a performance improvement in our
connectivity check.

Add a new option `--exclude-hidden=` that excludes any hidden refs from
the next pseudo-ref like `--all` or `--branches`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/rev-list-options.txt |   7 ++
 builtin/rev-list.c                 |   1 +
 revision.c                         |  55 +++++++++-
 revision.h                         |  16 +++
 t/t6021-rev-list-exclude-hidden.sh | 163 +++++++++++++++++++++++++++++
 5 files changed, 241 insertions(+), 1 deletion(-)
 create mode 100755 t/t6021-rev-list-exclude-hidden.sh

diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt
index 1837509566..ff68e48406 100644
--- a/Documentation/rev-list-options.txt
+++ b/Documentation/rev-list-options.txt
@@ -195,6 +195,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[receive|uploadpack]::
+	Do not include refs that would be hidden by `git-receive-pack` or
+	`git-upload-pack` by consulting the appropriate `receive.hideRefs` or
+	`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see
+	linkgit:git-config[1]). This option affects the next pseudo-ref option
+	`--all` or `--glob` and is cleared after processing them.
+
 --reflog::
 	Pretend as if all objects mentioned by reflogs are listed on the
 	command line as `<commit>`.
diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index 3acd93f71e..d42db0b0cc 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -38,6 +38,7 @@ static const char rev_list_usage[] =
 "    --tags\n"
 "    --remotes\n"
 "    --stdin\n"
+"    --exclude-hidden=[receive|uploadpack]\n"
 "    --quiet\n"
 "  ordering output:\n"
 "    --topo-order\n"
diff --git a/revision.c b/revision.c
index fe3ec98f46..bc32fb819a 100644
--- a/revision.c
+++ b/revision.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "config.h"
 #include "object-store.h"
 #include "tag.h"
 #include "blob.h"
@@ -1519,11 +1520,17 @@ static void add_rev_cmdline_list(struct rev_info *revs,
 
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path)
 {
+	const char *stripped_path = strip_namespace(path);
 	struct string_list_item *item;
+
 	for_each_string_list_item(item, &exclusions->excluded_refs) {
 		if (!wildmatch(item->string, path, 0))
 			return 1;
 	}
+
+	if (ref_is_hidden(stripped_path, path, &exclusions->hidden_refs))
+		return 1;
+
 	return 0;
 }
 
@@ -1536,6 +1543,8 @@ void init_ref_exclusions(struct ref_exclusions *exclusions)
 void clear_ref_exclusions(struct ref_exclusions *exclusions)
 {
 	string_list_clear(&exclusions->excluded_refs, 0);
+	string_list_clear(&exclusions->hidden_refs, 0);
+	exclusions->hidden_refs_configured = 0;
 }
 
 void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
@@ -1543,6 +1552,35 @@ void add_ref_exclusion(struct ref_exclusions *exclusions, const char *exclude)
 	string_list_append(&exclusions->excluded_refs, exclude);
 }
 
+struct exclude_hidden_refs_cb {
+	struct ref_exclusions *exclusions;
+	const char *section;
+};
+
+static int hide_refs_config(const char *var, const char *value, void *cb_data)
+{
+	struct exclude_hidden_refs_cb *cb = cb_data;
+	cb->exclusions->hidden_refs_configured = 1;
+	return parse_hide_refs_config(var, value, cb->section,
+				      &cb->exclusions->hidden_refs);
+}
+
+void exclude_hidden_refs(struct ref_exclusions *exclusions, const char *section)
+{
+	struct exclude_hidden_refs_cb cb;
+
+	if (strcmp(section, "receive") && strcmp(section, "uploadpack"))
+		die(_("unsupported section for hidden refs: %s"), section);
+
+	if (exclusions->hidden_refs_configured)
+		die(_("--exclude-hidden= passed more than once"));
+
+	cb.exclusions = exclusions;
+	cb.section = section;
+
+	git_config(hide_refs_config, &cb);
+}
+
 struct all_refs_cb {
 	int all_flags;
 	int warned_bad_reflog;
@@ -2221,7 +2259,7 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg
 	    !strcmp(arg, "--bisect") || starts_with(arg, "--glob=") ||
 	    !strcmp(arg, "--indexed-objects") ||
 	    !strcmp(arg, "--alternate-refs") ||
-	    starts_with(arg, "--exclude=") ||
+	    starts_with(arg, "--exclude=") || starts_with(arg, "--exclude-hidden=") ||
 	    starts_with(arg, "--branches=") || starts_with(arg, "--tags=") ||
 	    starts_with(arg, "--remotes=") || starts_with(arg, "--no-walk="))
 	{
@@ -2687,6 +2725,8 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 		}
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--branches")) {
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --branches"));
 		handle_refs(refs, revs, *flags, refs_for_each_branch_ref);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--bisect")) {
@@ -2696,9 +2736,13 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 			    for_each_good_bisect_ref);
 		revs->bisect = 1;
 	} else if (!strcmp(arg, "--tags")) {
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --tags"));
 		handle_refs(refs, revs, *flags, refs_for_each_tag_ref);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (!strcmp(arg, "--remotes")) {
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --remotes"));
 		handle_refs(refs, revs, *flags, refs_for_each_remote_ref);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if ((argcount = parse_long_opt("glob", argv, &optarg))) {
@@ -2710,18 +2754,27 @@ static int handle_revision_pseudo_opt(struct rev_info *revs,
 	} else if ((argcount = parse_long_opt("exclude", argv, &optarg))) {
 		add_ref_exclusion(&revs->ref_excludes, optarg);
 		return argcount;
+	} else if ((argcount = parse_long_opt("exclude-hidden", argv, &optarg))) {
+		exclude_hidden_refs(&revs->ref_excludes, optarg);
+		return argcount;
 	} else if (skip_prefix(arg, "--branches=", &optarg)) {
 		struct all_refs_cb cb;
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --branches"));
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/heads/", &cb);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--tags=", &optarg)) {
 		struct all_refs_cb cb;
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --tags"));
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/tags/", &cb);
 		clear_ref_exclusions(&revs->ref_excludes);
 	} else if (skip_prefix(arg, "--remotes=", &optarg)) {
 		struct all_refs_cb cb;
+		if (revs->ref_excludes.hidden_refs_configured)
+			return error(_("--exclude-hidden cannot be used together with --remotes"));
 		init_all_refs_cb(&cb, revs, *flags);
 		for_each_glob_ref_in(handle_one_ref, optarg, "refs/remotes/", &cb);
 		clear_ref_exclusions(&revs->ref_excludes);
diff --git a/revision.h b/revision.h
index 5c8ab16047..adb810c2f8 100644
--- a/revision.h
+++ b/revision.h
@@ -87,6 +87,19 @@ struct ref_exclusions {
 	 * patterns matches, the reference will be excluded.
 	 */
 	struct string_list excluded_refs;
+
+	/*
+	 * Hidden refs is a list of patterns that is to be hidden via
+	 * `ref_is_hidden()`.
+	 */
+	struct string_list hidden_refs;
+
+	/*
+	 * Indicates whether hidden refs have been configured. This is to
+	 * distinguish between no hidden refs existing and hidden refs not
+	 * being parsed.
+	 */
+	char hidden_refs_configured;
 };
 
 /**
@@ -94,6 +107,7 @@ struct ref_exclusions {
  */
 #define REF_EXCLUSIONS_INIT { \
 	.excluded_refs = STRING_LIST_INIT_DUP, \
+	.hidden_refs = STRING_LIST_INIT_DUP, \
 }
 
 struct oidset;
@@ -456,10 +470,12 @@ void show_object_with_name(FILE *, struct object *, const char *);
 /**
  * Helpers to check if a reference should be excluded.
  */
+
 int ref_excluded(const struct ref_exclusions *exclusions, const char *path);
 void init_ref_exclusions(struct ref_exclusions *);
 void clear_ref_exclusions(struct ref_exclusions *);
 void add_ref_exclusion(struct ref_exclusions *, const char *exclude);
+void exclude_hidden_refs(struct ref_exclusions *, const char *section);
 
 /**
  * This function can be used if you want to add commit objects as revision
diff --git a/t/t6021-rev-list-exclude-hidden.sh b/t/t6021-rev-list-exclude-hidden.sh
new file mode 100755
index 0000000000..32b2b09413
--- /dev/null
+++ b/t/t6021-rev-list-exclude-hidden.sh
@@ -0,0 +1,163 @@
+#!/bin/sh
+
+test_description='git rev-list --exclude-hidden test'
+
+. ./test-lib.sh
+
+test_expect_success 'setup' '
+	test_commit_bulk --id=commit --ref=refs/heads/branch 1 &&
+	COMMIT=$(git rev-parse refs/heads/branch) &&
+	test_commit_bulk --id=tag --ref=refs/tags/lightweight 1 &&
+	TAG=$(git rev-parse refs/tags/lightweight) &&
+	test_commit_bulk --id=hidden --ref=refs/hidden/commit 1 &&
+	HIDDEN=$(git rev-parse refs/hidden/commit) &&
+	test_commit_bulk --id=namespace --ref=refs/namespaces/namespace/refs/namespaced/commit 1 &&
+	NAMESPACE=$(git rev-parse refs/namespaces/namespace/refs/namespaced/commit)
+'
+
+test_expect_success 'invalid section' '
+	echo "fatal: unsupported section for hidden refs: unsupported" >expected &&
+	test_must_fail git rev-list --exclude-hidden=unsupported 2>err &&
+	test_cmp expected err
+'
+
+for section in receive uploadpack
+do
+	test_expect_success "$section: passed multiple times" '
+		echo "fatal: --exclude-hidden= passed more than once" >expected &&
+		test_must_fail git rev-list --exclude-hidden=$section --exclude-hidden=$section 2>err &&
+		test_cmp expected err
+	'
+
+	test_expect_success "$section: without hiddenRefs" '
+		git rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden via transfer.hideRefs" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden via $section.hideRefs" '
+		git -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: respects both transfer.hideRefs and $section.hideRefs" '
+		git -c transfer.hideRefs=refs/tags/ -c $section.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: negation without hidden refs marks everything as uninteresting" '
+		git rev-list --all --exclude-hidden=$section --not --all >out &&
+		test_must_be_empty out
+	'
+
+	test_expect_success "$section: negation with hidden refs marks them as interesting" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --all --exclude-hidden=$section --not --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: hidden refs and excludes work together" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude=refs/tags/* --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: excluded hidden refs get reset" '
+		git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: excluded hidden refs can be used with multiple pseudo-refs" '
+		git -c transfer.hideRefs=refs/ rev-list --exclude-hidden=$section --all --exclude-hidden=$section --all >out &&
+		test_must_be_empty out
+	'
+
+	test_expect_success "$section: works with --glob" '
+		git -c transfer.hideRefs=refs/hidden/ rev-list --exclude-hidden=$section --glob=refs/h* >out &&
+		cat >expected <<-EOF &&
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: operates on stripped refs by default" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaced/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: does not hide namespace by default" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$NAMESPACE
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	test_expect_success "$section: can operate on unstripped refs" '
+		GIT_NAMESPACE=namespace git -c transfer.hideRefs=^refs/namespaces/namespace/ rev-list --exclude-hidden=$section --all >out &&
+		cat >expected <<-EOF &&
+		$HIDDEN
+		$TAG
+		$COMMIT
+		EOF
+		test_cmp expected out
+	'
+
+	for pseudoopt in remotes branches tags
+	do
+		test_expect_success "$section: fails with --$pseudoopt" '
+			test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt 2>err &&
+			test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err
+		'
+
+		test_expect_success "$section: fails with --$pseudoopt=pattern" '
+			test_must_fail git rev-list --exclude-hidden=$section --$pseudoopt=pattern 2>err &&
+			test_i18ngrep "error: --exclude-hidden cannot be used together with --$pseudoopt" err
+		'
+	done
+done
+
+test_done
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 6/7] rev-parse: add `--exclude-hidden=` option

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 5452 bytes --]

Add a new `--exclude-hidden=` option that is similar to the one we just
added to git-rev-list(1). Given a section name `uploadpack` or `receive`
as argument, it causes us to exclude all references that would be hidden
by the respective `$section.hideRefs` configuration.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-rev-parse.txt |  7 ++++++
 builtin/rev-parse.c             | 10 +++++++++
 t/t6018-rev-list-glob.sh        | 40 +++++++++++++++++++++++++++++++++
 3 files changed, 57 insertions(+)

diff --git a/Documentation/git-rev-parse.txt b/Documentation/git-rev-parse.txt
index 6b8ca085aa..bcd8069287 100644
--- a/Documentation/git-rev-parse.txt
+++ b/Documentation/git-rev-parse.txt
@@ -197,6 +197,13 @@ respectively, and they must begin with `refs/` when applied to `--glob`
 or `--all`. If a trailing '/{asterisk}' is intended, it must be given
 explicitly.
 
+--exclude-hidden=[receive|uploadpack]::
+	Do not include refs that would be hidden by `git-receive-pack` or
+	`git-upload-pack` by consulting the appropriate `receive.hideRefs` or
+	`uploadpack.hideRefs` configuration along with `transfer.hideRefs` (see
+	linkgit:git-config[1]). This option affects the next pseudo-ref option
+	`--all` or `--glob` and is cleared after processing them.
+
 --disambiguate=<prefix>::
 	Show every object whose name begins with the given prefix.
 	The <prefix> must be at least 4 hexadecimal digits long to
diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 7fa5b6991b..b5666a03bd 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -876,10 +876,14 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (opt_with_value(arg, "--branches", &arg)) {
+				if (ref_excludes.hidden_refs_configured)
+					return error(_("--exclude-hidden cannot be used together with --branches"));
 				handle_ref_opt(arg, "refs/heads/");
 				continue;
 			}
 			if (opt_with_value(arg, "--tags", &arg)) {
+				if (ref_excludes.hidden_refs_configured)
+					return error(_("--exclude-hidden cannot be used together with --tags"));
 				handle_ref_opt(arg, "refs/tags/");
 				continue;
 			}
@@ -888,6 +892,8 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (opt_with_value(arg, "--remotes", &arg)) {
+				if (ref_excludes.hidden_refs_configured)
+					return error(_("--exclude-hidden cannot be used together with --remotes"));
 				handle_ref_opt(arg, "refs/remotes/");
 				continue;
 			}
@@ -895,6 +901,10 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				add_ref_exclusion(&ref_excludes, arg);
 				continue;
 			}
+			if (skip_prefix(arg, "--exclude-hidden=", &arg)) {
+				exclude_hidden_refs(&ref_excludes, arg);
+				continue;
+			}
 			if (!strcmp(arg, "--show-toplevel")) {
 				const char *work_tree = get_git_work_tree();
 				if (work_tree)
diff --git a/t/t6018-rev-list-glob.sh b/t/t6018-rev-list-glob.sh
index e1abc5c2b3..aabf590dda 100755
--- a/t/t6018-rev-list-glob.sh
+++ b/t/t6018-rev-list-glob.sh
@@ -187,6 +187,46 @@ test_expect_success 'rev-parse --exclude=ref with --remotes=glob' '
 	compare rev-parse "--exclude=upstream/x --remotes=upstream/*" "upstream/one upstream/two"
 '
 
+for section in receive uploadpack
+do
+	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
+		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags" "--exclude-hidden=$section --all"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section with --all" '
+		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --all" "--exclude-hidden=$section --all"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section with --glob" '
+		compare "-c transfer.hideRefs=refs/heads/subspace/ rev-parse" "--exclude=refs/heads/subspace/* --glob=refs/heads/*" "--exclude-hidden=$section --glob=refs/heads/*"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section can be passed once per pseudo-ref" '
+		compare "-c transfer.hideRefs=refs/remotes/ rev-parse" "--branches --tags --branches --tags" "--exclude-hidden=$section --all --exclude-hidden=$section --all"
+	'
+
+	test_expect_success "rev-parse --exclude-hidden=$section can only be passed once per pseudo-ref" '
+		echo "fatal: --exclude-hidden= passed more than once" >expected &&
+		test_must_fail git rev-parse --exclude-hidden=$section --exclude-hidden=$section 2>err &&
+		test_cmp expected err
+	'
+
+	for pseudoopt in branches tags remotes
+	do
+		test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt" '
+			echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected &&
+			test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt 2>err &&
+			test_cmp expected err
+		'
+
+		test_expect_success "rev-parse --exclude-hidden=$section fails with --$pseudoopt=pattern" '
+			echo "error: --exclude-hidden cannot be used together with --$pseudoopt" >expected &&
+			test_must_fail git rev-parse --exclude-hidden=$section --$pseudoopt=pattern 2>err &&
+			test_cmp expected err
+		'
+	done
+done
+
 test_expect_success 'rev-list --exclude=glob with --branches=glob' '
 	compare rev-list "--exclude=subspace-* --branches=sub*" "subspace/one subspace/two"
 '
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v6 7/7] receive-pack: only use visible refs for connectivity check

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4675 bytes --]

When serving a push, git-receive-pack(1) needs to verify that the
packfile sent by the client contains all objects that are required by
the updated references. This connectivity check works by marking all
preexisting references as uninteresting and using the new reference tips
as starting point for a graph walk.

Marking all preexisting references as uninteresting can be a problem
when it comes to performance. Git forges tend to do internal bookkeeping
to keep alive sets of objects for internal use or make them easy to find
via certain references. These references are typically hidden away from
the user so that they are neither advertised nor writeable. At GitLab,
we have one particular repository that contains a total of 7 million
references, of which 6.8 million are indeed internal references. With
the current connectivity check we are forced to load all these
references in order to mark them as uninteresting, and this alone takes
around 15 seconds to compute.

We can optimize this by only taking into account the set of visible refs
when marking objects as uninteresting. This means that we may now walk
more objects until we hit any object that is marked as uninteresting.
But it is rather unlikely that clients send objects that make large
parts of objects reachable that have previously only ever been hidden,
whereas the common case is to push incremental changes that build on top
of the visible object graph.

This provides a huge boost to performance in the mentioned repository,
where the vast majority of its refs hidden. Pushing a new commit into
this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs
as it is configured in Gitaly leads to a 4.5-fold speedup:

    Benchmark 1: main
      Time (mean ± σ):     30.977 s ±  0.157 s    [User: 30.226 s, System: 1.083 s]
      Range (min … max):   30.796 s … 31.071 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):      6.799 s ±  0.063 s    [User: 6.803 s, System: 0.354 s]
      Range (min … max):    6.729 s …  6.850 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        4.56 ± 0.05 times faster than 'main'

As we mostly go through the same codepaths even in the case where there
are no hidden refs at all compared to the code before there is no change
in performance when no refs are hidden:

    Benchmark 1: main
      Time (mean ± σ):     48.188 s ±  0.432 s    [User: 49.326 s, System: 5.009 s]
      Range (min … max):   47.706 s … 48.539 s    3 runs

    Benchmark 2: pks-connectivity-check-hide-refs
      Time (mean ± σ):     48.027 s ±  0.500 s    [User: 48.934 s, System: 5.025 s]
      Range (min … max):   47.504 s … 48.500 s    3 runs

    Summary
      'pks-connectivity-check-hide-refs' ran
        1.00 ± 0.01 times faster than 'main'

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/receive-pack.c | 2 ++
 connected.c            | 3 +++
 connected.h            | 7 +++++++
 3 files changed, 12 insertions(+)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 1e24b31a0a..a90af30363 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1929,6 +1929,8 @@ static void execute_commands(struct command *commands,
 	opt.err_fd = err_fd;
 	opt.progress = err_fd && !quiet;
 	opt.env = tmp_objdir_env(tmp_objdir);
+	opt.exclude_hidden_refs_section = "receive";
+
 	if (check_connected(iterate_receive_command_list, &data, &opt))
 		set_connectivity_errors(commands, si);
 
diff --git a/connected.c b/connected.c
index 74a20cb32e..4f6388eed7 100644
--- a/connected.c
+++ b/connected.c
@@ -100,6 +100,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data,
 		strvec_push(&rev_list.args, "--exclude-promisor-objects");
 	if (!opt->is_deepening_fetch) {
 		strvec_push(&rev_list.args, "--not");
+		if (opt->exclude_hidden_refs_section)
+			strvec_pushf(&rev_list.args, "--exclude-hidden=%s",
+				     opt->exclude_hidden_refs_section);
 		strvec_push(&rev_list.args, "--all");
 	}
 	strvec_push(&rev_list.args, "--quiet");
diff --git a/connected.h b/connected.h
index 6e59c92aa3..16b2c84f2e 100644
--- a/connected.h
+++ b/connected.h
@@ -46,6 +46,13 @@ struct check_connected_options {
 	 * during a fetch.
 	 */
 	unsigned is_deepening_fetch : 1;
+
+	/*
+	 * If not NULL, use `--exclude-hidden=$section` to exclude all refs
+	 * hidden via the `$section.hideRefs` config from the set of
+	 * already-reachable refs.
+	 */
+	const char *exclude_hidden_refs_section;
 };
 
 #define CHECK_CONNECTED_INIT { 0 }
-- 
2.38.1


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v6 0/7] receive-pack: only use visible refs for connectivity check

[Jeff King]

On Thu, Nov 17, 2022 at 06:46:35AM +0100, Patrick Steinhardt wrote:

> this is the sixth version of my patch series that tries to improve
> performance of the connectivity check by only considering preexisting
> refs as uninteresting that could actually have been advertised to the
> client.
> 
> There are only two changes in this version compared to v5:
> 
>     - A fix to the test setup in commit 5/7 so that tests pass when
>       GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main.
> 
>     - A typo fix in the commit message of patch 6/7.

Thanks. Looking at the range diff, this seems good to me!

-Peff

Re: [PATCH v6 0/7] receive-pack: only use visible refs for connectivity check

[Taylor Blau]

On Thu, Nov 17, 2022 at 10:03:46AM -0500, Jeff King wrote:
> On Thu, Nov 17, 2022 at 06:46:35AM +0100, Patrick Steinhardt wrote:
>
> > this is the sixth version of my patch series that tries to improve
> > performance of the connectivity check by only considering preexisting
> > refs as uninteresting that could actually have been advertised to the
> > client.
> >
> > There are only two changes in this version compared to v5:
> >
> >     - A fix to the test setup in commit 5/7 so that tests pass when
> >       GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main.
> >
> >     - A typo fix in the commit message of patch 6/7.
>
> Thanks. Looking at the range diff, this seems good to me!

Yep, I concur. Let's make sure that it passes CI, too, and then start
merging this down. Thanks, both.

Thanks,
Taylor