git for-each-ref - sorting by multiple keys

git for-each-ref - sorting by multiple keys

[clime]

Hello,

I have the following command:

/usr/bin/git for-each-ref --merged="${GIT_HEAD-HEAD}"
--sort='-taggerdate' --sort='-*committerdate'
--format="%(*committerdate)|%(taggerdate)|%(tag)" refs/tags

I thought this will use: -*commiterdate as a primary key and
-taggerdate as a secondary. According to man page for --sort: "You may
use the --sort=<key> option multiple times, in which case the last key
becomes the primary key."

But that doesn't seem to be the case. I created a repo with a single
commit and created annotated tags on the commit in the following
order:

$ git tag -a -m "foo" B
$ git tag -a -m "foo" C
$ git tag -a -m "foo" A

Yet the order I am getting after running the command is:

Sat May 2 22:10:30 2020 +0200|Sat May 2 22:14:49 2020 +0200|C
Sat May 2 22:10:30 2020 +0200|Sat May 2 22:14:45 2020 +0200|B
Sat May 2 22:10:30 2020 +0200|Sat May 2 22:14:51 2020 +0200|A

Is it a mistake in man pages? Is there any way to sort by multiple keys?

Thank you
clime

Re: git for-each-ref - sorting by multiple keys

[Jeff King]

On Sat, May 02, 2020 at 10:31:50PM +0200, clime wrote:

> I have the following command:
> 
> /usr/bin/git for-each-ref --merged="${GIT_HEAD-HEAD}"
> --sort='-taggerdate' --sort='-*committerdate'
> --format="%(*committerdate)|%(taggerdate)|%(tag)" refs/tags
> 
> I thought this will use: -*commiterdate as a primary key and
> -taggerdate as a secondary. According to man page for --sort: "You may
> use the --sort=<key> option multiple times, in which case the last key
> becomes the primary key."
> 
> But that doesn't seem to be the case. I created a repo with a single
> commit and created annotated tags on the commit in the following
> order:

It looks like this has been quite broken since 2015 and nobody noticed. :(

Thanks for your reproduction recipe; using times is a critical part of
the bug. Here's a fix, plus a fix for a related bug I noticed while
working on it (the second one fixes your bug).

  [1/2]: ref-filter: apply --ignore-case to all sorting keys
  [2/2]: ref-filter: apply fallback refname sort only after all user sorts

 builtin/branch.c        |  2 +-
 builtin/for-each-ref.c  |  2 +-
 builtin/tag.c           |  2 +-
 ref-filter.c            | 13 +++++-
 ref-filter.h            |  2 +
 t/t6300-for-each-ref.sh | 94 ++++++++++++++++++++++++++++++++++++++---
 6 files changed, 104 insertions(+), 11 deletions(-)

-Peff

[PATCH 1/2] ref-filter: apply --ignore-case to all sorting keys

[Jeff King]

All of the ref-filter users (for-each-ref, branch, and tag) take an
--ignore-case option which makes filtering and sorting case-insensitive.
However, this option was applied only to the first element of the
ref_sorting list. So:

  git for-each-ref --ignore-case --sort=refname

would do what you expect, but:

  git for-each-ref --ignore-case --sort=refname --sort=taggername

would sort the primary key (taggername) case-insensitively, but sort the
refname case-sensitively. We have two options here:

  - teach callers to set ignore_case on the whole list

  - replace the ref_sorting list with a struct that contains both the
    list of sorting keys, as well as options that apply to _all_
    keys

I went with the first one here, as it gives more flexibility if we later
want to let the users set the flag per-key (presumably through some
special syntax when defining the key; for now it's all or nothing
through --ignore-case).

The new test covers this by sorting on both tagger and subject
case-insensitively, which should compare "a" and "A" identically, but
still sort them before "b" and "B". We'll break ties by sorting on the
refname to give ourselves a stable output (this is actually supposed to
be done automatically, but there's another bug which will be fixed in
the next commit).

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/branch.c        |  2 +-
 builtin/for-each-ref.c  |  2 +-
 builtin/tag.c           |  2 +-
 ref-filter.c            |  6 ++++++
 ref-filter.h            |  2 ++
 t/t6300-for-each-ref.sh | 40 ++++++++++++++++++++++++++++++++++++++++
 6 files changed, 51 insertions(+), 3 deletions(-)

diff --git a/builtin/branch.c b/builtin/branch.c
index d8297f80ff..86341cc835 100644
--- a/builtin/branch.c
+++ b/builtin/branch.c
@@ -739,7 +739,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
 		 */
 		if (!sorting)
 			sorting = ref_default_sorting();
-		sorting->ignore_case = icase;
+		ref_sorting_icase_all(sorting, icase);
 		print_ref_list(&filter, sorting, &format);
 		print_columns(&output, colopts, NULL);
 		string_list_clear(&output, 0);
diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c
index 465153e853..57489e4eab 100644
--- a/builtin/for-each-ref.c
+++ b/builtin/for-each-ref.c
@@ -70,7 +70,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
 
 	if (!sorting)
 		sorting = ref_default_sorting();
-	sorting->ignore_case = icase;
+	ref_sorting_icase_all(sorting, icase);
 	filter.ignore_case = icase;
 
 	filter.name_patterns = argv;
diff --git a/builtin/tag.c b/builtin/tag.c
index dd160b49c7..ff7610b5c8 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -485,7 +485,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	}
 	if (!sorting)
 		sorting = ref_default_sorting();
-	sorting->ignore_case = icase;
+	ref_sorting_icase_all(sorting, icase);
 	filter.ignore_case = icase;
 	if (cmdmode == 'l') {
 		int ret;
diff --git a/ref-filter.c b/ref-filter.c
index 35776838f4..bdb3535ce5 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -2317,6 +2317,12 @@ static int compare_refs(const void *a_, const void *b_, void *ref_sorting)
 	return 0;
 }
 
+void ref_sorting_icase_all(struct ref_sorting *sorting, int flag)
+{
+	for (; sorting; sorting = sorting->next)
+		sorting->ignore_case = !!flag;
+}
+
 void ref_array_sort(struct ref_sorting *sorting, struct ref_array *array)
 {
 	QSORT_S(array->items, array->nr, compare_refs, sorting);
diff --git a/ref-filter.h b/ref-filter.h
index 64330e9601..8ecc33cdfa 100644
--- a/ref-filter.h
+++ b/ref-filter.h
@@ -114,6 +114,8 @@ void ref_array_clear(struct ref_array *array);
 int verify_ref_format(struct ref_format *format);
 /*  Sort the given ref_array as per the ref_sorting provided */
 void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);
+/*  Set the ignore_case flag for all elements of a sorting list */
+void ref_sorting_icase_all(struct ref_sorting *sorting, int flag);
 /*  Based on the given format and quote_style, fill the strbuf */
 int format_ref_array_item(struct ref_array_item *info,
 			  const struct ref_format *format,
diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh
index b3c1092338..c9caf26327 100755
--- a/t/t6300-for-each-ref.sh
+++ b/t/t6300-for-each-ref.sh
@@ -895,4 +895,44 @@ test_expect_success 'for-each-ref --ignore-case ignores case' '
 	test_cmp expect actual
 '
 
+test_expect_success 'for-each-ref --ignore-case works on multiple sort keys' '
+	# name refs numerically to avoid case-insensitive filesystem conflicts
+	nr=0 &&
+	for email in a A b B
+	do
+		for subject in a A b B
+		do
+			GIT_COMMITTER_EMAIL="$email@example.com" \
+			git tag -m "tag $subject" icase-$(printf %02d $nr) &&
+			nr=$((nr+1))||
+			return 1
+		done
+	done &&
+	git for-each-ref --ignore-case \
+		--format="%(taggeremail) %(subject) %(refname)" \
+		--sort=refname \
+		--sort=subject \
+		--sort=taggeremail \
+		refs/tags/icase-* >actual &&
+	cat >expect <<-\EOF &&
+	<a@example.com> tag a refs/tags/icase-00
+	<a@example.com> tag A refs/tags/icase-01
+	<A@example.com> tag a refs/tags/icase-04
+	<A@example.com> tag A refs/tags/icase-05
+	<a@example.com> tag b refs/tags/icase-02
+	<a@example.com> tag B refs/tags/icase-03
+	<A@example.com> tag b refs/tags/icase-06
+	<A@example.com> tag B refs/tags/icase-07
+	<b@example.com> tag a refs/tags/icase-08
+	<b@example.com> tag A refs/tags/icase-09
+	<B@example.com> tag a refs/tags/icase-12
+	<B@example.com> tag A refs/tags/icase-13
+	<b@example.com> tag b refs/tags/icase-10
+	<b@example.com> tag B refs/tags/icase-11
+	<B@example.com> tag b refs/tags/icase-14
+	<B@example.com> tag B refs/tags/icase-15
+	EOF
+	test_cmp expect actual
+'
+
 test_done
-- 
2.26.2.957.g6dc93e954a

Re: [PATCH 1/2] ref-filter: apply --ignore-case to all sorting keys

[Danh Doan]

On 2020-05-03 05:11:57-0400, Jeff King <peff@peff.net> wrote:
> +test_expect_success 'for-each-ref --ignore-case works on multiple sort keys' '
> +	# name refs numerically to avoid case-insensitive filesystem conflicts
> +	nr=0 &&
> +	for email in a A b B
> +	do
> +		for subject in a A b B
> +		do
> +			GIT_COMMITTER_EMAIL="$email@example.com" \
> +			git tag -m "tag $subject" icase-$(printf %02d $nr) &&
> +			nr=$((nr+1))||

The CodingGuidelines said we want to spell `$nr` instead of `nr`
inside arithmetic expansion for dash older than 0.5.4

I'm not sure if we should go with just `$((nr+1))` or it's better to
loosen our Guidelines. Since Debian Jessie (oldest supported Debian)
ships 0.5.7. I don't know about other systems.


-- 
Danh

Re: [PATCH 1/2] ref-filter: apply --ignore-case to all sorting keys

[Jeff King]

On Sun, May 03, 2020 at 06:44:02PM +0700, Danh Doan wrote:

> On 2020-05-03 05:11:57-0400, Jeff King <peff@peff.net> wrote:
> > +test_expect_success 'for-each-ref --ignore-case works on multiple sort keys' '
> > +	# name refs numerically to avoid case-insensitive filesystem conflicts
> > +	nr=0 &&
> > +	for email in a A b B
> > +	do
> > +		for subject in a A b B
> > +		do
> > +			GIT_COMMITTER_EMAIL="$email@example.com" \
> > +			git tag -m "tag $subject" icase-$(printf %02d $nr) &&
> > +			nr=$((nr+1))||
> 
> The CodingGuidelines said we want to spell `$nr` instead of `nr`
> inside arithmetic expansion for dash older than 0.5.4
> 
> I'm not sure if we should go with just `$((nr+1))` or it's better to
> loosen our Guidelines. Since Debian Jessie (oldest supported Debian)
> ships 0.5.7. I don't know about other systems.

Hmm, somehow I didn't know about that rule. We have many cases already
in the test suite and elsewhere (try grepping for '$(([a-z]', which
isn't exhaustive but turns up many examples).

Maybe it's time to loosen the rule?

I've actually seen style guides suggesting to never use "$" there for a
few reasons:

  - it's slightly cleaner to read (this is the recommendation and
    rationale in Google's shell style guide)

  - it's less surprising if you somehow end up with a non-number in your
    variable:

      $ foo=bar
      $ bar=41
      $ echo $((foo + 1))
      dash: 8: Illegal number: bar
      $ echo $(($foo + 1))
      42

    That's using dash. With bash, both produce the answer 42! Clearly
    this isn't something we should be doing either way, but I'd much
    rather see "illegal number" in some cases which would alert us that
    something confusing is going on.

-Peff

Re: [PATCH 1/2] ref-filter: apply --ignore-case to all sorting keys

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Hmm, somehow I didn't know about that rule. We have many cases already
> in the test suite and elsewhere (try grepping for '$(([a-z]', which
> isn't exhaustive but turns up many examples).
>
> Maybe it's time to loosen the rule?

Let's do that.  It's time.

[PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Jeff King]

On Mon, May 04, 2020 at 08:37:38AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > Hmm, somehow I didn't know about that rule. We have many cases already
> > in the test suite and elsewhere (try grepping for '$(([a-z]', which
> > isn't exhaustive but turns up many examples).
> >
> > Maybe it's time to loosen the rule?
> 
> Let's do that.  It's time.

Here it is in patch form for your convenience.

-- >8 --
Subject: CodingGuidelines: drop arithmetic expansion advice to use "$x"

The advice to use "$x" rather than "x" in arithmetric expansion was
working around a dash bug fixed in 0.5.4. Even Debian oldstable has
0.5.7 these days. And in the meantime, we've added almost two dozen
instances of the "x" form which you can find with:

  git grep '$(([a-z]'

and nobody seems to have complained. Let's declare this workaround
obsolete and simplify our style guide.

Helped-by: Danh Doan <congdanhqx@gmail.com>
Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/CodingGuidelines | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines
index 390ceece52..a89e8dcfbc 100644
--- a/Documentation/CodingGuidelines
+++ b/Documentation/CodingGuidelines
@@ -95,10 +95,6 @@ For shell scripts specifically (not exhaustive):
 
  - We use Arithmetic Expansion $(( ... )).
 
- - Inside Arithmetic Expansion, spell shell variables with $ in front
-   of them, as some shells do not grok $((x)) while accepting $(($x))
-   just fine (e.g. dash older than 0.5.4).
-
  - We do not use Process Substitution <(list) or >(list).
 
  - Do not write control structures on a single line with semicolon.
-- 
2.26.2.957.g6dc93e954a

Re: [PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Carlo Marcelo Arenas Belón]

On Mon, May 04, 2020 at 12:07:09PM -0400, Jeff King wrote:
> -- >8 --
> Subject: CodingGuidelines: drop arithmetic expansion advice to use "$x"
> 
> The advice to use "$x" rather than "x" in arithmetric expansion was
> working around a dash bug fixed in 0.5.4. Even Debian oldstable has
> 0.5.7 these days.

that would be oldoldstable, oldstable is actually in 0.5.8 ;)

> Helped-by: Danh Doan <congdanhqx@gmail.com>
> Signed-off-by: Jeff King <peff@peff.net>

Reviewed-by: Carlo Marcelo Arenas Belón <carenas@gmail.com>

Carlo

Re: [PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Jeff King]

On Mon, May 04, 2020 at 09:28:44AM -0700, Carlo Marcelo Arenas Belón wrote:

> On Mon, May 04, 2020 at 12:07:09PM -0400, Jeff King wrote:
> > -- >8 --
> > Subject: CodingGuidelines: drop arithmetic expansion advice to use "$x"
> > 
> > The advice to use "$x" rather than "x" in arithmetric expansion was
> > working around a dash bug fixed in 0.5.4. Even Debian oldstable has
> > 0.5.7 these days.
> 
> that would be oldoldstable, oldstable is actually in 0.5.8 ;)

Oh, you're right. I forgot they're doing an extra layer these days. It
will officially become obsolete in less than 2 months. :)

-Peff

Re: [PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, May 04, 2020 at 09:28:44AM -0700, Carlo Marcelo Arenas Belón wrote:
>
>> On Mon, May 04, 2020 at 12:07:09PM -0400, Jeff King wrote:
>> > -- >8 --
>> > Subject: CodingGuidelines: drop arithmetic expansion advice to use "$x"
>> > 
>> > The advice to use "$x" rather than "x" in arithmetric expansion was
>> > working around a dash bug fixed in 0.5.4. Even Debian oldstable has
>> > 0.5.7 these days.
>> 
>> that would be oldoldstable, oldstable is actually in 0.5.8 ;)
>
> Oh, you're right. I forgot they're doing an extra layer these days. It
> will officially become obsolete in less than 2 months. :)

Thanks, both.  I'll just do s/0.5.7/0.5.8/ when I add Carlo's
reviewed-by to queue the patch.

Re: [PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Danh Doan]

On 2020-05-04 12:07:09-0400, Jeff King <peff@peff.net> wrote:
> On Mon, May 04, 2020 at 08:37:38AM -0700, Junio C Hamano wrote:
> 
> Subject: CodingGuidelines: drop arithmetic expansion advice to use "$x"
> 
> The advice to use "$x" rather than "x" in arithmetric expansion was
> working around a dash bug fixed in 0.5.4. Even Debian oldstable has
> 0.5.7 these days. And in the meantime, we've added almost two dozen
> instances of the "x" form which you can find with:
> 
>   git grep '$(([a-z]'
> 
> and nobody seems to have complained. Let's declare this workaround
> obsolete and simplify our style guide.
> 
> Helped-by: Danh Doan <congdanhqx@gmail.com>
> Signed-off-by: Jeff King <peff@peff.net>

I see this patch hasn't been merged to pu yet.

Please have my name as (if it's not too much trouble for you):

	Đoàn Trần Công Danh <congdanhqx@gmail.com>

(I'm going to change my name in email setting)

-- 
Danh

Re: [PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines
> index 390ceece52..a89e8dcfbc 100644
> --- a/Documentation/CodingGuidelines
> +++ b/Documentation/CodingGuidelines
> @@ -95,10 +95,6 @@ For shell scripts specifically (not exhaustive):
>  
>   - We use Arithmetic Expansion $(( ... )).
>  
> - - Inside Arithmetic Expansion, spell shell variables with $ in front
> -   of them, as some shells do not grok $((x)) while accepting $(($x))
> -   just fine (e.g. dash older than 0.5.4).
> -
>   - We do not use Process Substitution <(list) or >(list).
>  
>   - Do not write control structures on a single line with semicolon.

A new entry in the "What's cooking" report has this:

    * jk/arith-expansion-coding-guidelines (2020-05-04) 1 commit
     - CodingGuidelines: drop arithmetic expansion advice to use "$x"

     The coding guideline for shell scripts instructed to refer to a
     variable with dollar-sign inside airthmetic expansion to work
     around a bug in old versions of bash, which is a thing of the past.
     Now we are not forbidden from writing $((var+1)).

Writing the last sentence made me wonder if we should go one step
further and actually encourage actively omitting the dollar-sign
from variable reference instead.

Re: [PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Jeff King]

On Tue, May 05, 2020 at 01:40:03PM -0700, Junio C Hamano wrote:

> A new entry in the "What's cooking" report has this:
> 
>     * jk/arith-expansion-coding-guidelines (2020-05-04) 1 commit
>      - CodingGuidelines: drop arithmetic expansion advice to use "$x"
> 
>      The coding guideline for shell scripts instructed to refer to a
>      variable with dollar-sign inside airthmetic expansion to work
>      around a bug in old versions of bash, which is a thing of the past.
>      Now we are not forbidden from writing $((var+1)).

s/bash/dash/, I think

> Writing the last sentence made me wonder if we should go one step
> further and actually encourage actively omitting the dollar-sign
> from variable reference instead.

I don't have a strong preference either way. I gave a few reasons to
prefer the dollar-less version in:

  https://lore.kernel.org/git/20200504151351.GC11373@coredump.intra.peff.net/

but I couldn't find a case where the difference really matters in
practice for otherwise-correct code. If we don't care much either way,
I'd just as soon not have a rule. We have enough rules as it is, and I
don't think either is obvious enough that somebody who is used to one
style will get confused by the other.

-Peff

Re: [PATCH] CodingGuidelines: drop arithmetic expansion advice to use "$x"

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I'd just as soon not have a rule. We have enough rules as it is, and I
> don't think either is obvious enough that somebody who is used to one
> style will get confused by the other.

OK.  I very much welcome one fewer paragraphs in the file ;-)

Re: [PATCH 1/2] ref-filter: apply --ignore-case to all sorting keys

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> would sort the primary key (taggername) case-insensitively, but sort the
> refname case-sensitively. We have two options here:
>
>   - teach callers to set ignore_case on the whole list
>
>   - replace the ref_sorting list with a struct that contains both the
>     list of sorting keys, as well as options that apply to _all_
>     keys
>
> I went with the first one here, as it gives more flexibility if we later
> want to let the users set the flag per-key (presumably through some
> special syntax when defining the key; for now it's all or nothing
> through --ignore-case).

A good design decision I would fully support.

> +test_expect_success 'for-each-ref --ignore-case works on multiple sort keys' '
> +	# name refs numerically to avoid case-insensitive filesystem conflicts

Very considerate.  If I were writing these nested loops, I am sure I
would have used "tag-$email-$subject" to be cute.

Queued.  Thanks.

> +	nr=0 &&
> +	for email in a A b B
> +	do
> +		for subject in a A b B
> +		do
> +			GIT_COMMITTER_EMAIL="$email@example.com" \
> +			git tag -m "tag $subject" icase-$(printf %02d $nr) &&
> +			nr=$((nr+1))||
> +			return 1
> +		done
> +	done &&
> +	git for-each-ref --ignore-case \
> +		--format="%(taggeremail) %(subject) %(refname)" \
> +		--sort=refname \
> +		--sort=subject \
> +		--sort=taggeremail \
> +		refs/tags/icase-* >actual &&
> +	cat >expect <<-\EOF &&
> +	<a@example.com> tag a refs/tags/icase-00
> +	<a@example.com> tag A refs/tags/icase-01
> +	<A@example.com> tag a refs/tags/icase-04
> +	<A@example.com> tag A refs/tags/icase-05
> +	<a@example.com> tag b refs/tags/icase-02
> +	<a@example.com> tag B refs/tags/icase-03
> +	<A@example.com> tag b refs/tags/icase-06
> +	<A@example.com> tag B refs/tags/icase-07
> +	<b@example.com> tag a refs/tags/icase-08
> +	<b@example.com> tag A refs/tags/icase-09
> +	<B@example.com> tag a refs/tags/icase-12
> +	<B@example.com> tag A refs/tags/icase-13
> +	<b@example.com> tag b refs/tags/icase-10
> +	<b@example.com> tag B refs/tags/icase-11
> +	<B@example.com> tag b refs/tags/icase-14
> +	<B@example.com> tag B refs/tags/icase-15
> +	EOF
> +	test_cmp expect actual
> +'
> +
>  test_done

Re: [PATCH 1/2] ref-filter: apply --ignore-case to all sorting keys

[Jeff King]

On Mon, May 04, 2020 at 02:00:12PM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > would sort the primary key (taggername) case-insensitively, but sort the
> > refname case-sensitively. We have two options here:
> >
> >   - teach callers to set ignore_case on the whole list
> >
> >   - replace the ref_sorting list with a struct that contains both the
> >     list of sorting keys, as well as options that apply to _all_
> >     keys
> >
> > I went with the first one here, as it gives more flexibility if we later
> > want to let the users set the flag per-key (presumably through some
> > special syntax when defining the key; for now it's all or nothing
> > through --ignore-case).
> 
> A good design decision I would fully support.

I admit I had second thoughts when dealing with the "oops, we have to
choose ignore_case from the first one" part of the second patch. But I
think it works OK in practice, and I did like having a less invasive
diff. :)

-Peff

Re: [PATCH 1/2] ref-filter: apply --ignore-case to all sorting keys

[Taylor Blau]

On Sun, May 03, 2020 at 05:11:57AM -0400, Jeff King wrote:
> All of the ref-filter users (for-each-ref, branch, and tag) take an
> --ignore-case option which makes filtering and sorting case-insensitive.
> However, this option was applied only to the first element of the
> ref_sorting list. So:
>
>   git for-each-ref --ignore-case --sort=refname
>
> would do what you expect, but:
>
>   git for-each-ref --ignore-case --sort=refname --sort=taggername
>
> would sort the primary key (taggername) case-insensitively, but sort the
> refname case-sensitively. We have two options here:
>
>   - teach callers to set ignore_case on the whole list
>
>   - replace the ref_sorting list with a struct that contains both the
>     list of sorting keys, as well as options that apply to _all_
>     keys
>
> I went with the first one here, as it gives more flexibility if we later
> want to let the users set the flag per-key (presumably through some
> special syntax when defining the key; for now it's all or nothing
> through --ignore-case).

Makes sense, I think that this will provide us more flexibility in the
future in case we want to have per-flag keys or some such.

> The new test covers this by sorting on both tagger and subject
> case-insensitively, which should compare "a" and "A" identically, but
> still sort them before "b" and "B". We'll break ties by sorting on the
> refname to give ourselves a stable output (this is actually supposed to
> be done automatically, but there's another bug which will be fixed in
> the next commit).

Thanks for adding a test.

> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  builtin/branch.c        |  2 +-
>  builtin/for-each-ref.c  |  2 +-
>  builtin/tag.c           |  2 +-
>  ref-filter.c            |  6 ++++++
>  ref-filter.h            |  2 ++
>  t/t6300-for-each-ref.sh | 40 ++++++++++++++++++++++++++++++++++++++++
>  6 files changed, 51 insertions(+), 3 deletions(-)
>
> diff --git a/builtin/branch.c b/builtin/branch.c
> index d8297f80ff..86341cc835 100644
> --- a/builtin/branch.c
> +++ b/builtin/branch.c
> @@ -739,7 +739,7 @@ int cmd_branch(int argc, const char **argv, const char *prefix)
>  		 */
>  		if (!sorting)
>  			sorting = ref_default_sorting();
> -		sorting->ignore_case = icase;
> +		ref_sorting_icase_all(sorting, icase);
>  		print_ref_list(&filter, sorting, &format);
>  		print_columns(&output, colopts, NULL);
>  		string_list_clear(&output, 0);
> diff --git a/builtin/for-each-ref.c b/builtin/for-each-ref.c
> index 465153e853..57489e4eab 100644
> --- a/builtin/for-each-ref.c
> +++ b/builtin/for-each-ref.c
> @@ -70,7 +70,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix)
>
>  	if (!sorting)
>  		sorting = ref_default_sorting();
> -	sorting->ignore_case = icase;
> +	ref_sorting_icase_all(sorting, icase);
>  	filter.ignore_case = icase;
>
>  	filter.name_patterns = argv;
> diff --git a/builtin/tag.c b/builtin/tag.c
> index dd160b49c7..ff7610b5c8 100644
> --- a/builtin/tag.c
> +++ b/builtin/tag.c
> @@ -485,7 +485,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
>  	}
>  	if (!sorting)
>  		sorting = ref_default_sorting();
> -	sorting->ignore_case = icase;
> +	ref_sorting_icase_all(sorting, icase);
>  	filter.ignore_case = icase;
>  	if (cmdmode == 'l') {
>  		int ret;
> diff --git a/ref-filter.c b/ref-filter.c
> index 35776838f4..bdb3535ce5 100644
> --- a/ref-filter.c
> +++ b/ref-filter.c
> @@ -2317,6 +2317,12 @@ static int compare_refs(const void *a_, const void *b_, void *ref_sorting)
>  	return 0;
>  }
>
> +void ref_sorting_icase_all(struct ref_sorting *sorting, int flag)
> +{
> +	for (; sorting; sorting = sorting->next)
> +		sorting->ignore_case = !!flag;
> +}
> +
>  void ref_array_sort(struct ref_sorting *sorting, struct ref_array *array)
>  {
>  	QSORT_S(array->items, array->nr, compare_refs, sorting);
> diff --git a/ref-filter.h b/ref-filter.h
> index 64330e9601..8ecc33cdfa 100644
> --- a/ref-filter.h
> +++ b/ref-filter.h
> @@ -114,6 +114,8 @@ void ref_array_clear(struct ref_array *array);
>  int verify_ref_format(struct ref_format *format);
>  /*  Sort the given ref_array as per the ref_sorting provided */
>  void ref_array_sort(struct ref_sorting *sort, struct ref_array *array);
> +/*  Set the ignore_case flag for all elements of a sorting list */
> +void ref_sorting_icase_all(struct ref_sorting *sorting, int flag);
>  /*  Based on the given format and quote_style, fill the strbuf */
>  int format_ref_array_item(struct ref_array_item *info,
>  			  const struct ref_format *format,
> diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh
> index b3c1092338..c9caf26327 100755
> --- a/t/t6300-for-each-ref.sh
> +++ b/t/t6300-for-each-ref.sh
> @@ -895,4 +895,44 @@ test_expect_success 'for-each-ref --ignore-case ignores case' '
>  	test_cmp expect actual
>  '
>
> +test_expect_success 'for-each-ref --ignore-case works on multiple sort keys' '
> +	# name refs numerically to avoid case-insensitive filesystem conflicts
> +	nr=0 &&
> +	for email in a A b B
> +	do
> +		for subject in a A b B
> +		do
> +			GIT_COMMITTER_EMAIL="$email@example.com" \
> +			git tag -m "tag $subject" icase-$(printf %02d $nr) &&
> +			nr=$((nr+1))||
> +			return 1
> +		done
> +	done &&
> +	git for-each-ref --ignore-case \
> +		--format="%(taggeremail) %(subject) %(refname)" \
> +		--sort=refname \
> +		--sort=subject \
> +		--sort=taggeremail \
> +		refs/tags/icase-* >actual &&
> +	cat >expect <<-\EOF &&
> +	<a@example.com> tag a refs/tags/icase-00
> +	<a@example.com> tag A refs/tags/icase-01
> +	<A@example.com> tag a refs/tags/icase-04
> +	<A@example.com> tag A refs/tags/icase-05
> +	<a@example.com> tag b refs/tags/icase-02
> +	<a@example.com> tag B refs/tags/icase-03
> +	<A@example.com> tag b refs/tags/icase-06
> +	<A@example.com> tag B refs/tags/icase-07
> +	<b@example.com> tag a refs/tags/icase-08
> +	<b@example.com> tag A refs/tags/icase-09
> +	<B@example.com> tag a refs/tags/icase-12
> +	<B@example.com> tag A refs/tags/icase-13
> +	<b@example.com> tag b refs/tags/icase-10
> +	<b@example.com> tag B refs/tags/icase-11
> +	<B@example.com> tag b refs/tags/icase-14
> +	<B@example.com> tag B refs/tags/icase-15
> +	EOF
> +	test_cmp expect actual
> +'
> +
>  test_done
> --
> 2.26.2.957.g6dc93e954a

All looks very reasonable, so:

  Reviewed-by: Taylor Blau <me@ttaylorr.com>

Thanks,
Taylor

[PATCH 2/2] ref-filter: apply fallback refname sort only after all user sorts

[Jeff King]

Commit 9e468334b4 (ref-filter: fallback on alphabetical comparison,
2015-10-30) taught ref-filter's sort to fallback to comparing refnames.
But it did it at the wrong level, overriding the comparison result for a
single "--sort" key from the user, rather than after all sort keys have
been exhausted.

This worked correctly for a single "--sort" option, but not for multiple
ones. We'd break any ties in the first key with the refname and never
evaluate the second key at all.

To make matters even more interesting, we only applied this fallback
sometimes! For a field like "taggeremail" which requires a string
comparison, we'd truly return the result of strcmp(), even if it was 0.
But for numerical "value" fields like "taggerdate", we did apply the
fallback. And that's why our multiple-sort test missed this: it uses
taggeremail as the main comparison.

So let's start by adding a much more rigorous test. We'll have a set of
commits expressing every combination of two tagger emails, dates, and
refnames. Then we can confirm that our sort is applied with the correct
precedence, and we'll be hitting both the string and value comparators.

That does show the bug, and the fix is simple: moving the fallback to
the outer compare_refs() function, after all ref_sorting keys have been
exhausted.

Note that in the outer function we don't have an "ignore_case" flag, as
it's part of each individual ref_sorting element. It's debatable what
such a fallback should do, since we didn't use the user's keys to match.
But until now we have been trying to respect that flag, so the
least-invasive thing is to try to continue to do so. Since all callers
in the current code either set the flag for all keys or for none, we can
just pull the flag from the first key. In a hypothetical world where the
user really can flip the case-insensitivity of keys separately, we may
want to extend the code to distinguish that case from a blanket
"--ignore-case".

Signed-off-by: Jeff King <peff@peff.net>
---
 ref-filter.c            |  7 ++++--
 t/t6300-for-each-ref.sh | 54 ++++++++++++++++++++++++++++++++++++-----
 2 files changed, 53 insertions(+), 8 deletions(-)

diff --git a/ref-filter.c b/ref-filter.c
index bdb3535ce5..bf7b70299b 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -2295,7 +2295,7 @@ static int cmp_ref_sorting(struct ref_sorting *s, struct ref_array_item *a, stru
 		if (va->value < vb->value)
 			cmp = -1;
 		else if (va->value == vb->value)
-			cmp = cmp_fn(a->refname, b->refname);
+			cmp = 0;
 		else
 			cmp = 1;
 	}
@@ -2314,7 +2314,10 @@ static int compare_refs(const void *a_, const void *b_, void *ref_sorting)
 		if (cmp)
 			return cmp;
 	}
-	return 0;
+	s = ref_sorting;
+	return s && s->ignore_case ?
+		strcasecmp(a->refname, b->refname) :
+		strcmp(a->refname, b->refname);
 }
 
 void ref_sorting_icase_all(struct ref_sorting *sorting, int flag)
diff --git a/t/t6300-for-each-ref.sh b/t/t6300-for-each-ref.sh
index c9caf26327..da59fadc5d 100755
--- a/t/t6300-for-each-ref.sh
+++ b/t/t6300-for-each-ref.sh
@@ -650,17 +650,59 @@ test_atom refs/tags/signed-long contents "subject line
 body contents
 $sig"
 
-sort >expected <<EOF
-$(git rev-parse refs/tags/bogo) <committer@example.com> refs/tags/bogo
-$(git rev-parse refs/tags/master) <committer@example.com> refs/tags/master
-EOF
+test_expect_success 'set up multiple-sort tags' '
+	for when in 100000 200000
+	do
+		for email in user1 user2
+		do
+			for ref in ref1 ref2
+			do
+				GIT_COMMITTER_DATE="@$when +0000" \
+				GIT_COMMITTER_EMAIL="$email@example.com" \
+				git tag -m "tag $ref-$when-$email" \
+				multi-$ref-$when-$email || return 1
+			done
+		done
+	done
+'
 
 test_expect_success 'Verify sort with multiple keys' '
-	git for-each-ref --format="%(objectname) %(taggeremail) %(refname)" --sort=objectname --sort=taggeremail \
-		refs/tags/bogo refs/tags/master > actual &&
+	cat >expected <<-\EOF &&
+	100000 <user1@example.com> refs/tags/multi-ref2-100000-user1
+	100000 <user1@example.com> refs/tags/multi-ref1-100000-user1
+	100000 <user2@example.com> refs/tags/multi-ref2-100000-user2
+	100000 <user2@example.com> refs/tags/multi-ref1-100000-user2
+	200000 <user1@example.com> refs/tags/multi-ref2-200000-user1
+	200000 <user1@example.com> refs/tags/multi-ref1-200000-user1
+	200000 <user2@example.com> refs/tags/multi-ref2-200000-user2
+	200000 <user2@example.com> refs/tags/multi-ref1-200000-user2
+	EOF
+	git for-each-ref \
+		--format="%(taggerdate:unix) %(taggeremail) %(refname)" \
+		--sort=-refname \
+		--sort=taggeremail \
+		--sort=taggerdate \
+		"refs/tags/multi-*" >actual &&
 	test_cmp expected actual
 '
 
+test_expect_success 'equivalent sorts fall back on refname' '
+	cat >expected <<-\EOF &&
+	100000 <user1@example.com> refs/tags/multi-ref1-100000-user1
+	100000 <user2@example.com> refs/tags/multi-ref1-100000-user2
+	100000 <user1@example.com> refs/tags/multi-ref2-100000-user1
+	100000 <user2@example.com> refs/tags/multi-ref2-100000-user2
+	200000 <user1@example.com> refs/tags/multi-ref1-200000-user1
+	200000 <user2@example.com> refs/tags/multi-ref1-200000-user2
+	200000 <user1@example.com> refs/tags/multi-ref2-200000-user1
+	200000 <user2@example.com> refs/tags/multi-ref2-200000-user2
+	EOF
+	git for-each-ref \
+		--format="%(taggerdate:unix) %(taggeremail) %(refname)" \
+		--sort=taggerdate \
+		"refs/tags/multi-*" >actual &&
+	test_cmp expected actual
+'
 
 test_expect_success 'do not dereference NULL upon %(HEAD) on unborn branch' '
 	test_when_finished "git checkout master" &&
-- 
2.26.2.957.g6dc93e954a

Re: [PATCH 2/2] ref-filter: apply fallback refname sort only after all user sorts

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Commit 9e468334b4 (ref-filter: fallback on alphabetical comparison,
> 2015-10-30) taught ref-filter's sort to fallback to comparing refnames.
> But it did it at the wrong level, overriding the comparison result for a
> single "--sort" key from the user, rather than after all sort keys have
> been exhausted.

Good find.  It does look like that the fallback was broken from the
very first version when it was introduced, as we did have multiple
keys support back in that version already.

The fix looks good to me.

Thanks.

Re: [PATCH 2/2] ref-filter: apply fallback refname sort only after all user sorts

[Taylor Blau]

On Mon, May 04, 2020 at 02:05:16PM -0700, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
>
> > Commit 9e468334b4 (ref-filter: fallback on alphabetical comparison,
> > 2015-10-30) taught ref-filter's sort to fallback to comparing refnames.
> > But it did it at the wrong level, overriding the comparison result for a
> > single "--sort" key from the user, rather than after all sort keys have
> > been exhausted.
>
> Good find.  It does look like that the fallback was broken from the
> very first version when it was introduced, as we did have multiple
> keys support back in that version already.
>
> The fix looks good to me.

Me too.

  Reviewed-by: Taylor Blau <me@ttaylorr.com>

> Thanks.

Thanks,
Taylor

Re: git for-each-ref - sorting by multiple keys

[clime]

Awesome, thanks!

On Sun, 3 May 2020 at 11:09, Jeff King <peff@peff.net> wrote:
>
> On Sat, May 02, 2020 at 10:31:50PM +0200, clime wrote:
>
> > I have the following command:
> >
> > /usr/bin/git for-each-ref --merged="${GIT_HEAD-HEAD}"
> > --sort='-taggerdate' --sort='-*committerdate'
> > --format="%(*committerdate)|%(taggerdate)|%(tag)" refs/tags
> >
> > I thought this will use: -*commiterdate as a primary key and
> > -taggerdate as a secondary. According to man page for --sort: "You may
> > use the --sort=<key> option multiple times, in which case the last key
> > becomes the primary key."
> >
> > But that doesn't seem to be the case. I created a repo with a single
> > commit and created annotated tags on the commit in the following
> > order:
>
> It looks like this has been quite broken since 2015 and nobody noticed. :(
>
> Thanks for your reproduction recipe; using times is a critical part of
> the bug. Here's a fix, plus a fix for a related bug I noticed while
> working on it (the second one fixes your bug).
>
>   [1/2]: ref-filter: apply --ignore-case to all sorting keys
>   [2/2]: ref-filter: apply fallback refname sort only after all user sorts
>
>  builtin/branch.c        |  2 +-
>  builtin/for-each-ref.c  |  2 +-
>  builtin/tag.c           |  2 +-
>  ref-filter.c            | 13 +++++-
>  ref-filter.h            |  2 +
>  t/t6300-for-each-ref.sh | 94 ++++++++++++++++++++++++++++++++++++++---
>  6 files changed, 104 insertions(+), 11 deletions(-)
>
> -Peff