[PATCH v2 00/23] SHA-256 test fixes, part 8

[PATCH v2 00/23] SHA-256 test fixes, part 8

[brian m. carlson]

This is the second-to-last series of test fixes for SHA-256.

As mentioned previously, the patch for t3305 seems to indicate a bug in
the notes code and I'm not familiar enough with that code to apply a
fix.  This is a band-aid to get it working with SHA-256, but any
comments on a more robust approach would of course be welcome.

Changes from v1:
* Drop patch for t3404 in favor of Dscho's fix.
* Drop patch for t5616 in favor of Jonathan Tan's fix.
* Add missing sign-off.
* Move test_oid_init into the correct patch.

brian m. carlson (22):
  t/lib-pack: support SHA-256
  t3206: make hash size independent
  t3305: annotate with SHA1 prerequisite
  t3308: make test work with SHA-256
  t3309: make test work with SHA-256
  t3310: make test work with SHA-256
  t3311: make test work with SHA-256
  t4013: make test hash independent
  t4060: make test work with SHA-256
  t4211: make test hash independent
  t5302: make hash size independent
  t5309: make test hash independent
  t5313: make test hash independent
  t5321: make test hash independent
  t5515: make test hash independent
  t5318: update for SHA-256
  t5607: make hash size independent
  t5703: make test work with SHA-256
  t5703: switch tests to use test_oid
  t6000: abstract away SHA-1-specific constants
  t6006: make hash size independent
  t6024: update for SHA-256

 t/lib-pack.sh                                |  35 ++-
 t/t3206-range-diff.sh                        |  14 +-
 t/t3305-notes-fanout.sh                      |   2 +-
 t/t3308-notes-merge.sh                       |  83 ++++---
 t/t3309-notes-merge-auto-resolve.sh          | 228 ++++++++++++-------
 t/t3310-notes-merge-manual-resolve.sh        |  84 ++++---
 t/t3311-notes-merge-fanout.sh                |  60 +++--
 t/t4013-diff-various.sh                      |  44 +++-
 t/t4060-diff-submodule-option-diff-format.sh | 126 +++++-----
 t/t4211-line-log.sh                          |  14 +-
 t/t5302-pack-index.sh                        |  18 +-
 t/t5309-pack-delta-cycles.sh                 |  10 +-
 t/t5313-pack-bounds-checks.sh                |  19 +-
 t/t5318-commit-graph.sh                      |   4 +-
 t/t5321-pack-large-objects.sh                |   4 +-
 t/t5515-fetch-merge-logic.sh                 |  51 ++++-
 t/t5607-clone-bundle.sh                      |   2 +-
 t/t5703-upload-pack-ref-in-want.sh           |   7 +-
 t/t6000-rev-list-misc.sh                     |  13 +-
 t/t6006-rev-list-format.sh                   |   4 +-
 t/t6024-recursive-merge.sh                   |  15 +-
 21 files changed, 559 insertions(+), 278 deletions(-)

[PATCH v2 01/22] t/lib-pack: support SHA-256

[brian m. carlson]

Update the support routines for generating packs to support both SHA-1
and SHA-256.  Compute the trailing pack checksum and its length
correctly depending on the algorithm, and look up the object names based
on the algorithm as well.  Ensure we initialize the algorithm facts so
that our callers need not do so.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/lib-pack.sh | 35 ++++++++++++++++++++++++++++++-----
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/t/lib-pack.sh b/t/lib-pack.sh
index c4d907a450..f3463170b3 100644
--- a/t/lib-pack.sh
+++ b/t/lib-pack.sh
@@ -35,9 +35,11 @@ pack_header () {
 # have hardcoded some well-known objects. See the case statements below for the
 # complete list.
 pack_obj () {
+	test_oid_init
+
 	case "$1" in
 	# empty blob
-	e69de29bb2d1d6434b8b29ae775ad8c2e48c5391)
+	$EMPTY_BLOB)
 		case "$2" in
 		'')
 			printf '\060\170\234\003\0\0\0\0\1'
@@ -47,7 +49,7 @@ pack_obj () {
 		;;
 
 	# blob containing "\7\76"
-	e68fe8129b546b101aee9510c5328e7f21ca1d18)
+	$(test_oid packlib_7_76))
 		case "$2" in
 		'')
 			printf '\062\170\234\143\267\3\0\0\116\0\106'
@@ -59,11 +61,18 @@ pack_obj () {
 			printf '\234\143\142\142\142\267\003\0\0\151\0\114'
 			return
 			;;
+		37c8e2c15bb22b912e59b43fd51a4f7e9465ed0b5084c5a1411d991cbe630683)
+			printf '\165\67\310\342\301\133\262\53\221\56\131' &&
+			printf '\264\77\325\32\117\176\224\145\355\13\120' &&
+			printf '\204\305\241\101\35\231\34\276\143\6\203\170' &&
+			printf '\234\143\142\142\142\267\003\0\0\151\0\114'
+			return
+			;;
 		esac
 		;;
 
 	# blob containing "\7\0"
-	01d7713666f4de822776c7622c10f1b07de280dc)
+	$(test_oid packlib_7_0))
 		case "$2" in
 		'')
 			printf '\062\170\234\143\147\0\0\0\20\0\10'
@@ -75,6 +84,13 @@ pack_obj () {
 			printf '\143\142\142\142\147\0\0\0\53\0\16'
 			return
 			;;
+		5d8e6fc40f2dab00e6983a48523fe57e621f46434cb58dbd4422fba03380d886)
+			printf '\165\135\216\157\304\17\55\253\0\346\230\72' &&
+			printf '\110\122\77\345\176\142\37\106\103\114\265' &&
+			printf '\215\275\104\42\373\240\63\200\330\206\170\234' &&
+			printf '\143\142\142\142\147\0\0\0\53\0\16'
+			return
+			;;
 		esac
 		;;
 	esac
@@ -86,7 +102,7 @@ pack_obj () {
 	then
 		echo "$1" | git pack-objects --stdout >pack_obj.tmp &&
 		size=$(wc -c <pack_obj.tmp) &&
-		dd if=pack_obj.tmp bs=1 count=$((size - 20 - 12)) skip=12 &&
+		dd if=pack_obj.tmp bs=1 count=$((size - $(test_oid rawsz) - 12)) skip=12 &&
 		rm -f pack_obj.tmp
 		return
 	fi
@@ -97,7 +113,8 @@ pack_obj () {
 
 # Compute and append pack trailer to "$1"
 pack_trailer () {
-	test-tool sha1 -b <"$1" >trailer.tmp &&
+	test_oid_init &&
+	test-tool $(test_oid algo) -b <"$1" >trailer.tmp &&
 	cat trailer.tmp >>"$1" &&
 	rm -f trailer.tmp
 }
@@ -108,3 +125,11 @@ pack_trailer () {
 clear_packs () {
 	rm -f .git/objects/pack/*
 }
+
+test_oid_cache <<-EOF
+packlib_7_0 sha1:01d7713666f4de822776c7622c10f1b07de280dc
+packlib_7_0 sha256:37c8e2c15bb22b912e59b43fd51a4f7e9465ed0b5084c5a1411d991cbe630683
+
+packlib_7_76 sha1:e68fe8129b546b101aee9510c5328e7f21ca1d18
+packlib_7_76 sha256:5d8e6fc40f2dab00e6983a48523fe57e621f46434cb58dbd4422fba03380d886
+EOF

[PATCH v2 02/22] t3206: make hash size independent

[brian m. carlson]

Fix the one assertion in this test that still uses SHA-1 to use test_oid
to be independent of the hash.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3206-range-diff.sh | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/t/t3206-range-diff.sh b/t/t3206-range-diff.sh
index 0575dd72b1..bd808f87ed 100755
--- a/t/t3206-range-diff.sh
+++ b/t/t3206-range-diff.sh
@@ -102,6 +102,14 @@ test_expect_success 'setup' '
 	n3 sha256:3b0a644
 	n4 sha256:e461653
 
+	# mode change
+	o1 sha1:4d39cb3
+	o2 sha1:26c107f
+	o3 sha1:4c1e0f5
+	o1 sha256:d0dd598
+	o2 sha256:c4a279e
+	o3 sha256:78459d7
+
 	# added and removed
 	s1 sha1:096b1ba
 	s2 sha1:d92e698
@@ -336,7 +344,7 @@ test_expect_success 'renamed file' '
 test_expect_success 'file with mode only change' '
 	git range-diff --no-color --submodule=log topic...mode-only-change >actual &&
 	sed s/Z/\ /g >expect <<-EOF &&
-	1:  fccce22 ! 1:  4d39cb3 s/4/A/
+	1:  $(test_oid t2) ! 1:  $(test_oid o1) s/4/A/
 	    @@ Metadata
 	    ZAuthor: Thomas Rast <trast@inf.ethz.ch>
 	    Z
@@ -352,7 +360,7 @@ test_expect_success 'file with mode only change' '
 	    Z 7
 	    +
 	    + ## other-file (new) ##
-	2:  147e64e ! 2:  26c107f s/11/B/
+	2:  $(test_oid t3) ! 2:  $(test_oid o2) s/11/B/
 	    @@ Metadata
 	    ZAuthor: Thomas Rast <trast@inf.ethz.ch>
 	    Z
@@ -368,7 +376,7 @@ test_expect_success 'file with mode only change' '
 	    Z 14
 	    +
 	    + ## other-file (mode change 100644 => 100755) ##
-	3:  a63e992 = 3:  4c1e0f5 s/12/B/
+	3:  $(test_oid t4) = 3:  $(test_oid o3) s/12/B/
 	EOF
 	test_cmp expect actual
 '

[PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[brian m. carlson]

This test relies on a roughly equal distribution of hashes for notes in
order to ensure that fanouts are compressed.  If there are subtrees with
only one item left after removing notes, they'll end up still with one
level of fanout, causing the test to fail.  The test happens to pass
with SHA-1, but doesn't necessarily with other hash algorithms, so
annotate it with the SHA1 prerequisite.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3305-notes-fanout.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
index 831f83d211..3520402bb8 100755
--- a/t/t3305-notes-fanout.sh
+++ b/t/t3305-notes-fanout.sh
@@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
 	test_cmp expect output
 '
 
-test_expect_success 'deleting most notes triggers fanout consolidation' '
+test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
 	# Expect entire notes tree to have a fanout == 0
 	git ls-tree -r --name-only refs/notes/commits |
 	while read path

[PATCH v2 04/22] t3308: make test work with SHA-256

[brian m. carlson]

Replace the hard-coded SHA-1 constants with the use of test_oid to look
up an appropriate constant for each hash algorithm.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3308-notes-merge.sh | 83 ++++++++++++++++++++++++++++--------------
 1 file changed, 55 insertions(+), 28 deletions(-)

diff --git a/t/t3308-notes-merge.sh b/t/t3308-notes-merge.sh
index d60588ec8f..790e292966 100755
--- a/t/t3308-notes-merge.sh
+++ b/t/t3308-notes-merge.sh
@@ -20,7 +20,34 @@ test_expect_success setup '
 	git notes add -m "Notes on 3rd commit" 3rd &&
 	git notes add -m "Notes on 4th commit" 4th &&
 	# Copy notes to remote-notes
-	git fetch . refs/notes/*:refs/remote-notes/origin/*
+	git fetch . refs/notes/*:refs/remote-notes/origin/* &&
+
+	test_oid_init &&
+	test_oid_cache <<-EOF
+	hash4a sha1:5e93d24084d32e1cb61f7070505b9d2530cca987
+	hash3a sha1:8366731eeee53787d2bdf8fc1eff7d94757e8da0
+	hash2a sha1:eede89064cd42441590d6afec6c37b321ada3389
+	hash1a sha1:daa55ffad6cb99bf64226532147ffcaf5ce8bdd1
+	hash5b sha1:0f2efbd00262f2fd41dfae33df8765618eeacd99
+	hash4b sha1:dec2502dac3ea161543f71930044deff93fa945c
+	hash3b sha1:4069cdb399fd45463ec6eef8e051a16a03592d91
+	hash2c sha1:d000d30e6ddcfce3a8122c403226a2ce2fd04d9d
+	hash1c sha1:43add6bd0c8c0bc871ac7991e0f5573cfba27804
+	hash4d sha1:1f257a3a90328557c452f0817d6cc50c89d315d4
+	hash3d sha1:05a4927951bcef347f51486575b878b2b60137f2
+
+	hash4a sha256:eef876be1d32ac2e2e42240e0429325cec116e55e88cb2969899fac695aa762f
+	hash3a sha256:cf7cd1bc091d7ba4166a86df864110e42087cd893a5ae96bc50d637e0290939d
+	hash2a sha256:21ddde7ebce2c285213898cb04deca0fd3209610cf7aaf8222e4e2f45262fae2
+	hash1a sha256:f9fe0eda16c6027732ed9d4295689a03abd16f893be69b3dcbf4037ddb191921
+	hash5b sha256:20046f2244577797a9e3d3f790ea9eca4d8a6bafb2a5570bcb0e03aa02ce100b
+	hash4b sha256:f90563d134c61a95bb88afbd45d48ccc9e919c62aa6fbfcd483302b3e4d8dbcb
+	hash3b sha256:988f2aca9f2d87e93e6a73197c2bb99560cc44a2f92d18653968f956f01221e0
+	hash2c sha256:84153b777b4d42827a756c6578dcdb59d8ae5d1360b874fb37c430150c825c26
+	hash1c sha256:9beb2bc4eef72e4c4087be168a20573e34d993d9ab1883055f23e322afa06567
+	hash4d sha256:32de39dc06e679a7abb2d4a55ede7709b3124340a4a90aa305971b1c72ac319d
+	hash3d sha256:fa73b20e41cbb7541c4c81d1535016131dbfbeb05bf6a71f6115e9cad31c7af5
+	EOF
 '
 
 commit_sha1=$(git rev-parse 1st^{commit})
@@ -40,10 +67,10 @@ verify_notes () {
 }
 
 cat <<EOF | sort >expect_notes_x
-5e93d24084d32e1cb61f7070505b9d2530cca987 $commit_sha4
-8366731eeee53787d2bdf8fc1eff7d94757e8da0 $commit_sha3
-eede89064cd42441590d6afec6c37b321ada3389 $commit_sha2
-daa55ffad6cb99bf64226532147ffcaf5ce8bdd1 $commit_sha1
+$(test_oid hash4a) $commit_sha4
+$(test_oid hash3a) $commit_sha3
+$(test_oid hash2a) $commit_sha2
+$(test_oid hash1a) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -126,10 +153,10 @@ test_expect_success 'merge previous notes commit (y^ => y) => No-op' '
 '
 
 cat <<EOF | sort >expect_notes_y
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-dec2502dac3ea161543f71930044deff93fa945c $commit_sha4
-4069cdb399fd45463ec6eef8e051a16a03592d91 $commit_sha3
-daa55ffad6cb99bf64226532147ffcaf5ce8bdd1 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4b) $commit_sha4
+$(test_oid hash3b) $commit_sha3
+$(test_oid hash1a) $commit_sha1
 EOF
 
 cat >expect_log_y <<EOF
@@ -193,11 +220,11 @@ test_expect_success 'merge empty notes ref (z => y)' '
 '
 
 cat <<EOF | sort >expect_notes_y
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-dec2502dac3ea161543f71930044deff93fa945c $commit_sha4
-4069cdb399fd45463ec6eef8e051a16a03592d91 $commit_sha3
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
-43add6bd0c8c0bc871ac7991e0f5573cfba27804 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4b) $commit_sha4
+$(test_oid hash3b) $commit_sha3
+$(test_oid hash2c) $commit_sha2
+$(test_oid hash1c) $commit_sha1
 EOF
 
 cat >expect_log_y <<EOF
@@ -231,9 +258,9 @@ test_expect_success 'change notes on other notes ref (y)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-1f257a3a90328557c452f0817d6cc50c89d315d4 $commit_sha4
-daa55ffad6cb99bf64226532147ffcaf5ce8bdd1 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4d) $commit_sha4
+$(test_oid hash1a) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -262,10 +289,10 @@ test_expect_success 'change notes on notes ref (x)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-1f257a3a90328557c452f0817d6cc50c89d315d4 $commit_sha4
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
-43add6bd0c8c0bc871ac7991e0f5573cfba27804 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4d) $commit_sha4
+$(test_oid hash2c) $commit_sha2
+$(test_oid hash1c) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -296,8 +323,8 @@ test_expect_success 'merge y into x => Non-conflicting 3-way merge' '
 '
 
 cat <<EOF | sort >expect_notes_w
-05a4927951bcef347f51486575b878b2b60137f2 $commit_sha3
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
+$(test_oid hash3d) $commit_sha3
+$(test_oid hash2c) $commit_sha2
 EOF
 
 cat >expect_log_w <<EOF
@@ -326,11 +353,11 @@ test_expect_success 'create notes on new, separate notes ref (w)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-0f2efbd00262f2fd41dfae33df8765618eeacd99 $commit_sha5
-1f257a3a90328557c452f0817d6cc50c89d315d4 $commit_sha4
-05a4927951bcef347f51486575b878b2b60137f2 $commit_sha3
-d000d30e6ddcfce3a8122c403226a2ce2fd04d9d $commit_sha2
-43add6bd0c8c0bc871ac7991e0f5573cfba27804 $commit_sha1
+$(test_oid hash5b) $commit_sha5
+$(test_oid hash4d) $commit_sha4
+$(test_oid hash3d) $commit_sha3
+$(test_oid hash2c) $commit_sha2
+$(test_oid hash1c) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF

[PATCH v2 05/22] t3309: make test work with SHA-256

[brian m. carlson]

Replace the hard-coded SHA-1 constants with the use of test_oid to look
up an appropriate constant for each hash algorithm.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3309-notes-merge-auto-resolve.sh | 228 ++++++++++++++++++----------
 1 file changed, 144 insertions(+), 84 deletions(-)

diff --git a/t/t3309-notes-merge-auto-resolve.sh b/t/t3309-notes-merge-auto-resolve.sh
index 14c2adf970..141d3e4ca4 100755
--- a/t/t3309-notes-merge-auto-resolve.sh
+++ b/t/t3309-notes-merge-auto-resolve.sh
@@ -23,7 +23,67 @@ test_expect_success 'setup commits' '
 	test_commit 12th &&
 	test_commit 13th &&
 	test_commit 14th &&
-	test_commit 15th
+	test_commit 15th &&
+
+	test_oid_cache <<-EOF
+	hash15a sha1:457a85d6c814ea208550f15fcc48f804ac8dc023
+	hash14a sha1:b0c95b954301d69da2bc3723f4cb1680d355937c
+	hash13a sha1:5d30216a129eeffa97d9694ffe8c74317a560315
+	hash12a sha1:dd161bc149470fd890dd4ab52a4cbd79bbd18c36
+	hash11a sha1:7abbc45126d680336fb24294f013a7cdfa3ed545
+	hash10a sha1:b8d03e173f67f6505a76f6e00cf93440200dd9be
+	hash09a sha1:20c613c835011c48a5abe29170a2402ca6354910
+	hash08a sha1:a3daf8a1e4e5dc3409a303ad8481d57bfea7f5d6
+	hash07a sha1:897003322b53bc6ca098e9324ee508362347e734
+	hash06a sha1:11d97fdebfa5ceee540a3da07bce6fa0222bc082
+	hash15b sha1:68b8630d25516028bed862719855b3d6768d7833
+	hash14b sha1:5de7ea7ad4f47e7ff91989fb82234634730f75df
+	hash13b sha1:3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc
+	hash12b sha1:a66055fa82f7a03fe0c02a6aba3287a85abf7c62
+	hash05b sha1:154508c7a0bcad82b6fe4b472bc4c26b3bf0825b
+	hash04b sha1:e2bfd06a37dd2031684a59a6e2b033e212239c78
+	hash03b sha1:5772f42408c0dd6f097a7ca2d24de0e78d1c46b1
+	hash15c sha1:9b4b2c61f0615412da3c10f98ff85b57c04ec765
+	hash11c sha1:7e3c53503a3db8dd996cb62e37c66e070b44b54d
+	hash08c sha1:851e1638784a884c7dd26c5d41f3340f6387413a
+	hash05c sha1:99fc34adfc400b95c67b013115e37e31aa9a6d23
+	hash02c sha1:283b48219aee9a4105f6cab337e789065c82c2b9
+	hash15d sha1:7c4e546efd0fe939f876beb262ece02797880b54
+	hash05d sha1:6c841cc36ea496027290967ca96bd2bef54dbb47
+	hash15e sha1:d682107b8bf7a7aea1e537a8d5cb6a12b60135f1
+	hash05e sha1:357b6ca14c7afd59b7f8b8aaaa6b8b723771135b
+	hash15f sha1:6be90240b5f54594203e25d9f2f64b7567175aee
+	hash05f sha1:660311d7f78dc53db12ac373a43fca7465381a7e
+
+	hash15a sha256:45b1558e5c1b75f570010fa48aaa67bb2289fcd431b34ad81cb4c8b95f4f872a
+	hash14a sha256:6e7af179ea4dd28afdc83ae6912ba0098cdeff764b26a8b750b157dd81749092
+	hash13a sha256:7353089961baf555388e1bac68c67c8ea94b08ccbd97532201cf7f6790703052
+	hash12a sha256:5863e4521689ee1879ceab3b38d39e93ab5b51ec70aaf6a96ad388fbdedfa25e
+	hash11a sha256:82a0ec0338b4ecf8b44304badf4ad38d7469dc41827f38d7ba6c42e3bae3ee98
+	hash10a sha256:e84f2564e92de9792c93b8d197262c735d7ccb1de6025cef8759af8f6c3308eb
+	hash09a sha256:4dd07764bcec696f195c0ea71ae89e174876403af1637e4642b8f4453fd23028
+	hash08a sha256:02132c4546cd88a1d0aa5854dd55da120927f7904ba16afe36fe03e91a622067
+	hash07a sha256:369baf7d00c6720efdc10273493555f943051f84a4706fb24caeb353fa4789db
+	hash06a sha256:52d32c10353583b2d96a5849b1f1f43c8018e76f3e8ef1b0d46eb5cff7cdefaf
+	hash15b sha256:345e6660b345fa174738a31a7a59423c394bdf414804e200bc510c65d971ae96
+	hash14b sha256:7653a6596021c52e405cba979eea15a729993e7102b9a61ba4667e34f0ead4a1
+	hash13b sha256:0f202a0b6b9690de2349c173dfd766a37e82744f61c14f1c389306f1d69f470b
+	hash12b sha256:eb00f219c026136ea6535b16ff8ec3efa510e6bf50098ca041e1a2a1d4b79840
+	hash05b sha256:993b2290cd0c24c27c849d99f1904f3b590f77af0f539932734ad05679ac5a2f
+	hash04b sha256:c7fba0d6104917fbf35258f40b9fa4fc697cfa992deecd1570a3b08d0a5587a9
+	hash03b sha256:7287a2d78a3766c181b08df38951d784b08b72a44f571ed6d855bd0be22c70f6
+	hash15c sha256:62316660a22bf97857dc4a16709ec4d93a224e8c9f37d661ef91751e1f4c4166
+	hash11c sha256:51c3763de9b08309370adc5036d58debb331980e73097902957c444602551daa
+	hash08c sha256:22cf1fa29599898a7218c51135d66ed85d22aad584f77db3305dedce4c3d4798
+	hash05c sha256:2508fd86db980f0508893a1c1571bdf3b2ee113dc25ddb1a3a2fb94bd6cd0d58
+	hash02c sha256:63bb527e0b4e1c8e1dd0d54dd778ca7c3718689fd6e37c473044cfbcf1cacfdb
+	hash15d sha256:667acb4e2d5f8df15e5aea4506dfd16d25bc7feca70fdb0d965a7222f983bb88
+	hash05d sha256:09e6b5a6fe666c4a027674b6611a254b7d2528cd211c6b5288d1b4db6c741dfa
+	hash15e sha256:e8cbf52f6fcadc6de3c7761e64a89e9fe38d19a03d3e28ef6ca8596d93fc4f3a
+	hash05e sha256:cdb1e19f7ba1539f95af51a57edeb88a7ecc97d3c2f52da8c4c86af308595607
+	hash15f sha256:29c14cb92da448a923963b8a43994268b19c2e57913de73f3667421fd2c0eeec
+	hash05f sha256:14a6e641b2c0a9f398ebac6b4d34afa5efea4c52d2631382f45f8f662266903b
+	EOF
 '
 
 commit_sha1=$(git rev-parse 1st^{commit})
@@ -68,16 +128,16 @@ test_expect_success 'setup merge base (x)' '
 '
 
 cat <<EOF | sort >expect_notes_x
-457a85d6c814ea208550f15fcc48f804ac8dc023 $commit_sha15
-b0c95b954301d69da2bc3723f4cb1680d355937c $commit_sha14
-5d30216a129eeffa97d9694ffe8c74317a560315 $commit_sha13
-dd161bc149470fd890dd4ab52a4cbd79bbd18c36 $commit_sha12
-7abbc45126d680336fb24294f013a7cdfa3ed545 $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-20c613c835011c48a5abe29170a2402ca6354910 $commit_sha9
-a3daf8a1e4e5dc3409a303ad8481d57bfea7f5d6 $commit_sha8
-897003322b53bc6ca098e9324ee508362347e734 $commit_sha7
-11d97fdebfa5ceee540a3da07bce6fa0222bc082 $commit_sha6
+$(test_oid hash15a) $commit_sha15
+$(test_oid hash14a) $commit_sha14
+$(test_oid hash13a) $commit_sha13
+$(test_oid hash12a) $commit_sha12
+$(test_oid hash11a) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash09a) $commit_sha9
+$(test_oid hash08a) $commit_sha8
+$(test_oid hash07a) $commit_sha7
+$(test_oid hash06a) $commit_sha6
 EOF
 
 cat >expect_log_x <<EOF
@@ -141,16 +201,16 @@ test_expect_success 'setup local branch (y)' '
 '
 
 cat <<EOF | sort >expect_notes_y
-68b8630d25516028bed862719855b3d6768d7833 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7abbc45126d680336fb24294f013a7cdfa3ed545 $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-20c613c835011c48a5abe29170a2402ca6354910 $commit_sha9
-154508c7a0bcad82b6fe4b472bc4c26b3bf0825b $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
+$(test_oid hash15b) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11a) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash09a) $commit_sha9
+$(test_oid hash05b) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
 EOF
 
 cat >expect_log_y <<EOF
@@ -214,16 +274,16 @@ test_expect_success 'setup remote branch (z)' '
 '
 
 cat <<EOF | sort >expect_notes_z
-9b4b2c61f0615412da3c10f98ff85b57c04ec765 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-5d30216a129eeffa97d9694ffe8c74317a560315 $commit_sha13
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-897003322b53bc6ca098e9324ee508362347e734 $commit_sha7
-99fc34adfc400b95c67b013115e37e31aa9a6d23 $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15c) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13a) $commit_sha13
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash07a) $commit_sha7
+$(test_oid hash05c) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_z <<EOF
@@ -306,16 +366,16 @@ test_expect_success 'merge z into y with invalid configuration option => Fail/No
 '
 
 cat <<EOF | sort >expect_notes_ours
-68b8630d25516028bed862719855b3d6768d7833 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-154508c7a0bcad82b6fe4b472bc4c26b3bf0825b $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15b) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash05b) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_ours <<EOF
@@ -395,16 +455,16 @@ test_expect_success 'reset to pre-merge state (y)' '
 '
 
 cat <<EOF | sort >expect_notes_theirs
-9b4b2c61f0615412da3c10f98ff85b57c04ec765 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-99fc34adfc400b95c67b013115e37e31aa9a6d23 $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15c) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05c) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_theirs <<EOF
@@ -473,17 +533,17 @@ test_expect_success 'reset to pre-merge state (y)' '
 '
 
 cat <<EOF | sort >expect_notes_union
-7c4e546efd0fe939f876beb262ece02797880b54 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-6c841cc36ea496027290967ca96bd2bef54dbb47 $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15d) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05d) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_union <<EOF
@@ -574,17 +634,17 @@ test_expect_success 'merge z into y with "manual" per-ref only checks specific r
 '
 
 cat <<EOF | sort >expect_notes_union2
-d682107b8bf7a7aea1e537a8d5cb6a12b60135f1 $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-357b6ca14c7afd59b7f8b8aaaa6b8b723771135b $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15e) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05e) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_union2 <<EOF
@@ -648,17 +708,17 @@ test_expect_success 'reset to pre-merge state (z)' '
 '
 
 cat <<EOF | sort >expect_notes_cat_sort_uniq
-6be90240b5f54594203e25d9f2f64b7567175aee $commit_sha15
-5de7ea7ad4f47e7ff91989fb82234634730f75df $commit_sha14
-3a631fdb6f41b05b55d8f4baf20728ba8f6fccbc $commit_sha13
-a66055fa82f7a03fe0c02a6aba3287a85abf7c62 $commit_sha12
-7e3c53503a3db8dd996cb62e37c66e070b44b54d $commit_sha11
-b8d03e173f67f6505a76f6e00cf93440200dd9be $commit_sha10
-851e1638784a884c7dd26c5d41f3340f6387413a $commit_sha8
-660311d7f78dc53db12ac373a43fca7465381a7e $commit_sha5
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
+$(test_oid hash15f) $commit_sha15
+$(test_oid hash14b) $commit_sha14
+$(test_oid hash13b) $commit_sha13
+$(test_oid hash12b) $commit_sha12
+$(test_oid hash11c) $commit_sha11
+$(test_oid hash10a) $commit_sha10
+$(test_oid hash08c) $commit_sha8
+$(test_oid hash05f) $commit_sha5
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
 EOF
 
 cat >expect_log_cat_sort_uniq <<EOF

[PATCH v2 06/22] t3310: make test work with SHA-256

[brian m. carlson]

Replace the hard-coded SHA-1 constants with the use of test_oid to look
up an appropriate constant for each hash algorithm.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3310-notes-merge-manual-resolve.sh | 84 ++++++++++++++++++---------
 1 file changed, 58 insertions(+), 26 deletions(-)

diff --git a/t/t3310-notes-merge-manual-resolve.sh b/t/t3310-notes-merge-manual-resolve.sh
index 2dea846e25..806d812a17 100755
--- a/t/t3310-notes-merge-manual-resolve.sh
+++ b/t/t3310-notes-merge-manual-resolve.sh
@@ -13,7 +13,39 @@ test_expect_success 'setup commits' '
 	test_commit 2nd &&
 	test_commit 3rd &&
 	test_commit 4th &&
-	test_commit 5th
+	test_commit 5th &&
+
+	test_oid_cache <<-EOF
+	hash04a sha1:6e8e3febca3c2bb896704335cc4d0c34cb2f8715
+	hash03a sha1:e5388c10860456ee60673025345fe2e153eb8cf8
+	hash02a sha1:ceefa674873670e7ecd131814d909723cce2b669
+	hash04b sha1:e2bfd06a37dd2031684a59a6e2b033e212239c78
+	hash03b sha1:5772f42408c0dd6f097a7ca2d24de0e78d1c46b1
+	hash01b sha1:b0a6021ec006d07e80e9b20ec9b444cbd9d560d3
+	hash04c sha1:cff59c793c20bb49a4e01bc06fb06bad642e0d54
+	hash02c sha1:283b48219aee9a4105f6cab337e789065c82c2b9
+	hash01c sha1:0a81da8956346e19bcb27a906f04af327e03e31b
+	hash04d sha1:00494adecf2d9635a02fa431308d67993f853968
+	hash01e sha1:f75d1df88cbfe4258d49852f26cfc83f2ad4494b
+	hash04f sha1:021faa20e931fb48986ffc6282b4bb05553ac946
+	hash01f sha1:0a59e787e6d688aa6309e56e8c1b89431a0fc1c1
+	hash05g sha1:304dfb4325cf243025b9957486eb605a9b51c199
+
+	hash04a	sha256:f18a935e65866345098b3b754071dbf9f3aa3520eb27a7b036b278c5e2f1ed7e
+	hash03a	sha256:713035dc94067a64e5fa6e4e1821b7c3bde49a77c7cb3f80eaadefa1ca41b3d2
+	hash02a	sha256:f160a67e048b6fa75bec3952184154045076692cf5dccd3da21e3fd34b7a3f0f
+	hash04b sha256:c7fba0d6104917fbf35258f40b9fa4fc697cfa992deecd1570a3b08d0a5587a9
+	hash03b sha256:7287a2d78a3766c181b08df38951d784b08b72a44f571ed6d855bd0be22c70f6
+	hash01b sha256:da96cf778c15d0a2bb76f98b2a62f6c9c01730fa7030e8f08ef0191048e7d620
+	hash04c sha256:cb615d2def4b834d5f55b2351df97dc92bee4f5009d285201427f349081c8aca
+	hash02c sha256:63bb527e0b4e1c8e1dd0d54dd778ca7c3718689fd6e37c473044cfbcf1cacfdb
+	hash01c sha256:5b87237ac1fbae0246256fed9f9a1f077c4140fb7e6444925f8dbfa5ae406cd8
+	hash04d sha256:eeddc9f9f6cb3d6b39b861659853f10891dc373e0b6eecb09e03e39b6ce64714
+	hash01e sha256:108f521b1a74c2e6d0b52a4eda87e09162bf847f7d190cfce496ee1af0b29a5a
+	hash04f sha256:901acda0454502b3bbd281f130c419e6c8de78afcf72a8def8d45ad31462bce4
+	hash01f sha256:a2d99d1b8bf23c8af7d9d91368454adc110dfd5cc068a4cebb486ee8f5a1e16c
+	hash05g sha256:4fef015b01da8efe929a68e3bb9b8fbad81f53995f097befe8ebc93f12ab98ec
+	EOF
 '
 
 commit_sha1=$(git rev-parse 1st^{commit})
@@ -33,9 +65,9 @@ verify_notes () {
 }
 
 cat <<EOF | sort >expect_notes_x
-6e8e3febca3c2bb896704335cc4d0c34cb2f8715 $commit_sha4
-e5388c10860456ee60673025345fe2e153eb8cf8 $commit_sha3
-ceefa674873670e7ecd131814d909723cce2b669 $commit_sha2
+$(test_oid hash04a) $commit_sha4
+$(test_oid hash03a) $commit_sha3
+$(test_oid hash02a) $commit_sha2
 EOF
 
 cat >expect_log_x <<EOF
@@ -63,9 +95,9 @@ test_expect_success 'setup merge base (x)' '
 '
 
 cat <<EOF | sort >expect_notes_y
-e2bfd06a37dd2031684a59a6e2b033e212239c78 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-b0a6021ec006d07e80e9b20ec9b444cbd9d560d3 $commit_sha1
+$(test_oid hash04b) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash01b) $commit_sha1
 EOF
 
 cat >expect_log_y <<EOF
@@ -95,9 +127,9 @@ test_expect_success 'setup local branch (y)' '
 '
 
 cat <<EOF | sort >expect_notes_z
-cff59c793c20bb49a4e01bc06fb06bad642e0d54 $commit_sha4
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a81da8956346e19bcb27a906f04af327e03e31b $commit_sha1
+$(test_oid hash04c) $commit_sha4
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01c) $commit_sha1
 EOF
 
 cat >expect_log_z <<EOF
@@ -193,9 +225,9 @@ test_expect_success 'merge z into m (== y) with default ("manual") resolver => C
 '
 
 cat <<EOF | sort >expect_notes_z
-00494adecf2d9635a02fa431308d67993f853968 $commit_sha4
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a81da8956346e19bcb27a906f04af327e03e31b $commit_sha1
+$(test_oid hash04d) $commit_sha4
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01c) $commit_sha1
 EOF
 
 cat >expect_log_z <<EOF
@@ -231,8 +263,8 @@ test_expect_success 'cannot do merge w/conflicts when previous merge is unfinish
 # Setup non-conflicting merge between x and new notes ref w
 
 cat <<EOF | sort >expect_notes_w
-ceefa674873670e7ecd131814d909723cce2b669 $commit_sha2
-f75d1df88cbfe4258d49852f26cfc83f2ad4494b $commit_sha1
+$(test_oid hash02a) $commit_sha2
+$(test_oid hash01e) $commit_sha1
 EOF
 
 cat >expect_log_w <<EOF
@@ -258,10 +290,10 @@ test_expect_success 'setup unrelated notes ref (w)' '
 '
 
 cat <<EOF | sort >expect_notes_w
-6e8e3febca3c2bb896704335cc4d0c34cb2f8715 $commit_sha4
-e5388c10860456ee60673025345fe2e153eb8cf8 $commit_sha3
-ceefa674873670e7ecd131814d909723cce2b669 $commit_sha2
-f75d1df88cbfe4258d49852f26cfc83f2ad4494b $commit_sha1
+$(test_oid hash04a) $commit_sha4
+$(test_oid hash03a) $commit_sha3
+$(test_oid hash02a) $commit_sha2
+$(test_oid hash01e) $commit_sha1
 EOF
 
 cat >expect_log_w <<EOF
@@ -291,10 +323,10 @@ test_expect_success 'can do merge without conflicts even if previous merge is un
 '
 
 cat <<EOF | sort >expect_notes_m
-021faa20e931fb48986ffc6282b4bb05553ac946 $commit_sha4
-5772f42408c0dd6f097a7ca2d24de0e78d1c46b1 $commit_sha3
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a59e787e6d688aa6309e56e8c1b89431a0fc1c1 $commit_sha1
+$(test_oid hash04f) $commit_sha4
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01f) $commit_sha1
 EOF
 
 cat >expect_log_m <<EOF
@@ -430,9 +462,9 @@ test_expect_success 'redo merge of z into m (== y) with default ("manual") resol
 '
 
 cat <<EOF | sort >expect_notes_m
-304dfb4325cf243025b9957486eb605a9b51c199 $commit_sha5
-283b48219aee9a4105f6cab337e789065c82c2b9 $commit_sha2
-0a59e787e6d688aa6309e56e8c1b89431a0fc1c1 $commit_sha1
+$(test_oid hash05g) $commit_sha5
+$(test_oid hash02c) $commit_sha2
+$(test_oid hash01f) $commit_sha1
 EOF
 
 cat >expect_log_m <<EOF

[PATCH v2 07/22] t3311: make test work with SHA-256

[brian m. carlson]

Replace the hard-coded SHA-1 constants with the use of test_oid to look
up an appropriate constant for each hash algorithm.  In addition, adjust
the fanout checks to look for either zero or one slashes in the filename
without needing to check for an explicit length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t3311-notes-merge-fanout.sh | 60 ++++++++++++++++++++---------------
 1 file changed, 35 insertions(+), 25 deletions(-)

diff --git a/t/t3311-notes-merge-fanout.sh b/t/t3311-notes-merge-fanout.sh
index 37151a3adc..5b675417e9 100755
--- a/t/t3311-notes-merge-fanout.sh
+++ b/t/t3311-notes-merge-fanout.sh
@@ -29,15 +29,10 @@ verify_fanout () {
 	git ls-tree -r --name-only "refs/notes/$notes_ref" |
 	while read path
 	do
-		case "$path" in
-		??/??????????????????????????????????????)
-			: true
-			;;
-		*)
+		echo "$path" | grep "^../[0-9a-f]*$" || {
 			echo "Invalid path \"$path\"" &&
-			return 1
-			;;
-		esac
+			return 1;
+		}
 	done
 }
 
@@ -48,15 +43,10 @@ verify_no_fanout () {
 	git ls-tree -r --name-only "refs/notes/$notes_ref" |
 	while read path
 	do
-		case "$path" in
-		????????????????????????????????????????)
-			: true
-			;;
-		*)
+		echo "$path" | grep -v "^../.*" || {
 			echo "Invalid path \"$path\"" &&
-			return 1
-			;;
-		esac
+			return 1;
+		}
 	done
 }
 
@@ -67,7 +57,27 @@ test_expect_success 'setup a few initial commits with notes (notes ref: x)' '
 	do
 		test_commit "commit$i" >/dev/null &&
 		git notes add -m "notes for commit$i" || return 1
-	done
+	done &&
+
+	git log --format=oneline &&
+
+	test_oid_cache <<-EOF
+	hash05a sha1:aed91155c7a72c2188e781fdf40e0f3761b299db
+	hash04a sha1:99fab268f9d7ee7b011e091a436c78def8eeee69
+	hash03a sha1:953c20ae26c7aa0b428c20693fe38bc687f9d1a9
+	hash02a sha1:6358796131b8916eaa2dde6902642942a1cb37e1
+	hash01a sha1:b02d459c32f0e68f2fe0981033bb34f38776ba47
+	hash03b sha1:9f506ee70e20379d7f78204c77b334f43d77410d
+	hash02b sha1:23a47d6ea7d589895faf800752054818e1e7627b
+
+	hash05a sha256:3aae5d26619d96dba93795f66325716e4cbc486884f95a6adee8fb0615a76d12
+	hash04a sha256:07e43dd3d89fe634d3252e253b426aacc7285a995dcdbcf94ac284060a1122cf
+	hash03a sha256:26fb52eaa7f4866bf735254587be7b31209ec10e525912ffd8e8ba549ba892ff
+	hash02a sha256:b57ebdf23634e750dcbc4b9a37991d70f90830d568a0e4529ce9de0a3f8d605c
+	hash01a sha256:377903b1572bd5117087a5518fcb1011b5053cccbc59e3c7c823a8615204173b
+	hash03b sha256:04e7b392fda7c185bfa17c9179b56db732edc2dc2b3bf887308dcaabb717270d
+	hash02b sha256:66099aaaec49a485ed990acadd9a9b81232ea592079964113d8f581ff69ef50b
+	EOF
 '
 
 commit_sha1=$(git rev-parse commit1^{commit})
@@ -77,11 +87,11 @@ commit_sha4=$(git rev-parse commit4^{commit})
 commit_sha5=$(git rev-parse commit5^{commit})
 
 cat <<EOF | sort >expect_notes_x
-aed91155c7a72c2188e781fdf40e0f3761b299db $commit_sha5
-99fab268f9d7ee7b011e091a436c78def8eeee69 $commit_sha4
-953c20ae26c7aa0b428c20693fe38bc687f9d1a9 $commit_sha3
-6358796131b8916eaa2dde6902642942a1cb37e1 $commit_sha2
-b02d459c32f0e68f2fe0981033bb34f38776ba47 $commit_sha1
+$(test_oid hash05a) $commit_sha5
+$(test_oid hash04a) $commit_sha4
+$(test_oid hash03a) $commit_sha3
+$(test_oid hash02a) $commit_sha2
+$(test_oid hash01a) $commit_sha1
 EOF
 
 cat >expect_log_x <<EOF
@@ -145,9 +155,9 @@ test_expect_success 'Fast-forward merge (y => x)' '
 '
 
 cat <<EOF | sort >expect_notes_z
-9f506ee70e20379d7f78204c77b334f43d77410d $commit_sha3
-23a47d6ea7d589895faf800752054818e1e7627b $commit_sha2
-b02d459c32f0e68f2fe0981033bb34f38776ba47 $commit_sha1
+$(test_oid hash03b) $commit_sha3
+$(test_oid hash02b) $commit_sha2
+$(test_oid hash01a) $commit_sha1
 EOF
 
 cat >expect_log_z <<EOF

[PATCH v2 08/22] t4013: make test hash independent

[brian m. carlson]

This test produces a large number of diff formats and compares the
output with test files that have content specific to SHA-1. Since we are
more interested in the format of the diffs, and not their specific
values, which are tested elsewhere, add a function which uses sed to
transform these specific object IDs into generic ones of the right size,
which we can then compare.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t4013-diff-various.sh | 44 +++++++++++++++++++++++++++++++++++------
 1 file changed, 38 insertions(+), 6 deletions(-)

diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
index 5ac94b390d..6f5f05c3a8 100755
--- a/t/t4013-diff-various.sh
+++ b/t/t4013-diff-various.sh
@@ -120,6 +120,30 @@ test_expect_success setup '
 +*++ [initial] Initial
 EOF
 
+process_diffs () {
+	x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
+	x07="$_x05[0-9a-f][0-9a-f]" &&
+	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
+	    -e "s/From $_x40 /From $ZERO_OID /" \
+	    -e "s/from $_x40)/from $ZERO_OID)/" \
+	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
+	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
+	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
+	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
+	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
+	    -e "s/^$_x40 /$ZERO_OID /" \
+	    -e "s/^$_x40$/$ZERO_OID/" \
+	    -e "s/$x07\.\.$x07/fffffff..fffffff/g" \
+	    -e "s/$x07,$x07\.\.$x07/fffffff,fffffff..fffffff/g" \
+	    -e "s/$x07 $x07 $x07/fffffff fffffff fffffff/g" \
+	    -e "s/$x07 $x07 /fffffff fffffff /g" \
+	    -e "s/Merge: $x07 $x07/Merge: fffffff fffffff/g" \
+	    -e "s/$x07\.\.\./fffffff.../g" \
+	    -e "s/ $x04\.\.\./ ffff.../g" \
+	    -e "s/ $x04/ ffff/g" \
+	    "$1"
+}
+
 V=$(git version | sed -e 's/^git version //' -e 's/\./\\./g')
 while read magic cmd
 do
@@ -158,13 +182,15 @@ do
 		} >"$actual" &&
 		if test -f "$expect"
 		then
+			process_diffs "$actual" >actual &&
+			process_diffs "$expect" >expect &&
 			case $cmd in
 			*format-patch* | *-stat*)
-				test_i18ncmp "$expect" "$actual";;
+				test_i18ncmp expect actual;;
 			*)
-				test_cmp "$expect" "$actual";;
+				test_cmp expect actual;;
 			esac &&
-			rm -f "$actual"
+			rm -f "$actual" actual expect
 		else
 			# this is to help developing new tests.
 			cp "$actual" "$expect"
@@ -383,16 +409,22 @@ test_expect_success 'log -S requires an argument' '
 test_expect_success 'diff --cached on unborn branch' '
 	echo ref: refs/heads/unborn >.git/HEAD &&
 	git diff --cached >result &&
-	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached" result
+	process_diffs result >actual &&
+	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached" >expected &&
+	test_cmp expected actual
 '
 
 test_expect_success 'diff --cached -- file on unborn branch' '
 	git diff --cached -- file0 >result &&
-	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" result
+	process_diffs result >actual &&
+	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" >expected &&
+	test_cmp expected actual
 '
 test_expect_success 'diff --line-prefix with spaces' '
 	git diff --line-prefix="| | | " --cached -- file0 >result &&
-	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" result
+	process_diffs result >actual &&
+	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" >expected &&
+	test_cmp expected actual
 '
 
 test_expect_success 'diff-tree --stdin with log formatting' '

[PATCH v2 09/22] t4060: make test work with SHA-256

[brian m. carlson]

In this test, there are two main types of object IDs we see in the
diffs: the ones for the submodules, which we care about, and the ones
for the individual files, which are unrelated to what we're testing.
Much of the test already computes the former, so extend the rest of the
test to do so as well.  Add a diff comparison function that normalizes
the differences in the latter, since they're not explicitly what we're
testing.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t4060-diff-submodule-option-diff-format.sh | 126 ++++++++++---------
 1 file changed, 70 insertions(+), 56 deletions(-)

diff --git a/t/t4060-diff-submodule-option-diff-format.sh b/t/t4060-diff-submodule-option-diff-format.sh
index 9dcb69df5c..fc8229c726 100755
--- a/t/t4060-diff-submodule-option-diff-format.sh
+++ b/t/t4060-diff-submodule-option-diff-format.sh
@@ -42,6 +42,17 @@ commit_file () {
 	git commit "$@" -m "Commit $*" >/dev/null
 }
 
+diff_cmp () {
+       for i in "$1" "$2"
+       do
+		sed -e 's/^index 0000000\.\.[0-9a-f]*/index 0000000..1234567/' \
+		-e 's/^index [0-9a-f]*\.\.[0-9a-f]*/index 1234567..89abcde/' \
+		"$i" >"$i.compare" || return 1
+       done &&
+       test_cmp "$1.compare" "$2.compare" &&
+       rm -f "$1.compare" "$2.compare"
+}
+
 test_expect_success 'setup repository' '
 	test_create_repo sm1 &&
 	add_file . foo &&
@@ -69,7 +80,7 @@ test_expect_success 'added submodule' '
 	@@ -0,0 +1 @@
 	+foo2
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'added submodule, set diff.submodule' '
@@ -93,7 +104,7 @@ test_expect_success 'added submodule, set diff.submodule' '
 	@@ -0,0 +1 @@
 	+foo2
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success '--submodule=short overrides diff.submodule' '
@@ -109,7 +120,7 @@ test_expect_success '--submodule=short overrides diff.submodule' '
 	@@ -0,0 +1 @@
 	+Subproject commit $fullhead1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'diff.submodule does not affect plumbing' '
@@ -124,7 +135,7 @@ test_expect_success 'diff.submodule does not affect plumbing' '
 	@@ -0,0 +1 @@
 	+Subproject commit $fullhead1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
@@ -142,7 +153,7 @@ test_expect_success 'modified submodule(forward)' '
 	@@ -0,0 +1 @@
 	+foo3
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule(forward)' '
@@ -157,7 +168,7 @@ test_expect_success 'modified submodule(forward)' '
 	@@ -0,0 +1 @@
 	+foo3
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule(forward) --submodule' '
@@ -166,7 +177,7 @@ test_expect_success 'modified submodule(forward) --submodule' '
 	Submodule sm1 $head1..$head2:
 	  > Add foo3 ($added foo3)
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 fullhead2=$(cd sm1; git rev-parse --verify HEAD)
@@ -181,7 +192,7 @@ test_expect_success 'modified submodule(forward) --submodule=short' '
 	-Subproject commit $fullhead1
 	+Subproject commit $fullhead2
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
@@ -210,7 +221,7 @@ test_expect_success 'modified submodule(backward)' '
 	@@ -1 +0,0 @@
 	-foo3
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 head4=$(add_file sm1 foo4 foo5)
@@ -247,7 +258,7 @@ test_expect_success 'modified submodule(backward and forward)' '
 	@@ -0,0 +1 @@
 	+foo5
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
@@ -291,7 +302,7 @@ test_expect_success 'typechanged submodule(submodule->blob), --cached' '
 	@@ -0,0 +1 @@
 	+sm1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'typechanged submodule(submodule->blob)' '
@@ -327,7 +338,7 @@ test_expect_success 'typechanged submodule(submodule->blob)' '
 	@@ -0,0 +1 @@
 	+foo5
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 rm -rf sm1 &&
@@ -344,7 +355,7 @@ test_expect_success 'typechanged submodule(submodule->blob)' '
 	@@ -0,0 +1 @@
 	+sm1
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 rm -f sm1 &&
@@ -356,7 +367,7 @@ test_expect_success 'nonexistent commit' '
 	cat >expected <<-EOF &&
 	Submodule sm1 $head4...$head6 (commits not present)
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file
@@ -386,11 +397,12 @@ test_expect_success 'typechanged submodule(blob->submodule)' '
 	@@ -0,0 +1 @@
 	+foo7
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm1 &&
 test_expect_success 'submodule is up to date' '
+	head7=$(git -C sm1 rev-parse --short --verify HEAD) &&
 	git diff-index -p --submodule=diff HEAD >actual &&
 	test_must_be_empty actual
 '
@@ -401,7 +413,7 @@ test_expect_success 'submodule contains untracked content' '
 	cat >expected <<-EOF &&
 	Submodule sm1 contains untracked content
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'submodule contains untracked content (untracked ignored)' '
@@ -433,7 +445,7 @@ test_expect_success 'submodule contains untracked and modified content' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 # NOT OK
@@ -450,7 +462,7 @@ test_expect_success 'submodule contains untracked and modified content (untracke
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'submodule contains untracked and modified content (dirty ignored)' '
@@ -478,7 +490,7 @@ test_expect_success 'submodule contains modified content' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 (cd sm1; git commit -mchange foo6 >/dev/null) &&
@@ -486,7 +498,7 @@ head8=$(cd sm1; git rev-parse --short --verify HEAD) &&
 test_expect_success 'submodule is modified' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..$head8:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -495,7 +507,7 @@ test_expect_success 'submodule is modified' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content' '
@@ -503,7 +515,7 @@ test_expect_success 'modified submodule contains untracked content' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
 	Submodule sm1 contains untracked content
-	Submodule sm1 17243c9..$head8:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -512,13 +524,13 @@ test_expect_success 'modified submodule contains untracked content' '
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content (untracked ignored)' '
 	git diff-index -p --ignore-submodules=untracked --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..$head8:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -527,13 +539,13 @@ test_expect_success 'modified submodule contains untracked content (untracked ig
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content (dirty ignored)' '
 	git diff-index -p --ignore-submodules=dirty --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -542,7 +554,7 @@ test_expect_success 'modified submodule contains untracked content (dirty ignore
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked content (all ignored)' '
@@ -556,7 +568,7 @@ test_expect_success 'modified submodule contains untracked and modified content'
 	cat >expected <<-EOF &&
 	Submodule sm1 contains untracked content
 	Submodule sm1 contains modified content
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..dfda541 100644
 	--- a/sm1/foo6
@@ -566,7 +578,7 @@ test_expect_success 'modified submodule contains untracked and modified content'
 	+new
 	+modification
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked and modified content (untracked ignored)' '
@@ -574,7 +586,7 @@ test_expect_success 'modified submodule contains untracked and modified content
 	git diff-index -p --ignore-submodules=untracked --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
 	Submodule sm1 contains modified content
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..e20e2d9 100644
 	--- a/sm1/foo6
@@ -585,14 +597,14 @@ test_expect_success 'modified submodule contains untracked and modified content
 	+modification
 	+modification
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked and modified content (dirty ignored)' '
 	echo modification >> sm1/foo6 &&
 	git diff-index -p --ignore-submodules=dirty --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..3e75765 100644
 	--- a/sm1/foo6
@@ -601,7 +613,7 @@ test_expect_success 'modified submodule contains untracked and modified content
 	-foo6
 	+new
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'modified submodule contains untracked and modified content (all ignored)' '
@@ -616,7 +628,7 @@ test_expect_success 'modified submodule contains modified content' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
 	Submodule sm1 contains modified content
-	Submodule sm1 17243c9..cfce562:
+	Submodule sm1 $head7..$head8:
 	diff --git a/sm1/foo6 b/sm1/foo6
 	index 462398b..ac466ca 100644
 	--- a/sm1/foo6
@@ -629,29 +641,29 @@ test_expect_success 'modified submodule contains modified content' '
 	+modification
 	+modification
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 rm -rf sm1
 test_expect_success 'deleted submodule' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
+	Submodule sm1 $head7...0000000 (submodule deleted)
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'create second submodule' '
 	test_create_repo sm2 &&
-	head7=$(add_file sm2 foo8 foo9) &&
+	head9=$(add_file sm2 foo8 foo9) &&
 	git add sm2
 '
 
 test_expect_success 'multiple submodules' '
 	git diff-index -p --submodule=diff HEAD >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm1 $head7...0000000 (submodule deleted)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -667,13 +679,13 @@ test_expect_success 'multiple submodules' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'path filter' '
 	git diff-index -p --submodule=diff HEAD sm2 >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -689,15 +701,15 @@ test_expect_success 'path filter' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 commit_file sm2
 test_expect_success 'given commit' '
 	git diff-index -p --submodule=diff HEAD^ >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm1 $head7...0000000 (submodule deleted)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -713,7 +725,7 @@ test_expect_success 'given commit' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'setup .git file for sm2' '
@@ -726,8 +738,8 @@ test_expect_success 'setup .git file for sm2' '
 test_expect_success 'diff --submodule=diff with .git file' '
 	git diff --submodule=diff HEAD^ >actual &&
 	cat >expected <<-EOF &&
-	Submodule sm1 17243c9...0000000 (submodule deleted)
-	Submodule sm2 0000000...a5a65c9 (new submodule)
+	Submodule sm1 $head7...0000000 (submodule deleted)
+	Submodule sm2 0000000...$head9 (new submodule)
 	diff --git a/sm2/foo8 b/sm2/foo8
 	new file mode 100644
 	index 0000000..db9916b
@@ -743,25 +755,27 @@ test_expect_success 'diff --submodule=diff with .git file' '
 	@@ -0,0 +1 @@
 	+foo9
 	EOF
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'setup nested submodule' '
 	git submodule add -f ./sm2 &&
 	git commit -a -m "add sm2" &&
 	git -C sm2 submodule add ../sm2 nested &&
-	git -C sm2 commit -a -m "nested sub"
+	git -C sm2 commit -a -m "nested sub" &&
+	head10=$(git -C sm2 rev-parse --short --verify HEAD)
 '
 
 test_expect_success 'move nested submodule HEAD' '
 	echo "nested content" >sm2/nested/file &&
 	git -C sm2/nested add file &&
-	git -C sm2/nested commit --allow-empty -m "new HEAD"
+	git -C sm2/nested commit --allow-empty -m "new HEAD" &&
+	head11=$(git -C sm2/nested rev-parse --short --verify HEAD)
 '
 
 test_expect_success 'diff --submodule=diff with moved nested submodule HEAD' '
 	cat >expected <<-EOF &&
-	Submodule nested a5a65c9..b55928c:
+	Submodule nested $head9..$head11:
 	diff --git a/nested/file b/nested/file
 	new file mode 100644
 	index 0000000..ca281f5
@@ -772,13 +786,13 @@ test_expect_success 'diff --submodule=diff with moved nested submodule HEAD' '
 	EOF
 	git -C sm2 diff --submodule=diff >actual 2>err &&
 	test_must_be_empty err &&
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_expect_success 'diff --submodule=diff recurses into nested submodules' '
 	cat >expected <<-EOF &&
 	Submodule sm2 contains modified content
-	Submodule sm2 a5a65c9..280969a:
+	Submodule sm2 $head9..$head10:
 	diff --git a/sm2/.gitmodules b/sm2/.gitmodules
 	new file mode 100644
 	index 0000000..3a816b8
@@ -788,7 +802,7 @@ test_expect_success 'diff --submodule=diff recurses into nested submodules' '
 	+[submodule "nested"]
 	+	path = nested
 	+	url = ../sm2
-	Submodule nested 0000000...b55928c (new submodule)
+	Submodule nested 0000000...$head11 (new submodule)
 	diff --git a/sm2/nested/file b/sm2/nested/file
 	new file mode 100644
 	index 0000000..ca281f5
@@ -813,7 +827,7 @@ test_expect_success 'diff --submodule=diff recurses into nested submodules' '
 	EOF
 	git diff --submodule=diff >actual 2>err &&
 	test_must_be_empty err &&
-	test_cmp expected actual
+	diff_cmp expected actual
 '
 
 test_done

[PATCH v2 10/22] t4211: make test hash independent

[brian m. carlson]

This test uses several test files that contain hard-coded SHA-1 object
IDs. Replace these values with generic ones of the correct size so that
the test works with either SHA-1 or SHA-256.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t4211-line-log.sh | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/t/t4211-line-log.sh b/t/t4211-line-log.sh
index 8319163744..2cbfd8dd9e 100755
--- a/t/t4211-line-log.sh
+++ b/t/t4211-line-log.sh
@@ -8,10 +8,20 @@ test_expect_success 'setup (import history)' '
 	git reset --hard
 '
 
+process_output () {
+	x07="$_x05[0-9a-f][0-9a-f]"
+	sed -e "s/commit $OID_REGEX/commit $ZERO_OID/" \
+	    -e "s/commit $_x40$/commit $ZERO_OID/" \
+	    -e "s/Merge: $x07 $x07$/Merge: 0000000 0000000/" \
+	    "$1"
+}
+
 canned_test_1 () {
 	test_expect_$1 "$2" "
-		git log $2 >actual &&
-		test_cmp \"\$TEST_DIRECTORY\"/t4211/expect.$3 actual
+		git log $2 >result &&
+		process_output result >actual &&
+		process_output \"\$TEST_DIRECTORY\"/t4211/expect.$3 >expected &&
+		test_cmp expected actual
 	"
 }
 

[PATCH v2 11/22] t5302: make hash size independent

[brian m. carlson]

Compute the length of object IDs and pack offsets instead of hard-coding
constants.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5302-pack-index.sh | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
index 91d51b35f9..93ac003639 100755
--- a/t/t5302-pack-index.sh
+++ b/t/t5302-pack-index.sh
@@ -8,7 +8,8 @@ test_description='pack index with 64-bit offsets and object CRC'
 
 test_expect_success \
     'setup' \
-    'rm -rf .git &&
+    'test_oid_init &&
+     rm -rf .git &&
      git init &&
      git config pack.threads 1 &&
      i=1 &&
@@ -32,7 +33,9 @@ test_expect_success \
 	 echo $tree &&
 	 git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\)	.*/\\1/"
      } >obj-list &&
-     git update-ref HEAD $commit'
+     git update-ref HEAD $commit &&
+     rawsz=$(test_oid rawsz)
+'
 
 test_expect_success \
     'pack-objects with index version 1' \
@@ -152,6 +155,7 @@ test_expect_success \
     '[index v1] 2) create a stealth corruption in a delta base reference' \
     '# This test assumes file_101 is a delta smaller than 16 bytes.
      # It should be against file_100 but we substitute its base for file_099
+     offset=$((rawsz + 4)) &&
      sha1_101=$(git hash-object file_101) &&
      sha1_099=$(git hash-object file_099) &&
      offs_101=$(index_obj_offset 1.idx $sha1_101) &&
@@ -159,8 +163,8 @@ test_expect_success \
      chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
      dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
         if=".git/objects/pack/pack-${pack1}.idx" \
-        skip=$((4 + 256 * 4 + $nr_099 * 24)) \
-        bs=1 count=20 conv=notrunc &&
+        skip=$((4 + 256 * 4 + $nr_099 * offset)) \
+        bs=1 count=$rawsz conv=notrunc &&
      git cat-file blob $sha1_101 > file_101_foo1'
 
 test_expect_success \
@@ -200,8 +204,8 @@ test_expect_success \
      chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
      dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
         if=".git/objects/pack/pack-${pack1}.idx" \
-        skip=$((8 + 256 * 4 + $nr_099 * 20)) \
-        bs=1 count=20 conv=notrunc &&
+        skip=$((8 + 256 * 4 + $nr_099 * rawsz)) \
+        bs=1 count=$rawsz conv=notrunc &&
      git cat-file blob $sha1_101 > file_101_foo2'
 
 test_expect_success \
@@ -226,7 +230,7 @@ test_expect_success \
      nr=$(index_obj_nr ".git/objects/pack/pack-${pack1}.idx" $obj) &&
      chmod +w ".git/objects/pack/pack-${pack1}.idx" &&
      printf xxxx | dd of=".git/objects/pack/pack-${pack1}.idx" conv=notrunc \
-        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * 20 + $nr * 4)) &&
+        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * rawsz + $nr * 4)) &&
      ( while read obj
        do git cat-file -p $obj >/dev/null || exit 1
        done <obj-list ) &&

[PATCH v2 12/22] t5309: make test hash independent

[brian m. carlson]

Use the proper pack constants defined in lib-pack.sh to make this test
work with SHA-256.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5309-pack-delta-cycles.sh | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/t/t5309-pack-delta-cycles.sh b/t/t5309-pack-delta-cycles.sh
index 491556dad9..b6b9792913 100755
--- a/t/t5309-pack-delta-cycles.sh
+++ b/t/t5309-pack-delta-cycles.sh
@@ -4,15 +4,9 @@ test_description='test index-pack handling of delta cycles in packfiles'
 . ./test-lib.sh
 . "$TEST_DIRECTORY"/lib-pack.sh
 
-if ! test_have_prereq SHA1
-then
-       skip_all='not using SHA-1 for objects'
-       test_done
-fi
-
 # Two similar-ish objects that we have computed deltas between.
-A=01d7713666f4de822776c7622c10f1b07de280dc
-B=e68fe8129b546b101aee9510c5328e7f21ca1d18
+A=$(test_oid packlib_7_0)
+B=$(test_oid packlib_7_76)
 
 # double-check our hand-constucted packs
 test_expect_success 'index-pack works with a single delta (A->B)' '

[PATCH v2 13/22] t5313: make test hash independent

[brian m. carlson]

Make this test hash independent by computing the length of the object
offsets and looking up values which will hash to object IDs with the
right properties.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5313-pack-bounds-checks.sh | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh
index f1708d415e..8d805f845a 100755
--- a/t/t5313-pack-bounds-checks.sh
+++ b/t/t5313-pack-bounds-checks.sh
@@ -38,16 +38,27 @@ munge () {
 # for the initial, and another ofs(4*nr) past that for the extended.
 #
 ofs_table () {
-	echo $((4 + 4 + 4*256 + 20*$1 + 4*$1))
+	echo $((4 + 4 + 4*256 + $(test_oid rawsz)*$1 + 4*$1))
 }
 extended_table () {
 	echo $(($(ofs_table "$1") + 4*$1))
 }
 
+test_expect_success 'setup' '
+	test_oid_init &&
+	test_oid_cache <<-EOF
+	oid000 sha1:1485
+	oid000 sha256:4222
+
+	oidfff sha1:74
+	oidfff sha256:1350
+	EOF
+'
+
 test_expect_success 'set up base packfile and variables' '
 	# the hash of this content starts with ff, which
 	# makes some later computations much simpler
-	echo 74 >file &&
+	echo $(test_oid oidfff) >file &&
 	git add file &&
 	git commit -m base &&
 	git repack -ad &&
@@ -140,10 +151,10 @@ test_expect_success 'bogus offset inside v2 extended table' '
 	# an extended table (if the first object were larger than 2^31).
 	#
 	# Note that the value is important here. We want $object as
-	# the second entry in sorted-sha1 order. The sha1 of 1485 starts
+	# the second entry in sorted-sha1 order. The hash of this object starts
 	# with "000", which sorts before that of $object (which starts
 	# with "fff").
-	second=$(echo 1485 | git hash-object -w --stdin) &&
+	second=$(test_oid oid000 | git hash-object -w --stdin) &&
 	do_pack "$object $second" --index-version=2 &&
 
 	# We have to make extra room for the table, so we cannot

[PATCH v2 14/22] t5321: make test hash independent

[brian m. carlson]

Use the proper pack constants defined in lib-pack.sh to make this test
work with SHA-256.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5321-pack-large-objects.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/t5321-pack-large-objects.sh b/t/t5321-pack-large-objects.sh
index a75eab87d3..8a56d98a0e 100755
--- a/t/t5321-pack-large-objects.sh
+++ b/t/t5321-pack-large-objects.sh
@@ -10,8 +10,8 @@ test_description='git pack-object with "large" deltas
 . "$TEST_DIRECTORY"/lib-pack.sh
 
 # Two similar-ish objects that we have computed deltas between.
-A=01d7713666f4de822776c7622c10f1b07de280dc
-B=e68fe8129b546b101aee9510c5328e7f21ca1d18
+A=$(test_oid packlib_7_0)
+B=$(test_oid packlib_7_76)
 
 test_expect_success 'setup' '
 	clear_packs &&

[PATCH v2 15/22] t5515: make test hash independent

[brian m. carlson]

This test contains a large number of data files, mostly using the same
object ID values for refs. Instead of producing two separate sets of
test files, keep the test files using SHA-1 and translate them on the
fly by replacing the SHA-1 values with the values for the current hash
algorithm in use.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5515-fetch-merge-logic.sh | 51 +++++++++++++++++++++++++++++++++---
 1 file changed, 47 insertions(+), 4 deletions(-)

diff --git a/t/t5515-fetch-merge-logic.sh b/t/t5515-fetch-merge-logic.sh
index 961eb35c99..ed49df582d 100755
--- a/t/t5515-fetch-merge-logic.sh
+++ b/t/t5515-fetch-merge-logic.sh
@@ -12,11 +12,50 @@ GIT_TEST_PROTOCOL_VERSION=
 
 . ./test-lib.sh
 
+convert_expected () {
+	file="$1" &&
+	for i in one three_file master master2 one_tree three two two2 three2
+	do
+		sed -e "s/$(test_oid --hash=sha1 "$i")/$(test_oid "$i")/g" \
+			"$file" >"$file.tmp" &&
+		mv "$file.tmp" "$file"
+	done
+}
+
 test_expect_success setup '
 	GIT_AUTHOR_DATE="2006-06-26 00:00:00 +0000" &&
 	GIT_COMMITTER_DATE="2006-06-26 00:00:00 +0000" &&
 	export GIT_AUTHOR_DATE GIT_COMMITTER_DATE &&
 
+	test_oid_cache <<-EOF &&
+	one sha1:8e32a6d901327a23ef831511badce7bf3bf46689
+	one sha256:8739546433ab1ac72ee93088dce611210effee072b2b586ceac6dde43ebec9ce
+
+	three_file sha1:0e3b14047d3ee365f4f2a1b673db059c3972589c
+	three_file sha256:bc4447d50c07497a8bfe6eef817f2364ecca9d471452e43b52756cc1a908bd32
+
+	master sha1:6c9dec2b923228c9ff994c6cfe4ae16c12408dc5
+	master sha256:8521c3072461fcfe8f32d67f95cc6e6b832a2db2fa29769ffc788bce85ebcd75
+
+	one_tree sha1:22feea448b023a2d864ef94b013735af34d238ba
+	one_tree sha256:6e4743f4ef2356b881dda5e91f5c7cdffe870faf350bf7b312f80a20935f5d83
+
+	three sha1:c61a82b60967180544e3c19f819ddbd0c9f89899
+	three sha256:0cc6d1eda617ded715170786e31ba4e2d0185404ec5a3508dd0d73b324860c6a
+
+	two sha1:525b7fb068d59950d185a8779dc957c77eed73ba
+	two sha256:3b21de3440cd38c2a9e9b464adb923f7054949ed4c918e1a0ac4c95cd52774db
+
+	master2 sha1:754b754407bf032e9a2f9d5a9ad05ca79a6b228f
+	master2 sha256:6c7abaea8a6d8ef4d89877e68462758dc6774690fbbbb0e6d7dd57415c9abde0
+
+	two2 sha1:6134ee8f857693b96ff1cc98d3e2fd62b199e5a8
+	two2 sha256:87a2d3ee29c83a3dc7afd41c0606b11f67603120b910a7be7840accdc18344d4
+
+	three2 sha1:0567da4d5edd2ff4bb292a465ba9e64dcad9536b
+	three2 sha256:cceb3e8eca364fa9a0a39a1efbebecacc664af86cbbd8070571f5faeb5f0e8c3
+	EOF
+
 	echo >file original &&
 	git add file &&
 	git commit -a -m One &&
@@ -137,6 +176,10 @@ do
 	actual_r="$pfx-refs.$test"
 
 	test_expect_success "$cmd" '
+		cp "$expect_f" expect_f &&
+		convert_expected expect_f &&
+		cp "$expect_r" expect_r &&
+		convert_expected expect_r &&
 		{
 			echo "# $cmd"
 			set x $cmd; shift
@@ -152,18 +195,18 @@ do
 			cat .git/FETCH_HEAD
 		} >"$actual_f" &&
 		git show-ref >"$actual_r" &&
-		if test -f "$expect_f"
+		if test -f "expect_f"
 		then
-			test_cmp "$expect_f" "$actual_f" &&
+			test_cmp "expect_f" "$actual_f" &&
 			rm -f "$actual_f"
 		else
 			# this is to help developing new tests.
 			cp "$actual_f" "$expect_f"
 			false
 		fi &&
-		if test -f "$expect_r"
+		if test -f "expect_r"
 		then
-			test_cmp "$expect_r" "$actual_r" &&
+			test_cmp "expect_r" "$actual_r" &&
 			rm -f "$actual_r"
 		else
 			# this is to help developing new tests.

[PATCH v2 16/22] t5318: update for SHA-256

[brian m. carlson]

Switch two tests to use $ZERO_OID to represent the all-zeros object ID.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5318-commit-graph.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh
index 3f03de6018..55a94072b1 100755
--- a/t/t5318-commit-graph.sh
+++ b/t/t5318-commit-graph.sh
@@ -629,7 +629,7 @@ test_expect_success 'corrupt commit-graph write (broken parent)' '
 		empty="$(git mktree </dev/null)" &&
 		cat >broken <<-EOF &&
 		tree $empty
-		parent 0000000000000000000000000000000000000000
+		parent $ZERO_OID
 		author whatever <whatever@example.com> 1234 -0000
 		committer whatever <whatever@example.com> 1234 -0000
 
@@ -650,7 +650,7 @@ test_expect_success 'corrupt commit-graph write (missing tree)' '
 		cd repo &&
 		tree="$(git mktree </dev/null)" &&
 		cat >broken <<-EOF &&
-		parent 0000000000000000000000000000000000000000
+		parent $ZERO_OID
 		author whatever <whatever@example.com> 1234 -0000
 		committer whatever <whatever@example.com> 1234 -0000
 

[PATCH v2 17/22] t5607: make hash size independent

[brian m. carlson]

Use $OID_REGEX instead of a hard-coded regular expression.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5607-clone-bundle.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh
index b7a3fdf02d..9108ff6fbd 100755
--- a/t/t5607-clone-bundle.sh
+++ b/t/t5607-clone-bundle.sh
@@ -64,7 +64,7 @@ test_expect_success 'ridiculously long subject in boundary' '
 	test -s heads &&
 	git fetch long-subject-bundle.bdl &&
 	sed -n "/^-/{p;q;}" long-subject-bundle.bdl >boundary &&
-	grep "^-[0-9a-f]\\{40\\} " boundary
+	grep "^-$OID_REGEX " boundary
 '
 
 test_expect_success 'prerequisites with an empty commit message' '

[PATCH v2 17/23] t5616: use correct filter syntax

[brian m. carlson]

In the setup steps for the promisor remote tests, we clone a repository
and filter out all trees with depth greater than or equal to zero, which
also filters out all blobs.

With SHA-1, this test passes because the object we happen to request
from the server is the blob that the promisor remote has.  However, due
to a different ordering with SHA-256, we request the tree containing
that blob, which the promisor remote does not have.  As a consequence,
we fail with a "not our ref" error.

Since what we want to test is that the blob is transferred, let's adjust
the filter to just filter out blobs, not trees.  That means that we'll
transfer the previously problematic tree as part of the normal clone,
and we can then test that the blob is fetched from the promisor remote
as expected.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5616-partial-clone.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh
index fea56cda6d..9fd6e780f9 100755
--- a/t/t5616-partial-clone.sh
+++ b/t/t5616-partial-clone.sh
@@ -317,7 +317,7 @@ setup_triangle () {
 	cp big-blob.txt server &&
 	git -C server add big-blob.txt &&
 	git -C server commit -m "initial" &&
-	git clone --bare --filter=tree:0 "file://$(pwd)/server" client &&
+	git clone --bare --filter=blob:none "file://$(pwd)/server" client &&
 	echo another line >>server/big-blob.txt &&
 	git -C server commit -am "append line to big blob" &&
 

[PATCH v2 18/23] t5607: make hash size independent

[brian m. carlson]

Use $OID_REGEX instead of a hard-coded regular expression.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5607-clone-bundle.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh
index b7a3fdf02d..9108ff6fbd 100755
--- a/t/t5607-clone-bundle.sh
+++ b/t/t5607-clone-bundle.sh
@@ -64,7 +64,7 @@ test_expect_success 'ridiculously long subject in boundary' '
 	test -s heads &&
 	git fetch long-subject-bundle.bdl &&
 	sed -n "/^-/{p;q;}" long-subject-bundle.bdl >boundary &&
-	grep "^-[0-9a-f]\\{40\\} " boundary
+	grep "^-$OID_REGEX " boundary
 '
 
 test_expect_success 'prerequisites with an empty commit message' '

[PATCH v2 18/22] t5703: make test work with SHA-256

[brian m. carlson]

This test used an object ID which was 40 hex characters in length,
causing the test not only not to pass, but to hang, when run with
SHA-256 as the hash.  Change this value to a fixed dummy object ID using
test_oid_init and test_oid.

Furthermore, ensure we extract an object ID of the appropriate length
using cut with fields instead of a fixed length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 1424fabd4a..5511cdcec2 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -19,7 +19,7 @@ get_actual_commits () {
 		}' <out | test-tool pkt-line unpack-sideband >o.pack &&
 	git index-pack o.pack &&
 	git verify-pack -v o.idx >objs &&
-	grep commit objs | cut -c-40 | sort >actual_commits
+	grep commit objs | cut -d" " -f1 | sort >actual_commits
 }
 
 check_output () {

[PATCH v2 19/23] t5703: make test work with SHA-256

[brian m. carlson]

This test used an object ID which was 40 hex characters in length,
causing the test not only not to pass, but to hang, when run with
SHA-256 as the hash.  Change this value to a fixed dummy object ID using
test_oid_init and test_oid.

Furthermore, ensure we extract an object ID of the appropriate length
using cut with fields instead of a fixed length.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 1424fabd4a..5511cdcec2 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -19,7 +19,7 @@ get_actual_commits () {
 		}' <out | test-tool pkt-line unpack-sideband >o.pack &&
 	git index-pack o.pack &&
 	git verify-pack -v o.idx >objs &&
-	grep commit objs | cut -c-40 | sort >actual_commits
+	grep commit objs | cut -d" " -f1 | sort >actual_commits
 }
 
 check_output () {

[PATCH v2 19/22] t5703: switch tests to use test_oid

[brian m. carlson]

To make this test work correctly with SHA-256, switch the hard-coded
invalid object IDs to use test_oid.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 5511cdcec2..8aeeaac509 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -37,6 +37,7 @@ check_output () {
 #             \ | /
 #               a
 test_expect_success 'setup repository' '
+	test_oid_init &&
 	test_commit a &&
 	git checkout -b o/foo &&
 	test_commit b &&
@@ -333,7 +334,7 @@ test_expect_success 'server is initially ahead - no ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant false &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	test_must_fail git -C local fetch 2>err &&
 	test_i18ngrep "fatal: remote error: upload-pack: not our ref" err
 '
@@ -342,7 +343,7 @@ test_expect_success 'server is initially ahead - ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant true &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	git -C local fetch &&
 
 	git -C "$REPO" rev-parse --verify master >expected &&

[PATCH v2 20/23] t5703: switch tests to use test_oid

[brian m. carlson]

To make this test work correctly with SHA-256, switch the hard-coded
invalid object IDs to use test_oid.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 5511cdcec2..8aeeaac509 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -37,6 +37,7 @@ check_output () {
 #             \ | /
 #               a
 test_expect_success 'setup repository' '
+	test_oid_init &&
 	test_commit a &&
 	git checkout -b o/foo &&
 	test_commit b &&
@@ -333,7 +334,7 @@ test_expect_success 'server is initially ahead - no ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant false &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	test_must_fail git -C local fetch 2>err &&
 	test_i18ngrep "fatal: remote error: upload-pack: not our ref" err
 '
@@ -342,7 +343,7 @@ test_expect_success 'server is initially ahead - ref in want' '
 	git -C "$REPO" config uploadpack.allowRefInWant true &&
 	rm -rf local &&
 	cp -r "$LOCAL_PRISTINE" local &&
-	inconsistency master 1234567890123456789012345678901234567890 &&
+	inconsistency master $(test_oid numeric) &&
 	git -C local fetch &&
 
 	git -C "$REPO" rev-parse --verify master >expected &&

[PATCH v2 20/22] t6000: abstract away SHA-1-specific constants

[brian m. carlson]

Adjust the test so that it computes variables for object IDs instead of
using hard-coded hashes.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6000-rev-list-misc.sh | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh
index b8cf82349b..a0baf9ee43 100755
--- a/t/t6000-rev-list-misc.sh
+++ b/t/t6000-rev-list-misc.sh
@@ -104,13 +104,16 @@ test_expect_success 'rev-list can show index objects' '
 	#   - we do not show the root tree; since we updated the index, it
 	#     does not have a valid cache tree
 	#
-	cat >expect <<-\EOF &&
-	8e4020bb5a8d8c873b25de15933e75cc0fc275df one
-	d9d3a7417b9605cfd88ee6306b28dadc29e6ab08 only-in-index
-	9200b628cf9dc883a85a7abc8d6e6730baee589c two
-	EOF
 	echo only-in-index >only-in-index &&
 	test_when_finished "git reset --hard" &&
+	rev1=$(git rev-parse HEAD:one) &&
+	rev2=$(git rev-parse HEAD:two) &&
+	revi=$(git hash-object only-in-index) &&
+	cat >expect <<-EOF &&
+	$rev1 one
+	$revi only-in-index
+	$rev2 two
+	EOF
 	git add only-in-index &&
 	git rev-list --objects --indexed-objects >actual &&
 	test_cmp expect actual

[PATCH v2 21/23] t6000: abstract away SHA-1-specific constants

[brian m. carlson]

Adjust the test so that it computes variables for object IDs instead of
using hard-coded hashes.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6000-rev-list-misc.sh | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh
index b8cf82349b..a0baf9ee43 100755
--- a/t/t6000-rev-list-misc.sh
+++ b/t/t6000-rev-list-misc.sh
@@ -104,13 +104,16 @@ test_expect_success 'rev-list can show index objects' '
 	#   - we do not show the root tree; since we updated the index, it
 	#     does not have a valid cache tree
 	#
-	cat >expect <<-\EOF &&
-	8e4020bb5a8d8c873b25de15933e75cc0fc275df one
-	d9d3a7417b9605cfd88ee6306b28dadc29e6ab08 only-in-index
-	9200b628cf9dc883a85a7abc8d6e6730baee589c two
-	EOF
 	echo only-in-index >only-in-index &&
 	test_when_finished "git reset --hard" &&
+	rev1=$(git rev-parse HEAD:one) &&
+	rev2=$(git rev-parse HEAD:two) &&
+	revi=$(git hash-object only-in-index) &&
+	cat >expect <<-EOF &&
+	$rev1 one
+	$revi only-in-index
+	$rev2 two
+	EOF
 	git add only-in-index &&
 	git rev-list --objects --indexed-objects >actual &&
 	test_cmp expect actual

[PATCH v2 21/22] t6006: make hash size independent

[brian m. carlson]

Instead of hard-coding the length of an object ID when creating a tree,
compute it for the hash in use using the translation tables.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6006-rev-list-format.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh
index ebdc49c496..7e82e43a63 100755
--- a/t/t6006-rev-list-format.sh
+++ b/t/t6006-rev-list-format.sh
@@ -32,6 +32,7 @@ changed_iso88591=$(echo "$changed" | iconv -f utf-8 -t $test_encoding)
 truncate_count=20
 
 test_expect_success 'setup' '
+	test_oid_init &&
 	: >foo &&
 	git add foo &&
 	git config i18n.commitEncoding $test_encoding &&
@@ -463,9 +464,10 @@ test_expect_success '--abbrev' '
 '
 
 test_expect_success '%H is not affected by --abbrev-commit' '
+	expected=$(($(test_oid hexsz) + 1)) &&
 	git log -1 --format=%H --abbrev-commit --abbrev=20 HEAD >actual &&
 	len=$(wc -c <actual) &&
-	test $len = 41
+	test $len = $expected
 '
 
 test_expect_success '%h is not affected by --abbrev-commit' '

[PATCH v2 22/23] t6006: make hash size independent

[brian m. carlson]

Instead of hard-coding the length of an object ID when creating a tree,
compute it for the hash in use using the translation tables.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6006-rev-list-format.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/t/t6006-rev-list-format.sh b/t/t6006-rev-list-format.sh
index ebdc49c496..7e82e43a63 100755
--- a/t/t6006-rev-list-format.sh
+++ b/t/t6006-rev-list-format.sh
@@ -32,6 +32,7 @@ changed_iso88591=$(echo "$changed" | iconv -f utf-8 -t $test_encoding)
 truncate_count=20
 
 test_expect_success 'setup' '
+	test_oid_init &&
 	: >foo &&
 	git add foo &&
 	git config i18n.commitEncoding $test_encoding &&
@@ -463,9 +464,10 @@ test_expect_success '--abbrev' '
 '
 
 test_expect_success '%H is not affected by --abbrev-commit' '
+	expected=$(($(test_oid hexsz) + 1)) &&
 	git log -1 --format=%H --abbrev-commit --abbrev=20 HEAD >actual &&
 	len=$(wc -c <actual) &&
-	test $len = 41
+	test $len = $expected
 '
 
 test_expect_success '%h is not affected by --abbrev-commit' '

[PATCH v2 22/22] t6024: update for SHA-256

[brian m. carlson]

To make this test work with SHA-256, compute two of the items in the
conflicted index entry.  The other entry is a conflict within a conflict
and computing it is difficult, so use test_oid_cache to specify the
proper values for both hash algorithms.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6024-recursive-merge.sh | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/t/t6024-recursive-merge.sh b/t/t6024-recursive-merge.sh
index 0c9e3c20e8..332cfc53fd 100755
--- a/t/t6024-recursive-merge.sh
+++ b/t/t6024-recursive-merge.sh
@@ -57,7 +57,12 @@ test_expect_success 'setup tests' '
 	git rev-parse C >.git/MERGE_HEAD &&
 	echo F >a1 &&
 	git update-index a1 &&
-	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F
+	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F &&
+
+	test_oid_cache <<-EOF
+	idxstage1 sha1:ec3fe2a791706733f2d8fa7ad45d9a9672031f5e
+	idxstage1 sha256:b3c8488929903aaebdeb22270cb6d36e5b8724b01ae0d4da24632f158c99676f
+	EOF
 '
 
 test_expect_success 'combined merge conflicts' '
@@ -79,10 +84,10 @@ test_expect_success 'result contains a conflict' '
 test_expect_success 'virtual trees were processed' '
 	git ls-files --stage >out &&
 
-	cat >expect <<-\EOF &&
-	100644 ec3fe2a791706733f2d8fa7ad45d9a9672031f5e 1	a1
-	100644 cf84443e49e1b366fac938711ddf4be2d4d1d9e9 2	a1
-	100644 fd7923529855d0b274795ae3349c5e0438333979 3	a1
+	cat >expect <<-EOF &&
+	100644 $(test_oid idxstage1) 1	a1
+	100644 $(git rev-parse F:a1) 2	a1
+	100644 $(git rev-parse G:a1) 3	a1
 	EOF
 
 	test_cmp expect out

[PATCH v2 23/23] t6024: update for SHA-256

[brian m. carlson]

To make this test work with SHA-256, compute two of the items in the
conflicted index entry.  The other entry is a conflict within a conflict
and computing it is difficult, so use test_oid_cache to specify the
proper values for both hash algorithms.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 t/t6024-recursive-merge.sh | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/t/t6024-recursive-merge.sh b/t/t6024-recursive-merge.sh
index 0c9e3c20e8..332cfc53fd 100755
--- a/t/t6024-recursive-merge.sh
+++ b/t/t6024-recursive-merge.sh
@@ -57,7 +57,12 @@ test_expect_success 'setup tests' '
 	git rev-parse C >.git/MERGE_HEAD &&
 	echo F >a1 &&
 	git update-index a1 &&
-	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F
+	GIT_AUTHOR_DATE="2006-12-12 23:00:08" git commit -m F &&
+
+	test_oid_cache <<-EOF
+	idxstage1 sha1:ec3fe2a791706733f2d8fa7ad45d9a9672031f5e
+	idxstage1 sha256:b3c8488929903aaebdeb22270cb6d36e5b8724b01ae0d4da24632f158c99676f
+	EOF
 '
 
 test_expect_success 'combined merge conflicts' '
@@ -79,10 +84,10 @@ test_expect_success 'result contains a conflict' '
 test_expect_success 'virtual trees were processed' '
 	git ls-files --stage >out &&
 
-	cat >expect <<-\EOF &&
-	100644 ec3fe2a791706733f2d8fa7ad45d9a9672031f5e 1	a1
-	100644 cf84443e49e1b366fac938711ddf4be2d4d1d9e9 2	a1
-	100644 fd7923529855d0b274795ae3349c5e0438333979 3	a1
+	cat >expect <<-EOF &&
+	100644 $(test_oid idxstage1) 1	a1
+	100644 $(git rev-parse F:a1) 2	a1
+	100644 $(git rev-parse G:a1) 3	a1
 	EOF
 
 	test_cmp expect out

Re: [PATCH v2 00/23] SHA-256 test fixes, part 8

[Johannes Schindelin]

Hi brian,

On Sat, 25 Jan 2020, brian m. carlson wrote:

> This is the second-to-last series of test fixes for SHA-256.
>
> As mentioned previously, the patch for t3305 seems to indicate a bug in
> the notes code and I'm not familiar enough with that code to apply a
> fix.  This is a band-aid to get it working with SHA-256, but any
> comments on a more robust approach would of course be welcome.
>
> Changes from v1:
> * Drop patch for t3404 in favor of Dscho's fix.
> * Drop patch for t5616 in favor of Jonathan Tan's fix.
> * Add missing sign-off.
> * Move test_oid_init into the correct patch.

Would you terribly mind pushing up your local branch to a public place? I
used the apply-from-public-inbox.sh script I maintain at
https://github.com/git-for-windows/build-extra to apply your patch series
on top of v2.25.0, but I got this:

	Applying: t5607: make hash size independent
	Using index info to reconstruct a base tree...
	M       t/t5607-clone-bundle.sh
	Falling back to patching base and 3-way merge...
	No changes -- Patch already applied.

(I want to have a look at the notes fanout. Will keep you posted on my
investigation there.)

Thanks,
Dscho

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[Johannes Schindelin]

Hi brian,

On Sat, 25 Jan 2020, brian m. carlson wrote:

> This test relies on a roughly equal distribution of hashes for notes in
> order to ensure that fanouts are compressed.  If there are subtrees with
> only one item left after removing notes, they'll end up still with one
> level of fanout, causing the test to fail.

That is _almost_ correct: The heuristic wants to see one bucket that has
a note in it. Or something like that.

See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)

> The test happens to pass with SHA-1, but doesn't necessarily with other
> hash algorithms, so annotate it with the SHA1 prerequisite.

I would rather see this tested, still, and reducing the number of notes
that are retained from 50 to 20 before testing that the fanout has been
reduced to 0 seems to do the trick. Therefore, I would love to submit this
for squashing:

-- snip --
diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
index 3520402bb81..39b12c9902c 100755
--- a/t/t3305-notes-fanout.sh
+++ b/t/t3305-notes-fanout.sh
@@ -43,7 +43,7 @@ test_expect_success 'many notes created with git-notes triggers fanout' '
 '

 test_expect_success 'deleting most notes with git-notes' '
-	num_notes=250 &&
+	num_notes=280 &&
 	i=0 &&
 	git rev-list HEAD |
 	while test $i -lt $num_notes && read sha1
@@ -56,8 +56,8 @@ test_expect_success 'deleting most notes with git-notes' '
 '

 test_expect_success 'most notes deleted correctly with git-notes' '
-	git log HEAD~250 | grep "^    " > output &&
-	i=50 &&
+	git log HEAD~280 | grep "^    " > output &&
+	i=20 &&
 	while test $i -gt 0
 	do
 		echo "    commit #$i" &&
@@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
 	test_cmp expect output
 '

-test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
+test_expect_success 'deleting most notes triggers fanout consolidation' '
 	# Expect entire notes tree to have a fanout == 0
 	git ls-tree -r --name-only refs/notes/commits |
 	while read path
-- snap --

Thanks,
Dscho

>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t3305-notes-fanout.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
> index 831f83d211..3520402bb8 100755
> --- a/t/t3305-notes-fanout.sh
> +++ b/t/t3305-notes-fanout.sh
> @@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
>  	test_cmp expect output
>  '
>
> -test_expect_success 'deleting most notes triggers fanout consolidation' '
> +test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
>  	# Expect entire notes tree to have a fanout == 0
>  	git ls-tree -r --name-only refs/notes/commits |
>  	while read path
>

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[Johan Herland]

On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> On Sat, 25 Jan 2020, brian m. carlson wrote:
> > This test relies on a roughly equal distribution of hashes for notes in
> > order to ensure that fanouts are compressed.  If there are subtrees with
> > only one item left after removing notes, they'll end up still with one
> > level of fanout, causing the test to fail.
>
> That is _almost_ correct: The heuristic wants to see one bucket that has
> a note in it. Or something like that.
>
> See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)

Something like that, yeah... Re-reading this code, I believe we stop
the fanout at the current level when we can find one or more notes
that do not share the high-nibble of their path with another note.

Here we're at the top level, so this corresponds to looking at the
very first hex character (0-9a-f) of the path (oid of annotated
object), and if there are at least two such objects for each hex
character, we will use a fanout of 1, otherwise, we collapse the
fanout to 0.

Hence we need an absolute minimum of 32 notes (and some rotten luck)
to get a fanout of 1. As the number of notes increase, the probably of
fanning out increases, passing 50% at ~79 notes, and reaching ~100%
somewhere north of 150 notes.

> > The test happens to pass with SHA-1, but doesn't necessarily with other
> > hash algorithms, so annotate it with the SHA1 prerequisite.
>
> I would rather see this tested, still, and reducing the number of notes
> that are retained from 50 to 20 before testing that the fanout has been
> reduced to 0 seems to do the trick. Therefore, I would love to submit this
> for squashing:

Yes, it seems that for SHA1 and the (deterministic) objects used in
the test, we got away with 50 notes, but that is not the case for
other hash algorithms. Lowering the number to 20 definitely results a
fanout of 0, as should any other number below 32.

+1 to Dscho's squash.

...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: [PATCH v2 00/23] SHA-256 test fixes, part 8

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]

On 2020-01-26 at 10:25:31, Johannes Schindelin wrote:
> Hi brian,
> 
> On Sat, 25 Jan 2020, brian m. carlson wrote:
> 
> > This is the second-to-last series of test fixes for SHA-256.
> >
> > As mentioned previously, the patch for t3305 seems to indicate a bug in
> > the notes code and I'm not familiar enough with that code to apply a
> > fix.  This is a band-aid to get it working with SHA-256, but any
> > comments on a more robust approach would of course be welcome.
> >
> > Changes from v1:
> > * Drop patch for t3404 in favor of Dscho's fix.
> > * Drop patch for t5616 in favor of Jonathan Tan's fix.
> > * Add missing sign-off.
> > * Move test_oid_init into the correct patch.
> 
> Would you terribly mind pushing up your local branch to a public place? I
> used the apply-from-public-inbox.sh script I maintain at
> https://github.com/git-for-windows/build-extra to apply your patch series
> on top of v2.25.0, but I got this:
> 
> 	Applying: t5607: make hash size independent
> 	Using index info to reconstruct a base tree...
> 	M       t/t5607-clone-bundle.sh
> 	Falling back to patching base and 3-way merge...
> 	No changes -- Patch already applied.
> 
> (I want to have a look at the notes fanout. Will keep you posted on my
> investigation there.)

The branch is test-fixes-part8 on https://github.com/bk2204/git.git.
It's currently based off master.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1793 bytes --]

On 2020-01-26 at 11:15:52, Johannes Schindelin wrote:
> I would rather see this tested, still, and reducing the number of notes
> that are retained from 50 to 20 before testing that the fanout has been
> reduced to 0 seems to do the trick. Therefore, I would love to submit this
> for squashing:
> 
> -- snip --
> diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
> index 3520402bb81..39b12c9902c 100755
> --- a/t/t3305-notes-fanout.sh
> +++ b/t/t3305-notes-fanout.sh
> @@ -43,7 +43,7 @@ test_expect_success 'many notes created with git-notes triggers fanout' '
>  '
> 
>  test_expect_success 'deleting most notes with git-notes' '
> -	num_notes=250 &&
> +	num_notes=280 &&
>  	i=0 &&
>  	git rev-list HEAD |
>  	while test $i -lt $num_notes && read sha1
> @@ -56,8 +56,8 @@ test_expect_success 'deleting most notes with git-notes' '
>  '
> 
>  test_expect_success 'most notes deleted correctly with git-notes' '
> -	git log HEAD~250 | grep "^    " > output &&
> -	i=50 &&
> +	git log HEAD~280 | grep "^    " > output &&
> +	i=20 &&
>  	while test $i -gt 0
>  	do
>  		echo "    commit #$i" &&
> @@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
>  	test_cmp expect output
>  '
> 
> -test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
> +test_expect_success 'deleting most notes triggers fanout consolidation' '
>  	# Expect entire notes tree to have a fanout == 0
>  	git ls-tree -r --name-only refs/notes/commits |
>  	while read path
> -- snap --

Sure, that's a fine solution instead.  I'll squash that in and update
the commit message.  I'll CC you for your sign-off once I've done that.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[Johannes Schindelin]

Hi Johan,

On Sun, 26 Jan 2020, Johan Herland wrote:

> On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > On Sat, 25 Jan 2020, brian m. carlson wrote:
> > > This test relies on a roughly equal distribution of hashes for notes in
> > > order to ensure that fanouts are compressed.  If there are subtrees with
> > > only one item left after removing notes, they'll end up still with one
> > > level of fanout, causing the test to fail.
> >
> > That is _almost_ correct: The heuristic wants to see one bucket that has
> > a note in it. Or something like that.
> >
> > See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> > tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)
>
> Something like that, yeah... Re-reading this code, I believe we stop
> the fanout at the current level when we can find one or more notes
> that do not share the high-nibble of their path with another note.
>
> Here we're at the top level, so this corresponds to looking at the
> very first hex character (0-9a-f) of the path (oid of annotated
> object), and if there are at least two such objects for each hex
> character, we will use a fanout of 1, otherwise, we collapse the
> fanout to 0.

That makes sense, but when I looked at the failed test, there seems
something else at play, at least in addition to what you said: for
_adding_ notes, your description is 100% accurate, but when deleting
notes, we are apparently not collapsing `PTR_TYPE_SUBTREE` nodes "quickly"
enough. Let me show you the part of `git ls-tree -r refs/notes/commits`
that starts with the hex digits 7, 8 and 9:

-- snip --
100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b 72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
100644 blob 2e3f3dad7043b2d02d09ca12b299acbe05d781357d3a56a2d012fdf787409459 76/62d4b264094ca479be86ef7aa66daae63e45afa633a3892dc787b13ecff495
100644 blob 22e7f0c5af7315d692f8a107a43ba0784e1ab00a20ea803fab1acad1319e5f79 7b/8c0a9c86815a94da7ef90b356b1f98d6a4099af3fdc3d8625a8fa793b63821
100644 blob b3ce67c9d5507dc00d95d8fb2000c1d5b70908ad1d2c034e5833f57b7bb85511 7f/0af9cf9259cd6e67c0af3324ca443dd3d56694fbdc94d28e300a768d3d0e6e
100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a 8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53 93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
100644 blob 0b1915ccb6f64cac64f2297893bce1ba7408660f79182302faaada71ee8c3c1c 96/b562b168f94e5c6e6a1e40ec4b817aa168c99769b1d9a46f7e048e93897fb4
-- snap --

This command was run in the worktree after running an unmodified t3305
with `GIT_TEST_DEFAULT_HASH=sha256` in brian's `transition-stage-4` branch
(well, I removed the `SHA1` prereq again, that's the modification I made).

As you can see, there _is_ a fanout of 1, but there is only a single note
for a commit whose ID starts with `8`.

Debugging a little, it seems that the `PTR_TYPE_SUBTREE` node for `8` was
not collapsed, even if there was only one item left.

So I formed a hypothesis that the subtrees are only removed when the
_last_ item is removed for any given leading nybble, but that turned out
to also be incorrect. It is a bit more tricky than that. This is the
smallest set to which I was able to reduce the notes without reducing the
fanout to 0:

-- snip --
100644 blob cc2ee5e00d8bc3805d704b67439dc8175bcf9497603288e6de2d4d8b3fc7be9f	05/b6e3d1394d020129f71fd8e41cf7ea8cbc58ae0f1332abd5da0c74ea194b71
100644 blob 1c6afe76a1bcd0103c36ab9707a2ca9e68974b6a6bbaae564c0509c43b4392bd	1e/311d64dc3ad5491964bcded60fee15b19d5b9c916a7e62a4f0746fa4e81fa6
100644 blob cf97d105e3970ef1cf9b12ac092be80abcd496c593bc8ca5550d059c3967630a	28/79e092d524b7ae9a42026ab2886094cce8ffc63175f8b3fd5de84faef10df3
100644 blob 78ef788b804dd0e5415c386b4a29668f61033483c35438f0471dfd7c4bfa093b	36/fe8fe67a2e9d0203c665d6e08ca454833ec32a97369769a7138d3938c0000b
100644 blob f46845c2d7e3272319aa5c18e359fdbc37731c88a945bb9632b8c4321983c75a	4a/a271a09d848f99d3fb978e5c156baa812a0fa1c30a88c885831641630be01a
100644 blob ecf5a4a178ca4b51cea457abe7c935761ca15a1d817f83c2da6816ede84db779	51/eaee3ca1a8698cb0aaa4de2d3f339985570da68b28e84af752e1cfd25f5197
100644 blob 2f4e9d6a4a1d050f8cb932ba545e53b48d9b669f6e00dc4a962d88ee9d92482b	62/dd63d43070c3ca7e3a6cdfa4ed970256a00a06e88d10fcb0532acd51419e0f
100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b	72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a	8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53	93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
100644 blob 589656c26944c471b5ac65739f8c7b96663a9f827a3d27beb49e39e3707b7294	a2/0e8f30856061125d479779755ef3238a7b561f9336e0143c437daac7d93f4c
100644 blob 7eaa9350d5c6bd6fd0fc4071c5d6a266949046c67987a9e1f665ba34d95f419d	a2/80d13a05aa68cc5ef948a8b69067807457fd37c00ce4a234fa4a0c0753ef4e
100644 blob 7a8378dd60d6024db645757eac7271a80b101a2df230ebd1a57ff52ff2d32e36	b9/2a3a7dc6db290d93f79150bfb31447f3e550cfe4a63c5ddbeac18fec755e86
100644 blob 86fff6007d9911249bee803a6630e182677355a5574e637d1a8301e219c5da86	c1/52ffd73d1c5e7c121c7c247682f1ee971f6f09101c96a84486d18be41d0dd0
100644 blob cd887698e1da81a76ae1caf0eaec19d60830a13ead152ec4700be511ceb8ee33	d0/3f742b8b95f68478946d7fe7495da9462801fadeaaa06c11bf54dbc46610f5
100644 blob 19f7bdfee9687311dbe1195e0a64954b677ae68e6d734fd5fb76ea4ad4f93782	ea/356ae2d38123b46639db98df14953f4c7cdd91738779174ec67876ce9487e3
100644 blob 948c0cba23ec0405c622c9dea8ed8dd7b3fa043c86b5e5a8b4de0d1c6a0e67b9	f7/802d4c716fed3c76fe58c86ac7c3ae3e19b8c0d3ea97c9f90f5939fe5a78d8
100644 blob 28690ad489e29e3607c82e1f626ec24d7f831555c802108bfe9b993fbd794a7e	f7/da03e811b7d9071ee19dabdc721e0f863e28c92dfa3257474282396a73bb44
-- snap --

You will note that there are two entries that start with `a2`, and two
entries that start with `f7`. If I remove any of those, the corresponding
subtree will be collapsed, and the fanout will be reduced to 0.

But it is only happenstance with SHA-256 that there are these entries that
agree not only in the first, but also in the second leading nybble.

Therefore...

> Hence we need an absolute minimum of 32 notes (and some rotten luck)
> to get a fanout of 1. As the number of notes increase, the probably of
> fanning out increases, passing 50% at ~79 notes, and reaching ~100%
> somewhere north of 150 notes.

... I would register that we need an absolute minimum of 16 notes (and
some rather crafty craft) to get a fanout of 1.

In that light, I think that I would prefer to retract my patch that
"only" reduces the remaining number of notes to 20: it should reduce them
to 15 or less. So why not reduce it to 10 (because it is only one changed
digit).

> > > The test happens to pass with SHA-1, but doesn't necessarily with other
> > > hash algorithms, so annotate it with the SHA1 prerequisite.
> >
> > I would rather see this tested, still, and reducing the number of notes
> > that are retained from 50 to 20 before testing that the fanout has been
> > reduced to 0 seems to do the trick. Therefore, I would love to submit this
> > for squashing:
>
> Yes, it seems that for SHA1 and the (deterministic) objects used in
> the test, we got away with 50 notes, but that is not the case for
> other hash algorithms. Lowering the number to 20 definitely results a
> fanout of 0, as should any other number below 32.
>
> +1 to Dscho's squash.
>
> ...Johan

Thank you so much for the analysis. To be honest, I did not quite
understand all the details of the comment added in 73f77b989f8 when I
wrote the patch I suggested, so I basically just picked that number "20"
out of thin air.

Together with your insights, I would like to propose this commit message
for the squashed commits (I left in the hunk that removes the `SHA1`
prerequisite, but of course that won't be part of the final commit):

-- snip --
t3305: make fanout test more robust (needed for SHA-256)

To make things more performant, notes are stored in a "fanout": when
there are enough commit notes, they are no longer stored as verbatim
commit IDs at the top-level tree of the notes ref, but instead the tree
is deepened much like the loose object cache: subtrees are introduced
whose names are the two hex digits they "chomp off" the commit IDs.

The test case 'deleting most notes triggers fanout consolidation' wants
to verify that the fanout level is reduced automatically when enough
notes have been deleted.

However, that test case expected that reduction to level 0 (i.e. _no_
fanout subtrees) to happen after reducing the originally-added 300 notes
to 50, which _happened_ to work with SHA-1-based commit IDs, but it is
no longer works with SHA-256-based ones.

The reason: The heuristic for the fanout looks at the number of entries
for leading nybbles (read: hex digits) of the commit IDs. If there are
more than a single annotated commit for all of the 16 hex digits, the
fanout is incremented. It is a bit more tricky when reducing the number
of notes: the fanout is reduced reliably only if there are less notes
than hex digits (i.e. less than 15 notes) for a given prefix.

For good measure, let's reduce the number of notes to 10 in the test
case 'deleting most notes with git-notes' so that the test case
'deleting most notes triggers fanout consolidation' is guaranteed to
succeed with _any_ hash algorithm.

Original-patch-by: brian m. carlson <sandals@crustytoothpaste.net>
Helped-by: Johan Herland <johan@herland.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
diff --git a/t/t3305-notes-fanout.sh b/t/t3305-notes-fanout.sh
index 3520402bb81..b83c2670ebe 100755
--- a/t/t3305-notes-fanout.sh
+++ b/t/t3305-notes-fanout.sh
@@ -43,7 +43,7 @@ test_expect_success 'many notes created with git-notes triggers fanout' '
 '

 test_expect_success 'deleting most notes with git-notes' '
-	num_notes=250 &&
+	num_notes=290 &&
 	i=0 &&
 	git rev-list HEAD |
 	while test $i -lt $num_notes && read sha1
@@ -56,8 +56,8 @@ test_expect_success 'deleting most notes with git-notes' '
 '

 test_expect_success 'most notes deleted correctly with git-notes' '
-	git log HEAD~250 | grep "^    " > output &&
-	i=50 &&
+	git log HEAD~290 | grep "^    " > output &&
+	i=10 &&
 	while test $i -gt 0
 	do
 		echo "    commit #$i" &&
@@ -67,7 +67,7 @@ test_expect_success 'most notes deleted correctly with git-notes' '
 	test_cmp expect output
 '

-test_expect_success SHA1 'deleting most notes triggers fanout consolidation' '
+test_expect_success 'deleting most notes triggers fanout consolidation' '
 	# Expect entire notes tree to have a fanout == 0
 	git ls-tree -r --name-only refs/notes/commits |
 	while read path
-- snap --

brian, would you mind adopting this patch into your patch series? For your
convenience, I pushed it up as `t3305-sha256-fanout` (based on your
`transition-stage-4`) to https://github.com/dscho/git.

Thanks, Dscho

Re: [PATCH v2 00/23] SHA-256 test fixes, part 8

[Johannes Schindelin]

Hi brian,

On Sun, 26 Jan 2020, brian m. carlson wrote:

> On 2020-01-26 at 10:25:31, Johannes Schindelin wrote:
> > Hi brian,
> >
> > On Sat, 25 Jan 2020, brian m. carlson wrote:
> >
> > > This is the second-to-last series of test fixes for SHA-256.
> > >
> > > As mentioned previously, the patch for t3305 seems to indicate a bug in
> > > the notes code and I'm not familiar enough with that code to apply a
> > > fix.  This is a band-aid to get it working with SHA-256, but any
> > > comments on a more robust approach would of course be welcome.
> > >
> > > Changes from v1:
> > > * Drop patch for t3404 in favor of Dscho's fix.
> > > * Drop patch for t5616 in favor of Jonathan Tan's fix.
> > > * Add missing sign-off.
> > > * Move test_oid_init into the correct patch.
> >
> > Would you terribly mind pushing up your local branch to a public place? I
> > used the apply-from-public-inbox.sh script I maintain at
> > https://github.com/git-for-windows/build-extra to apply your patch series
> > on top of v2.25.0, but I got this:
> >
> > 	Applying: t5607: make hash size independent
> > 	Using index info to reconstruct a base tree...
> > 	M       t/t5607-clone-bundle.sh
> > 	Falling back to patching base and 3-way merge...
> > 	No changes -- Patch already applied.
> >
> > (I want to have a look at the notes fanout. Will keep you posted on my
> > investigation there.)
>
> The branch is test-fixes-part8 on https://github.com/bk2204/git.git.
> It's currently based off master.

Thank you!
Dscho

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On 2020-01-26 at 21:28:58, Johannes Schindelin wrote:
> brian, would you mind adopting this patch into your patch series? For your
> convenience, I pushed it up as `t3305-sha256-fanout` (based on your
> `transition-stage-4`) to https://github.com/dscho/git.

Absolutely.  Thanks so much for both of your work on this, and for
confirming that it's just a heuristic difference and not a bug in the
code.

I'll send out a v3 soon.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH v2 08/22] t4013: make test hash independent

[Johannes Schindelin]

Hi brian,

On Sat, 25 Jan 2020, brian m. carlson wrote:

> This test produces a large number of diff formats and compares the
> output with test files that have content specific to SHA-1. Since we are
> more interested in the format of the diffs, and not their specific
> values, which are tested elsewhere, add a function which uses sed to
> transform these specific object IDs into generic ones of the right size,
> which we can then compare.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t4013-diff-various.sh | 44 +++++++++++++++++++++++++++++++++++------
>  1 file changed, 38 insertions(+), 6 deletions(-)
>
> diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
> index 5ac94b390d..6f5f05c3a8 100755
> --- a/t/t4013-diff-various.sh
> +++ b/t/t4013-diff-various.sh
> @@ -120,6 +120,30 @@ test_expect_success setup '
>  +*++ [initial] Initial
>  EOF
>
> +process_diffs () {
> +	x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
> +	x07="$_x05[0-9a-f][0-9a-f]" &&

Any reason not to stay with the convention, i.e. using `_x04` and `_x07`
here (with leading underscores)?

> +	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
> +	    -e "s/From $_x40 /From $ZERO_OID /" \
> +	    -e "s/from $_x40)/from $ZERO_OID)/" \
> +	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
> +	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
> +	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
> +	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
> +	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
> +	    -e "s/^$_x40 /$ZERO_OID /" \
> +	    -e "s/^$_x40$/$ZERO_OID/" \
> +	    -e "s/$x07\.\.$x07/fffffff..fffffff/g" \
> +	    -e "s/$x07,$x07\.\.$x07/fffffff,fffffff..fffffff/g" \
> +	    -e "s/$x07 $x07 $x07/fffffff fffffff fffffff/g" \
> +	    -e "s/$x07 $x07 /fffffff fffffff /g" \
> +	    -e "s/Merge: $x07 $x07/Merge: fffffff fffffff/g" \
> +	    -e "s/$x07\.\.\./fffffff.../g" \
> +	    -e "s/ $x04\.\.\./ ffff.../g" \
> +	    -e "s/ $x04/ ffff/g" \
> +	    "$1"
> +}
> +
>  V=$(git version | sed -e 's/^git version //' -e 's/\./\\./g')
>  while read magic cmd
>  do
> @@ -158,13 +182,15 @@ do
>  		} >"$actual" &&
>  		if test -f "$expect"
>  		then
> +			process_diffs "$actual" >actual &&
> +			process_diffs "$expect" >expect &&
>  			case $cmd in
>  			*format-patch* | *-stat*)
> -				test_i18ncmp "$expect" "$actual";;
> +				test_i18ncmp expect actual;;
>  			*)
> -				test_cmp "$expect" "$actual";;
> +				test_cmp expect actual;;
>  			esac &&
> -			rm -f "$actual"
> +			rm -f "$actual" actual expect
>  		else
>  			# this is to help developing new tests.
>  			cp "$actual" "$expect"
> @@ -383,16 +409,22 @@ test_expect_success 'log -S requires an argument' '
>  test_expect_success 'diff --cached on unborn branch' '
>  	echo ref: refs/heads/unborn >.git/HEAD &&
>  	git diff --cached >result &&
> -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached" result
> +	process_diffs result >actual &&
> +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached" >expected &&

I was about to suggest letting `process_diffs` work in-place, but this
line makes that idea moot.

Another idea I had was to implement a `test_cmp_diff` that processes the
diffs and then compares them, but I guess that would be _less_ concise
than this patch.

Looks good,
Dscho

> +	test_cmp expected actual
>  '
>
>  test_expect_success 'diff --cached -- file on unborn branch' '
>  	git diff --cached -- file0 >result &&
> -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" result
> +	process_diffs result >actual &&
> +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached_--_file0" >expected &&
> +	test_cmp expected actual
>  '
>  test_expect_success 'diff --line-prefix with spaces' '
>  	git diff --line-prefix="| | | " --cached -- file0 >result &&
> -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" result
> +	process_diffs result >actual &&
> +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--line-prefix_--cached_--_file0" >expected &&
> +	test_cmp expected actual
>  '
>
>  test_expect_success 'diff-tree --stdin with log formatting' '
>

Re: [PATCH v2 10/22] t4211: make test hash independent

[Johannes Schindelin]

Hi brian,

On Sat, 25 Jan 2020, brian m. carlson wrote:

> This test uses several test files that contain hard-coded SHA-1 object
> IDs. Replace these values with generic ones of the correct size so that
> the test works with either SHA-1 or SHA-256.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---

An alternative would be, of course, to change the files in t/t4211/ to
have SHA-256 variants, and then to implement a helper to replace those by
SHA-1 variants when needed.

It would be quite a bit more work (and I'd be willing to carry at least
some of it), but I think it would be worth it, in order to keep the
associated safety against regressions. Would you agree?

Ciao,
Dscho

>  t/t4211-line-log.sh | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/t/t4211-line-log.sh b/t/t4211-line-log.sh
> index 8319163744..2cbfd8dd9e 100755
> --- a/t/t4211-line-log.sh
> +++ b/t/t4211-line-log.sh
> @@ -8,10 +8,20 @@ test_expect_success 'setup (import history)' '
>  	git reset --hard
>  '
>
> +process_output () {
> +	x07="$_x05[0-9a-f][0-9a-f]"
> +	sed -e "s/commit $OID_REGEX/commit $ZERO_OID/" \
> +	    -e "s/commit $_x40$/commit $ZERO_OID/" \
> +	    -e "s/Merge: $x07 $x07$/Merge: 0000000 0000000/" \
> +	    "$1"
> +}
> +
>  canned_test_1 () {
>  	test_expect_$1 "$2" "
> -		git log $2 >actual &&
> -		test_cmp \"\$TEST_DIRECTORY\"/t4211/expect.$3 actual
> +		git log $2 >result &&
> +		process_output result >actual &&
> +		process_output \"\$TEST_DIRECTORY\"/t4211/expect.$3 >expected &&
> +		test_cmp expected actual
>  	"
>  }
>
>

Re: [PATCH v2 11/22] t5302: make hash size independent

[Johannes Schindelin]

Hi brian,

On Sat, 25 Jan 2020, brian m. carlson wrote:

> Compute the length of object IDs and pack offsets instead of hard-coding
> constants.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5302-pack-index.sh | 18 +++++++++++-------
>  1 file changed, 11 insertions(+), 7 deletions(-)
>
> diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
> index 91d51b35f9..93ac003639 100755
> --- a/t/t5302-pack-index.sh
> +++ b/t/t5302-pack-index.sh
> @@ -8,7 +8,8 @@ test_description='pack index with 64-bit offsets and object CRC'
>
>  test_expect_success \
>      'setup' \
> -    'rm -rf .git &&
> +    'test_oid_init &&
> +     rm -rf .git &&

Why not consolidate the `test_expect_success` line into the current
convention at the same time ("while at it")? I.e.

	test_expect_success 'setup' '

>       git init &&
>       git config pack.threads 1 &&
>       i=1 &&
> @@ -32,7 +33,9 @@ test_expect_success \
>  	 echo $tree &&
>  	 git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\)	.*/\\1/"
>       } >obj-list &&
> -     git update-ref HEAD $commit'
> +     git update-ref HEAD $commit &&
> +     rawsz=$(test_oid rawsz)

Since the `rawsz` assignment has a lot to do with `test_oid_init`, I would
coddle this added line with the `test_oid_init` line above instead of
adding it here.

> +'
>
>  test_expect_success \
>      'pack-objects with index version 1' \
> @@ -152,6 +155,7 @@ test_expect_success \
>      '[index v1] 2) create a stealth corruption in a delta base reference' \
>      '# This test assumes file_101 is a delta smaller than 16 bytes.
>       # It should be against file_100 but we substitute its base for file_099
> +     offset=$((rawsz + 4)) &&
>       sha1_101=$(git hash-object file_101) &&
>       sha1_099=$(git hash-object file_099) &&
>       offs_101=$(index_obj_offset 1.idx $sha1_101) &&
> @@ -159,8 +163,8 @@ test_expect_success \
>       chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
>       dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
>          if=".git/objects/pack/pack-${pack1}.idx" \
> -        skip=$((4 + 256 * 4 + $nr_099 * 24)) \
> -        bs=1 count=20 conv=notrunc &&
> +        skip=$((4 + 256 * 4 + $nr_099 * offset)) \
> +        bs=1 count=$rawsz conv=notrunc &&

Similarly, the `offset` variable is only used here, so I would assign it
just before the `dd` call. The name `offset` might be a bit to generic not
to be reused, either, maybe `recordsz` or `index_entry_size` or `entrysz`?

Apart from that, the patch looks obviously good to me.

Ciao,
Dscho

P.S.: I'll stop reviewing here for now (It is not that I am tired of
looking at your patches, it is that I am just tired).

>       git cat-file blob $sha1_101 > file_101_foo1'
>
>  test_expect_success \
> @@ -200,8 +204,8 @@ test_expect_success \
>       chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
>       dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
>          if=".git/objects/pack/pack-${pack1}.idx" \
> -        skip=$((8 + 256 * 4 + $nr_099 * 20)) \
> -        bs=1 count=20 conv=notrunc &&
> +        skip=$((8 + 256 * 4 + $nr_099 * rawsz)) \
> +        bs=1 count=$rawsz conv=notrunc &&
>       git cat-file blob $sha1_101 > file_101_foo2'
>
>  test_expect_success \
> @@ -226,7 +230,7 @@ test_expect_success \
>       nr=$(index_obj_nr ".git/objects/pack/pack-${pack1}.idx" $obj) &&
>       chmod +w ".git/objects/pack/pack-${pack1}.idx" &&
>       printf xxxx | dd of=".git/objects/pack/pack-${pack1}.idx" conv=notrunc \
> -        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * 20 + $nr * 4)) &&
> +        bs=1 count=4 seek=$((8 + 256 * 4 + $(wc -l <obj-list) * rawsz + $nr * 4)) &&
>       ( while read obj
>         do git cat-file -p $obj >/dev/null || exit 1
>         done <obj-list ) &&
>

Re: [PATCH v2 08/22] t4013: make test hash independent

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3583 bytes --]

On 2020-01-26 at 22:08:18, Johannes Schindelin wrote:
> > diff --git a/t/t4013-diff-various.sh b/t/t4013-diff-various.sh
> > index 5ac94b390d..6f5f05c3a8 100755
> > --- a/t/t4013-diff-various.sh
> > +++ b/t/t4013-diff-various.sh
> > @@ -120,6 +120,30 @@ test_expect_success setup '
> >  +*++ [initial] Initial
> >  EOF
> >
> > +process_diffs () {
> > +	x04="[0-9a-f][0-9a-f][0-9a-f][0-9a-f]" &&
> > +	x07="$_x05[0-9a-f][0-9a-f]" &&
> 
> Any reason not to stay with the convention, i.e. using `_x04` and `_x07`
> here (with leading underscores)?

None in particular.  I have a slight bias against initial underscores
from C, where that has a specific meaning, but I agree consistency is
good, so I'll make that change.

> > +	sed -e "s/$OID_REGEX/$ZERO_OID/g" \
> > +	    -e "s/From $_x40 /From $ZERO_OID /" \
> > +	    -e "s/from $_x40)/from $ZERO_OID)/" \
> > +	    -e "s/commit $_x40\$/commit $ZERO_OID/" \
> > +	    -e "s/commit $_x40 (/commit $ZERO_OID (/" \
> > +	    -e "s/$_x40 $_x40 $_x40/$ZERO_OID $ZERO_OID $ZERO_OID/" \
> > +	    -e "s/$_x40 $_x40 /$ZERO_OID $ZERO_OID /" \
> > +	    -e "s/^$_x40 $_x40$/$ZERO_OID $ZERO_OID/" \
> > +	    -e "s/^$_x40 /$ZERO_OID /" \
> > +	    -e "s/^$_x40$/$ZERO_OID/" \
> > +	    -e "s/$x07\.\.$x07/fffffff..fffffff/g" \
> > +	    -e "s/$x07,$x07\.\.$x07/fffffff,fffffff..fffffff/g" \
> > +	    -e "s/$x07 $x07 $x07/fffffff fffffff fffffff/g" \
> > +	    -e "s/$x07 $x07 /fffffff fffffff /g" \
> > +	    -e "s/Merge: $x07 $x07/Merge: fffffff fffffff/g" \
> > +	    -e "s/$x07\.\.\./fffffff.../g" \
> > +	    -e "s/ $x04\.\.\./ ffff.../g" \
> > +	    -e "s/ $x04/ ffff/g" \
> > +	    "$1"
> > +}
> > +
> >  V=$(git version | sed -e 's/^git version //' -e 's/\./\\./g')
> >  while read magic cmd
> >  do
> > @@ -158,13 +182,15 @@ do
> >  		} >"$actual" &&
> >  		if test -f "$expect"
> >  		then
> > +			process_diffs "$actual" >actual &&
> > +			process_diffs "$expect" >expect &&
> >  			case $cmd in
> >  			*format-patch* | *-stat*)
> > -				test_i18ncmp "$expect" "$actual";;
> > +				test_i18ncmp expect actual;;
> >  			*)
> > -				test_cmp "$expect" "$actual";;
> > +				test_cmp expect actual;;
> >  			esac &&
> > -			rm -f "$actual"
> > +			rm -f "$actual" actual expect
> >  		else
> >  			# this is to help developing new tests.
> >  			cp "$actual" "$expect"
> > @@ -383,16 +409,22 @@ test_expect_success 'log -S requires an argument' '
> >  test_expect_success 'diff --cached on unborn branch' '
> >  	echo ref: refs/heads/unborn >.git/HEAD &&
> >  	git diff --cached >result &&
> > -	test_cmp "$TEST_DIRECTORY/t4013/diff.diff_--cached" result
> > +	process_diffs result >actual &&
> > +	process_diffs "$TEST_DIRECTORY/t4013/diff.diff_--cached" >expected &&
> 
> I was about to suggest letting `process_diffs` work in-place, but this
> line makes that idea moot.

If I could have done that, I would.

> Another idea I had was to implement a `test_cmp_diff` that processes the
> diffs and then compares them, but I guess that would be _less_ concise
> than this patch.

Yeah, this is a tricky test to work with because it does so many
different things and trying to handle all of them in a tidy way is hard
(as one can intuit from the giant sed statement).  As part of the patch
you quoted above, we sometimes use test_i18ncmp and sometimes use
test_cmp here, so it's not easy hard to pick something that works for
all of these cases without some duplication.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH v2 11/22] t5302: make hash size independent

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3160 bytes --]

On 2020-01-26 at 22:23:32, Johannes Schindelin wrote:
> On Sat, 25 Jan 2020, brian m. carlson wrote:
> 
> > Compute the length of object IDs and pack offsets instead of hard-coding
> > constants.
> >
> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> > ---
> >  t/t5302-pack-index.sh | 18 +++++++++++-------
> >  1 file changed, 11 insertions(+), 7 deletions(-)
> >
> > diff --git a/t/t5302-pack-index.sh b/t/t5302-pack-index.sh
> > index 91d51b35f9..93ac003639 100755
> > --- a/t/t5302-pack-index.sh
> > +++ b/t/t5302-pack-index.sh
> > @@ -8,7 +8,8 @@ test_description='pack index with 64-bit offsets and object CRC'
> >
> >  test_expect_success \
> >      'setup' \
> > -    'rm -rf .git &&
> > +    'test_oid_init &&
> > +     rm -rf .git &&
> 
> Why not consolidate the `test_expect_success` line into the current
> convention at the same time ("while at it")? I.e.
> 
> 	test_expect_success 'setup' '

Sounds good.

> >       git init &&
> >       git config pack.threads 1 &&
> >       i=1 &&
> > @@ -32,7 +33,9 @@ test_expect_success \
> >  	 echo $tree &&
> >  	 git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\)	.*/\\1/"
> >       } >obj-list &&
> > -     git update-ref HEAD $commit'
> > +     git update-ref HEAD $commit &&
> > +     rawsz=$(test_oid rawsz)
> 
> Since the `rawsz` assignment has a lot to do with `test_oid_init`, I would
> coddle this added line with the `test_oid_init` line above instead of
> adding it here.

Sure, that seems like a good idea.

> > +'
> >
> >  test_expect_success \
> >      'pack-objects with index version 1' \
> > @@ -152,6 +155,7 @@ test_expect_success \
> >      '[index v1] 2) create a stealth corruption in a delta base reference' \
> >      '# This test assumes file_101 is a delta smaller than 16 bytes.
> >       # It should be against file_100 but we substitute its base for file_099
> > +     offset=$((rawsz + 4)) &&
> >       sha1_101=$(git hash-object file_101) &&
> >       sha1_099=$(git hash-object file_099) &&
> >       offs_101=$(index_obj_offset 1.idx $sha1_101) &&
> > @@ -159,8 +163,8 @@ test_expect_success \
> >       chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
> >       dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
> >          if=".git/objects/pack/pack-${pack1}.idx" \
> > -        skip=$((4 + 256 * 4 + $nr_099 * 24)) \
> > -        bs=1 count=20 conv=notrunc &&
> > +        skip=$((4 + 256 * 4 + $nr_099 * offset)) \
> > +        bs=1 count=$rawsz conv=notrunc &&
> 
> Similarly, the `offset` variable is only used here, so I would assign it
> just before the `dd` call. The name `offset` might be a bit to generic not
> to be reused, either, maybe `recordsz` or `index_entry_size` or `entrysz`?

Yeah, those would be better names.  As I will freely admit, I'm bad at
naming things.

> P.S.: I'll stop reviewing here for now (It is not that I am tired of
> looking at your patches, it is that I am just tired).

Sure.  It's late where you are.  I appreciate your review.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[Johan Herland]

Hi Johannes,

On Sun, Jan 26, 2020 at 10:29 PM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> On Sun, 26 Jan 2020, Johan Herland wrote:
> > On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > On Sat, 25 Jan 2020, brian m. carlson wrote:
> > > > This test relies on a roughly equal distribution of hashes for notes in
> > > > order to ensure that fanouts are compressed.  If there are subtrees with
> > > > only one item left after removing notes, they'll end up still with one
> > > > level of fanout, causing the test to fail.
> > >
> > > That is _almost_ correct: The heuristic wants to see one bucket that has
> > > a note in it. Or something like that.
> > >
> > > See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> > > tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)
> >
> > Something like that, yeah... Re-reading this code, I believe we stop
> > the fanout at the current level when we can find one or more notes
> > that do not share the high-nibble of their path with another note.
> >
> > Here we're at the top level, so this corresponds to looking at the
> > very first hex character (0-9a-f) of the path (oid of annotated
> > object), and if there are at least two such objects for each hex
> > character, we will use a fanout of 1, otherwise, we collapse the
> > fanout to 0.
>
> That makes sense, but when I looked at the failed test, there seems
> something else at play, at least in addition to what you said: for
> _adding_ notes, your description is 100% accurate, but when deleting
> notes, we are apparently not collapsing `PTR_TYPE_SUBTREE` nodes "quickly"
> enough. Let me show you the part of `git ls-tree -r refs/notes/commits`
> that starts with the hex digits 7, 8 and 9:
>
> -- snip --
> 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b 72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> 100644 blob 2e3f3dad7043b2d02d09ca12b299acbe05d781357d3a56a2d012fdf787409459 76/62d4b264094ca479be86ef7aa66daae63e45afa633a3892dc787b13ecff495
> 100644 blob 22e7f0c5af7315d692f8a107a43ba0784e1ab00a20ea803fab1acad1319e5f79 7b/8c0a9c86815a94da7ef90b356b1f98d6a4099af3fdc3d8625a8fa793b63821
> 100644 blob b3ce67c9d5507dc00d95d8fb2000c1d5b70908ad1d2c034e5833f57b7bb85511 7f/0af9cf9259cd6e67c0af3324ca443dd3d56694fbdc94d28e300a768d3d0e6e
> 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a 8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53 93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> 100644 blob 0b1915ccb6f64cac64f2297893bce1ba7408660f79182302faaada71ee8c3c1c 96/b562b168f94e5c6e6a1e40ec4b817aa168c99769b1d9a46f7e048e93897fb4
> -- snap --
>
> This command was run in the worktree after running an unmodified t3305
> with `GIT_TEST_DEFAULT_HASH=sha256` in brian's `transition-stage-4` branch
> (well, I removed the `SHA1` prereq again, that's the modification I made).
>
> As you can see, there _is_ a fanout of 1, but there is only a single note
> for a commit whose ID starts with `8`.

Ah, yes, I glossed over the subtrees in my analysis, blindly assuming
that they would collapse more "quickly" than they do...

It seems the code values not unpacking subtrees "unnecessarily", and
this causes them to remain for longer than perhaps ideal.

> Debugging a little, it seems that the `PTR_TYPE_SUBTREE` node for `8` was
> not collapsed, even if there was only one item left.
>
> So I formed a hypothesis that the subtrees are only removed when the
> _last_ item is removed for any given leading nybble, but that turned out
> to also be incorrect. It is a bit more tricky than that. This is the
> smallest set to which I was able to reduce the notes without reducing the
> fanout to 0:
>
> -- snip --
> 100644 blob cc2ee5e00d8bc3805d704b67439dc8175bcf9497603288e6de2d4d8b3fc7be9f    05/b6e3d1394d020129f71fd8e41cf7ea8cbc58ae0f1332abd5da0c74ea194b71
> 100644 blob 1c6afe76a1bcd0103c36ab9707a2ca9e68974b6a6bbaae564c0509c43b4392bd    1e/311d64dc3ad5491964bcded60fee15b19d5b9c916a7e62a4f0746fa4e81fa6
> 100644 blob cf97d105e3970ef1cf9b12ac092be80abcd496c593bc8ca5550d059c3967630a    28/79e092d524b7ae9a42026ab2886094cce8ffc63175f8b3fd5de84faef10df3
> 100644 blob 78ef788b804dd0e5415c386b4a29668f61033483c35438f0471dfd7c4bfa093b    36/fe8fe67a2e9d0203c665d6e08ca454833ec32a97369769a7138d3938c0000b
> 100644 blob f46845c2d7e3272319aa5c18e359fdbc37731c88a945bb9632b8c4321983c75a    4a/a271a09d848f99d3fb978e5c156baa812a0fa1c30a88c885831641630be01a
> 100644 blob ecf5a4a178ca4b51cea457abe7c935761ca15a1d817f83c2da6816ede84db779    51/eaee3ca1a8698cb0aaa4de2d3f339985570da68b28e84af752e1cfd25f5197
> 100644 blob 2f4e9d6a4a1d050f8cb932ba545e53b48d9b669f6e00dc4a962d88ee9d92482b    62/dd63d43070c3ca7e3a6cdfa4ed970256a00a06e88d10fcb0532acd51419e0f
> 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b    72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a    8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53    93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> 100644 blob 589656c26944c471b5ac65739f8c7b96663a9f827a3d27beb49e39e3707b7294    a2/0e8f30856061125d479779755ef3238a7b561f9336e0143c437daac7d93f4c
> 100644 blob 7eaa9350d5c6bd6fd0fc4071c5d6a266949046c67987a9e1f665ba34d95f419d    a2/80d13a05aa68cc5ef948a8b69067807457fd37c00ce4a234fa4a0c0753ef4e
> 100644 blob 7a8378dd60d6024db645757eac7271a80b101a2df230ebd1a57ff52ff2d32e36    b9/2a3a7dc6db290d93f79150bfb31447f3e550cfe4a63c5ddbeac18fec755e86
> 100644 blob 86fff6007d9911249bee803a6630e182677355a5574e637d1a8301e219c5da86    c1/52ffd73d1c5e7c121c7c247682f1ee971f6f09101c96a84486d18be41d0dd0
> 100644 blob cd887698e1da81a76ae1caf0eaec19d60830a13ead152ec4700be511ceb8ee33    d0/3f742b8b95f68478946d7fe7495da9462801fadeaaa06c11bf54dbc46610f5
> 100644 blob 19f7bdfee9687311dbe1195e0a64954b677ae68e6d734fd5fb76ea4ad4f93782    ea/356ae2d38123b46639db98df14953f4c7cdd91738779174ec67876ce9487e3
> 100644 blob 948c0cba23ec0405c622c9dea8ed8dd7b3fa043c86b5e5a8b4de0d1c6a0e67b9    f7/802d4c716fed3c76fe58c86ac7c3ae3e19b8c0d3ea97c9f90f5939fe5a78d8
> 100644 blob 28690ad489e29e3607c82e1f626ec24d7f831555c802108bfe9b993fbd794a7e    f7/da03e811b7d9071ee19dabdc721e0f863e28c92dfa3257474282396a73bb44
> -- snap --
>
> You will note that there are two entries that start with `a2`, and two
> entries that start with `f7`. If I remove any of those, the corresponding
> subtree will be collapsed, and the fanout will be reduced to 0.

I'm looking at note_tree_remove() and note_tree_consolidate() to
explain this behavior. AFAICS, at the point where you remove, say, the
a20e8f entry, the note tree structure should consist of a single
int_node containing 16 subtree entries. note_tree_remove() will search
for the a20e8f entry which will cause the a2 subtree to be unpacked
and replaced with an int_node (representing 'a') referencing another
int_node (representing '2') containing two leaf_nodes. One of the leaf
nodes will be removed, and note_tree_consolidate() will then replace
the 2 int_nodes with the last remaining leaf_node. At this point the
root int_node should now contain 15 subtrees (not yet unpacked) and 1
"regular" leaf_node.

We then move on to write_notes_tree() to write out the resulting tree.
This ends up calling determine_fanout() (via for_each_note_helper()),
which will look at the root int_node, find the one "regular" leaf_node
there, and use that to return fanout = 0. (At which point the other
subtrees will be unpacked and the entire tree is "flattened".)

I now wonder why this did not happen before we got down to 17 notes.
Let's assume (as is most probable) that the previous notes removed
only shared _one_ leading nibble (not two). The root int_node would
have an int_node for the first-nibble which would contain two
subtrees, one for each second-nibble. One of these would be unpacked
and promptly removed, and then note_tree_consolidate() would be left
with the int_node for the first-nibble containing a single subtree
entry. AFAICS note_tree_consolidate() does _not_ collapse this
int_node (and move the subtree into the root node), although I can't
immediately see why it could/should not do this. Even if it did,
though, that would not be enough to trigger a lower fanout, as
determine_fanout() does not distinguish between int_nodes and subtree
nodes.

However, I suspect there may be room for further improvement here: If
we find ourselves consolidating an int_node whose _only_ non-NULL
entry is a subtree, it is a fairly safe assumption that the subtree
itself is probably close to empty as well, and we can probably bear
the cost of more eagerly unpacking it, as we are then more likely to
trigger a lower fanout when writing out the notes tree. At that point
going from fanout 1 to fanout 0 should certainly happen at a less
ridiculous total note count than 17...

> But it is only happenstance with SHA-256 that there are these entries that
> agree not only in the first, but also in the second leading nybble.

True.

> Therefore...
>
> > Hence we need an absolute minimum of 32 notes (and some rotten luck)
> > to get a fanout of 1. As the number of notes increase, the probably of
> > fanning out increases, passing 50% at ~79 notes, and reaching ~100%
> > somewhere north of 150 notes.
>
> ... I would register that we need an absolute minimum of 16 notes (and
> some rather crafty craft) to get a fanout of 1.
>
> In that light, I think that I would prefer to retract my patch that
> "only" reduces the remaining number of notes to 20: it should reduce them
> to 15 or less. So why not reduce it to 10 (because it is only one changed
> digit).

Agreed. The thing we're looking for in the test (and what is certainly
more important) is that we _do_ consolidate the fanout when the note
count decreases. The details around exactly when that happens is more
of a performance tuning issue, and not something that should break the
test.

> > > > The test happens to pass with SHA-1, but doesn't necessarily with other
> > > > hash algorithms, so annotate it with the SHA1 prerequisite.
> > >
> > > I would rather see this tested, still, and reducing the number of notes
> > > that are retained from 50 to 20 before testing that the fanout has been
> > > reduced to 0 seems to do the trick. Therefore, I would love to submit this
> > > for squashing:
> >
> > Yes, it seems that for SHA1 and the (deterministic) objects used in
> > the test, we got away with 50 notes, but that is not the case for
> > other hash algorithms. Lowering the number to 20 definitely results a
> > fanout of 0, as should any other number below 32.
> >
> > +1 to Dscho's squash.
> >
> > ...Johan
>
> Thank you so much for the analysis. To be honest, I did not quite
> understand all the details of the comment added in 73f77b989f8 when I
> wrote the patch I suggested, so I basically just picked that number "20"
> out of thin air.
>
> Together with your insights, I would like to propose this commit message
> for the squashed commits (I left in the hunk that removes the `SHA1`
> prerequisite, but of course that won't be part of the final commit):
>
> -- snip --
> t3305: make fanout test more robust (needed for SHA-256)
>
> To make things more performant, notes are stored in a "fanout": when
> there are enough commit notes, they are no longer stored as verbatim
> commit IDs at the top-level tree of the notes ref, but instead the tree
> is deepened much like the loose object cache: subtrees are introduced
> whose names are the two hex digits they "chomp off" the commit IDs.
>
> The test case 'deleting most notes triggers fanout consolidation' wants
> to verify that the fanout level is reduced automatically when enough
> notes have been deleted.
>
> However, that test case expected that reduction to level 0 (i.e. _no_
> fanout subtrees) to happen after reducing the originally-added 300 notes
> to 50, which _happened_ to work with SHA-1-based commit IDs, but it is
> no longer works with SHA-256-based ones.
>
> The reason: The heuristic for the fanout looks at the number of entries
> for leading nybbles (read: hex digits) of the commit IDs. If there are
> more than a single annotated commit for all of the 16 hex digits, the
> fanout is incremented. It is a bit more tricky when reducing the number
> of notes: the fanout is reduced reliably only if there are less notes
> than hex digits (i.e. less than 15 notes) for a given prefix.
>
> For good measure, let's reduce the number of notes to 10 in the test
> case 'deleting most notes with git-notes' so that the test case
> 'deleting most notes triggers fanout consolidation' is guaranteed to
> succeed with _any_ hash algorithm.

Great commit message.

> Original-patch-by: brian m. carlson <sandals@crustytoothpaste.net>
> Helped-by: Johan Herland <johan@herland.net>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johan Herland <johan@herland.net>


Have fun!
...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: [PATCH v2 03/22] t3305: annotate with SHA1 prerequisite

[Johannes Schindelin]

Hi Johan,

On Mon, 27 Jan 2020, Johan Herland wrote:

> On Sun, Jan 26, 2020 at 10:29 PM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > On Sun, 26 Jan 2020, Johan Herland wrote:
> > > On Sun, Jan 26, 2020 at 12:16 PM Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > > > On Sat, 25 Jan 2020, brian m. carlson wrote:
> > > > > This test relies on a roughly equal distribution of hashes for notes in
> > > > > order to ensure that fanouts are compressed.  If there are subtrees with
> > > > > only one item left after removing notes, they'll end up still with one
> > > > > level of fanout, causing the test to fail.
> > > >
> > > > That is _almost_ correct: The heuristic wants to see one bucket that has
> > > > a note in it. Or something like that.
> > > >
> > > > See 73f77b909f8 (Notes API: for_each_note(): Traverse the entire notes
> > > > tree with a callback, 2010-02-13) for details. (Cc:ing Johan.)
> > >
> > > Something like that, yeah... Re-reading this code, I believe we stop
> > > the fanout at the current level when we can find one or more notes
> > > that do not share the high-nibble of their path with another note.
> > >
> > > Here we're at the top level, so this corresponds to looking at the
> > > very first hex character (0-9a-f) of the path (oid of annotated
> > > object), and if there are at least two such objects for each hex
> > > character, we will use a fanout of 1, otherwise, we collapse the
> > > fanout to 0.
> >
> > That makes sense, but when I looked at the failed test, there seems
> > something else at play, at least in addition to what you said: for
> > _adding_ notes, your description is 100% accurate, but when deleting
> > notes, we are apparently not collapsing `PTR_TYPE_SUBTREE` nodes "quickly"
> > enough. Let me show you the part of `git ls-tree -r refs/notes/commits`
> > that starts with the hex digits 7, 8 and 9:
> >
> > -- snip --
> > 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b 72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> > 100644 blob 2e3f3dad7043b2d02d09ca12b299acbe05d781357d3a56a2d012fdf787409459 76/62d4b264094ca479be86ef7aa66daae63e45afa633a3892dc787b13ecff495
> > 100644 blob 22e7f0c5af7315d692f8a107a43ba0784e1ab00a20ea803fab1acad1319e5f79 7b/8c0a9c86815a94da7ef90b356b1f98d6a4099af3fdc3d8625a8fa793b63821
> > 100644 blob b3ce67c9d5507dc00d95d8fb2000c1d5b70908ad1d2c034e5833f57b7bb85511 7f/0af9cf9259cd6e67c0af3324ca443dd3d56694fbdc94d28e300a768d3d0e6e
> > 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a 8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> > 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53 93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> > 100644 blob 0b1915ccb6f64cac64f2297893bce1ba7408660f79182302faaada71ee8c3c1c 96/b562b168f94e5c6e6a1e40ec4b817aa168c99769b1d9a46f7e048e93897fb4
> > -- snap --
> >
> > This command was run in the worktree after running an unmodified t3305
> > with `GIT_TEST_DEFAULT_HASH=sha256` in brian's `transition-stage-4` branch
> > (well, I removed the `SHA1` prereq again, that's the modification I made).
> >
> > As you can see, there _is_ a fanout of 1, but there is only a single note
> > for a commit whose ID starts with `8`.
>
> Ah, yes, I glossed over the subtrees in my analysis, blindly assuming
> that they would collapse more "quickly" than they do...
>
> It seems the code values not unpacking subtrees "unnecessarily", and
> this causes them to remain for longer than perhaps ideal.
>
> > Debugging a little, it seems that the `PTR_TYPE_SUBTREE` node for `8` was
> > not collapsed, even if there was only one item left.
> >
> > So I formed a hypothesis that the subtrees are only removed when the
> > _last_ item is removed for any given leading nybble, but that turned out
> > to also be incorrect. It is a bit more tricky than that. This is the
> > smallest set to which I was able to reduce the notes without reducing the
> > fanout to 0:
> >
> > -- snip --
> > 100644 blob cc2ee5e00d8bc3805d704b67439dc8175bcf9497603288e6de2d4d8b3fc7be9f    05/b6e3d1394d020129f71fd8e41cf7ea8cbc58ae0f1332abd5da0c74ea194b71
> > 100644 blob 1c6afe76a1bcd0103c36ab9707a2ca9e68974b6a6bbaae564c0509c43b4392bd    1e/311d64dc3ad5491964bcded60fee15b19d5b9c916a7e62a4f0746fa4e81fa6
> > 100644 blob cf97d105e3970ef1cf9b12ac092be80abcd496c593bc8ca5550d059c3967630a    28/79e092d524b7ae9a42026ab2886094cce8ffc63175f8b3fd5de84faef10df3
> > 100644 blob 78ef788b804dd0e5415c386b4a29668f61033483c35438f0471dfd7c4bfa093b    36/fe8fe67a2e9d0203c665d6e08ca454833ec32a97369769a7138d3938c0000b
> > 100644 blob f46845c2d7e3272319aa5c18e359fdbc37731c88a945bb9632b8c4321983c75a    4a/a271a09d848f99d3fb978e5c156baa812a0fa1c30a88c885831641630be01a
> > 100644 blob ecf5a4a178ca4b51cea457abe7c935761ca15a1d817f83c2da6816ede84db779    51/eaee3ca1a8698cb0aaa4de2d3f339985570da68b28e84af752e1cfd25f5197
> > 100644 blob 2f4e9d6a4a1d050f8cb932ba545e53b48d9b669f6e00dc4a962d88ee9d92482b    62/dd63d43070c3ca7e3a6cdfa4ed970256a00a06e88d10fcb0532acd51419e0f
> > 100644 blob 22939636f79a53181ea58f04f2136bd745976edd0018465e60a8df89816a9c2b    72/67b1c94ab5ddf005fd4b0e50b6a7816a62e7ed0459cec6aa1a00577b2111ba
> > 100644 blob fafb40e32e87a2c481df6ceb37804d80c995faec3d7772c071b129fd47c2ba8a    8a/f1ad99a5e559f5835007f2bcb1b07de0e8c7434e7fbaa676a2edbd796a7f60
> > 100644 blob b811e8da7c7acc83ef025753504ff6ed2d1eb8d2bb832d6b7487a7786d67aa53    93/f364fafaee6a178d8e8939f15d5b260f71940f663c3731396ee43082fd6551
> > 100644 blob 589656c26944c471b5ac65739f8c7b96663a9f827a3d27beb49e39e3707b7294    a2/0e8f30856061125d479779755ef3238a7b561f9336e0143c437daac7d93f4c
> > 100644 blob 7eaa9350d5c6bd6fd0fc4071c5d6a266949046c67987a9e1f665ba34d95f419d    a2/80d13a05aa68cc5ef948a8b69067807457fd37c00ce4a234fa4a0c0753ef4e
> > 100644 blob 7a8378dd60d6024db645757eac7271a80b101a2df230ebd1a57ff52ff2d32e36    b9/2a3a7dc6db290d93f79150bfb31447f3e550cfe4a63c5ddbeac18fec755e86
> > 100644 blob 86fff6007d9911249bee803a6630e182677355a5574e637d1a8301e219c5da86    c1/52ffd73d1c5e7c121c7c247682f1ee971f6f09101c96a84486d18be41d0dd0
> > 100644 blob cd887698e1da81a76ae1caf0eaec19d60830a13ead152ec4700be511ceb8ee33    d0/3f742b8b95f68478946d7fe7495da9462801fadeaaa06c11bf54dbc46610f5
> > 100644 blob 19f7bdfee9687311dbe1195e0a64954b677ae68e6d734fd5fb76ea4ad4f93782    ea/356ae2d38123b46639db98df14953f4c7cdd91738779174ec67876ce9487e3
> > 100644 blob 948c0cba23ec0405c622c9dea8ed8dd7b3fa043c86b5e5a8b4de0d1c6a0e67b9    f7/802d4c716fed3c76fe58c86ac7c3ae3e19b8c0d3ea97c9f90f5939fe5a78d8
> > 100644 blob 28690ad489e29e3607c82e1f626ec24d7f831555c802108bfe9b993fbd794a7e    f7/da03e811b7d9071ee19dabdc721e0f863e28c92dfa3257474282396a73bb44
> > -- snap --
> >
> > You will note that there are two entries that start with `a2`, and two
> > entries that start with `f7`. If I remove any of those, the corresponding
> > subtree will be collapsed, and the fanout will be reduced to 0.
>
> I'm looking at note_tree_remove() and note_tree_consolidate() to
> explain this behavior. AFAICS, at the point where you remove, say, the
> a20e8f entry, the note tree structure should consist of a single
> int_node containing 16 subtree entries. note_tree_remove() will search
> for the a20e8f entry which will cause the a2 subtree to be unpacked
> and replaced with an int_node (representing 'a') referencing another
> int_node (representing '2') containing two leaf_nodes. One of the leaf
> nodes will be removed, and note_tree_consolidate() will then replace
> the 2 int_nodes with the last remaining leaf_node. At this point the
> root int_node should now contain 15 subtrees (not yet unpacked) and 1
> "regular" leaf_node.
>
> We then move on to write_notes_tree() to write out the resulting tree.
> This ends up calling determine_fanout() (via for_each_note_helper()),
> which will look at the root int_node, find the one "regular" leaf_node
> there, and use that to return fanout = 0. (At which point the other
> subtrees will be unpacked and the entire tree is "flattened".)
>
> I now wonder why this did not happen before we got down to 17 notes.
> Let's assume (as is most probable) that the previous notes removed
> only shared _one_ leading nibble (not two). The root int_node would
> have an int_node for the first-nibble which would contain two
> subtrees, one for each second-nibble. One of these would be unpacked
> and promptly removed, and then note_tree_consolidate() would be left
> with the int_node for the first-nibble containing a single subtree
> entry. AFAICS note_tree_consolidate() does _not_ collapse this
> int_node (and move the subtree into the root node), although I can't
> immediately see why it could/should not do this. Even if it did,
> though, that would not be enough to trigger a lower fanout, as
> determine_fanout() does not distinguish between int_nodes and subtree
> nodes.
>
> However, I suspect there may be room for further improvement here: If
> we find ourselves consolidating an int_node whose _only_ non-NULL
> entry is a subtree, it is a fairly safe assumption that the subtree
> itself is probably close to empty as well, and we can probably bear
> the cost of more eagerly unpacking it, as we are then more likely to
> trigger a lower fanout when writing out the notes tree. At that point
> going from fanout 1 to fanout 0 should certainly happen at a less
> ridiculous total note count than 17...

Thank you for digging in even further!

> > But it is only happenstance with SHA-256 that there are these entries that
> > agree not only in the first, but also in the second leading nybble.
>
> True.
>
> > Therefore...
> >
> > > Hence we need an absolute minimum of 32 notes (and some rotten luck)
> > > to get a fanout of 1. As the number of notes increase, the probably of
> > > fanning out increases, passing 50% at ~79 notes, and reaching ~100%
> > > somewhere north of 150 notes.
> >
> > ... I would register that we need an absolute minimum of 16 notes (and
> > some rather crafty craft) to get a fanout of 1.
> >
> > In that light, I think that I would prefer to retract my patch that
> > "only" reduces the remaining number of notes to 20: it should reduce them
> > to 15 or less. So why not reduce it to 10 (because it is only one changed
> > digit).
>
> Agreed. The thing we're looking for in the test (and what is certainly
> more important) is that we _do_ consolidate the fanout when the note
> count decreases. The details around exactly when that happens is more
> of a performance tuning issue, and not something that should break the
> test.
>
> > > > > The test happens to pass with SHA-1, but doesn't necessarily with other
> > > > > hash algorithms, so annotate it with the SHA1 prerequisite.
> > > >
> > > > I would rather see this tested, still, and reducing the number of notes
> > > > that are retained from 50 to 20 before testing that the fanout has been
> > > > reduced to 0 seems to do the trick. Therefore, I would love to submit this
> > > > for squashing:
> > >
> > > Yes, it seems that for SHA1 and the (deterministic) objects used in
> > > the test, we got away with 50 notes, but that is not the case for
> > > other hash algorithms. Lowering the number to 20 definitely results a
> > > fanout of 0, as should any other number below 32.
> > >
> > > +1 to Dscho's squash.
> > >
> > > ...Johan
> >
> > Thank you so much for the analysis. To be honest, I did not quite
> > understand all the details of the comment added in 73f77b989f8 when I
> > wrote the patch I suggested, so I basically just picked that number "20"
> > out of thin air.
> >
> > Together with your insights, I would like to propose this commit message
> > for the squashed commits (I left in the hunk that removes the `SHA1`
> > prerequisite, but of course that won't be part of the final commit):
> >
> > -- snip --
> > t3305: make fanout test more robust (needed for SHA-256)
> >
> > To make things more performant, notes are stored in a "fanout": when
> > there are enough commit notes, they are no longer stored as verbatim
> > commit IDs at the top-level tree of the notes ref, but instead the tree
> > is deepened much like the loose object cache: subtrees are introduced
> > whose names are the two hex digits they "chomp off" the commit IDs.
> >
> > The test case 'deleting most notes triggers fanout consolidation' wants
> > to verify that the fanout level is reduced automatically when enough
> > notes have been deleted.
> >
> > However, that test case expected that reduction to level 0 (i.e. _no_
> > fanout subtrees) to happen after reducing the originally-added 300 notes
> > to 50, which _happened_ to work with SHA-1-based commit IDs, but it is
> > no longer works with SHA-256-based ones.
> >
> > The reason: The heuristic for the fanout looks at the number of entries
> > for leading nybbles (read: hex digits) of the commit IDs. If there are
> > more than a single annotated commit for all of the 16 hex digits, the
> > fanout is incremented. It is a bit more tricky when reducing the number
> > of notes: the fanout is reduced reliably only if there are less notes
> > than hex digits (i.e. less than 15 notes) for a given prefix.
> >
> > For good measure, let's reduce the number of notes to 10 in the test
> > case 'deleting most notes with git-notes' so that the test case
> > 'deleting most notes triggers fanout consolidation' is guaranteed to
> > succeed with _any_ hash algorithm.
>
> Great commit message.
>
> > Original-patch-by: brian m. carlson <sandals@crustytoothpaste.net>
> > Helped-by: Johan Herland <johan@herland.net>
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> Signed-off-by: Johan Herland <johan@herland.net>

Excellent!

> Have fun!
> ...Johan

You, too. Thank you!
Dscho

>
> --
> Johan Herland, <johan@herland.net>
> www.herland.net
>

Re: [PATCH v2 13/22] t5313: make test hash independent

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> Make this test hash independent by computing the length of the object
> offsets and looking up values which will hash to object IDs with the
> right properties.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5313-pack-bounds-checks.sh | 19 +++++++++++++++----
>  1 file changed, 15 insertions(+), 4 deletions(-)
>
> diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh
> index f1708d415e..8d805f845a 100755
> --- a/t/t5313-pack-bounds-checks.sh
> +++ b/t/t5313-pack-bounds-checks.sh
> @@ -38,16 +38,27 @@ munge () {
>  # for the initial, and another ofs(4*nr) past that for the extended.
>  #
>  ofs_table () {
> -	echo $((4 + 4 + 4*256 + 20*$1 + 4*$1))
> +	echo $((4 + 4 + 4*256 + $(test_oid rawsz)*$1 + 4*$1))
>  }
>  extended_table () {
>  	echo $(($(ofs_table "$1") + 4*$1))
>  }
>  
> +test_expect_success 'setup' '
> +	test_oid_init &&
> +	test_oid_cache <<-EOF
> +	oid000 sha1:1485
> +	oid000 sha256:4222
> +
> +	oidfff sha1:74
> +	oidfff sha256:1350
> +	EOF
> +'
> +
>  test_expect_success 'set up base packfile and variables' '
>  	# the hash of this content starts with ff, which
>  	# makes some later computations much simpler
> -	echo 74 >file &&
> +	echo $(test_oid oidfff) >file &&
>  	git add file &&
>  	git commit -m base &&
>  	git repack -ad &&
> @@ -140,10 +151,10 @@ test_expect_success 'bogus offset inside v2 extended table' '
>  	# an extended table (if the first object were larger than 2^31).
>  	#
>  	# Note that the value is important here. We want $object as
> -	# the second entry in sorted-sha1 order. The sha1 of 1485 starts
> +	# the second entry in sorted-sha1 order. The hash of this object starts

Micronit: "shorted-hash order", no?

>  	# with "000", which sorts before that of $object (which starts
>  	# with "fff").
> -	second=$(echo 1485 | git hash-object -w --stdin) &&
> +	second=$(test_oid oid000 | git hash-object -w --stdin) &&
>  	do_pack "$object $second" --index-version=2 &&
>  
>  	# We have to make extra room for the table, so we cannot

Re: [PATCH v2 15/22] t5515: make test hash independent

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> +convert_expected () {
> +	file="$1" &&
> +	for i in one three_file master master2 one_tree three two two2 three2
> +	do
> +		sed -e "s/$(test_oid --hash=sha1 "$i")/$(test_oid "$i")/g" \
> +			"$file" >"$file.tmp" &&
> +		mv "$file.tmp" "$file"
> +	done
> +}

Perhaps we can avoid rewriting the same file that many times, by
feeding the mapping to a single invocation of sed?  E.g.

	sedScript=
	for i in one three_file master master2 one_tree three two two2 three2
	do
		i="s/$(test_oid --hash=sha1 $i/$(test_oid $i)/g"
		sedScript=$sedScript${sedScript:+;}$i"
	done &&
	sed -e "$sedScript" "$file" >"$file.new" &&
	mv "$file.new" "$file"

If somebody's "sed" does not like multiple command concatenated with
";", we can take advantage of the fact that we are just replacing
hexadecimal string without anything funny and go eval, e.g.

	sedCmd="sed"
	for i in one three_file master master2 one_tree three two two2 three2
	do
		sedCmd="$sedCmd -e s/$(test_oid --hash=sha1 $i/$(test_oid $i)/g"
	done &&
	eval "$sedCmd" "$file" >"$file.new" &&
	mv "$file.new" "$file"

Re: [PATCH v2 17/23] t5616: use correct filter syntax

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> In the setup steps for the promisor remote tests, we clone a repository
> and filter out all trees with depth greater than or equal to zero, which
> also filters out all blobs.
>
> With SHA-1, this test passes because the object we happen to request
> from the server is the blob that the promisor remote has.  However, due
> to a different ordering with SHA-256, we request the tree containing
> that blob, which the promisor remote does not have.  As a consequence,
> we fail with a "not our ref" error.

Sorry, but I do not understand this part.

The object name of the original blob (which is the only thing
"promisor-remote" is given) may sort earlier or later than other
objects that are missing in the "client" repository, but it is not
clear how it makes difference in the final outcome---even if the
blob is asked first (in the SHA-1 version), wouldn't we need to
fetch the tree after that, and wouldn't that fail?  If the SHA-256
version that happens to ask for the tree first and fails, wouldn't
that mean we need to fetch both anyway?

Is it that the current test with SHA-1 is broken in that it lets the
lazy fetch fail (due to missing tree) but because the failure happens
after the blob gets feteched, and it ignores the failure of the lazy
fetch, and only checks if the blob got fetched, it happens to "pass"?

> Since what we want to test is that the blob is transferred, let's adjust
> the filter to just filter out blobs, not trees.  That means that we'll
> transfer the previously problematic tree as part of the normal clone,
> and we can then test that the blob is fetched from the promisor remote
> as expected.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5616-partial-clone.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh
> index fea56cda6d..9fd6e780f9 100755
> --- a/t/t5616-partial-clone.sh
> +++ b/t/t5616-partial-clone.sh
> @@ -317,7 +317,7 @@ setup_triangle () {
>  	cp big-blob.txt server &&
>  	git -C server add big-blob.txt &&
>  	git -C server commit -m "initial" &&
> -	git clone --bare --filter=tree:0 "file://$(pwd)/server" client &&
> +	git clone --bare --filter=blob:none "file://$(pwd)/server" client &&
>  	echo another line >>server/big-blob.txt &&
>  	git -C server commit -am "append line to big blob" &&
>  

Re: [PATCH v2 18/22] t5703: make test work with SHA-256

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> This test used an object ID which was 40 hex characters in length,
> causing the test not only not to pass, but to hang, when run with
> SHA-256 as the hash.  Change this value to a fixed dummy object ID using
> test_oid_init and test_oid.

Has the above part been split into another patch?

> Furthermore, ensure we extract an object ID of the appropriate length
> using cut with fields instead of a fixed length.

This one makes sense.

> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  t/t5703-upload-pack-ref-in-want.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
> index 1424fabd4a..5511cdcec2 100755
> --- a/t/t5703-upload-pack-ref-in-want.sh
> +++ b/t/t5703-upload-pack-ref-in-want.sh
> @@ -19,7 +19,7 @@ get_actual_commits () {
>  		}' <out | test-tool pkt-line unpack-sideband >o.pack &&
>  	git index-pack o.pack &&
>  	git verify-pack -v o.idx >objs &&
> -	grep commit objs | cut -c-40 | sort >actual_commits
> +	grep commit objs | cut -d" " -f1 | sort >actual_commits
>  }
>  
>  check_output () {

Re: [PATCH v2 18/22] t5703: make test work with SHA-256

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 620 bytes --]

On 2020-01-28 at 19:09:52, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > This test used an object ID which was 40 hex characters in length,
> > causing the test not only not to pass, but to hang, when run with
> > SHA-256 as the hash.  Change this value to a fixed dummy object ID using
> > test_oid_init and test_oid.
> 
> Has the above part been split into another patch?

I think it has been.  I'll just squash them together, since it doesn't
really make sense to separate them out.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH v2 17/23] t5616: use correct filter syntax

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2134 bytes --]

On 2020-01-28 at 19:06:01, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > In the setup steps for the promisor remote tests, we clone a repository
> > and filter out all trees with depth greater than or equal to zero, which
> > also filters out all blobs.
> >
> > With SHA-1, this test passes because the object we happen to request
> > from the server is the blob that the promisor remote has.  However, due
> > to a different ordering with SHA-256, we request the tree containing
> > that blob, which the promisor remote does not have.  As a consequence,
> > we fail with a "not our ref" error.
> 
> Sorry, but I do not understand this part.
> 
> The object name of the original blob (which is the only thing
> "promisor-remote" is given) may sort earlier or later than other
> objects that are missing in the "client" repository, but it is not
> clear how it makes difference in the final outcome---even if the
> blob is asked first (in the SHA-1 version), wouldn't we need to
> fetch the tree after that, and wouldn't that fail?  If the SHA-256
> version that happens to ask for the tree first and fails, wouldn't
> that mean we need to fetch both anyway?
> 
> Is it that the current test with SHA-1 is broken in that it lets the
> lazy fetch fail (due to missing tree) but because the failure happens
> after the blob gets feteched, and it ignores the failure of the lazy
> fetch, and only checks if the blob got fetched, it happens to "pass"?

I think Jonathan Tan figured out that my analysis was wrong, and that
the issue is that the larger object ID length causes deltification to
happen.  The test assumes that the tree is sent as a non-delta object,
and when it's sent as a deltaed object instead, we fail.  He explains
this quite well in <20200113202823.228062-1-jonathantanmy@google.com>,
which you seem to have picked up.

Since my analysis was wrong here and he's provided a patch which fixes
the issue in a much more robust way, I'm dropping this patch in v3.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: [PATCH v2 15/22] t5515: make test hash independent

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

On 2020-01-28 at 18:28:55, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > +convert_expected () {
> > +	file="$1" &&
> > +	for i in one three_file master master2 one_tree three two two2 three2
> > +	do
> > +		sed -e "s/$(test_oid --hash=sha1 "$i")/$(test_oid "$i")/g" \
> > +			"$file" >"$file.tmp" &&
> > +		mv "$file.tmp" "$file"
> > +	done
> > +}
> 
> Perhaps we can avoid rewriting the same file that many times, by
> feeding the mapping to a single invocation of sed?  E.g.
> 
> 	sedScript=
> 	for i in one three_file master master2 one_tree three two two2 three2
> 	do
> 		i="s/$(test_oid --hash=sha1 $i/$(test_oid $i)/g"
> 		sedScript=$sedScript${sedScript:+;}$i"
> 	done &&
> 	sed -e "$sedScript" "$file" >"$file.new" &&
> 	mv "$file.new" "$file"

This is a good idea.  We could also write a small sed script to a file
at the beginning and invoke it with sed -f.  That's portable (POSIX
supports it) and should be easy and efficient to do.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]