git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
* [PATCH v4 00/39] SHA-256, part 3/3
@ 2020-07-26 19:53 brian m. carlson
  2020-07-26 19:53 ` [PATCH v4 01/39] t: make test-bloom initialize repository brian m. carlson
                   ` (38 more replies)
  0 siblings, 39 replies; 58+ messages in thread
From: brian m. carlson @ 2020-07-26 19:53 UTC (permalink / raw)
  To: git; +Cc: Eric Sunshine

This is the final part required for the stage 4 implementation of
SHA-256.

Changes from v3:

Changes from v2:
* Move bundle signature parsing into a function.
* Improve parsing of bundle capabilities.
* Improve t5510 to select only the interesting lines with sed.
* Use cat and a heredoc instead of multiple echos.
* Use test_i18ngrep where appropriate.
* Reject extensions.objectFormat for repo v0.
* Update documentation accordingly.

Changes from v1:
* Add a patch to add a bundle v3 format with capabilities that include
  the hash algorithm.
* Remove bundle detection of hash algorithm using hash algorithm length.
* Fix typo in the extensions.objectFormat documentation.
* Document that extensions.objectFormat is ignored for repo v0.
* Use test instead of brackets.
* Use a command group instead of a subshell for efficiency.
* Drop the bizarre test_might_fail, since it isn't needed (and is
  bizarre and wrong).
* Write "SHA-256" instead of "NewHash" in commit messages.
* Split SHA-256 CI tests to an independent run and avoid running a third
  test run to a single CI job.

Range-diff below.

Johannes Schindelin (1):
  t3404: prepare 'short SHA-1 collision' tests for SHA-256

brian m. carlson (38):
  t: make test-bloom initialize repository
  t1001: use $ZERO_OID
  t3305: make hash agnostic
  t6100: make hash size independent
  t6101: make hash size independent
  t6301: make hash size independent
  t6500: specify test values for SHA-256
  t6501: avoid hard-coded objects
  t7003: compute appropriate length constant
  t7063: make hash size independent
  t7201: abstract away SHA-1-specific constants
  t7102: abstract away SHA-1-specific constants
  t7400: make hash size independent
  t7405: make hash size independent
  t7506: avoid checking for SHA-1-specific constants
  t7508: use $ZERO_OID instead of hard-coded constant
  t8002: make hash size independent
  t8003: make hash size independent
  t8011: make hash size independent
  t9300: abstract away SHA-1-specific constants
  t9300: use $ZERO_OID instead of hard-coded object ID
  t9301: make hash size independent
  t9350: make hash size independent
  t9500: ensure that algorithm info is preserved in config
  t9700: make hash size independent
  t5308: make test work with SHA-256
  t0410: mark test with SHA1 prerequisite
  http-fetch: set up git directory before parsing pack hashes
  builtin/verify-pack: implement an --object-format option
  bundle: add new version for use with SHA-256
  setup: add support for reading extensions.objectformat
  Enable SHA-256 support by default
  t: add test_oid option to select hash algorithm
  t: allow testing different hash algorithms via environment
  t: make SHA1 prerequisite depend on default hash
  ci: run tests with SHA-256
  docs: add documentation for extensions.objectFormat
  t: remove test_oid_init in tests

 Documentation/config.txt                  |   2 +
 Documentation/config/extensions.txt       |   8 ++
 Documentation/git-bundle.txt              |   9 +-
 Documentation/technical/bundle-format.txt |  30 ++++-
 builtin/bundle.c                          |   5 +-
 builtin/init-db.c                         |   5 -
 builtin/verify-pack.c                     |  29 +++--
 bundle.c                                  |  85 ++++++++----
 bundle.h                                  |   4 +-
 ci/run-build-and-tests.sh                 |   5 +
 config.mak.dev                            |   2 -
 http-fetch.c                              |   4 +-
 repository.c                              |   4 -
 setup.c                                   |  16 ++-
 t/helper/test-bloom.c                     |   2 +
 t/lib-pack.sh                             |  11 +-
 t/lib-submodule-update.sh                 |   1 -
 t/t0000-basic.sh                          |  15 ++-
 t/t0001-init.sh                           |  29 +++++
 t/t0410-partial-clone.sh                  |   4 +-
 t/t1006-cat-file.sh                       |   2 -
 t/t1050-large.sh                          |   1 -
 t/t1091-sparse-checkout-builtin.sh        |   4 +-
 t/t1410-reflog.sh                         |   1 -
 t/t1450-fsck.sh                           |   1 -
 t/t1500-rev-parse.sh                      |   1 -
 t/t3305-notes-fanout.sh                   |   2 +-
 t/t3308-notes-merge.sh                    |   1 -
 t/t3404-rebase-interactive.sh             |  49 +++++--
 t/t3600-rm.sh                             |   1 -
 t/t3800-mktag.sh                          |   1 -
 t/t4002-diff-basic.sh                     |   2 -
 t/t4027-diff-submodule.sh                 |   1 -
 t/t4134-apply-submodule.sh                |   1 -
 t/t4200-rerere.sh                         |   1 -
 t/t4211-line-log.sh                       |   1 -
 t/t5300-pack-object.sh                    |   3 +-
 t/t5302-pack-index.sh                     |   1 -
 t/t5308-pack-detect-duplicates.sh         |  20 +--
 t/t5313-pack-bounds-checks.sh             |   1 -
 t/t5318-commit-graph.sh                   |   3 +-
 t/t5319-multi-pack-index.sh               |   1 -
 t/t5324-split-commit-graph.sh             |   1 -
 t/t5504-fetch-receive-strict.sh           |   1 -
 t/t5510-fetch.sh                          |  15 ++-
 t/t5530-upload-pack-error.sh              |   1 -
 t/t5562-http-backend-content-length.sh    |   1 -
 t/t5607-clone-bundle.sh                   |  31 +++++
 t/t5702-protocol-v2.sh                    |   3 +-
 t/t5703-upload-pack-ref-in-want.sh        |   1 -
 t/t6006-rev-list-format.sh                |   1 -
 t/t6100-rev-list-in-order.sh              |   4 +-
 t/t6101-rev-parse-parents.sh              |   2 +-
 t/t6301-for-each-ref-errors.sh            |   2 +-
 t/t6500-gc.sh                             |  27 +++-
 t/t6501-freshen-objects.sh                |  14 +-
 t/t7003-filter-branch.sh                  |   3 +-
 t/t7063-status-untracked-cache.sh         | 151 ++++++++++++----------
 t/t7102-reset.sh                          |  93 +++++++------
 t/t7201-co.sh                             |   5 +-
 t/t7400-submodule-basic.sh                |  26 ++--
 t/t7405-submodule-merge.sh                |   4 +-
 t/t7506-status-submodule.sh               |  12 +-
 t/t7508-status.sh                         |   2 +-
 t/t8002-blame.sh                          |  15 ++-
 t/t8003-blame-corner-cases.sh             |   3 +-
 t/t8011-blame-split-file.sh               |   2 +-
 t/t9300-fast-import.sh                    | 117 +++++++++--------
 t/t9301-fast-import-notes.sh              |  13 +-
 t/t9350-fast-export.sh                    |  15 +--
 t/t9500-gitweb-standalone-no-errors.sh    |  22 +++-
 t/t9700/test.pl                           |   6 +-
 t/test-lib-functions.sh                   |  16 ++-
 t/test-lib.sh                             |   6 +-
 74 files changed, 629 insertions(+), 355 deletions(-)
 create mode 100644 Documentation/config/extensions.txt

Diff-intervalle contre v3 :
 1:  792a7e6b99 =  1:  2d720a6beb t: make test-bloom initialize repository
 2:  652259edaa =  2:  530704736a t1001: use $ZERO_OID
 3:  b719726393 =  3:  33e7822d0e t3305: make hash agnostic
 4:  137c86a4e0 =  4:  82a6827f45 t3404: prepare 'short SHA-1 collision' tests for SHA-256
 5:  307b90fa2a =  5:  23cbd25060 t6100: make hash size independent
 6:  f5f4028291 =  6:  9e2553b1f2 t6101: make hash size independent
 7:  654b95494c =  7:  c2c5dd407e t6301: make hash size independent
 8:  787bde1ffc =  8:  1ab4ed032a t6500: specify test values for SHA-256
 9:  b63ed599fe =  9:  54fcc025bb t6501: avoid hard-coded objects
10:  40847a1490 = 10:  a2f420fcd4 t7003: compute appropriate length constant
11:  3dbd65993c = 11:  766bba42f6 t7063: make hash size independent
12:  bba7969017 = 12:  66d242ccd9 t7201: abstract away SHA-1-specific constants
13:  bf5d06f1fc = 13:  090041ee54 t7102: abstract away SHA-1-specific constants
14:  a272eafa86 = 14:  4a6991d6a1 t7400: make hash size independent
15:  4dc14d1ded = 15:  d39bbeb5a2 t7405: make hash size independent
16:  f291df2f20 = 16:  5b0d52ae0e t7506: avoid checking for SHA-1-specific constants
17:  ded35216ce = 17:  54d1706107 t7508: use $ZERO_OID instead of hard-coded constant
18:  a197d7f09a = 18:  f3f4239ce9 t8002: make hash size independent
19:  7c49d9a8e8 = 19:  719db3822e t8003: make hash size independent
20:  4ab13f691a = 20:  803e5665e9 t8011: make hash size independent
21:  81edf64b72 = 21:  d1c4351bc9 t9300: abstract away SHA-1-specific constants
22:  d8253d9aca = 22:  db74bf3c8c t9300: use $ZERO_OID instead of hard-coded object ID
23:  7b40da5753 = 23:  fa7daed2bf t9301: make hash size independent
24:  1890f22a5b = 24:  84aef86ac0 t9350: make hash size independent
25:  708f599a71 = 25:  541108fc64 t9500: ensure that algorithm info is preserved in config
26:  33c84a2bc6 = 26:  2de5d2da58 t9700: make hash size independent
27:  3d7476f28b = 27:  c916058c38 t5308: make test work with SHA-256
28:  0d80e419e5 = 28:  b0781430ef t0410: mark test with SHA1 prerequisite
29:  bd812fb0e4 = 29:  230a933e87 http-fetch: set up git directory before parsing pack hashes
30:  667537e1aa ! 30:  d3dbe9e760 builtin/verify-pack: implement an --object-format option
    @@ Commit message
         Since we're not in a repository here, we need to provide git verify-pack
         help to set things up properly.  git index-pack already knows an
         --object-format option, so let's accept one as well and pass it down to
    -    our git index-pack invocation.  Since this argument is optional, let's
    -    dynamically determine the proper location to insert it into the array.
    -    Finally, let's make t5702 pass the proper argument on down to its git
    -    verify-pack caller.
    +    our git index-pack invocation.  Since we're now dynamically adjusting
    +    the elements in argv, let's switch to using struct argv_array to manage
    +    them.  Finally, let's make t5702 pass the proper argument on down to its
    +    git verify-pack caller.
     
         Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
     
    @@ builtin/verify-pack.c
      	struct child_process index_pack = CHILD_PROCESS_INIT;
     -	const char *argv[] = {"index-pack", NULL, NULL, NULL };
     -	struct strbuf arg = STRBUF_INIT;
    -+	const char *argv[] = {"index-pack", NULL, NULL, NULL, NULL };
    ++	struct argv_array argv = ARGV_ARRAY_INIT;
     +	struct strbuf arg = STRBUF_INIT, hash_arg = STRBUF_INIT;
      	int verbose = flags & VERIFY_PACK_VERBOSE;
      	int stat_only = flags & VERIFY_PACK_STAT_ONLY;
      	int err;
    -+	int argno = 1;
      
    ++	argv_array_push(&argv, "index-pack");
    ++
      	if (stat_only)
     -		argv[1] = "--verify-stat-only";
    -+		argv[argno++] = "--verify-stat-only";
    ++		argv_array_push(&argv, "--verify-stat-only");
      	else if (verbose)
     -		argv[1] = "--verify-stat";
    -+		argv[argno++] = "--verify-stat";
    ++		argv_array_push(&argv, "--verify-stat");
      	else
     -		argv[1] = "--verify";
    -+		argv[argno++] = "--verify";
    ++		argv_array_push(&argv, "--verify");
     +
     +	if (hash_algo) {
     +		strbuf_addf(&hash_arg, "--object-format=%s", hash_algo);
    -+		argv[argno++] = hash_arg.buf;
    ++		argv_array_push(&argv, hash_arg.buf);
     +	}
      
      	/*
    @@ builtin/verify-pack.c: static int verify_one_pack(const char *path, unsigned int
      	    !ends_with(arg.buf, ".pack"))
      		strbuf_addstr(&arg, ".pack");
     -	argv[2] = arg.buf;
    -+	argv[argno++] = arg.buf;
    ++	argv_array_push(&argv, arg.buf);
      
    - 	index_pack.argv = argv;
    +-	index_pack.argv = argv;
    ++	index_pack.argv = argv.argv;
      	index_pack.git_cmd = 1;
    + 
    + 	err = run_command(&index_pack);
    +@@ builtin/verify-pack.c: static int verify_one_pack(const char *path, unsigned int flags)
    + 		}
    + 	}
    + 	strbuf_release(&arg);
    ++	argv_array_clear(&argv);
    + 
    + 	return err;
    + }
     @@ builtin/verify-pack.c: int cmd_verify_pack(int argc, const char **argv, const char *prefix)
      {
      	int err = 0;
31:  acdf1d9009 ! 31:  e3805101ce bundle: add new version for use with SHA-256
    @@ Commit message
         for SHA-256.
     
         In t5510, always use format v3 so we can be sure we produce consistent
    -    results across hash algorithms.  Note here that head -n N lists the top
    -    N lines, whereas we want to check only the given line, so while we're at
    -    it, switch to use sed to make sure we're checking only the line we care
    -    about for a match.
    +    results across hash algorithms.  Since head -n N lists the top N lines
    +    instead of the Nth line, let's run our output through sed to normalize
    +    it and compare it against a fixed value, which will make sure we get
    +    exactly what we're expecting.
     
         Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
     
    @@ Documentation/technical/bundle-format.txt: reference    = obj-id SP refname LF
      == Semantics
      
     -A Git bundle consists of three parts.
    -+A Git bundle consists of four parts.
    ++A Git bundle consists of several parts.
     +
     +* "Capabilities", which are only in the v3 format, indicate functionality that
     +	the bundle requires to be read properly.
    @@ t/t5510-fetch.sh: test_expect_success 'create bundle 1' '
     -	head -n 2 "$D"/bundle1 | grep "^-$OID_REGEX " &&
     -	head -n 3 "$D"/bundle1 | grep "^$OID_REGEX " &&
     -	head -n 4 "$D"/bundle1 | grep "^$"
    -+	sed -n -e 1p "$D"/bundle1 | grep "^#" &&
    -+	sed -n -e 2p "$D"/bundle1 | grep "^@object-format=" &&
    -+	sed -n -e 3p "$D"/bundle1 | grep "^-$OID_REGEX " &&
    -+	sed -n -e 4p "$D"/bundle1 | grep "^$OID_REGEX " &&
    -+	sed -n -e 5p "$D"/bundle1 | grep "^$"
    ++	cat >expect <<-EOF &&
    ++	# v3 git bundle
    ++	@object-format=$(test_oid algo)
    ++	-OID message
    ++	OID message
    ++
    ++	EOF
    ++	sed -e "s/$OID_REGEX .*/OID message/g" -e "5q" "$D"/bundle1 >actual &&
    ++	test_cmp expect actual
      '
      
      test_expect_success 'create bundle 2' '
    @@ t/t5607-clone-bundle.sh: test_expect_success 'fetch SHA-1 from bundle' '
     +	# v3 git bundle
     +	@object-format=$(test_oid algo)
     +	EOF
    -+	test_cmp actual expect &&
    ++	test_cmp expect actual &&
     +	git bundle verify bundle
     +'
     +
    -+test_expect_success 'git bundle v3 rejects unknown extensions' '
    -+	head -n2 bundle >new &&
    -+	echo "@unknown=silly" >>new &&
    -+	sed "1,2d" bundle >>new &&
    ++test_expect_success 'git bundle v3 rejects unknown capabilities' '
    ++	cat >new <<-EOF &&
    ++	# v3 git bundle
    ++	@object-format=$(test_oid algo)
    ++	@unknown=silly
    ++	EOF
     +	test_must_fail git bundle verify new 2>output &&
     +	test_i18ngrep "unknown capability .unknown=silly." output
     +'
32:  65310eaf1b = 32:  c941fcf63c setup: add support for reading extensions.objectformat
33:  f69cf8b1b5 = 33:  7c6264ac79 Enable SHA-256 support by default
34:  e41d139b78 ! 34:  2719ab94e6 t: add test_oid option to select hash algorithm
    @@ Commit message
     
         In order to do so, we'll need to read both the source algorithm and the
         current algorithm, so add an optional flag to the test_oid helper that
    -    lets us read look up a value for a specified hash algorithm. This should
    +    lets us look up a value for a specified hash algorithm. This should
         not cause any conflicts with existing tests, since key arguments to
         test_oid are allowed to contains only shell identifier characters.
     
    @@ t/t0000-basic.sh: test_expect_success 'test_oid can look up data for SHA-256' '
      	test "$hexsz" -eq 64
      '
      
    -+test_expect_success 'test_oid can look up data a specified algorithm' '
    ++test_expect_success 'test_oid can look up data for a specified algorithm' '
     +	rawsz="$(test_oid --hash=sha1 rawsz)" &&
     +	hexsz="$(test_oid --hash=sha1 hexsz)" &&
     +	test "$rawsz" -eq 20 &&
    @@ t/test-lib-functions.sh: test_oid_cache () {
     +	local algo="${test_hash_algo}" &&
     +
     +	case "$1" in
    -+		--hash=*)
    -+			algo="${1#--hash=}" &&
    -+			shift;;
    -+		*)
    -+			;;
    ++	--hash=*)
    ++		algo="${1#--hash=}" &&
    ++		shift;;
    ++	*)
    ++		;;
     +	esac &&
     +
     +	local var="test_oid_${algo}_$1" &&
35:  20dbc5ff66 = 35:  1c8e5041d1 t: allow testing different hash algorithms via environment
36:  4ee7f9eec8 = 36:  7213d59222 t: make SHA1 prerequisite depend on default hash
37:  0080a96f7b = 37:  caf6c026ba ci: run tests with SHA-256
38:  03e1203a08 = 38:  ec4e6ae166 docs: add documentation for extensions.objectFormat
39:  fbc015258f = 39:  3185c62cfc t: remove test_oid_init in tests

^ permalink raw reply	[flat|nested] 58+ messages in thread

end of thread, other threads:[~2020-07-26 23:57 UTC | newest]

Thread overview: 58+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-26 19:53 [PATCH v4 00/39] SHA-256, part 3/3 brian m. carlson
2020-07-26 19:53 ` [PATCH v4 01/39] t: make test-bloom initialize repository brian m. carlson
2020-07-26 19:53 ` [PATCH v4 02/39] t1001: use $ZERO_OID brian m. carlson
2020-07-26 19:53 ` [PATCH v4 03/39] t3305: make hash agnostic brian m. carlson
2020-07-26 19:53 ` [PATCH v4 04/39] t3404: prepare 'short SHA-1 collision' tests for SHA-256 brian m. carlson
2020-07-26 22:31   ` Eric Sunshine
2020-07-26 19:53 ` [PATCH v4 05/39] t6100: make hash size independent brian m. carlson
2020-07-26 19:53 ` [PATCH v4 06/39] t6101: " brian m. carlson
2020-07-26 19:53 ` [PATCH v4 07/39] t6301: " brian m. carlson
2020-07-26 19:53 ` [PATCH v4 08/39] t6500: specify test values for SHA-256 brian m. carlson
2020-07-26 19:53 ` [PATCH v4 09/39] t6501: avoid hard-coded objects brian m. carlson
2020-07-26 19:53 ` [PATCH v4 10/39] t7003: compute appropriate length constant brian m. carlson
2020-07-26 19:53 ` [PATCH v4 11/39] t7063: make hash size independent brian m. carlson
2020-07-26 22:40   ` Eric Sunshine
2020-07-26 19:53 ` [PATCH v4 12/39] t7201: abstract away SHA-1-specific constants brian m. carlson
2020-07-26 22:54   ` Eric Sunshine
2020-07-26 19:53 ` [PATCH v4 13/39] t7102: " brian m. carlson
2020-07-26 19:53 ` [PATCH v4 14/39] t7400: make hash size independent brian m. carlson
2020-07-26 19:54 ` [PATCH v4 15/39] t7405: " brian m. carlson
2020-07-26 19:54 ` [PATCH v4 16/39] t7506: avoid checking for SHA-1-specific constants brian m. carlson
2020-07-26 19:54 ` [PATCH v4 17/39] t7508: use $ZERO_OID instead of hard-coded constant brian m. carlson
2020-07-26 19:54 ` [PATCH v4 18/39] t8002: make hash size independent brian m. carlson
2020-07-26 22:49   ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 19/39] t8003: " brian m. carlson
2020-07-26 19:54 ` [PATCH v4 20/39] t8011: " brian m. carlson
2020-07-26 23:00   ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 21/39] t9300: abstract away SHA-1-specific constants brian m. carlson
2020-07-26 23:14   ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 22/39] t9300: use $ZERO_OID instead of hard-coded object ID brian m. carlson
2020-07-26 19:54 ` [PATCH v4 23/39] t9301: make hash size independent brian m. carlson
2020-07-26 19:54 ` [PATCH v4 24/39] t9350: " brian m. carlson
2020-07-26 19:54 ` [PATCH v4 25/39] t9500: ensure that algorithm info is preserved in config brian m. carlson
2020-07-26 19:54 ` [PATCH v4 26/39] t9700: make hash size independent brian m. carlson
2020-07-26 19:54 ` [PATCH v4 27/39] t5308: make test work with SHA-256 brian m. carlson
2020-07-26 19:54 ` [PATCH v4 28/39] t0410: mark test with SHA1 prerequisite brian m. carlson
2020-07-26 19:54 ` [PATCH v4 29/39] http-fetch: set up git directory before parsing pack hashes brian m. carlson
2020-07-26 23:28   ` Eric Sunshine
2020-07-26 23:30     ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 30/39] builtin/verify-pack: implement an --object-format option brian m. carlson
2020-07-26 21:29   ` Eric Sunshine
2020-07-26 22:56     ` brian m. carlson
2020-07-26 19:54 ` [PATCH v4 31/39] bundle: add new version for use with SHA-256 brian m. carlson
2020-07-26 22:18   ` Eric Sunshine
2020-07-26 22:59     ` brian m. carlson
2020-07-26 19:54 ` [PATCH v4 32/39] setup: add support for reading extensions.objectformat brian m. carlson
2020-07-26 23:34   ` Eric Sunshine
2020-07-26 23:41     ` brian m. carlson
2020-07-26 19:54 ` [PATCH v4 33/39] Enable SHA-256 support by default brian m. carlson
2020-07-26 23:41   ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 34/39] t: add test_oid option to select hash algorithm brian m. carlson
2020-07-26 19:54 ` [PATCH v4 35/39] t: allow testing different hash algorithms via environment brian m. carlson
2020-07-26 19:54 ` [PATCH v4 36/39] t: make SHA1 prerequisite depend on default hash brian m. carlson
2020-07-26 23:52   ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 37/39] ci: run tests with SHA-256 brian m. carlson
2020-07-26 23:54   ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 38/39] docs: add documentation for extensions.objectFormat brian m. carlson
2020-07-26 23:56   ` Eric Sunshine
2020-07-26 19:54 ` [PATCH v4 39/39] t: remove test_oid_init in tests brian m. carlson

Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).