What's cooking in git.git (Jul 2019, #05; Fri, 19)

What's cooking in git.git (Jul 2019, #05; Fri, 19)

[Junio C Hamano]

Here are the topics that have been cooking.  Commits prefixed with
'-' are only in 'pu' (proposed updates) while commits prefixed with
'+' are in 'next'.  The ones marked with '.' do not appear in any of
the integration branches, but I am still holding onto them.

The sixth batch is in and the 'next' branch has thinned down quite a
bit.

You can find the changes described here in the integration branches
of the repositories listed at

    http://git-blame.blogspot.com/p/git-public-repositories.html

--------------------------------------------------
[Graduated to "master"]

* br/blame-ignore (2019-07-01) 11 commits
  (merged to 'next' on 2019-07-01 at f4b79421c9)
 + t8014: remove unnecessary braces
 + blame: drop some unused function parameters
  (merged to 'next' on 2019-06-27 at 36b91a787f)
 + blame: add a test to cover blame_coalesce()
 + blame: use the fingerprint heuristic to match ignored lines
 + blame: add a fingerprint heuristic to match ignored lines
 + blame: optionally track line fingerprints during fill_blame_origin()
 + blame: add config options for the output of ignored or unblamable lines
 + blame: add the ability to ignore commits and their changes
 + blame: use a helper function in blame_chunk()
 + Move oidset_parse_file() to oidset.c
 + fsck: rename and touch up init_skiplist()

 "git blame" learned to "ignore" commits in the history, whose
 effects (as well as their presence) get ignored.


* cc/test-oidmap (2019-07-01) 6 commits
  (merged to 'next' on 2019-07-01 at 331a414a24)
 + t0016: add 'remove' subcommand test
 + test-oidmap: remove 'add' subcommand
  (merged to 'next' on 2019-06-27 at 247a4341ca)
 + test-hashmap: remove 'hash' command
 + oidmap: use sha1hash() instead of static hash() function
 + t: add t0016-oidmap.sh
 + t/helper: add test-oidmap.c

 Extend the test coverage a bit.


* ds/commit-graph-incremental (2019-06-19) 18 commits
  (merged to 'next' on 2019-07-03 at 5dee5edbdf)
 + commit-graph: test verify across alternates
 + commit-graph: normalize commit-graph filenames
 + commit-graph: test --split across alternate without --split
 + commit-graph: test octopus merges with --split
 + commit-graph: clean up chains after flattened write
 + commit-graph: verify chains with --shallow mode
 + commit-graph: create options for split files
 + commit-graph: expire commit-graph files
 + commit-graph: allow cross-alternate chains
 + commit-graph: merge commit-graph chains
 + commit-graph: add --split option to builtin
 + commit-graph: write commit-graph chains
 + commit-graph: rearrange chunk count logic
 + commit-graph: add base graphs chunk
 + commit-graph: load commit-graph chains
 + commit-graph: rename commit_compare to oid_compare
 + commit-graph: prepare for commit-graph chains
 + commit-graph: document commit-graph chains

 The commits in a repository can be described by multiple
 commit-graph files now, which allows the commit-graph files to be
 updated incrementally.


* ds/midx-expire-repack (2019-07-01) 12 commits
  (merged to 'next' on 2019-07-01 at c2a907f395)
 + t5319: use 'test-tool path-utils' instead of 'ls -l'
  (merged to 'next' on 2019-06-27 at 35e6e3b38d)
 + t5319-multi-pack-index.sh: test batch size zero
 + midx: add test that 'expire' respects .keep files
 + multi-pack-index: test expire while adding packs
 + midx: implement midx_repack()
 + multi-pack-index: prepare 'repack' subcommand
 + multi-pack-index: implement 'expire' subcommand
 + midx: refactor permutation logic and pack sorting
 + midx: simplify computation of pack name lengths
 + multi-pack-index: prepare for 'expire' subcommand
 + Docs: rearrange subcommands for multi-pack-index
 + repack: refactor pack deletion for future use

 "git multi-pack-index" learned expire and repack subcommands.


* ea/merge-code-cleanup (2019-07-09) 1 commit
  (merged to 'next' on 2019-07-11 at a7b831b03e)
 + builtin/merge.c - cleanup of code in for-cycle that tests strategies

 A loop has been rewritten for conciseness and clarity.


* ew/repack-with-bitmaps-by-default (2019-07-01) 1 commit
  (merged to 'next' on 2019-07-09 at 985419cb6e)
 + repack: disable bitmaps-by-default if .keep files exist

 Generation of pack bitmaps are now disabled when .keep files exist,
 as these are mutually exclusive features.


* jk/check-connected-with-alternates (2019-07-01) 2 commits
  (merged to 'next' on 2019-07-09 at b5888bd955)
 + check_everything_connected: assume alternate ref tips are valid
 + object-store.h: move for_each_alternate_ref() from transport.h

 The tips of refs from the alternate object store can be used as
 starting point for reachability computation now.


* jl/status-reduce-vertical-blank (2019-06-21) 1 commit
  (merged to 'next' on 2019-07-11 at 5716f37348)
 + status: remove the empty line after hints

 Extra blank lines in "git status" output have been reduced.


* js/trace2-signo-typofix (2019-07-09) 1 commit
  (merged to 'next' on 2019-07-11 at b2d2029b90)
 + trace2: correct trace2 field name documentation

 Documentation fix.


* kb/mingw-set-home (2019-07-08) 1 commit
  (merged to 'next' on 2019-07-11 at 51c7e23dab)
 + mingw: initialize HOME on startup

 Windows port update.


* ma/ref-filter-leakfix (2019-07-10) 1 commit
  (merged to 'next' on 2019-07-11 at a7a4bb81ac)
 + ref-filter: fix memory leak in `free_array_item()`

 Leakfix.


* nd/tree-walk-with-repo (2019-06-28) 7 commits
  (merged to 'next' on 2019-07-09 at 3e613f977c)
 + t7814: do not generate same commits in different repos
 + Use the right 'struct repository' instead of the_repository
 + match-trees.c: remove the_repo from shift_tree*()
 + tree-walk.c: remove the_repo from get_tree_entry_follow_symlinks()
 + tree-walk.c: remove the_repo from get_tree_entry()
 + tree-walk.c: remove the_repo from fill_tree_descriptor()
 + sha1-file.c: remove the_repo from read_object_with_reference()

 The tree-walk API learned to pass an in-core repository
 instance throughout more codepaths.


* pw/prompt-cherry-pick-revert-fix (2019-07-01) 1 commit
  (merged to 'next' on 2019-07-09 at bf8f0af685)
 + git-prompt: improve cherry-pick/revert detection

 When one step in multi step cherry-pick or revert is reset or
 committed, the command line prompt script failed to notice the
 current status, which has been improved.


* pw/rebase-progress-test-cleanup (2019-07-08) 1 commit
  (merged to 'next' on 2019-07-11 at 1363de0fe2)
 + t3420: remove progress lines before comparing output

 Test cleanup.


* pw/status-with-corrupt-sequencer-state (2019-06-27) 3 commits
  (merged to 'next' on 2019-07-03 at 273aee6b3c)
 + status: do not report errors in sequencer/todo
 + sequencer: factor out todo command name parsing
 + sequencer: always allow tab after command name

 The code to read state files used by the sequencer machinery for
 "git status" has been made more robust against a corrupt or stale
 state files.


* ra/cherry-pick-revert-skip (2019-07-02) 5 commits
  (merged to 'next' on 2019-07-09 at 4ffa502220)
 + cherry-pick/revert: advise using --skip
 + cherry-pick/revert: add --skip option
 + sequencer: use argv_array in reset_merge
 + sequencer: rename reset_for_rollback to reset_merge
 + sequencer: add advice for revert

 "git cherry-pick/revert" learned a new "--skip" action.


* tb/ref-filter-multiple-patterns (2019-06-27) 1 commit
  (merged to 'next' on 2019-07-03 at f59ad42c4a)
 + ref-filter.c: find disjoint pattern prefixes

 "git for-each-ref" with multiple patterns have been optimized.

--------------------------------------------------
[Stalled]

* nd/index-dump-in-json (2019-06-26) 11 commits
 - SQUASH???
 - t3008: use the new SINGLE_CPU prereq
 - read-cache.c: dump "IEOT" extension as json
 - read-cache.c: dump "EOIE" extension as json
 - resolve-undo.c: dump "REUC" extension as json
 - fsmonitor.c: dump "FSMN" extension as json
 - split-index.c: dump "link" extension as json
 - dir.c: dump "UNTR" extension as json
 - cache-tree.c: dump "TREE" extension as json
 - read-cache.c: dump common extension info in json
 - ls-files: add --json to dump the index

 "ls-files" learned "--debug-json" option to dump the contents and
 the extensions of the index file.

 At least the fixup at the tip needs to be squashed into the right
 commit.  Also the new test seems flaky.


* jn/unknown-index-extensions (2018-11-21) 2 commits
 - index: offer advice for unknown index extensions
 - index: do not warn about unrecognized extensions

 A bit too alarming warning given when unknown index extensions
 exist is getting revamped.

 Expecting a reroll.


* jc/format-patch-delay-message-id (2019-04-05) 1 commit
 - format-patch: move message-id and related headers to the end

 The location "git format-patch --thread" adds the Message-Id:
 header in the series of header fields has been moved down, which
 may help working around a suspected bug in GMail MSA, reported at
 <CAHk-=whP1stFZNAaJiMi5eZ9rj0MRt20Y_yHVczZPH+O01d+sA@mail.gmail.com>

 Waiting for feedback to see if it truly helps.
 Needs tests.


* jt/fetch-cdn-offload (2019-03-12) 9 commits
 - SQUASH???
 - upload-pack: send part of packfile response as uri
 - fetch-pack: support more than one pack lockfile
 - upload-pack: refactor reading of pack-objects out
 - Documentation: add Packfile URIs design doc
 - Documentation: order protocol v2 sections
 - http-fetch: support fetching packfiles by URL
 - http: improve documentation of http_pack_request
 - http: use --stdin when getting dumb HTTP pack

 WIP for allowing a response to "git fetch" to instruct the bulk of
 the pack contents to be instead taken from elsewhere (aka CDN).

 Stalled


* js/protocol-advertise-multi (2018-12-28) 1 commit
 - protocol: advertise multiple supported versions

 The transport layer has been updated so that the protocol version
 used can be negotiated between the parties, by the initiator
 listing the protocol versions it is willing to talk, and the other
 side choosing from one of them.

 Expecting a reroll.
 cf. <CANq=j3u-zdb_FvNJGPCmygNMScseav63GhVvBX3NcVS4f7TejA@mail.gmail.com>


* mk/use-size-t-in-zlib (2018-10-15) 1 commit
 - zlib.c: use size_t for size

 The wrapper to call into zlib followed our long tradition to use
 "unsigned long" for sizes of regions in memory, which have been
 updated to use "size_t".


* dl/remote-save-to-push (2018-12-11) 1 commit
 - remote: add --save-to-push option to git remote set-url

 "git remote set-url" learned a new option that moves existing value
 of the URL field to pushURL field of the remote before replacing
 the URL field with a new value.

 Anybody who wants to champion this topic?
 I am personally not yet quite convinced if this is worth pursuing.

--------------------------------------------------
[Cooking]

* ac/log-use-mailmap-by-default-transition (2019-07-15) 3 commits
  (merged to 'next' on 2019-07-19 at e5669de950)
 + tests: defang pager tests by explicitly disabling the log.mailmap warning
 + documentation: mention --no-use-mailmap and log.mailmap false setting
 + log: add warning for unspecified log.mailmap setting

 The "git log" command learns to issue a warning when log.mailmap
 configuration is not set and --[no-]mailmap option is not used, to
 prepare users for future versions of Git that uses the mailmap by
 default.

 Will merge to 'master'.


* vn/xmmap-gently (2019-07-14) 1 commit
  (merged to 'next' on 2019-07-19 at d95c1d2be3)
 + read-cache.c: do not die if mmap fails

 Clean-up an error codepath.

 Will merge to 'master'.


* jc/post-c89-rules-doc (2019-07-18) 1 commit
  (merged to 'next' on 2019-07-19 at 8acd58e189)
 + CodingGuidelines: spell out post-C89 rules

 We have been trying out a few language features outside c89; the
 coding guidelines document did not talk about them and instead had
 a blanket ban against them.

 Will merge to 'master'.


* js/mingw-spawn-with-spaces-in-path (2019-07-16) 1 commit
  (merged to 'next' on 2019-07-19 at 33dd6d0401)
 + mingw: support spawning programs containing spaces in their names

 Window 7 update ;-)

 Will merge to 'master'.


* sr/gpg-interface-stop-at-the-end (2019-07-16) 1 commit
  (merged to 'next' on 2019-07-19 at 5d38aa1236)
 + gpg-interface: do not scan past the end of buffer

 A codepath that reads from GPG for signed object verification read
 past the end of allocated buffer, which has been fixed.

 Will merge to 'master'.


* tg/stash-keep-index-with-removed-paths (2019-07-16) 1 commit
  (merged to 'next' on 2019-07-19 at d4ae24a939)
 + stash: fix handling removed files with --keep-index

 "git stash --keep-index" did not work correctly on paths that have
 been removed, which has been fixed.

 Will merge to 'master'.


* jc/denoise-rm-to-resolve (2019-07-18) 1 commit
  (merged to 'next' on 2019-07-19 at 12f7e5d413)
 + rm: resolving by removal is not a warning-worthy event

 "git rm" to resolve a conflicted path leaked an internal message
 "needs merge" before actually removing the path, which was
 confusing.  This has been corrected.

 Will merge to 'master'.


* js/clean-report-too-long-a-path (2019-07-19) 1 commit
  (merged to 'next' on 2019-07-19 at b7da0a821c)
 + clean: show an error message when the path is too long

 "git clean" silently skipped a path when it cannot lstat() it; now
 it gives a warning.

 Will merge to 'master'.


* di/readme-markup-fix (2019-07-18) 1 commit
  (merged to 'next' on 2019-07-19 at 339470d824)
 + README: fix rendering of text in angle brackets

 Docfix.

 Will merge to 'master'.


* js/builtin-add-i (2019-07-18) 11 commits
 - built-in add -i: implement the `help` command
 - built-in add -i: use color in the main loop
 - built-in add -i: support `?` (prompt help)
 - built-in add -i: show unique prefixes of the commands
 - Add a function to determine unique prefixes for a list of strings
 - built-in add -i: implement the main loop
 - built-in add -i: color the header in the `status` command
 - built-in add -i: refresh the index before running `status`
 - built-in add -i: implement the `status` command
 - diff: export diffstat interface
 - Start to implement a built-in version of `git add --interactive`

 The beginning of rewriting "git add -i" in C.


* js/visual-studio (2019-07-18) 24 commits
 - git: avoid calling aliased builtins via their dashed form
 - t5505,t5516: create .git/branches/ when needed
 - bin-wrappers: append `.exe` to target paths if necessary
 - .gitignore: ignore Visual Studio's temporary/generated files
 - .gitignore: touch up the entries regarding Visual Studio
 - vcxproj: also link-or-copy builtins
 - msvc: add a Makefile target to pre-generate the Visual Studio solution
 - contrib/buildsystems: add a backend for modern Visual Studio versions
 - contrib/buildsystems: handle options starting with a slash
 - contrib/buildsystems: also handle -lexpat
 - contrib/buildsystems: handle libiconv, too
 - contrib/buildsystems: handle the curl library option
 - contrib/buildsystems: error out on unknown option
 - contrib/buildsystems: optionally capture the dry-run in a file
 - contrib/buildsystems: redirect errors of the dry run into a log file
 - contrib/buildsystems: ignore gettext stuff
 - contrib/buildsystems: handle quoted spaces in filenames
 - contrib/buildsystems: fix misleading error message
 - contrib/buildsystems: ignore irrelevant files in Generators/
 - contrib/buildsystems: ignore invalidcontinue.obj
 - Vcproj.pm: urlencode '<' and '>' when generating VC projects
 - Vcproj.pm: do not configure VCWebServiceProxyGeneratorTool
 - Vcproj.pm: list git.exe first to be startup project
 - Vcproj.pm: auto-generate GUIDs

 Support building Git with Visual Studio


* js/unmap-before-ext-diff (2019-07-11) 1 commit
  (merged to 'next' on 2019-07-15 at 7aa292c66c)
 + diff: munmap() file contents before running external diff

 Windows update.

 Will merge to 'master'.


* rm/gpg-program-doc-fix (2019-07-12) 1 commit
  (merged to 'next' on 2019-07-15 at ef358ec2e9)
 + gpg(docs): use correct --verify syntax

 Docfix.

 Will merge to 'master'.


* mt/dir-iterator-updates (2019-07-11) 10 commits
  (merged to 'next' on 2019-07-19 at 2ebb586ce6)
 + clone: replace strcmp by fspathcmp
 + clone: use dir-iterator to avoid explicit dir traversal
 + clone: extract function from copy_or_link_directory
 + clone: copy hidden paths at local clone
 + dir-iterator: add flags parameter to dir_iterator_begin
 + dir-iterator: refactor state machine model
 + dir-iterator: use warning_errno when possible
 + dir-iterator: add tests for dir-iterator API
 + clone: better handle symlinked files at .git/objects/
 + clone: test for our behavior on odd objects/* content

 Adjust the dir-iterator API and apply it to the local clone
 optimization codepath.

 Will merge to 'master'.


* tg/range-diff-output-update (2019-07-11) 14 commits
  (merged to 'next' on 2019-07-15 at b847d206ed)
 + range-diff: add headers to the outer hunk header
 + range-diff: add filename to inner diff
 + range-diff: add section header instead of diff header
 + range-diff: suppress line count in outer diff
 + range-diff: don't remove funcname from inner diff
 + range-diff: split lines manually
 + range-diff: fix function parameter indentation
 + apply: make parse_git_diff_header public
 + apply: only pass required data to gitdiff_* functions
 + apply: only pass required data to find_name_*
 + apply: only pass required data to check_header_line
 + apply: only pass required data to git_header_name
 + apply: only pass required data to skip_tree_prefix
 + apply: replace marc.info link with public-inbox

 "git range-diff" output has been tweaked for easier identification
 of which part of what file the patch shown is about.

 Will merge to 'master'.


* bc/hash-independent-tests-part-4 (2019-07-01) 10 commits
 - t2203: avoid hard-coded object ID values
 - t1710: make hash independent
 - t1007: remove SHA1 prerequisites
 - t0090: make test pass with SHA-256
 - t0027: make hash size independent
 - t6030: make test work with SHA-256
 - t5000: make hash independent
 - t1450: make hash size independent
 - t1410: make hash size independent
 - t: add helper to convert object IDs to paths

 Update to the tests to help SHA-256 transition continues.

 Ready?  After cursory scan of this series I didn't find anything
 glaringly wrong.  Will merge to 'next' until I hear otherwise
 within a few days.


* es/local-atomic-push-failure-with-http (2019-07-16) 2 commits
  (merged to 'next' on 2019-07-19 at 8d5b776a96)
 + transport-helper: avoid var decl in for () loop control
  (merged to 'next' on 2019-07-15 at 960e92d24f)
 + transport-helper: enforce atomic in push_refs_with_push

 "git push --atomic" that goes over the transport-helper (namely,
 the smart http transport) failed to prevent refs to be pushed when
 it can locally tell that one of the ref update will fail without
 having to consult the other end, which has been corrected.

 Will merge to 'master'.


* es/walken-tutorial (2019-07-02) 1 commit
 - documentation: add tutorial for revision walking

 Yet another revision walker tutorial.


* jk/test-commit-bulk (2019-07-02) 7 commits
 - SQUASH???
 - t6200: use test_commit_bulk
 - t5703: use test_commit_bulk
 - t5702: use test_commit_bulk
 - t3311: use test_commit_bulk
 - t5310: increase the number of bitmapped commits
 - test-lib: introduce test_commit_bulk

 A test helper has been introduced to optimize preparation of test
 repositories with many simple commits, and a handful of test
 scripts have been updated to use it.

 Needs a bit more polishing.
 I think I spotted one unused command, which we may want to remove, though.


* ab/test-env (2019-07-11) 9 commits
  (merged to 'next' on 2019-07-15 at 42e86beb20)
 + env--helper: mark a file-local symbol as static
  (merged to 'next' on 2019-07-09 at 096658f382)
 + tests: make GIT_TEST_FAIL_PREREQS a boolean
 + tests: replace test_tristate with "git env--helper"
 + tests README: re-flow a previously changed paragraph
 + tests: make GIT_TEST_GETTEXT_POISON a boolean
 + t6040 test: stop using global "script" variable
 + config.c: refactor die_bad_number() to not call gettext() early
 + env--helper: new undocumented builtin wrapping git_env_*()
 + config tests: simplify include cycle test

 Many GIT_TEST_* environment variables control various aspects of
 how our tests are run, but a few followed "non-empty is true, empty
 or unset is false" while others followed the usual "there are a few
 ways to spell true, like yes, on, etc., and also ways to spell
 false, like no, off, etc." convention.

 Will merge to 'master'.


* ds/early-access (2019-07-01) 3 commits
 - repo-settings: pack.useSparse=true
 - repo-settings: use index.version=4 by default
 - repo-settings: create core.featureAdoptionRate setting

 A mechanism to enable newish configuration settings in bulk has
 been invented.

 The model may need to be rethought.  A linear scale with "early
 adoption" mindset may not match diverse use cases of the tool.


* ab/no-kwset (2019-07-01) 10 commits
  (merged to 'next' on 2019-07-15 at ed0479ce3d)
 + grep: use PCRE v2 for optimized fixed-string search
 + grep: remove the kwset optimization
 + grep: drop support for \0 in --fixed-strings <pattern>
 + grep: make the behavior for NUL-byte in patterns sane
 + grep tests: move binary pattern tests into their own file
 + grep tests: move "grep binary" alongside the rest
 + grep: inline the return value of a function call used only once
 + t4210: skip more command-line encoding tests on MinGW
 + grep: don't use PCRE2?_UTF8 with "log --encoding=<non-utf8>"
 + log tests: test regex backends in "--encode=<enc>" tests

 Retire use of kwset library, which is an optimization for looking
 for fixed strings, with use of pcre2 JIT.

 Will merge to 'master'.


* md/list-objects-filter-combo (2019-06-28) 10 commits
 - list-objects-filter-options: make parser void
 - list-objects-filter-options: clean up use of ALLOC_GROW
 - list-objects-filter-options: allow mult. --filter
 - strbuf: give URL-encoding API a char predicate fn
 - list-objects-filter-options: make filter_spec a string_list
 - list-objects-filter-options: move error check up
 - list-objects-filter: implement composite filters
 - list-objects-filter-options: always supply *errbuf
 - list-objects-filter: put omits set in filter struct
 - list-objects-filter: encapsulate filter components

 The list-objects-filter API (used to create a sparse/lazy clone)
 learned to take a combined filter specification.

 There is a bit of interaction with cc/multi-promisor topic, whose
 conflict resolution I have no confidence in X-<.  Extra sets of
 eyes are appreciated.


* cb/xdiff-no-system-includes-in-dot-c (2019-06-19) 1 commit
 - xdiff: avoid accidental redefinition of LFS feature in OpenIndiana

 Compilation fix.

 Will be rerolled together with patches from the
 jk/no-system-includes-in-dot-c topic.


* jk/no-system-includes-in-dot-c (2019-06-19) 2 commits
 - wt-status.h: drop stdio.h include
 - verify-tag: drop signal.h include

 Compilation fix.

 Will be rerolled with the above.


* cc/multi-promisor (2019-06-25) 15 commits
 - Move core_partial_clone_filter_default to promisor-remote.c
 - Move repository_format_partial_clone to promisor-remote.c
 - Remove fetch-object.{c,h} in favor of promisor-remote.{c,h}
 - remote: add promisor and partial clone config to the doc
 - partial-clone: add multiple remotes in the doc
 - t0410: test fetching from many promisor remotes
 - builtin/fetch: remove unique promisor remote limitation
 - promisor-remote: parse remote.*.partialclonefilter
 - Use promisor_remote_get_direct() and has_promisor_remote()
 - promisor-remote: use repository_format_partial_clone
 - promisor-remote: add promisor_remote_reinit()
 - promisor-remote: implement promisor_remote_get_direct()
 - Add initial support for many promisor remotes
 - fetch-object: make functions return an error code
 - t0410: remove pipes after git commands

 Teach the lazy clone machinery that there can be more than one
 promisor remote and consult them in order when downloading missing
 objects on demand.

 There is a bit of interaction with md/list-objects-filter-combo
 topic, whose conflict resolution I have no confidence in X-<.
 Extra sets of eyes are appreciated.


* jc/format-patch-noclobber (2019-02-22) 1 commit
 - format-patch: --no-clobber refrains from overwriting output files

 "git format-patch" used to overwrite an existing patch/cover-letter
 file.  A new "--no-clobber" option stops it.

 Undecided but inclined to discard.


* dl/rebase-i-keep-base (2019-04-25) 6 commits
 - rebase: teach rebase --keep-base
 - rebase: fast-forward --fork-point in more cases
 - rebase: fast-forward --onto in more cases
 - rebase: refactor can_fast_forward into goto tower
 - t3432: test rebase fast-forward behavior
 - t3431: add rebase --fork-point tests

 "git rebase --keep-base <upstream>" tries to find the original base
 of the topic being rebased and rebase on top of that same base,
 which is useful when running the "git rebase -i" (and its limited
 variant "git rebase -x").

 The command also has learned to fast-forward in more cases where it
 can instead of replaying to recreate identical commits.

 On hold.
 cf. <20190508001252.15752-1-avarab@gmail.com>
 cf. <xmqqa7fxionx.fsf@gitster-ct.c.googlers.com>

jk/test-commit-bulk, was: What's cooking in git.git (Jul 2019, #05; Fri, 19)

[Jeff King]

On Fri, Jul 19, 2019 at 01:27:38PM -0700, Junio C Hamano wrote:

> * jk/test-commit-bulk (2019-07-02) 7 commits
>  - SQUASH???
>  - t6200: use test_commit_bulk
>  - t5703: use test_commit_bulk
>  - t5702: use test_commit_bulk
>  - t3311: use test_commit_bulk
>  - t5310: increase the number of bitmapped commits
>  - test-lib: introduce test_commit_bulk
> 
>  A test helper has been introduced to optimize preparation of test
>  repositories with many simple commits, and a handful of test
>  scripts have been updated to use it.
> 
>  Needs a bit more polishing.
>  I think I spotted one unused command, which we may want to remove, though.

I think these comments are out-of-date. The unused command was fixed up
in v2 of the first patch, and then I did a little extra polishing in v3:

  https://public-inbox.org/git/20190702051649.GA16344@sigill.intra.peff.net/

Together with your SQUASH fixup, I think that covers all comments. For
convenience, here's v3 again as a whole series which can replace what
you have already (including the squash).

  [1/6]: test-lib: introduce test_commit_bulk
  [2/6]: t5310: increase the number of bitmapped commits
  [3/6]: t3311: use test_commit_bulk
  [4/6]: t5702: use test_commit_bulk
  [5/6]: t5703: use test_commit_bulk
  [6/6]: t6200: use test_commit_bulk

 t/t3311-notes-merge-fanout.sh      |  10 +--
 t/t5310-pack-bitmaps.sh            |  15 +---
 t/t5702-protocol-v2.sh             |  10 +--
 t/t5703-upload-pack-ref-in-want.sh |   4 +-
 t/t6200-fmt-merge-msg.sh           |   7 +-
 t/test-lib-functions.sh            | 123 +++++++++++++++++++++++++++++
 6 files changed, 136 insertions(+), 33 deletions(-)

-Peff

[PATCH v3 1/6] test-lib: introduce test_commit_bulk

[Jeff King]

Some tests need to create a string of commits. Doing this with
test_commit is very heavy-weight, as it needs at least one process per
commit (and in fact, uses several).

For bulk creation, we can do much better by using fast-import, but it's
often a pain to generate the input. Let's provide a helper to do so.

We'll use t5310 as a guinea pig, as it has three 10-commit loops. Here
are hyperfine results before and after:

  [before]
  Benchmark #1: ./t5310-pack-bitmaps.sh --root=/var/ram/git-tests
    Time (mean ± σ):      2.846 s ±  0.305 s    [User: 3.042 s, System: 0.919 s]
    Range (min … max):    2.250 s …  3.210 s    10 runs

  [after]
  Benchmark #1: ./t5310-pack-bitmaps.sh --root=/var/ram/git-tests
    Time (mean ± σ):      2.210 s ±  0.174 s    [User: 2.570 s, System: 0.604 s]
    Range (min … max):    1.999 s …  2.590 s    10 runs

So we're over 20% faster, while making the callers slightly shorter. We
added a lot more lines in test-lib-function.sh, of course, and the
helper is way more featureful than we need here. But my hope is that it
will be flexible enough to use in more places.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t5310-pack-bitmaps.sh |  15 +----
 t/test-lib-functions.sh | 123 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 126 insertions(+), 12 deletions(-)

diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index a26c8ba9a2..3aab7024ca 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -21,15 +21,9 @@ has_any () {
 }
 
 test_expect_success 'setup repo with moderate-sized history' '
-	for i in $(test_seq 1 10)
-	do
-		test_commit $i
-	done &&
+	test_commit_bulk --id=file 10 &&
 	git checkout -b other HEAD~5 &&
-	for i in $(test_seq 1 10)
-	do
-		test_commit side-$i
-	done &&
+	test_commit_bulk --id=side 10 &&
 	git checkout master &&
 	bitmaptip=$(git rev-parse master) &&
 	blob=$(echo tagged-blob | git hash-object -w --stdin) &&
@@ -106,10 +100,7 @@ test_expect_success 'clone from bitmapped repository' '
 '
 
 test_expect_success 'setup further non-bitmapped commits' '
-	for i in $(test_seq 1 10)
-	do
-		test_commit further-$i
-	done
+	test_commit_bulk --id=further 10
 '
 
 rev_list_tests 'partial bitmap'
diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh
index 7308f67922..c129cd3d69 100644
--- a/t/test-lib-functions.sh
+++ b/t/test-lib-functions.sh
@@ -233,6 +233,129 @@ test_merge () {
 	git tag "$1"
 }
 
+# Efficiently create <nr> commits, each with a unique number (from 1 to <nr>
+# by default) in the commit message.
+#
+# Usage: test_commit_bulk [options] <nr>
+#   -C <dir>:
+#	Run all git commands in directory <dir>
+#   --ref=<n>:
+#	ref on which to create commits (default: HEAD)
+#   --start=<n>:
+#	number commit messages from <n> (default: 1)
+#   --message=<msg>:
+#	use <msg> as the commit mesasge (default: "commit %s")
+#   --filename=<fn>:
+#	modify <fn> in each commit (default: %s.t)
+#   --contents=<string>:
+#	place <string> in each file (default: "content %s")
+#   --id=<string>:
+#	shorthand to use <string> and %s in message, filename, and contents
+#
+# The message, filename, and contents strings are evaluated by printf, with the
+# first "%s" replaced by the current commit number. So you can do:
+#
+#   test_commit_bulk --filename=file --contents="modification %s"
+#
+# to have every commit touch the same file, but with unique content.
+#
+test_commit_bulk () {
+	tmpfile=.bulk-commit.input
+	indir=.
+	ref=HEAD
+	n=1
+	message='commit %s'
+	filename='%s.t'
+	contents='content %s'
+	while test $# -gt 0
+	do
+		case "$1" in
+		-C)
+			indir=$2
+			shift
+			;;
+		--ref=*)
+			ref=${1#--*=}
+			;;
+		--start=*)
+			n=${1#--*=}
+			;;
+		--message=*)
+			message=${1#--*=}
+			;;
+		--filename=*)
+			filename=${1#--*=}
+			;;
+		--contents=*)
+			contents=${1#--*=}
+			;;
+		--id=*)
+			message="${1#--*=} %s"
+			filename="${1#--*=}-%s.t"
+			contents="${1#--*=} %s"
+			;;
+		-*)
+			BUG "invalid test_commit_bulk option: $1"
+			;;
+		*)
+			break
+			;;
+		esac
+		shift
+	done
+	total=$1
+
+	add_from=
+	if git -C "$indir" rev-parse --verify "$ref"
+	then
+		add_from=t
+	fi
+
+	while test "$total" -gt 0
+	do
+		test_tick &&
+		echo "commit $ref"
+		printf 'author %s <%s> %s\n' \
+			"$GIT_AUTHOR_NAME" \
+			"$GIT_AUTHOR_EMAIL" \
+			"$GIT_AUTHOR_DATE"
+		printf 'committer %s <%s> %s\n' \
+			"$GIT_COMMITTER_NAME" \
+			"$GIT_COMMITTER_EMAIL" \
+			"$GIT_COMMITTER_DATE"
+		echo "data <<EOF"
+		printf "$message\n" $n
+		echo "EOF"
+		if test -n "$add_from"
+		then
+			echo "from $ref^0"
+			add_from=
+		fi
+		printf "M 644 inline $filename\n" $n
+		echo "data <<EOF"
+		printf "$contents\n" $n
+		echo "EOF"
+		echo
+		n=$((n + 1))
+		total=$((total - 1))
+	done >"$tmpfile"
+
+	git -C "$indir" \
+	    -c fastimport.unpacklimit=0 \
+	    fast-import <"$tmpfile" || return 1
+
+	# This will be left in place on failure, which may aid debugging.
+	rm -f "$tmpfile"
+
+	# If we updated HEAD, then be nice and update the index and working
+	# tree, too.
+	if test "$ref" = "HEAD"
+	then
+		git -C "$indir" checkout -f HEAD || return 1
+	fi
+
+}
+
 # This function helps systems where core.filemode=false is set.
 # Use it instead of plain 'chmod +x' to set or unset the executable bit
 # of a file in the working directory and add it to the index.
-- 
2.22.0.993.gcc1030c86b

[PATCH v3 2/6] t5310: increase the number of bitmapped commits

[Jeff King]

The bitmap index we compute in t5310 has only 20 commits in it. This
gives poor coverage of bitmap_writer_select_commits(), which simply
writes a bitmap for everything when there are fewer than 100 commits.

Let's bump the number of commits in the test to cover the more complex
code paths (this does drop coverage of the individual lines of the
trivial path, but the complex path does everything it does and more).

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t5310-pack-bitmaps.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index 3aab7024ca..6640329ebf 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -21,7 +21,7 @@ has_any () {
 }
 
 test_expect_success 'setup repo with moderate-sized history' '
-	test_commit_bulk --id=file 10 &&
+	test_commit_bulk --id=file 100 &&
 	git checkout -b other HEAD~5 &&
 	test_commit_bulk --id=side 10 &&
 	git checkout master &&
-- 
2.22.0.993.gcc1030c86b

[PATCH v3 3/6] t3311: use test_commit_bulk

[Jeff King]

One of the tests in t3311 creates 300 commits by running "test_commit"
in a loop. This requires 900 processes. Instead, we can use
test_commit_bulk to do it with only four. This improves the runtime of
the script from:

  Benchmark #1: ./t3311-notes-merge-fanout.sh --root=/var/ram/git-tests
    Time (mean ± σ):      5.821 s ±  0.691 s    [User: 3.146 s, System: 2.782 s]
    Range (min … max):    4.783 s …  6.841 s    10 runs

to:

  Benchmark #1: ./t3311-notes-merge-fanout.sh --root=/var/ram/git-tests
    Time (mean ± σ):      1.743 s ±  0.116 s    [User: 1.144 s, System: 0.691 s]
    Range (min … max):    1.629 s …  1.994 s    10 runs

for an average speedup of over 70%.

Unfortunately we still have to run 300 instances of "git notes add",
since the point is to test the fanout that comes from adding notes one
by one.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t3311-notes-merge-fanout.sh | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/t/t3311-notes-merge-fanout.sh b/t/t3311-notes-merge-fanout.sh
index 93516ef67c..37151a3adc 100755
--- a/t/t3311-notes-merge-fanout.sh
+++ b/t/t3311-notes-merge-fanout.sh
@@ -114,12 +114,12 @@ cp expect_log_x expect_log_y
 test_expect_success 'Add a few hundred commits w/notes to trigger fanout (x -> y)' '
 	git update-ref refs/notes/y refs/notes/x &&
 	git config core.notesRef refs/notes/y &&
-	i=5 &&
-	while test $i -lt $num
+	test_commit_bulk --start=6 --id=commit $((num - 5)) &&
+	i=0 &&
+	while test $i -lt $((num - 5))
 	do
-		i=$(($i + 1)) &&
-		test_commit "commit$i" >/dev/null &&
-		git notes add -m "notes for commit$i" || return 1
+		git notes add -m "notes for commit$i" HEAD~$i || return 1
+		i=$((i + 1))
 	done &&
 	test "$(git rev-parse refs/notes/y)" != "$(git rev-parse refs/notes/x)" &&
 	# Expected number of commits and notes
-- 
2.22.0.993.gcc1030c86b

[PATCH v3 4/6] t5702: use test_commit_bulk

[Jeff King]

There are two loops that create 32 commits each using test_commit. Using
test_commit_bulk speeds this up from:

  Benchmark #1: ./t5702-protocol-v2.sh --root=/var/ram/git-tests
    Time (mean ± σ):      5.409 s ±  0.513 s    [User: 2.382 s, System: 2.466 s]
    Range (min … max):    4.633 s …  5.927 s    10 runs

to:

  Benchmark #1: ./t5702-protocol-v2.sh --root=/var/ram/git-tests
    Time (mean ± σ):      3.956 s ±  0.242 s    [User: 1.775 s, System: 1.627 s]
    Range (min … max):    3.449 s …  4.239 s    10 runs

for an average savings of over 25%.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t5702-protocol-v2.sh | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 5b33f625dd..011b81d4fc 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -499,10 +499,7 @@ test_expect_success 'upload-pack respects client shallows' '
 
 	# Add extra commits to the client so that the whole fetch takes more
 	# than 1 request (due to negotiation)
-	for i in $(test_seq 1 32)
-	do
-		test_commit -C client c$i
-	done &&
+	test_commit_bulk -C client --id=c 32 &&
 
 	git -C server checkout -b newbranch base &&
 	test_commit -C server client_wants &&
@@ -711,10 +708,7 @@ test_expect_success 'when server does not send "ready", expect FLUSH' '
 	# Create many commits to extend the negotiation phase across multiple
 	# requests, so that the server does not send "ready" in the first
 	# request.
-	for i in $(test_seq 1 32)
-	do
-		test_commit -C http_child c$i
-	done &&
+	test_commit_bulk -C http_child --id=c 32 &&
 
 	# After the acknowledgments section, pretend that a DELIM
 	# (0001) was sent instead of a FLUSH (0000).
-- 
2.22.0.993.gcc1030c86b

[PATCH v3 5/6] t5703: use test_commit_bulk

[Jeff King]

There are two loops that create 33 commits each using test_commit. Using
test_commit_bulk speeds this up from:

  Benchmark #1: ./t5703-upload-pack-ref-in-want.sh --root=/var/ram/git-tests
    Time (mean ± σ):      2.142 s ±  0.161 s    [User: 1.136 s, System: 0.974 s]
    Range (min … max):    1.903 s …  2.401 s    10 runs

to:

  Benchmark #1: ./t5703-upload-pack-ref-in-want.sh --root=/var/ram/git-tests
    Time (mean ± σ):      1.440 s ±  0.114 s    [User: 737.7 ms, System: 615.4 ms]
    Range (min … max):    1.230 s …  1.604 s    10 runs

for an average savings of almost 33%.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t5703-upload-pack-ref-in-want.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/t/t5703-upload-pack-ref-in-want.sh b/t/t5703-upload-pack-ref-in-want.sh
index 0951d1bbdc..de4b6106ef 100755
--- a/t/t5703-upload-pack-ref-in-want.sh
+++ b/t/t5703-upload-pack-ref-in-want.sh
@@ -176,7 +176,7 @@ test_expect_success 'setup repos for change-while-negotiating test' '
 		git clone "http://127.0.0.1:$LIB_HTTPD_PORT/smart/repo" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(test_seq 1 33); do test_commit s$i; done &&
+		test_commit_bulk --id=s 33 &&
 
 		# Add novel commits to upstream
 		git checkout master &&
@@ -287,7 +287,7 @@ test_expect_success 'setup repos for fetching with ref-in-want tests' '
 		git clone "file://$REPO" "$LOCAL_PRISTINE" &&
 		cd "$LOCAL_PRISTINE" &&
 		git checkout -b side &&
-		for i in $(test_seq 1 33); do test_commit s$i; done &&
+		test_commit_bulk --id=s 33 &&
 
 		# Add novel commits to upstream
 		git checkout master &&
-- 
2.22.0.993.gcc1030c86b

[PATCH v3 6/6] t6200: use test_commit_bulk

[Jeff King]

There's a loop that creates 30 commits using test_commit. Using
test_commit_bulk speeds this up from:

  Benchmark #1: ./t6200-fmt-merge-msg.sh --root=/var/ram/git-tests
    Time (mean ± σ):      1.926 s ±  0.240 s    [User: 1.055 s, System: 0.963 s]
    Range (min … max):    1.431 s …  2.166 s    10 runs

to:

  Benchmark #1: ./t6200-fmt-merge-msg.sh --root=/var/ram/git-tests
    Time (mean ± σ):      1.343 s ±  0.179 s    [User: 766.5 ms, System: 662.9 ms]
    Range (min … max):    1.032 s …  1.664 s    10 runs

for an average savings of over 30%.

Signed-off-by: Jeff King <peff@peff.net>
---
 t/t6200-fmt-merge-msg.sh | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)

diff --git a/t/t6200-fmt-merge-msg.sh b/t/t6200-fmt-merge-msg.sh
index 93f23cfa82..8a72b4c43a 100755
--- a/t/t6200-fmt-merge-msg.sh
+++ b/t/t6200-fmt-merge-msg.sh
@@ -66,12 +66,7 @@ test_expect_success setup '
 	git commit -a -m "Right #5" &&
 
 	git checkout -b long &&
-	i=0 &&
-	while test $i -lt 30
-	do
-		test_commit $i one &&
-		i=$(($i+1))
-	done &&
+	test_commit_bulk --start=0 --message=%s --filename=one 30 &&
 
 	git show-branch &&
 
-- 
2.22.0.993.gcc1030c86b

Re: jk/test-commit-bulk,

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>>  Needs a bit more polishing.
>>  I think I spotted one unused command, which we may want to remove, though.
>
> I think these comments are out-of-date. The unused command was fixed up
> in v2 of the first patch, and then I did a little extra polishing in v3:
>
>   https://public-inbox.org/git/20190702051649.GA16344@sigill.intra.peff.net/

Thanks, indeed.  I've been a bit under the weather and my responses
may be a bit slow (haven't caught up with my mailbox, either X-<).

Will replace.  Thanks for all the help.

>
> Together with your SQUASH fixup, I think that covers all comments. For
> convenience, here's v3 again as a whole series which can replace what
> you have already (including the squash).
>
>   [1/6]: test-lib: introduce test_commit_bulk
>   [2/6]: t5310: increase the number of bitmapped commits
>   [3/6]: t3311: use test_commit_bulk
>   [4/6]: t5702: use test_commit_bulk
>   [5/6]: t5703: use test_commit_bulk
>   [6/6]: t6200: use test_commit_bulk
>
>  t/t3311-notes-merge-fanout.sh      |  10 +--
>  t/t5310-pack-bitmaps.sh            |  15 +---
>  t/t5702-protocol-v2.sh             |  10 +--
>  t/t5703-upload-pack-ref-in-want.sh |   4 +-
>  t/t6200-fmt-merge-msg.sh           |   7 +-
>  t/test-lib-functions.sh            | 123 +++++++++++++++++++++++++++++
>  6 files changed, 136 insertions(+), 33 deletions(-)
>
> -Peff