What's cooking in git.git (Jun 2020, #01; Wed, 3)

What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Junio C Hamano]

Here are the topics that have been cooking.  Commits prefixed with
'-' are only in 'pu' (proposed updates) while commits prefixed with
'+' are in 'next'.  The ones marked with '.' do not appear in any of
the integration branches, but I am still holding onto them.

Git 2.27 has been tagged, and the first batch of topics (including
the "throw protocol v2 to the experimental group of features" thing)
have been merged to the 'master' branch.  I'm planning to rewind the
tip of 'next' in a not-so-distant future.

Seeing a handful of regression reports [*] immediately after a
feature release is made gives me a mixed feeling: people are eager
enough to help by reporting issues they encounter, but there are not
enough people who are eager enough to help by testing the tip of
'master' before the release.  Are there things we can do to help
them become early adopters so that they do not have to scramble
after the release?

    * sparse-checkout
    cf. <CAD3+_6CUX0RPr-dgfUnfGDNNfqu80SYCskioYnu=MS6aJv2dEQ@mail.gmail.com>

    * untracked with pathspec
    cf. <CAJmdR_pG74x0Zn43MSm7zXNcoitqjjOy+WnhyGBW+oFjVFLbRQ@mail.gmail.com>

    * shallow point adjustment
    cf. <20200603034213.GB253041@google.com>

You can find the changes described here in the integration branches
of the repositories listed at

    http://git-blame.blogspot.com/p/git-public-repositories.html

--------------------------------------------------
[Graduated to "master"]

* an/merge-single-strategy-optim (2020-05-19) 1 commit
  (merged to 'next' on 2020-05-20 at 8d5cacc8d1)
 + merge: optimization to skip evaluate_result for single strategy

 Code optimization for a common case.


* bk/p4-prepare-p4-only-fix (2020-05-12) 1 commit
  (merged to 'next' on 2020-05-24 at c1b4644b04)
 + git-p4.py: fix --prepare-p4-only error with multiple commits

 The "--prepare-p4-only" option is supposed to stop after replaying
 one changeset, but kept going (by mistake?)
 cf. <CAE5ih797YYxsR2H0TA65w9W-1jF4jQLayja_nGjQMGtc=PB6Jw@mail.gmail.com>


* cb/t5608-cleanup (2020-05-24) 1 commit
  (merged to 'next' on 2020-05-24 at 2bfa581890)
 + t5608: avoid say() and use "skip_all" instead for consistency

 Test fixup.


* en/fast-import-looser-date (2020-05-31) 1 commit
  (merged to 'next' on 2020-05-31 at 49de9c07c5)
 + fast-import: add new --date-format=raw-permissive format

 Some repositories in the wild have commits that record nonsense
 committer timezone (e.g. rails.git); "git fast-import" learned an
 option to pass these nonsense timestamps intact to allow recreating
 existing repositories as-is.


* jn/experimental-opts-into-proto-v2 (2020-05-21) 1 commit
  (merged to 'next' on 2020-05-24 at 53cd664dfe)
 + config: let feature.experimental imply protocol.version=2

 "feature.experimental" configuration variable is to let volunteers
 easily opt into a set of newer features, which use of the v2
 transport protocol is now a part of.


* jx/pkt-line-doc-count-fix (2020-05-21) 1 commit
  (merged to 'next' on 2020-05-24 at 7115240db4)
 + doc: fix wrong 4-byte length of pkt-line message

 Docfix.


* la/diff-relative-config (2020-05-24) 1 commit
  (merged to 'next' on 2020-05-26 at b4604e6abc)
 + diff: add config option relative

 The commands in the "diff" family learned to honor "diff.relative"
 configuration variable.


* lo/sparse-universal-zero-init (2020-05-24) 1 commit
  (merged to 'next' on 2020-05-24 at 1f4ea6b348)
 + sparse: allow '{ 0 }' to be used without warnings

 We've adopted a convention that any on-stack structure can be
 initialized to have zero values in all fields with "= { 0 }", even
 when the first field happens to be a pointer, but sparse complained
 that a null pointer should be spelled NULL for a long time.  Start
 using -Wno-universal-initializer option to squelch it.


* mt/zsh-completion-optim (2020-05-28) 1 commit
  (merged to 'next' on 2020-05-31 at f5d02a5498)
 + completion: use native ZSH array pattern matching

 Command line completion (incontrib/) update.


* rs/checkout-b-track-error (2020-05-24) 2 commits
  (merged to 'next' on 2020-05-26 at 9220e43203)
 + checkout: improve error messages for -b with extra argument
 + checkout: add tests for -b and --track

 The error message from "git checkout -b foo -t bar baz" was
 confusing.

--------------------------------------------------
[New Topics]

* jk/diff-memuse-optim-with-stat-unmatch (2020-06-02) 1 commit
 - diff: discard blob data from stat-unmatched pairs

 Reduce memory usage during "diff --quiet" in a worktree with too
 many stat-unmatched paths.

 Will merge to 'next'.


* js/reflog-anonymize-for-clone-and-fetch (2020-06-02) 1 commit
 - clone/fetch: anonymize URLs in the reflog

 The reflog entries for "git clone" and "git fetch" did not
 anonymize the URL they operated on.

 Will merge to 'next'.


* tb/t5318-cleanup (2020-06-02) 2 commits
 - t5318: test that '--stdin-commits' respects '--[no-]progress'
 - t5318: use 'test_must_be_empty'

 Code cleanup.

 Almost there.
 cf. <20200602180403.GA4791@szeder.dev>

--------------------------------------------------
[Stalled]

* dr/push-remoteref-fix (2020-04-23) 1 commit
 - remote.c: fix handling of %(push:remoteref)

 The "%(push:remoteref)" placeholder in the "--format=" argument of
 "git format-patch" (and friends) only showed what got explicitly
 configured, not what ref at the receiving end would be updated when
 "git push" was used, as it ignored the default behaviour (e.g. update
 the same ref as the source).

 Expecting a reroll.
 cf. <20200416152145.wp2zeibxmuyas6y6@feanor>


* dl/test-must-fail-fixes-5 (2020-05-21) 5 commits
 - lib-submodule-update: pass OVERWRITING_FAIL
 - SQUASH???  <20200521112545.GB581643@generichostname>
 - lib-submodule-update: prepend "git" to $command
 - lib-submodule-update: consolidate --recurse-submodules
 - lib-submodule-update: add space after function name

 The effort to avoid using test_must_fail on non-git command continues.

 Expecting a reroll.
 cf. <20200521182928.GA1308647@coredump.intra.peff.net>
 The last step is a bit too ugly to live; Peff seems to have better
 ideas.


* mr/bisect-in-c-2 (2020-04-23) 12 commits
 - bisect--helper: retire `--bisect-autostart` subcommand
 - bisect--helper: retire `--write-terms` subcommand
 - bisect--helper: retire `--check-expected-revs` subcommand
 - bisect--helper: reimplement `bisect_state` & `bisect_head` shell functions in C
 - bisect--helper: retire `--next-all` subcommand
 - bisect--helper: retire `--bisect-clean-state` subcommand
 - bisect--helper: finish porting `bisect_start()` to C
 - bisect--helper: reimplement `bisect_next` and `bisect_auto_next` shell functions in C
 - bisect--helper: reimplement `bisect_autostart` shell function in C
 - bisect--helper: introduce new `write_in_file()` function
 - bisect--helper: use '-res' in 'cmd_bisect__helper' return
 - bisect--helper: fix `cmd_*()` function switch default return

 Rewrite of the remainder of "git bisect" script in C continues.

 Expecting a response to reviews.
 cf. <nycvar.QRO.7.76.6.2005230007260.56@tvgsbejvaqbjf.bet>


* mk/use-size-t-in-zlib (2018-10-15) 1 commit
 - zlib.c: use size_t for size

 The wrapper to call into zlib followed our long tradition to use
 "unsigned long" for sizes of regions in memory, which have been
 updated to use "size_t".

--------------------------------------------------
[Cooking]

* cc/upload-pack-data-2 (2020-06-02) 14 commits
 - fixup! upload-pack: change multi_ack to an enum
 - upload-pack: move pack_objects_hook to upload_pack_data
 - upload-pack: move allow_sideband_all to upload_pack_data
 - upload-pack: move allow_ref_in_want to upload_pack_data
 - upload-pack: move allow_filter to upload_pack_data
 - upload-pack: move keepalive to upload_pack_data
 - upload-pack: pass upload_pack_data to upload_pack_config()
 - upload-pack: change multi_ack to an enum
 - upload-pack: move multi_ack to upload_pack_data
 - upload-pack: move filter_capability_requested to upload_pack_data
 - upload-pack: move use_sideband to upload_pack_data
 - upload-pack: move static vars to upload_pack_data
 - upload-pack: annotate upload_pack_data fields
 - upload-pack: actually use some upload_pack_data bitfields
 (this branch uses cc/upload-pack-data.)

 Further code clean-up.

 Almost there.


* js/checkout-p-new-file (2020-05-27) 1 commit
  (merged to 'next' on 2020-05-31 at 017c325bf6)
 + checkout -p: handle new files correctly

 "git checkout -p" did not handle a newly added path at all.

 Will merge to 'master'.


* xl/upgrade-repo-format (2020-05-29) 1 commit
 - fetch: allow adding a filter after initial clone

 Allow runtime upgrade of the repository format version, which needs
 to be done carefully.

 Almost there.


* jk/complete-git-switch (2020-05-28) 16 commits
 - completion: improve handling of --orphan option of switch/checkout
 - completion: improve handling of -c/-C and -b/-B in switch/checkout
 - completion: improve handling of --track in switch/checkout
 - completion: improve handling of --detach in checkout
 - completion: improve completion for git switch with no options
 - completion: improve handling of DWIM mode for switch/checkout
 - completion: perform DWIM logic directly in __git_complete_refs
 - completion: extract function __git_dwim_remote_heads
 - completion: replace overloaded track term for __git_complete_refs
 - completion: add tests showing subpar switch/checkout --orphan logic
 - completion: add tests showing subpar -c/C argument completion
 - completion: add tests showing subpar -c/-C startpoint completion
 - completion: add tests showing subpar switch/checkout --track logic
 - completion: add tests showing subar checkout --detach logic
 - completion: add tests showing subpar DWIM logic for switch/checkout
 - completion: add test showing subpar git switch completion

 The command line completion (in contrib/) learned to complete
 options that the "git switch" command takes.


* en/sparse-with-submodule-doc (2020-05-25) 1 commit
 - git-sparse-checkout: clarify interactions with submodules

 The effect of sparse checkout settings on submodules is documented.

 Needs review.


* bc/filter-process (2020-05-21) 2 commits
  (merged to 'next' on 2020-05-24 at 3d51328096)
 + t2060: add a test for switch with --orphan and --discard-changes
 + builtin/checkout: simplify metadata initialization

 Code simplification and test coverage enhancement.

 Will merge to 'master'.


* cb/bisect-helper-parser-fix (2020-05-24) 1 commit
  (merged to 'next' on 2020-05-24 at d30e10fa86)
 + bisect--helper: avoid segfault with bad syntax in `start --term-*`

 The code to parse "git bisect start" command line was lax in
 validating the arguments.

 Will merge to 'master'.


* rs/fsck-duplicate-names-in-trees (2020-05-21) 4 commits
  (merged to 'next' on 2020-05-24 at 6e1d6ba087)
 + fsck: detect more in-tree d/f conflicts
 + t1450: demonstrate undetected in-tree d/f conflict
 + t1450: increase test coverage of in-tree d/f detection
 + fsck: fix a typo in a comment

 The check in "git fsck" to ensure that the tree objects are sorted
 still had corner cases it missed unsorted entries.

 Will merge to 'master'.


* ss/submodule-set-branch-in-c (2020-06-02) 1 commit
 - submodule: port subcommand 'set-branch' from shell to C

 Rewrite of parts of the scripted "git submodule" Porcelain command
 continues; this time it is "git submodule set-branch" subcommand's
 turn.

 Almost there.
 cf. <1b851e49-3bb1-3b59-7f24-b903c5514391@gmail.com>


* vs/complete-stash-show-p-fix (2020-05-21) 1 commit
  (merged to 'next' on 2020-05-24 at fcbff29a0f)
 + completion: don't override given stash subcommand with -p

 The command line completion script (in contrib/) tried to complete
 "git stash -p" as if it were "git stash push -p", but it was too
 aggressive and also affected "git stash show -p", which has been
 corrected.

 Will merge to 'master'.


* es/config-hooks (2020-05-21) 4 commits
 . hook: add --porcelain to list command
 . hook: add list command
 . hook: scaffolding for git-hook subcommand
 . doc: propose hooks managed by the config


* pw/rebase-i-more-options (2020-05-27) 5 commits
 - rebase: add --reset-author-date
 - rebase -i: support --ignore-date
 - sequencer: rename amend_author to author_to_free
 - rebase -i: support --committer-date-is-author-date
 - rebase -i: add --ignore-whitespace flag

 "git rebase -i" learns a bit more options.

 Not quite
 cf. <nycvar.QRO.7.76.6.2005290437350.56@tvgsbejvaqbjf.bet>


* cc/upload-pack-data (2020-05-18) 13 commits
  (merged to 'next' on 2020-05-29 at 66008d9209)
 + upload-pack: use upload_pack_data fields in receive_needs()
 + upload-pack: pass upload_pack_data to create_pack_file()
 + upload-pack: remove static variable 'stateless_rpc'
 + upload-pack: pass upload_pack_data to check_non_tip()
 + upload-pack: pass upload_pack_data to send_ref()
 + upload-pack: move symref to upload_pack_data
 + upload-pack: use upload_pack_data writer in receive_needs()
 + upload-pack: pass upload_pack_data to receive_needs()
 + upload-pack: pass upload_pack_data to get_common_commits()
 + upload-pack: use 'struct upload_pack_data' in upload_pack()
 + upload-pack: move 'struct upload_pack_data' around
 + upload-pack: move {want,have}_obj to upload_pack_data
 + upload-pack: remove unused 'wants' from upload_pack_data
 (this branch is used by cc/upload-pack-data-2.)

 Code clean-up.

 Will merge to 'master'.


* dl/remote-curl-deadlock-fix (2020-05-24) 7 commits
  (merged to 'next' on 2020-05-26 at 072714ca7f)
 + stateless-connect: send response end packet
 + pkt-line: define PACKET_READ_RESPONSE_END
 + remote-curl: error on incomplete packet
 + pkt-line: extern packet_length()
 + transport: extract common fetch_pack() call
 + remote-curl: remove label indentation
 + remote-curl: fix typo

 On-the-wire protocol v2 easily falls into a deadlock between the
 remote-curl helper and the fetch-pack process when the server side
 prematurely throws an error and disconnects.  The communication has
 been updated to make it more robust.

 Will merge to 'master'.


* cb/t4210-illseq-auto-detect (2020-05-18) 2 commits
  (merged to 'next' on 2020-05-24 at c03b4095fa)
 + t4210: detect REG_ILLSEQ dynamically and skip affected tests
 + t/helper: teach test-regex to report pattern errors (like REG_ILLSEQ)

 As FreeBSD is not the only platform whose regexp library reports
 a REG_ILLSEQ error when fed invalid UTF-8, add logic to detect that
 automatically and skip the affected tests.

 Will merge to 'master'.


* jt/curl-verbose-on-trace-curl (2020-05-11) 2 commits
  (merged to 'next' on 2020-05-11 at 814e31b9d4)
 + http, imap-send: stop using CURLOPT_VERBOSE
 + t5551: test that GIT_TRACE_CURL redacts password

 Rewrite support for GIT_CURL_VERBOSE in terms of GIT_TRACE_CURL.

 Expecting further work on optionally disabling redacting authinfo


* mt/grep-sparse-checkout (2020-06-02) 6 commits
 - git.c: fix sparse warning
 - config: add setting to ignore sparsity patterns in some cmds
 - grep: honor sparse checkout patterns
 - config: correctly read worktree configs in submodules
 - t/helper/test-config: return exit codes consistently
 - doc: grep: unify info on configuration variables

 "git grep" has been tweaked to be limited to the sparse checkout
 paths.

 Expecting a reroll.
 cf. <cover.1590627264.git.matheus.bernardino@usp.br>
 cf. <CABPp-BFsCPPNOZ92JQRJeGyNd0e-TCW-LcLyr0i_+VSQJP+GCg@mail.gmail.com>


* bc/sha-256-part-2 (2020-05-27) 44 commits
 - remote-testgit: adapt for object-format
 - bundle: detect hash algorithm when reading refs
 - t5300: pass --object-format to git index-pack
 - t5704: send object-format capability with SHA-256
 - t5703: use object-format serve option
 - t5702: offer an object-format capability in the test
 - t/helper: initialize the repository for test-sha1-array
 - remote-curl: avoid truncating refs with ls-remote
 - t1050: pass algorithm to index-pack when outside repo
 - builtin/index-pack: add option to specify hash algorithm
 - remote-curl: detect algorithm for dumb HTTP by size
 - builtin/ls-remote: initialize repository based on fetch
 - t5500: make hash independent
 - serve: advertise object-format capability for protocol v2
 - connect: parse v2 refs with correct hash algorithm
 - connect: pass full packet reader when parsing v2 refs
 - Documentation/technical: document object-format for protocol v2
 - t1302: expect repo format version 1 for SHA-256
 - builtin/show-index: provide options to determine hash algo
 - t5302: modernize test formatting
 - packfile: compute and use the index CRC offset
 - t3200: mark assertion with SHA1 prerequisite
 - setup: set the_repository's hash algo when checking format
 - fetch-pack: parse and advertise the object-format capability
 - t5562: pass object-format in synthesized test data
 - builtin/clone: initialize hash algorithm properly
 - remote-curl: implement object-format extensions
 - transport-helper: implement object-format extensions
 - docs: update remote helper docs for object-format extensions
 - builtin/receive-pack: detect when the server doesn't support our hash
 - connect: detect algorithm when fetching refs
 - fetch-pack: detect when the server doesn't support our hash
 - connect: make parse_feature_value extern
 - send-pack: detect when the server doesn't support our hash
 - connect: add function to detect supported v1 hash functions
 - transport: add a hash algorithm member
 - pkt-line: add a member for hash algorithm
 - connect: add function to fetch value of a v2 server capability
 - connect: add function to parse multiple v1 capability values
 - remote: advertise the object-format capability on the server side
 - wrapper: add function to compare strings with different NUL termination
 - connect: have ref processing code take struct packet_reader
 - Documentation: document v1 protocol object-format capability
 - t1050: match object ID paths in a hash-insensitive way

 SHA-256 migration work continues.


* es/bugreport-shell (2020-05-12) 2 commits
  (merged to 'next' on 2020-05-24 at 79c5c8308b)
 + bugreport: include user interactive shell
 + help: add shell-path to --build-options

 "git bugreport" learns to report what shell is in use.

 Will merge to 'master'.
 We may want to learn more details than just the path, but
 that can come later.
 cf. <20200512235924.GC6605@camp.crustytoothpaste.net>


* ds/line-log-on-bloom (2020-05-11) 5 commits
  (merged to 'next' on 2020-05-11 at 046d49d455)
 + line-log: integrate with changed-path Bloom filters
 + line-log: try to use generation number-based topo-ordering
 + line-log: more responsive, incremental 'git log -L'
 + t4211-line-log: add tests for parent oids
 + line-log: remove unused fields from 'struct line_log_data'

 "git log -L..." now takes advantage of the "which paths are touched
 by this commit?" info stored in the commit-graph system.

 Will merge to 'master'.


* tb/commit-graph-no-check-oids (2020-05-18) 8 commits
  (merged to 'next' on 2020-05-24 at 72bd1063ca)
 + commit-graph: drop COMMIT_GRAPH_WRITE_CHECK_OIDS flag
 + t5318: reorder test below 'graph_read_expect'
 + commit-graph.c: simplify 'fill_oids_from_commits'
 + builtin/commit-graph.c: dereference tags in builtin
 + builtin/commit-graph.c: extract 'read_one_commit()'
 + commit-graph.c: peel refs in 'add_ref_to_set'
 + commit-graph.c: show progress of finding reachable commits
 + commit-graph.c: extract 'refs_cb_data'

 Clean-up the commit-graph codepath.

 Will merge to 'master'.


* jx/proc-receive-hook (2020-05-18) 11 commits
 - doc: add documentation for the proc-receive hook
 - transport: parse report options for tracking refs
 - t5411: test updates of remote-tracking branches
 - receive-pack: new config receive.procReceiveRefs
 - refs.c: refactor to reuse ref_is_hidden()
 - receive-pack: feed report options to post-receive
 - doc: add document for capability report-status-v2
 - New capability "report-status-v2" for git-push
 - receive-pack: add new proc-receive hook
 - t5411: add basic test cases for proc-receive hook
 - transport: not report a non-head push as a branch

 "git receive-pack" that accepts requests by "git push" learned to
 outsource most of the ref updates to the new "proc-receive" hook.

 Needs review.


* hn/refs-cleanup (2020-05-27) 7 commits
  (merged to 'next' on 2020-05-27 at aada2199e1)
 + reftable doc: use link: and urlencode to avoid dead links
  (merged to 'next' on 2020-05-24 at dd9b665698)
 + reftable: define version 2 of the spec to accomodate SHA256
 + reftable: clarify how empty tables should be written
 + reftable: file format documentation
 + refs: improve documentation for ref iterator
 + t: use update-ref and show-ref to reading/writing refs
 + refs.h: clarify reflog iteration order
 (this branch is used by hn/reftable.)

 Preliminary clean-ups around refs API, plus file format
 specification documentation for the reftable backend.

 Will merge to 'master' after squashing the fix in.
 We probably would want to squash the fix at the tip to an earlier
 step when we rewind the 'next' branch.


* hn/reftable (2020-05-28) 13 commits
 . Add reftable testing infrastructure
 . vcxproj: adjust for the reftable changes
 . Add GIT_DEBUG_REFS debugging mechanism
 . Reftable support for git-core
 . Add reftable library
 . Add .gitattributes for the reftable/ directory
 . Iterate over the "refs/" namespace in for_each_[raw]ref
 . Move REF_LOG_ONLY to refs-internal.h
 . Treat REVERT_HEAD as a pseudo ref
 . Treat CHERRY_PICK_HEAD as a pseudo ref
 . Treat BISECT_HEAD as a pseudo ref
 . Make refs_ref_exists public
 . Write pseudorefs through ref backends.
 (this branch uses hn/refs-cleanup.)

 A new refs backend "reftable" to replace the traditional
 combination of packed-refs files and one-file-per-ref loose refs
 has been implemented and integrated for improved performance and
 atomicity.

 Not quite--does not pass tests by itself.

--------------------------------------------------
[Discarded]

* jc/credential-store-file-format-doc (2020-04-27) 1 commit
 . credential-store: document the file format a bit more

 Now has become a part of Carlo's credential-store fix patches.


* js/ci-skip-on-github-workflow (2020-05-02) 1 commit
 . ci: respect the [skip ci] convention in our GitHub workflow "CI/PR"

 Allow contributors to mark a branch/push that it does not have to
 be built via GitHub actions, in a way similar to how Travis lets
 them mark the commits with an embedded "[skip ci]" string.

 Superseded by dd/ci-only-on-selective-branches topic.


* dd/ci-only-on-selective-branches (2020-05-05) 2 commits
 - CI: limit GitHub Actions to designated branches
 - SubmittingPatches: advertise GitHub Actions CI

 Instead of always building all branches of all forks of our project
 at GitHub via GitHub Actions, only build when branches with known
 and specific names are updated, and also a pull request.

 Superseded by jk/ci-only-on-selected-branches

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Elijah Newren]

Hi,

On Wed, Jun 3, 2020 at 2:01 PM Junio C Hamano <gitster@pobox.com> wrote:
> Seeing a handful of regression reports [*] immediately after a
> feature release is made gives me a mixed feeling: people are eager
> enough to help by reporting issues they encounter, but there are not
> enough people who are eager enough to help by testing the tip of
> 'master' before the release.  Are there things we can do to help
> them become early adopters so that they do not have to scramble
> after the release?

That's very diplomatically worded, but perhaps let me peel back that
deflection layer a bit and be more direct...

A disproportionate number of regressions that we've had in recent
releases have traced back to me.  2 of the 3 regressions from 2.27 do.
In the 2.26 release, we had a whole pile of regressions with rebase
due to the change in the default backend, which came from me.  And,
we've also had a bunch of "fun" with dir.c in _every_ _single_
_release_ (to the best of my memory anyway) since I got my foot caught
in that unrelenting trap[1], including 1 of the 3 reported regressions
in this latest release.  The regression reports after releases have
been weighing on me too; I was thinking about it a fair bit last week
as well as this week.

[1] https://kernel.googlesource.com/pub/scm/git/git/+/8d92fb292706fd8d13cfe55353b2ec9345153a3e


Now, it's possible these regressions could just be a reflection of the
fact that I'm focusing more on fixing inconsistent behaviors rather
than adding new features, which is a type of work where it's much
harder to avoid fallout and reported issues.  But it's also quite
possible that I'm going about these cleanups wrong or at least
suboptimally.  I'm open for suggestions of what I should change, or
even experiments to try.

Recent attempts I've made to make things better: (1) I have in the
past month or so gotten a company internal distribution of git
started, with a growing number of users.  This distribution uses
pre-release versions of git, mostly off master so far though I'm
considering moving to 'next' for it.  (2) I pushed hard during 2.27
for the dir.c changes to either merge early in that cycle or wait
until early in the 2.28 cycle -- hoping that an early merge would give
more time for testing.  (This was an attempt to learn from the 2.26
rebase issues, since that merged late in the 2.26 cycle).

Any other ideas I should try?


Thanks,
Elijah

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Christian Couder]

On Thu, Jun 4, 2020 at 11:20 AM Elijah Newren <newren@gmail.com> wrote:

> On Wed, Jun 3, 2020 at 2:01 PM Junio C Hamano <gitster@pobox.com> wrote:

> Now, it's possible these regressions could just be a reflection of the
> fact that I'm focusing more on fixing inconsistent behaviors

I agree with that. It's very difficult to implement big changes like
changing the rebase backend without any regression at all.

> rather
> than adding new features, which is a type of work where it's much
> harder to avoid fallout and reported issues.  But it's also quite
> possible that I'm going about these cleanups wrong or at least
> suboptimally.  I'm open for suggestions of what I should change, or
> even experiments to try.

It's possible to add more tests, but it's not possible to test
everything and it's hard to know it the tests we add are effective at
avoiding regressions. Not sure we can do much better.

> Recent attempts I've made to make things better: (1) I have in the
> past month or so gotten a company internal distribution of git
> started, with a growing number of users.  This distribution uses
> pre-release versions of git, mostly off master so far though I'm
> considering moving to 'next' for it.

Great! At GitLab we are slowly moving toward something like that, but
not there yet.

> (2) I pushed hard during 2.27
> for the dir.c changes to either merge early in that cycle or wait
> until early in the 2.28 cycle -- hoping that an early merge would give
> more time for testing.  (This was an attempt to learn from the 2.26
> rebase issues, since that merged late in the 2.26 cycle).

That's a good idea.

Now to go back to Junio's question:

> ... people are eager
> enough to help by reporting issues they encounter, but there are not
> enough people who are eager enough to help by testing the tip of
> 'master' before the release. Are there things we can do to help
> them become early adopters so that they do not have to scramble
> after the release?

Yeah, I agree that increasing the number of early adopters could be
the best way to avoid regressions report just after the release.

Maybe we could just ask people to test rc releases or 'master' in Git
Rev News? It might work better if someone wrote a small article about
what is coming in the next release before asking to test. Then there
is the issue of making it easier to build Git and to understand and
fix build issues. We could also perhaps coordinate Git Rev News
editions and Git releases, so that the editions are published for
example between rc-1 and rc-2  releases.

Best,
Christian.

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Christian Couder]

On Thu, Jun 4, 2020 at 4:32 PM Christian Couder
<christian.couder@gmail.com> wrote:
>
> Maybe we could just ask people to test rc releases or 'master' in Git
> Rev News? It might work better if someone wrote a small article about
> what is coming in the next release before asking to test.

Such articles and asking users to test more could also be done on
GitLab's, GitHub's, etc, corporate blogs or on Git developer's blogs
too. The content of such articles could also be shared between
different company blogs and Git Rev News to reach as many users as
possible.

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> On Wed, Jun 3, 2020 at 2:01 PM Junio C Hamano <gitster@pobox.com> wrote:
>> Seeing a handful of regression reports [*] immediately after a
>> feature release is made gives me a mixed feeling: people are eager
>> enough to help by reporting issues they encounter, but there are not
>> enough people who are eager enough to help by testing the tip of
>> 'master' before the release.  Are there things we can do to help
>> them become early adopters so that they do not have to scramble
>> after the release?
>
> That's very diplomatically worded, but perhaps let me peel back that
> deflection layer a bit and be more direct...

Sorry, but you are reading too much into it.

I am not worried about individual bugs; bugs happen and that is an
inevitable part of development.  I have enough confidence in our
developers to feel that we can promptly fix a regression once an
issue is raised and clarified sufficiently to be actionable.

> Now, it's possible these regressions could just be a reflection of the
> fact that I'm focusing more on fixing inconsistent behaviors rather
> than adding new features, which is a type of work where it's much
> harder to avoid fallout and reported issues.

Yes, and as I said already, bugs happen and they tend to happen to
those who touch the trickiest part of the existing code---it is
inevitable.  It is OK to be initially buggy as long as we can fix
breakages promptly.

I am however worried about how the issues often are not raised until
a regression hits a tagged version.  As a piece of software gets
used more and more widely, it would be used in more and more
different ways, and some of these ways may not be covered by
developers' minds; by definition, some regressions are noticed only
after a new version hits end users' hands.

Basing in-house releases a Git developer has control over on
'master' or 'next' would reach audiences that the Git developer has
access to, and as long as the in-house users use Git sufficiently
differently from the Git developer (is that a realistic expectation,
though?), we'd gain more coverage before a tagged release happens
that way.  Basing the version placed in a distro's 'testing' track
on 'master' or 'next' may give us even wider exposure but it would
be much harder to arrange, I am afraid.

People from time to time suggest to merge trickier topics early in
the cycle, and in principle I do agree that it may not be a bad
idea, but given that we do not have enough folks on 'master', I am
not so sure how much that would actially help.

Thanks.

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1906 bytes --]

On 2020-06-03 at 20:59:39, Junio C Hamano wrote:
> Here are the topics that have been cooking.  Commits prefixed with
> '-' are only in 'pu' (proposed updates) while commits prefixed with
> '+' are in 'next'.  The ones marked with '.' do not appear in any of
> the integration branches, but I am still holding onto them.
> 
> Git 2.27 has been tagged, and the first batch of topics (including
> the "throw protocol v2 to the experimental group of features" thing)
> have been merged to the 'master' branch.  I'm planning to rewind the
> tip of 'next' in a not-so-distant future.
> 
> Seeing a handful of regression reports [*] immediately after a
> feature release is made gives me a mixed feeling: people are eager
> enough to help by reporting issues they encounter, but there are not
> enough people who are eager enough to help by testing the tip of
> 'master' before the release.  Are there things we can do to help
> them become early adopters so that they do not have to scramble
> after the release?

For folks that are using Debian, Jonathan Nieder kindly keeps Debian
experimental generally within about a week or so of the latest next
(with the -rc merged into it during that period).  (As of today, the
version is from 2020-05-31.)  This is usually what I use and it's
reasonably stable, so folks who want to test things may find it
convenient to rely on existing packages.

I don't happen to use a lot of these features, so I don't notice any
issues with them, but I have seen issues in the past with other features
and reported them.  Maybe if there are companies with folks who are
using sparse-checkout and similar features it may be helpful to build
and deploy an -rc or two if they have some way of standard deployments
to developer machines (which I admit most places don't).
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Elijah Newren]

On Thu, Jun 4, 2020 at 7:49 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Elijah Newren <newren@gmail.com> writes:
>
> > On Wed, Jun 3, 2020 at 2:01 PM Junio C Hamano <gitster@pobox.com> wrote:
> >> Seeing a handful of regression reports [*] immediately after a
> >> feature release is made gives me a mixed feeling: people are eager
> >> enough to help by reporting issues they encounter, but there are not
> >> enough people who are eager enough to help by testing the tip of
> >> 'master' before the release.  Are there things we can do to help
> >> them become early adopters so that they do not have to scramble
> >> after the release?
> >
> > That's very diplomatically worded, but perhaps let me peel back that
> > deflection layer a bit and be more direct...
>
> Sorry, but you are reading too much into it.
>
> I am not worried about individual bugs; bugs happen and that is an
> inevitable part of development.  I have enough confidence in our
> developers to feel that we can promptly fix a regression once an
> issue is raised and clarified sufficiently to be actionable.
>
> > Now, it's possible these regressions could just be a reflection of the
> > fact that I'm focusing more on fixing inconsistent behaviors rather
> > than adding new features, which is a type of work where it's much
> > harder to avoid fallout and reported issues.
>
> Yes, and as I said already, bugs happen and they tend to happen to
> those who touch the trickiest part of the existing code---it is
> inevitable.  It is OK to be initially buggy as long as we can fix
> breakages promptly.

Thanks, I'm glad you're not worried about that.

2.26 drained me pretty bad, enough that I was somewhat apprehensive of
2.27 being released.  Then two quick reports after 2.27 was released
was quickly filling me with dread and had me considering how I could
change it.  Maybe I was just in an overly worried and drained mindset
when your email happened to arrive.

> I am however worried about how the issues often are not raised until
> a regression hits a tagged version.  As a piece of software gets
> used more and more widely, it would be used in more and more
> different ways, and some of these ways may not be covered by
> developers' minds; by definition, some regressions are noticed only
> after a new version hits end users' hands.
>
> Basing in-house releases a Git developer has control over on
> 'master' or 'next' would reach audiences that the Git developer has
> access to, and as long as the in-house users use Git sufficiently
> differently from the Git developer (is that a realistic expectation,
> though?), we'd gain more coverage before a tagged release happens
> that way.  Basing the version placed in a distro's 'testing' track
> on 'master' or 'next' may give us even wider exposure but it would
> be much harder to arrange, I am afraid.
>
> People from time to time suggest to merge trickier topics early in
> the cycle, and in principle I do agree that it may not be a bad
> idea, but given that we do not have enough folks on 'master', I am
> not so sure how much that would actially help.

True, though I thought jrnieder's reports from Google's internal
testing were pretty cool; they caught some important bugs even if they
didn't catch everything.  The more of us doing stuff like that, the
better off we'll be, even if it's not as good as getting on a distro's
testing track.

Another consideration we could make is just scheduling .1 releases
automatically, and not limiting them to security fixes but instead
considering the same kinds of things that would be allowed around the
time of -rc1 or -rc2.  I still want to push to make the quality of .0
releases as high as we can, but given that human nature seems to lean
towards avoiding things until they are labelled as official, and since
we benefit from that wider feedback, maybe we adjust slightly to take
advantage of natural behavior and do .1 releases?  Anyway, just a
thought.

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Philip Oakley]

On 05/06/2020 02:46, Elijah Newren wrote:
> Another consideration we could make is just scheduling .1 releases
> automatically, and not limiting them to security fixes but instead
> considering the same kinds of things that would be allowed around the
> time of -rc1 or -rc2.  I still want to push to make the quality of .0
> releases as high as we can, but given that human nature seems to lean
> towards avoiding things until they are labelled as official, and since
> we benefit from that wider feedback, maybe we adjust slightly to take
> advantage of natural behavior and do .1 releases?  Anyway, just a
> thought.

Maybe simply tag the current master every 2 weeks of the cycle (.wk2,
.wk4, wk6), as appropriate, just to give potential users/testers a stake
in the ground to reference to.

Or maybe a 'mid cycle' .mc1 release for a similar 'nudge' effect [1] of
'try this one'.

Philip

[1] https://en.wikipedia.org/wiki/Nudge_theory

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Kaartic Sivaraam]

On 04-06-2020 20:02, Christian Couder wrote:
>
> Now to go back to Junio's question:
> 
>> ... people are eager
>> enough to help by reporting issues they encounter, but there are not
>> enough people who are eager enough to help by testing the tip of
>> 'master' before the release. Are there things we can do to help
>> them become early adopters so that they do not have to scramble
>> after the release?
> 
> Yeah, I agree that increasing the number of early adopters could be
> the best way to avoid regressions report just after the release.
> 
> Maybe we could just ask people to test rc releases or 'master' in Git
> Rev News?

That sounds like a good idea.

> It might work better if someone wrote a small article about
> what is coming in the next release before asking to test.

I think writing an article wouldn't need much effort thanks to
the wonderful summaries of the topics already found in the '-rc'
release e-mails. It might need curation as there would be a lot
of topics.

> Then there
> is the issue of making it easier to build Git and to understand and
> fix build issues.

Yeah, that would be nice. As a first step we could link to steps for
building Git from the hypothetical article about the '-rc'.
The "Install Dependencies" section of MyFirstContribution is
an option[1].

[1]: https://git-scm.com/docs/MyFirstContribution#dependencies

> We could also perhaps coordinate Git Rev News
> editions and Git releases, so that the editions are published for
> example between rc-1 and rc-2  releases.
>

That would be necessary. Though, I'm wondering how the eight to ten
week feature release cycle of Git and monthly frequency of Rev News
would play along. This might need some thought.

-- 
Sivaraam

Re: What's cooking in git.git (Jun 2020, #01; Wed, 3)

[Christian Couder]

On Sat, Jun 6, 2020 at 4:53 PM Kaartic Sivaraam
<kaartic.sivaraam@gmail.com> wrote:
>
> On 04-06-2020 20:02, Christian Couder wrote:

> > Then there
> > is the issue of making it easier to build Git and to understand and
> > fix build issues.
>
> Yeah, that would be nice. As a first step we could link to steps for
> building Git from the hypothetical article about the '-rc'.
> The "Install Dependencies" section of MyFirstContribution is
> an option[1].
>
> [1]: https://git-scm.com/docs/MyFirstContribution#dependencies

A related issue (https://github.com/git/git.github.io/issues/435) was
opened recently about https://git.github.io/ not helping with
debugging Git, so there is a new page to try to help with this. You
can see it at:

https://git.github.io/Hacking-Git/

and:

https://github.com/git/git.github.io/blob/master/Hacking-Git.md