What's cooking in git.git (Nov 2016, #06; Mon, 28)

What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Junio C Hamano]

Here are the topics that have been cooking.  Commits prefixed with
'-' are only in 'pu' (proposed updates) while commits prefixed with
'+' are in 'next'.  The ones marked with '.' do not appear in any of
the integration branches, but I am still holding onto them.

You can find the changes described here in the integration branches
of the repositories listed at

    http://git-blame.blogspot.com/p/git-public-repositories.html

--------------------------------------------------
[New Topics]

* jc/renormalize-merge-kill-safer-crlf (2016-11-28) 2 commits
 - merge-recursive: handle NULL in add_cacheinfo() correctly
 - cherry-pick: demonstrate a segmentation fault

 Fix a corner case in merge-recursive regression that crept in
 during 2.10 development cycle.

 Will merge to 'next'.


* js/difftool-builtin (2016-11-28) 2 commits
 - difftool: implement the functionality in the builtin
 - difftool: add a skeleton for the upcoming builtin

 Rewrite a scripted porcelain "git difftool" in C.

 Under discussion.


* nd/qsort-in-merge-recursive (2016-11-28) 1 commit
 - merge-recursive.c: use string_list_sort instead of qsort

 Code simplification.

 Will merge to 'next'.

--------------------------------------------------
[Stalled]

* jc/retire-compaction-heuristics (2016-11-02) 3 commits
 - SQUASH???
 - SQUASH???
 - diff: retire the original experimental "compaction" heuristics

 Waiting for a reroll.


* jc/abbrev-autoscale-config (2016-11-01) 1 commit
 - config.abbrev: document the new default that auto-scales

 Waiting for a reroll.


* jk/nofollow-attr-ignore (2016-11-02) 5 commits
 - exclude: do not respect symlinks for in-tree .gitignore
 - attr: do not respect symlinks for in-tree .gitattributes
 - exclude: convert "check_index" into a flags field
 - attr: convert "macro_ok" into a flags field
 - add open_nofollow() helper

 As we do not follow symbolic links when reading control files like
 .gitignore and .gitattributes from the index, match the behaviour
 and not follow symbolic links when reading them from the working
 tree.  This also tightens security a bit by not leaking contents of
 an unrelated file in the error messages when it is pointed at by
 one of these files that is a symbolic link.

 Perhaps we want to cover .gitmodules too with the same mechanism?


* nd/worktree-move (2016-11-28) 11 commits
 . worktree remove: new command
 . worktree move: refuse to move worktrees with submodules
 . worktree move: accept destination as directory
 . worktree move: new command
 . worktree.c: add update_worktree_location()
 . worktree.c: add validate_worktree()
 . copy.c: convert copy_file() to copy_dir_recursively()
 . copy.c: style fix
 . copy.c: convert bb_(p)error_msg to error(_errno)
 . copy.c: delete unused code in copy_file()
 . copy.c: import copy_file() from busybox
 (this branch uses nd/worktree-list-fixup.)

 "git worktree" learned move and remove subcommands.

 Reported to break builds on Windows.


* jc/bundle (2016-03-03) 6 commits
 - index-pack: --clone-bundle option
 - Merge branch 'jc/index-pack' into jc/bundle
 - bundle v3: the beginning
 - bundle: keep a copy of bundle file name in the in-core bundle header
 - bundle: plug resource leak
 - bundle doc: 'verify' is not about verifying the bundle

 The beginning of "split bundle", which could be one of the
 ingredients to allow "git clone" traffic off of the core server
 network to CDN.

 While I think it would make it easier for people to experiment and
 build on if the topic is merged to 'next', I am at the same time a
 bit reluctant to merge an unproven new topic that introduces a new
 file format, which we may end up having to support til the end of
 time.  It is likely that to support a "prime clone from CDN", it
 would need a lot more than just "these are the heads and the pack
 data is over there", so this may not be sufficient.

 Will discard.


* mh/connect (2016-06-06) 10 commits
 - connect: [host:port] is legacy for ssh
 - connect: move ssh command line preparation to a separate function
 - connect: actively reject git:// urls with a user part
 - connect: change the --diag-url output to separate user and host
 - connect: make parse_connect_url() return the user part of the url as a separate value
 - connect: group CONNECT_DIAG_URL handling code
 - connect: make parse_connect_url() return separated host and port
 - connect: re-derive a host:port string from the separate host and port variables
 - connect: call get_host_and_port() earlier
 - connect: document why we sometimes call get_port after get_host_and_port

 Rewrite Git-URL parsing routine (hopefully) without changing any
 behaviour.

 It has been two months without any support.  We may want to discard
 this.


* ec/annotate-deleted (2015-11-20) 1 commit
 - annotate: skip checking working tree if a revision is provided

 Usability fix for annotate-specific "<file> <rev>" syntax with deleted
 files.

 Has been waiting for a review for too long without seeing anything.

 Will discard.


* dk/gc-more-wo-pack (2016-01-13) 4 commits
 - gc: clean garbage .bitmap files from pack dir
 - t5304: ensure non-garbage files are not deleted
 - t5304: test .bitmap garbage files
 - prepare_packed_git(): find more garbage

 Follow-on to dk/gc-idx-wo-pack topic, to clean up stale
 .bitmap and .keep files.

 Has been waiting for a reroll for too long.
 cf. <xmqq60ypbeng.fsf@gitster.mtv.corp.google.com>

 Will discard.


* jc/diff-b-m (2015-02-23) 5 commits
 . WIPWIP
 . WIP: diff-b-m
 - diffcore-rename: allow easier debugging
 - diffcore-rename.c: add locate_rename_src()
 - diffcore-break: allow debugging

 "git diff -B -M" produced incorrect patch when the postimage of a
 completely rewritten file is similar to the preimage of a removed
 file; such a resulting file must not be expressed as a rename from
 other place.

 The fix in this patch is broken, unfortunately.

 Will discard.

--------------------------------------------------
[Cooking]

* bw/push-dry-run (2016-11-23) 2 commits
 - push: fix --dry-run to not push submodules
 - push: --dry-run updates submodules when --recurse-submodules=on-demand
 (this branch uses hv/submodule-not-yet-pushed-fix.)

 "git push --dry-run --recurse-submodule=on-demand" wasn't
 "--dry-run" in the submodules.

 Will merge to 'next'.


* sb/push-make-submodule-check-the-default (2016-10-10) 2 commits
 - push: change submodule default to check when submodules exist
 - submodule add: extend force flag to add existing repos

 Turn the default of "push.recurseSubmodules" to "check" when
 submodules seem to be in use.

 Need to rebase on hv/submodule-not-yet-pushed-fix and then consider
 merging to 'next'.


* jk/rev-parse-symbolic-parents-fix (2016-11-16) 1 commit
 - rev-parse: fix parent shorthands with --symbolic

 "git rev-parse --symbolic" failed with a more recent notation like
 "HEAD^-1" and "HEAD^!".

 Will merge to 'next'.


* nd/worktree-list-fixup (2016-11-28) 5 commits
 - worktree list: keep the list sorted
 - worktree.c: get_worktrees() takes a new flag argument
 - get_worktrees() must return main worktree as first item even on error
 - worktree: reorder an if statement
 - worktree.c: zero new 'struct worktree' on allocation
 (this branch is used by nd/worktree-move.)

 The output from "git worktree list" was made in readdir() order,
 and was unstable.

 Will merge to 'next'.


* jk/trailers-placeholder-in-pretty (2016-11-21) 2 commits
 - ref-filter: add support to display trailers as part of contents
 - pretty: add %(trailers) format for displaying trailers of a commit message

 In addition to %(subject), %(body), "log --pretty=format:..."
 learned a new placeholder %(trailers).

 Will merge to 'next'.


* sb/submodule-intern-gitdir (2016-11-22) 5 commits
 - SQUASH
 - submodule: add embed-git-dir function
 - test-lib-functions.sh: teach test_commit -C <dir>
 - submodule helper: support super prefix
 - submodule: use absolute path for computing relative path connecting

 A new submodule helper "git submodule embedgitdirs" to make it
 easier to move embedded .git/ directory for submodules in a
 superproject to .git/modules/ (and point the latter with the former
 that is turned into a "gitdir:" file) has been added.

 Need to read it over again, deal with SQUASH, and may ask for a
 reroll.


* dt/empty-submodule-in-merge (2016-11-17) 1 commit
 - submodules: allow empty working-tree dirs in merge/cherry-pick

 An empty directory in a working tree that can simply be nuked used
 to interfere while merging or cherry-picking a change to create a
 submodule directory there, which has been fixed..

 Waiting for review.


* bw/grep-recurse-submodules (2016-11-22) 6 commits
 - grep: search history of moved submodules
 - grep: enable recurse-submodules to work on <tree> objects
 - grep: optionally recurse into submodules
 - grep: add submodules as a grep source type
 - submodules: load gitmodules file from commit sha1
 - submodules: add helper functions to determine presence of submodules

 "git grep" learns to optionally recurse into submodules

 Has anybody else seen t7814 being flakey with this series?


* dt/smart-http-detect-server-going-away (2016-11-18) 2 commits
  (merged to 'next' on 2016-11-21 at d502523347)
 + upload-pack: optionally allow fetching any sha1
 + remote-curl: don't hang when a server dies before any output

 When the http server gives an incomplete response to a smart-http
 rpc call, it could lead to client waiting for a full response that
 will never come.  Teach the client side to notice this condition
 and abort the transfer.

 An improvement counterproposal has failed.
 cf. <20161114194049.mktpsvgdhex2f4zv@sigill.intra.peff.net>

 Will cook in 'next'.


* mm/push-social-engineering-attack-doc (2016-11-14) 1 commit
  (merged to 'next' on 2016-11-16 at b7c1b27563)
 + doc: mention transfer data leaks in more places

 Doc update on fetching and pushing.

 Will cook in 'next'.


* jc/compression-config (2016-11-15) 1 commit
  (merged to 'next' on 2016-11-23 at b3c9254897)
 + compression: unify pack.compression configuration parsing

 Compression setting for producing packfiles were spread across
 three codepaths, one of which did not honor any configuration.
 Unify these so that all of them honor core.compression and
 pack.compression variables the same way.

 Will cook in 'next'.


* mm/gc-safety-doc (2016-11-16) 1 commit
  (merged to 'next' on 2016-11-17 at fc0d225b6b)
 + git-gc.txt: expand discussion of races with other processes

 Doc update.

 Will cook in 'next'.


* hv/submodule-not-yet-pushed-fix (2016-11-16) 4 commits
  (merged to 'next' on 2016-11-21 at 1a599af962)
 + submodule_needs_pushing(): explain the behaviour when we cannot answer
 + batch check whether submodule needs pushing into one call
 + serialize collection of refs that contain submodule changes
 + serialize collection of changed submodules
 (this branch is used by bw/push-dry-run.)

 The code in "git push" to compute if any commit being pushed in the
 superproject binds a commit in a submodule that hasn't been pushed
 out was overly inefficient, making it unusable even for a small
 project that does not have any submodule but have a reasonable
 number of refs.

 Will cook in 'next'.


* kn/ref-filter-branch-list (2016-11-15) 18 commits
 - for-each-ref: do not segv with %(HEAD) on an unborn branch
 - branch: implement '--format' option
 - branch: use ref-filter printing APIs
 - branch, tag: use porcelain output
 - ref-filter: allow porcelain to translate messages in the output
 - ref-filter: add `:dir` and `:base` options for ref printing atoms
 - ref-filter: make remote_ref_atom_parser() use refname_atom_parser_internal()
 - ref-filter: introduce symref_atom_parser() and refname_atom_parser()
 - ref-filter: introduce refname_atom_parser_internal()
 - ref-filter: make "%(symref)" atom work with the ':short' modifier
 - ref-filter: add support for %(upstream:track,nobracket)
 - ref-filter: make %(upstream:track) prints "[gone]" for invalid upstreams
 - ref-filter: introduce format_ref_array_item()
 - ref-filter: move get_head_description() from branch.c
 - ref-filter: modify "%(objectname:short)" to take length
 - ref-filter: implement %(if:equals=<string>) and %(if:notequals=<string>)
 - ref-filter: include reference to 'used_atom' within 'atom_value'
 - ref-filter: implement %(if), %(then), and %(else) atoms

 The code to list branches in "git branch" has been consolidated
 with the more generic ref-filter API.

 Rerolled, reviewed, looking good.
 Expecting a reroll.
 cf. <20161108201211.25213-1-Karthik.188@gmail.com>
 cf. <CAOLa=ZQqe3vEj_428d41vd_4kfjzsm87Wam6Zm2dhXWkPdJ8Rw@mail.gmail.com>
 cf. <xmqq7f84tqa7.fsf_-_@gitster.mtv.corp.google.com>


* bw/transport-protocol-policy (2016-11-09) 2 commits
  (merged to 'next' on 2016-11-16 at 1391d3eeed)
 + transport: add protocol policy config option
 + lib-proto-disable: variable name fix

 Finer-grained control of what protocols are allowed for transports
 during clone/fetch/push have been enabled via a new configuration
 mechanism.

 Will cook in 'next'.


* jt/fetch-no-redundant-tag-fetch-map (2016-11-11) 1 commit
  (merged to 'next' on 2016-11-16 at 5846c27cc5)
 + fetch: do not redundantly calculate tag refmap

 Code cleanup to avoid using redundant refspecs while fetching with
 the --tags option.

 Will cook in 'next'.


* sb/submodule-config-cleanup (2016-11-22) 3 commits
  (merged to 'next' on 2016-11-23 at c02e578b49)
 + submodule-config: clarify parsing of null_sha1 element
 + submodule-config: rename commit_sha1 to treeish_name
 + submodule config: inline config_from_{name, path}

 Minor code clean-up.

 Will cook in 'next'.


* jc/push-default-explicit (2016-10-31) 2 commits
  (merged to 'next' on 2016-11-01 at 8dc3a6cf25)
 + push: test pushing ambiguously named branches
 + push: do not use potentially ambiguous default refspec

 A lazy "git push" without refspec did not internally use a fully
 specified refspec to perform 'current', 'simple', or 'upstream'
 push, causing unnecessary "ambiguous ref" errors.

 Will cook in 'next'.


* jt/use-trailer-api-in-commands (2016-11-02) 6 commits
 - sequencer: use trailer's trailer layout
 - trailer: have function to describe trailer layout
 - trailer: avoid unnecessary splitting on lines
 - commit: make ignore_non_trailer take buf/len
 - SQUASH???
 - trailer: be stricter in parsing separators

 Commands that operate on a log message and add lines to the trailer
 blocks, such as "format-patch -s", "cherry-pick (-x|-s)", and
 "commit -s", have been taught to use the logic of and share the
 code with "git interpret-trailer".

 Will merge to 'next' after dealing with the SQUASH???


* nd/rebase-forget (2016-11-28) 1 commit
 - rebase: add --forget to cleanup rebase, leave everything else untouched

 "git rebase" learned "--forget" option, which allows a user to
 remove the metadata left by an earlier "git rebase" that was
 manually aborted without using "git rebase --abort".

 Will merge to 'next'.


* jc/git-open-cloexec (2016-11-02) 3 commits
 - sha1_file: stop opening files with O_NOATIME
 - git_open_cloexec(): use fcntl(2) w/ FD_CLOEXEC fallback
 - git_open(): untangle possible NOATIME and CLOEXEC interactions

 The codeflow of setting NOATIME and CLOEXEC on file descriptors Git
 opens has been simplified.

 We may want to drop the tip one.


* jk/no-looking-at-dotgit-outside-repo-final (2016-10-26) 1 commit
  (merged to 'next' on 2016-10-26 at 220e160451)
 + setup_git_env: avoid blind fall-back to ".git"

 This is the endgame of the topic to avoid blindly falling back to
 ".git" when the setup sequence said we are _not_ in Git repository.
 A corner case that happens to work right now may be broken by a
 call to die("BUG").

 Will cook in 'next'.


* jc/reset-unmerge (2016-10-24) 1 commit
 - reset: --unmerge

 After "git add" is run prematurely during a conflict resolution,
 "git diff" can no longer be used as a way to sanity check by
 looking at the combined diff.  "git reset" learned a new
 "--unmerge" option to recover from this situation.

 Will discard.
 This may not be needed, given that update-index has a similar
 option.


* jc/merge-base-fp-only (2016-10-19) 8 commits
 . merge-base: fp experiment
 - merge: allow to use only the fp-only merge bases
 - merge-base: limit the output to bases that are on first-parent chain
 - merge-base: mark bases that are on first-parent chain
 - merge-base: expose get_merge_bases_many_0() a bit more
 - merge-base: stop moving commits around in remove_redundant()
 - sha1_name: remove ONELINE_SEEN bit
 - commit: simplify fastpath of merge-base

 An experiment of merge-base that ignores common ancestors that are
 not on the first parent chain.

 Will discard.
 The whole premise feels wrong.


* tb/convert-stream-check (2016-10-27) 2 commits
 - convert.c: stream and fast search for binary
 - read-cache: factor out get_sha1_from_index() helper

 End-of-line conversion sometimes needs to see if the current blob
 in the index has NULs and CRs to base its decision.  We used to
 always get a full statistics over the blob, but in many cases we
 can return early when we have seen "enough" (e.g. if we see a
 single NUL, the blob will be handled as binary).  The codepaths
 have been optimized by using streaming interface.

 Will discard.
 Retracted.
 cf. <20161102071646.GA5094@tb-raspi>


* pb/bisect (2016-10-18) 27 commits
 - bisect--helper: remove the dequote in bisect_start()
 - bisect--helper: retire `--bisect-auto-next` subcommand
 - bisect--helper: retire `--bisect-autostart` subcommand
 - bisect--helper: retire `--bisect-write` subcommand
 - bisect--helper: `bisect_replay` shell function in C
 - bisect--helper: `bisect_log` shell function in C
 - bisect--helper: retire `--write-terms` subcommand
 - bisect--helper: retire `--check-expected-revs` subcommand
 - bisect--helper: `bisect_state` & `bisect_head` shell function in C
 - bisect--helper: `bisect_autostart` shell function in C
 - bisect--helper: retire `--next-all` subcommand
 - bisect--helper: retire `--bisect-clean-state` subcommand
 - bisect--helper: `bisect_next` and `bisect_auto_next` shell function in C
 - t6030: no cleanup with bad merge base
 - bisect--helper: `bisect_start` shell function partially in C
 - bisect--helper: `get_terms` & `bisect_terms` shell function in C
 - bisect--helper: `bisect_next_check` & bisect_voc shell function in C
 - bisect--helper: `check_and_set_terms` shell function in C
 - bisect--helper: `bisect_write` shell function in C
 - bisect--helper: `is_expected_rev` & `check_expected_revs` shell function in C
 - bisect--helper: `bisect_reset` shell function in C
 - wrapper: move is_empty_file() and rename it as is_empty_or_missing_file()
 - t6030: explicitly test for bisection cleanup
 - bisect--helper: `bisect_clean_state` shell function in C
 - bisect--helper: `write_terms` shell function in C
 - bisect: rewrite `check_term_format` shell function in C
 - bisect--helper: use OPT_CMDMODE instead of OPT_BOOL

 Move more parts of "git bisect" to C.

 Waiting for review.


* st/verify-tag (2016-10-10) 7 commits
 - t/t7004-tag: Add --format specifier tests
 - t/t7030-verify-tag: Add --format specifier tests
 - builtin/tag: add --format argument for tag -v
 - builtin/verify-tag: add --format to verify-tag
 - tag: add format specifier to gpg_verify_tag
 - ref-filter: add function to print single ref_array_item
 - gpg-interface, tag: add GPG_VERIFY_QUIET flag

 "git tag" and "git verify-tag" learned to put GPG verification
 status in their "--format=<placeholders>" output format.

 Waiting for a reroll.
 cf. <20161007210721.20437-1-santiago@nyu.edu>


* sb/attr (2016-11-11) 35 commits
 - completion: clone can initialize specific submodules
 - clone: add --init-submodule=<pathspec> switch
 - submodule update: add `--init-default-path` switch
 - pathspec: allow escaped query values
 - pathspec: allow querying for attributes
 - pathspec: move prefix check out of the inner loop
 - pathspec: move long magic parsing out of prefix_pathspec
 - Documentation: fix a typo
 - attr: keep attr stack for each check
 - attr: convert to new threadsafe API
 - attr: make git_check_attr_counted static
 - attr.c: outline the future plans by heavily commenting
 - attr.c: always pass check[] to collect_some_attrs()
 - attr.c: introduce empty_attr_check_elems()
 - attr.c: correct ugly hack for git_all_attrs()
 - attr.c: rename a local variable check
 - attr.c: pass struct git_attr_check down the callchain
 - attr.c: add push_stack() helper
 - attr: support quoting pathname patterns in C style
 - attr: expose validity check for attribute names
 - attr: add counted string version of git_check_attr()
 - attr: retire git_check_attrs() API
 - attr: convert git_check_attrs() callers to use the new API
 - attr: convert git_all_attrs() to use "struct git_attr_check"
 - attr: (re)introduce git_check_attr() and struct git_attr_check
 - attr: rename function and struct related to checking attributes
 - attr.c: plug small leak in parse_attr_line()
 - attr.c: tighten constness around "git_attr" structure
 - attr.c: simplify macroexpand_one()
 - attr.c: mark where #if DEBUG ends more clearly
 - attr.c: complete a sentence in a comment
 - attr.c: explain the lack of attr-name syntax check in parse_attr()
 - attr.c: update a stale comment on "struct match_attr"
 - attr.c: use strchrnul() to scan for one line
 - commit.c: use strchrnul() to scan for one line

 The attributes API has been updated so that it can later be
 optimized using the knowledge of which attributes are queried.
 Building on top of the updated API, the pathspec machinery learned
 to select only paths with given attributes set.

 Waiting for review.


* va/i18n-perl-scripts (2016-11-11) 16 commits
 - i18n: difftool: mark warnings for translation
 - i18n: send-email: mark composing message for translation
 - i18n: send-email: mark string with interpolation for translation
 - i18n: send-email: mark warnings and errors for translation
 - i18n: send-email: mark strings for translation
 - i18n: add--interactive: mark status words for translation
 - i18n: add--interactive: remove %patch_modes entries
 - i18n: add--interactive: mark edit_hunk_manually message for translation
 - i18n: add--interactive: i18n of help_patch_cmd
 - i18n: add--interactive: mark patch prompt for translation
 - i18n: add--interactive: mark plural strings
 - i18n: clean.c: match string with git-add--interactive.perl
 - i18n: add--interactive: mark strings with interpolation for translation
 - i18n: add--interactive: mark simple here-documents for translation
 - i18n: add--interactive: mark strings for translation
 - Git.pm: add subroutines for commenting lines

 Porcelain scripts written in Perl are getting internationalized.

 Waiting for review.


* jc/latin-1 (2016-09-26) 2 commits
  (merged to 'next' on 2016-09-28 at c8673e03c2)
 + utf8: accept "latin-1" as ISO-8859-1
 + utf8: refactor code to decide fallback encoding

 Some platforms no longer understand "latin-1" that is still seen in
 the wild in e-mail headers; replace them with "iso-8859-1" that is
 more widely known when conversion fails from/to it.

 Will cook in 'next'.


* sg/fix-versioncmp-with-common-suffix (2016-09-08) 5 commits
 - versioncmp: cope with common leading parts in versionsort.prereleaseSuffix
 - versioncmp: pass full tagnames to swap_prereleases()
 - t7004-tag: add version sort tests to show prerelease reordering issues
 - t7004-tag: use test_config helper
 - t7004-tag: delete unnecessary tags with test_when_finished

 The prereleaseSuffix feature of version comparison that is used in
 "git tag -l" did not correctly when two or more prereleases for the
 same release were present (e.g. when 2.0, 2.0-beta1, and 2.0-beta2
 are there and the code needs to compare 2.0-beta1 and 2.0-beta2).

 Waiting for a reroll.
 cf. <20160908223727.Horde.jVOOJ278ssZ3qkyjkmyqZD-@webmail.informatik.kit.edu>


* jc/pull-rebase-ff (2016-07-28) 1 commit
 - pull: fast-forward "pull --rebase=true"

 "git pull --rebase", when there is no new commits on our side since
 we forked from the upstream, should be able to fast-forward without
 invoking "git rebase", but it didn't.

 Needs a real log message and a few tests.


* jc/merge-drop-old-syntax (2015-04-29) 1 commit
  (merged to 'next' on 2016-10-11 at 8928c8b9b3)
 + merge: drop 'git merge <message> HEAD <commit>' syntax

 Stop supporting "git merge <message> HEAD <commit>" syntax that has
 been deprecated since October 2007, and issues a deprecation
 warning message since v2.5.0.

 Will cook in 'next'.

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/28, Junio C Hamano wrote:
> * bw/grep-recurse-submodules (2016-11-22) 6 commits
>  - grep: search history of moved submodules
>  - grep: enable recurse-submodules to work on <tree> objects
>  - grep: optionally recurse into submodules
>  - grep: add submodules as a grep source type
>  - submodules: load gitmodules file from commit sha1
>  - submodules: add helper functions to determine presence of submodules
> 
>  "git grep" learns to optionally recurse into submodules
> 
>  Has anybody else seen t7814 being flakey with this series?

Which tests in particular are you seeing issues with?  I can't see any
issues running it locally.

-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Mon, Nov 28, 2016 at 05:05:38PM -0800, Brandon Williams wrote:

> On 11/28, Junio C Hamano wrote:
> > * bw/grep-recurse-submodules (2016-11-22) 6 commits
> >  - grep: search history of moved submodules
> >  - grep: enable recurse-submodules to work on <tree> objects
> >  - grep: optionally recurse into submodules
> >  - grep: add submodules as a grep source type
> >  - submodules: load gitmodules file from commit sha1
> >  - submodules: add helper functions to determine presence of submodules
> > 
> >  "git grep" learns to optionally recurse into submodules
> > 
> >  Has anybody else seen t7814 being flakey with this series?
> 
> Which tests in particular are you seeing issues with?  I can't see any
> issues running it locally.

It looks like tests 14 and 15 are racy. The whole script usually passes,
but if I run it under my stress script[1], it fails within 5-10 seconds
on one of those two.

The failures always look like (this one is from test 15, but the one in
test 14 is similar):

  --- expect      2016-11-29 06:26:37.874664486 +0000
  +++ actual      2016-11-29 06:26:37.878664486 +0000
  @@ -1,2 +1 @@
  -file:foobar
   sub-moved/file:foobar

I haven't dug into it, but I don't see anything obviously racy-looking
in the test, so presumably it's in the code. Without looking, but
knowing the nature of the code, I'd guess the probable avenues are:

  1. A read() or write() that gets split under load (just because
     there's processes piping to other processes here).

  2. Grep threads doing more complicated stuff that needs to take a
     lock. You might try building with -fsanitize=thread to see if it
     turns up anything.

-Peff

[1] https://github.com/peff/git/blob/meta/stress

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Tue, Nov 29, 2016 at 01:37:59AM -0500, Jeff King wrote:

>   2. Grep threads doing more complicated stuff that needs to take a
>      lock. You might try building with -fsanitize=thread to see if it
>      turns up anything.

I tried this and it didn't find anything useful. It complains about
multiple threads calling want_color() at the same time, which you can
silence with something like:

diff --git a/builtin/grep.c b/builtin/grep.c
index 2c727ef49..d48846f40 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -207,6 +207,12 @@ static void start_threads(struct grep_opt *opt)
 {
 	int i;
 
+	/*
+	 * trigger want_color() for its side effect of caching the result;
+	 * otherwise the threads will fight over setting the cache
+	 */
+	want_color(GIT_COLOR_AUTO);
+
 	pthread_mutex_init(&grep_mutex, NULL);
 	pthread_mutex_init(&grep_read_mutex, NULL);
 	pthread_mutex_init(&grep_attr_mutex, NULL);

But the problem persists even with that patch, so it is something else.
It may still be a threading problem; -fsanitize=thread isn't perfect. I
also couldn't get the stress-test to fail when compiled with it. But
that may simply mean that the timing of the resulting binary is changed
enough not to trigger the issue.

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Mon, Nov 28, 2016 at 04:15:08PM -0800, Junio C Hamano wrote:

> * jk/nofollow-attr-ignore (2016-11-02) 5 commits
>  - exclude: do not respect symlinks for in-tree .gitignore
>  - attr: do not respect symlinks for in-tree .gitattributes
>  - exclude: convert "check_index" into a flags field
>  - attr: convert "macro_ok" into a flags field
>  - add open_nofollow() helper
> 
>  As we do not follow symbolic links when reading control files like
>  .gitignore and .gitattributes from the index, match the behaviour
>  and not follow symbolic links when reading them from the working
>  tree.  This also tightens security a bit by not leaking contents of
>  an unrelated file in the error messages when it is pointed at by
>  one of these files that is a symbolic link.
> 
>  Perhaps we want to cover .gitmodules too with the same mechanism?

Yes, sorry I haven't pushed that forward. I started on covering
.gitmodules, too, but it's much more complicated than the other two,
because we sometimes read them via "git config -f". So we have to
somehow teach git-config to start using O_NOFOLLOW in those cases.

I'm actually considering scrapping the approach you've queued above, and
just teaching verify_path() to reject any index entry starting with
".git" that is a symlink.

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I'm actually considering scrapping the approach you've queued above, and
> just teaching verify_path() to reject any index entry starting with
> ".git" that is a symlink.

Hmph, that's a thought.

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Tue, Nov 29, 2016 at 10:31:51AM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > I'm actually considering scrapping the approach you've queued above, and
> > just teaching verify_path() to reject any index entry starting with
> > ".git" that is a symlink.
> 
> Hmph, that's a thought.

I was resistant to it at first because we'll have to deal with all of
the headaches of matching case-folding, but if we just match ".git*" and
not ".gitmodules", ".gitattributes", etc, it actually gets easier. I
think we can basically build off of the existing is_hfs_dotgit() and
is_ntfs_dotgit() functions.

I haven't written the code yet, though, so there may be complications.

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Stefan Beller]

On Mon, Nov 28, 2016 at 4:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
> * sb/push-make-submodule-check-the-default (2016-10-10) 2 commits
>  - push: change submodule default to check when submodules exist
>  - submodule add: extend force flag to add existing repos
>
>  Turn the default of "push.recurseSubmodules" to "check" when
>  submodules seem to be in use.
>
>  Need to rebase on hv/submodule-not-yet-pushed-fix and then consider
>  merging to 'next'.

The rebase is without merge conflicts, so I assume there is no
work needed by me here.

> * sb/submodule-intern-gitdir (2016-11-22) 5 commits
>  - SQUASH
>  - submodule: add embed-git-dir function
>  - test-lib-functions.sh: teach test_commit -C <dir>
>  - submodule helper: support super prefix
>  - submodule: use absolute path for computing relative path connecting
>
>  A new submodule helper "git submodule embedgitdirs" to make it
>  easier to move embedded .git/ directory for submodules in a
>  superproject to .git/modules/ (and point the latter with the former
>  that is turned into a "gitdir:" file) has been added.
>
>  Need to read it over again, deal with SQUASH, and may ask for a
>  reroll.

Ok, I looked over it again and I may see some issues:
* it is applicable to all submodules recursive by default, i.e. really
  "all submodules reachable from this superproject". I anticipate
  this to be the most relevant use case (i.e. as a preparatory step
  for having e.g. git-checkout work), but there are no other commands
  yet that are recursing into submodules recursively by default. So
  a discussion/disagreement on the default may come up.
  (We also may want to see a --[no-]recursive flag)

* The output is okay-ish, but could be better as it is a mix of
  relative and absolute path:

    Migrating git directory of plugins/cookbook from
    '/absolute/path/here/gerrit/plugins/cookbook/.git' to
    '/absolute/path/here/gerrit/.git/modules/plugins/cookbook/'

  On the other hand this seems like what the user may need,
  as it is the maximum for trouble shooting

* As this is a subcommand we do not need to add it to command-list.txt
  However we may want to discuss if some submodule commands are
  porcelain (all except for the new embedgitdirs?) and if this new command
  is plumbing. We could also argue the submodule--helper (which needs
  listing in command-list.txt as a plumbing command?) is the plumbing
  equivalent and the "submodule embedgitdirs" is the porcelain.

* any other part where we need to add documentation for a new command?

FYI: I have a series cooking internally that adds a new page in
Documentation/submodules that introduces the concept of submodules,
which then allows Documentation/git-submodule.txt to be focused on the
actual command and its options.

>
> * dt/empty-submodule-in-merge (2016-11-17) 1 commit
>  - submodules: allow empty working-tree dirs in merge/cherry-pick
>
>  Waiting for review

That slipped by me. Will review.

> * sb/attr (2016-11-11) 35 commits
>  - completion: clone can initialize specific submodules
>  - clone: add --init-submodule=<pathspec> switch
>  - submodule update: add `--init-default-path` switch

I may end up rerolling these top three patches as its own series
again without the underlying pathspec support.

I will investigate if we need the mutex at all for the attribute
code or if we can initialize all attrs (in the pathspecs) before the
threaded preload_index takes place. That sounds cleaner to me,
but I do not prioritize it as high.

>  Waiting for review.

There was some review by Duy and Brandon, I may reroll with just their
issues addressed.

Thanks,
Stefan

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Mon, Nov 28, 2016 at 4:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>
>> * sb/push-make-submodule-check-the-default (2016-10-10) 2 commits
>>  - push: change submodule default to check when submodules exist
>>  - submodule add: extend force flag to add existing repos
>>
>>  Turn the default of "push.recurseSubmodules" to "check" when
>>  submodules seem to be in use.
>>
>>  Need to rebase on hv/submodule-not-yet-pushed-fix and then consider
>>  merging to 'next'.
>
> The rebase is without merge conflicts, so I assume there is no
> work needed by me here.

Correct.

>> * sb/submodule-intern-gitdir (2016-11-22) 5 commits
>>  - SQUASH
>>  - submodule: add embed-git-dir function
>>  - test-lib-functions.sh: teach test_commit -C <dir>
>>  - submodule helper: support super prefix
>>  - submodule: use absolute path for computing relative path connecting
>>
>>  A new submodule helper "git submodule embedgitdirs" to make it
>>  easier to move embedded .git/ directory for submodules in a
>>  superproject to .git/modules/ (and point the latter with the former
>>  that is turned into a "gitdir:" file) has been added.
>>
>>  Need to read it over again, deal with SQUASH, and may ask for a
>>  reroll.
>
> Ok, I looked over it again and I may see some issues:

OK then I'll procrastinate on this.

> That slipped by me. Will review.
>
>> * sb/attr (2016-11-11) 35 commits
>>  - completion: clone can initialize specific submodules
>>  - clone: add --init-submodule=<pathspec> switch
>>  - submodule update: add `--init-default-path` switch
>
> I may end up rerolling these top three patches as its own series
> again without the underlying pathspec support.
>
> I will investigate if we need the mutex at all for the attribute
> code or if we can initialize all attrs (in the pathspecs) before the
> threaded preload_index takes place. That sounds cleaner to me,
> but I do not prioritize it as high.

OK.

Thanks.

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Stefan Beller]

On Tue, Nov 29, 2016 at 11:26 AM, Junio C Hamano <gitster@pobox.com> wrote:

>>>  Need to read it over again, deal with SQUASH, and may ask for a
>>>  reroll.
>>
>> Ok, I looked over it again and I may see some issues:
>
> OK then I'll procrastinate on this.

This is not what I had in mind when typing out the issues.
I'd rather wanted to hear "these issues are all non-issues,
your design is actually correct as a first solution". ;)

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Stefan Beller]

On Tue, Nov 29, 2016 at 11:21 AM, Stefan Beller <sbeller@google.com> wrote:
>
>>
>> * dt/empty-submodule-in-merge (2016-11-17) 1 commit
>>  - submodules: allow empty working-tree dirs in merge/cherry-pick
>>
>>  Waiting for review
>
> That slipped by me. Will review.
>

I reviewed what you have queued and it still looks good to me.

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/29, Jeff King wrote:
> On Tue, Nov 29, 2016 at 01:37:59AM -0500, Jeff King wrote:
> 
> >   2. Grep threads doing more complicated stuff that needs to take a
> >      lock. You might try building with -fsanitize=thread to see if it
> >      turns up anything.
> 
> I tried this and it didn't find anything useful. It complains about
> multiple threads calling want_color() at the same time, which you can
> silence with something like:
> 
> diff --git a/builtin/grep.c b/builtin/grep.c
> index 2c727ef49..d48846f40 100644
> --- a/builtin/grep.c
> +++ b/builtin/grep.c
> @@ -207,6 +207,12 @@ static void start_threads(struct grep_opt *opt)
>  {
>  	int i;
>  
> +	/*
> +	 * trigger want_color() for its side effect of caching the result;
> +	 * otherwise the threads will fight over setting the cache
> +	 */
> +	want_color(GIT_COLOR_AUTO);
> +
>  	pthread_mutex_init(&grep_mutex, NULL);
>  	pthread_mutex_init(&grep_read_mutex, NULL);
>  	pthread_mutex_init(&grep_attr_mutex, NULL);
> 
> But the problem persists even with that patch, so it is something else.
> It may still be a threading problem; -fsanitize=thread isn't perfect. I
> also couldn't get the stress-test to fail when compiled with it. But
> that may simply mean that the timing of the resulting binary is changed
> enough not to trigger the issue.
> 
> -Peff

With you're stress script I'm able to see the failures.  The interesting
thing is that the entry missing is always from the non-submodule file.

-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/30, Brandon Williams wrote:
> On 11/29, Jeff King wrote:
> > On Tue, Nov 29, 2016 at 01:37:59AM -0500, Jeff King wrote:
> > 
> > >   2. Grep threads doing more complicated stuff that needs to take a
> > >      lock. You might try building with -fsanitize=thread to see if it
> > >      turns up anything.
> > 
> > I tried this and it didn't find anything useful. It complains about
> > multiple threads calling want_color() at the same time, which you can
> > silence with something like:
> > 
> > diff --git a/builtin/grep.c b/builtin/grep.c
> > index 2c727ef49..d48846f40 100644
> > --- a/builtin/grep.c
> > +++ b/builtin/grep.c
> > @@ -207,6 +207,12 @@ static void start_threads(struct grep_opt *opt)
> >  {
> >  	int i;
> >  
> > +	/*
> > +	 * trigger want_color() for its side effect of caching the result;
> > +	 * otherwise the threads will fight over setting the cache
> > +	 */
> > +	want_color(GIT_COLOR_AUTO);
> > +
> >  	pthread_mutex_init(&grep_mutex, NULL);
> >  	pthread_mutex_init(&grep_read_mutex, NULL);
> >  	pthread_mutex_init(&grep_attr_mutex, NULL);
> > 
> > But the problem persists even with that patch, so it is something else.
> > It may still be a threading problem; -fsanitize=thread isn't perfect. I
> > also couldn't get the stress-test to fail when compiled with it. But
> > that may simply mean that the timing of the resulting binary is changed
> > enough not to trigger the issue.
> > 
> > -Peff
> 
> With you're stress script I'm able to see the failures.  The interesting
> thing is that the entry missing is always from the non-submodule file.

So I couldn't find a race condition in the code.  I tracked the problem
to grep_source_load_file which attempts to run lstat on the file so that
it can read it into a buffer.  The lstat call fails with ENOENT (which
conveniently is skipped by the if statement which calls error_errno).  So
for some reason the file cannot be found and read into memory resulting
in nothing being grep'ed for that particular file (since the buffer is
NULL).

Maybe there is something wrong with the way I wrote the tests
themselves?  So I could try rewriting them.  Any other ideas?

-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Wed, Nov 30, 2016 at 03:28:23PM -0800, Brandon Williams wrote:

> So I couldn't find a race condition in the code.  I tracked the problem
> to grep_source_load_file which attempts to run lstat on the file so that
> it can read it into a buffer.  The lstat call fails with ENOENT (which
> conveniently is skipped by the if statement which calls error_errno).  So
> for some reason the file cannot be found and read into memory resulting
> in nothing being grep'ed for that particular file (since the buffer is
> NULL).

That's definitely weird. Is it possible that any of the underlying calls
from another thread are using chdir()? I think realpath() make do that
behind the scenes, and there may be others.

A full strace from a failing case would be interesting reading. In
theory we should be able to get that by running the stress script for
long enough. :)

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Wed, Nov 30, 2016 at 06:32:04PM -0500, Jeff King wrote:

> On Wed, Nov 30, 2016 at 03:28:23PM -0800, Brandon Williams wrote:
> 
> > So I couldn't find a race condition in the code.  I tracked the problem
> > to grep_source_load_file which attempts to run lstat on the file so that
> > it can read it into a buffer.  The lstat call fails with ENOENT (which
> > conveniently is skipped by the if statement which calls error_errno).  So
> > for some reason the file cannot be found and read into memory resulting
> > in nothing being grep'ed for that particular file (since the buffer is
> > NULL).
> 
> That's definitely weird. Is it possible that any of the underlying calls
> from another thread are using chdir()? I think realpath() make do that
> behind the scenes, and there may be others.
> 
> A full strace from a failing case would be interesting reading. In
> theory we should be able to get that by running the stress script for
> long enough. :)

Actually, it failed pretty much immediately. I guess the extra stracing
changes the timing to make the problem _more_ likely.

And indeed, I see:

20867 lstat("fi:le",  <unfinished ...>
20813 <... read resumed> "", 232)       = 0
20871 futex(0x558cdec8b164, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
20813 close(7 <unfinished ...>
20870 <... futex resumed> )             = 0
20869 lstat(".gitmodules",  <unfinished ...>
20813 <... close resumed> )             = 0
20865 set_robust_list(0x7f1df92579e0, 24 <unfinished ...>
20813 lstat("su:b/../.git/modules/su:b/commondir", 0x7ffecc8b3ac0) = -1 ENOENT (No such file or directory)
20865 <... set_robust_list resumed> )   = 0
20868 set_robust_list(0x7f1df7a549e0, 24 <unfinished ...>
20813 access("su:b/../.git/modules/su:b/objects", X_OK) = 0
20813 access("su:b/../.git/modules/su:b/refs", X_OK) = 0
20813 stat("su:b/../.git/modules/su:b", {st_mode=S_IFDIR|0755, st_size=280, ...}) = 0
20813 getcwd("/var/ram/git-stress/root-4/trash directory.t7814-grep-recurse-submodules/parent", 129) = 80
20869 <... lstat resumed> {st_mode=S_IFREG|0644, st_size=47, ...}) = 0
20813 chdir("su:b/../.git/modules/su:b") = 0
20869 open(".gitmodules", O_RDONLY <unfinished ...>
20813 getcwd( <unfinished ...>
20867 <... lstat resumed> 0x7f1df8254cf0) = -1 ENOENT (No such file or directory)

where 20813 and 20867 are two threads of the main process. One is doing
the lstat and the other calls chdir at the same moment.

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/30, Jeff King wrote:
> On Wed, Nov 30, 2016 at 06:32:04PM -0500, Jeff King wrote:
> 
> > On Wed, Nov 30, 2016 at 03:28:23PM -0800, Brandon Williams wrote:
> > 
> > > So I couldn't find a race condition in the code.  I tracked the problem
> > > to grep_source_load_file which attempts to run lstat on the file so that
> > > it can read it into a buffer.  The lstat call fails with ENOENT (which
> > > conveniently is skipped by the if statement which calls error_errno).  So
> > > for some reason the file cannot be found and read into memory resulting
> > > in nothing being grep'ed for that particular file (since the buffer is
> > > NULL).
> > 
> > That's definitely weird. Is it possible that any of the underlying calls
> > from another thread are using chdir()? I think realpath() make do that
> > behind the scenes, and there may be others.
> > 
> > A full strace from a failing case would be interesting reading. In
> > theory we should be able to get that by running the stress script for
> > long enough. :)
> 
> Actually, it failed pretty much immediately. I guess the extra stracing
> changes the timing to make the problem _more_ likely.
> 
> And indeed, I see:
> 
> 20867 lstat("fi:le",  <unfinished ...>
> 20813 <... read resumed> "", 232)       = 0
> 20871 futex(0x558cdec8b164, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> 20813 close(7 <unfinished ...>
> 20870 <... futex resumed> )             = 0
> 20869 lstat(".gitmodules",  <unfinished ...>
> 20813 <... close resumed> )             = 0
> 20865 set_robust_list(0x7f1df92579e0, 24 <unfinished ...>
> 20813 lstat("su:b/../.git/modules/su:b/commondir", 0x7ffecc8b3ac0) = -1 ENOENT (No such file or directory)
> 20865 <... set_robust_list resumed> )   = 0
> 20868 set_robust_list(0x7f1df7a549e0, 24 <unfinished ...>
> 20813 access("su:b/../.git/modules/su:b/objects", X_OK) = 0
> 20813 access("su:b/../.git/modules/su:b/refs", X_OK) = 0
> 20813 stat("su:b/../.git/modules/su:b", {st_mode=S_IFDIR|0755, st_size=280, ...}) = 0
> 20813 getcwd("/var/ram/git-stress/root-4/trash directory.t7814-grep-recurse-submodules/parent", 129) = 80
> 20869 <... lstat resumed> {st_mode=S_IFREG|0644, st_size=47, ...}) = 0
> 20813 chdir("su:b/../.git/modules/su:b") = 0
> 20869 open(".gitmodules", O_RDONLY <unfinished ...>
> 20813 getcwd( <unfinished ...>
> 20867 <... lstat resumed> 0x7f1df8254cf0) = -1 ENOENT (No such file or directory)
> 
> where 20813 and 20867 are two threads of the main process. One is doing
> the lstat and the other calls chdir at the same moment.
> 
> -Peff

Yeah so it looks like the start_command function calls chdir.  Which
means any uses of the run-command interface are not thread safe....

For now the work around could be to just pass "-C <dir>" to the child
process instead of relying on run-command to chdir.

-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Stefan Beller]

On Wed, Nov 30, 2016 at 3:40 PM, Jeff King <peff@peff.net> wrote:
> On Wed, Nov 30, 2016 at 06:32:04PM -0500, Jeff King wrote:
>
>> On Wed, Nov 30, 2016 at 03:28:23PM -0800, Brandon Williams wrote:
>>
>> > So I couldn't find a race condition in the code.  I tracked the problem
>> > to grep_source_load_file which attempts to run lstat on the file so that
>> > it can read it into a buffer.  The lstat call fails with ENOENT (which
>> > conveniently is skipped by the if statement which calls error_errno).  So
>> > for some reason the file cannot be found and read into memory resulting
>> > in nothing being grep'ed for that particular file (since the buffer is
>> > NULL).
>>
>> That's definitely weird. Is it possible that any of the underlying calls
>> from another thread are using chdir()? I think realpath() make do that
>> behind the scenes, and there may be others.
>>
>> A full strace from a failing case would be interesting reading. In
>> theory we should be able to get that by running the stress script for
>> long enough. :)
>
> Actually, it failed pretty much immediately. I guess the extra stracing
> changes the timing to make the problem _more_ likely.
>
> And indeed, I see:
>
> 20867 lstat("fi:le",  <unfinished ...>
> 20813 <... read resumed> "", 232)       = 0
> 20871 futex(0x558cdec8b164, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished ...>
> 20813 close(7 <unfinished ...>
> 20870 <... futex resumed> )             = 0
> 20869 lstat(".gitmodules",  <unfinished ...>
> 20813 <... close resumed> )             = 0
> 20865 set_robust_list(0x7f1df92579e0, 24 <unfinished ...>
> 20813 lstat("su:b/../.git/modules/su:b/commondir", 0x7ffecc8b3ac0) = -1 ENOENT (No such file or directory)
> 20865 <... set_robust_list resumed> )   = 0
> 20868 set_robust_list(0x7f1df7a549e0, 24 <unfinished ...>
> 20813 access("su:b/../.git/modules/su:b/objects", X_OK) = 0
> 20813 access("su:b/../.git/modules/su:b/refs", X_OK) = 0
> 20813 stat("su:b/../.git/modules/su:b", {st_mode=S_IFDIR|0755, st_size=280, ...}) = 0
> 20813 getcwd("/var/ram/git-stress/root-4/trash directory.t7814-grep-recurse-submodules/parent", 129) = 80
> 20869 <... lstat resumed> {st_mode=S_IFREG|0644, st_size=47, ...}) = 0
> 20813 chdir("su:b/../.git/modules/su:b") = 0
> 20869 open(".gitmodules", O_RDONLY <unfinished ...>
> 20813 getcwd( <unfinished ...>
> 20867 <... lstat resumed> 0x7f1df8254cf0) = -1 ENOENT (No such file or directory)
>
> where 20813 and 20867 are two threads of the main process. One is doing
> the lstat and the other calls chdir at the same moment.
>

Lessons learned here:
The run-command API is not thread safe when used with setting the directory.
If you need to run a thing in a threaded environment run
git -C <dir> ... such that the child chdirs.

Are there any other threaded environments that run things with .dir set?

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Wed, Nov 30, 2016 at 03:42:48PM -0800, Brandon Williams wrote:

> > where 20813 and 20867 are two threads of the main process. One is doing
> > the lstat and the other calls chdir at the same moment.
>
> Yeah so it looks like the start_command function calls chdir.  Which
> means any uses of the run-command interface are not thread safe....

That seems crazy. The chdir should be happening on the child side of the
fork (and looking at the code, it seems to be the case). And on the
Windows side, without fork, it's an option to the spawn call, which
makes sense.

> For now the work around could be to just pass "-C <dir>" to the child
> process instead of relying on run-command to chdir.

Yeah, that would push it after the exec. I just don't understand why
that would be necessary.

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/30, Jeff King wrote:
> On Wed, Nov 30, 2016 at 03:42:48PM -0800, Brandon Williams wrote:
> 
> > > where 20813 and 20867 are two threads of the main process. One is doing
> > > the lstat and the other calls chdir at the same moment.
> >
> > Yeah so it looks like the start_command function calls chdir.  Which
> > means any uses of the run-command interface are not thread safe....
> 
> That seems crazy. The chdir should be happening on the child side of the
> fork (and looking at the code, it seems to be the case). And on the
> Windows side, without fork, it's an option to the spawn call, which
> makes sense.
> 
> > For now the work around could be to just pass "-C <dir>" to the child
> > process instead of relying on run-command to chdir.
> 
> Yeah, that would push it after the exec. I just don't understand why
> that would be necessary.
> 
> -Peff

You're right, I jumped the gun.  That doesn't seem to fix the problem
(as I'm still seeing the same failure).


-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Wed, Nov 30, 2016 at 06:46:36PM -0500, Jeff King wrote:

> > For now the work around could be to just pass "-C <dir>" to the child
> > process instead of relying on run-command to chdir.
> 
> Yeah, that would push it after the exec. I just don't understand why
> that would be necessary.

Hmm. It still seems to fail, even with the workaround (the patch in
run-command is there to make sure there's not some other call that we're
not catching):

diff --git a/builtin/grep.c b/builtin/grep.c
index 2c727ef49..3323a3e7f 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -553,6 +553,7 @@ static int grep_submodule_launch(struct grep_opt *opt,
 	argv_array_pushf(&cp.args, "--super-prefix=%s%s/",
 			 super_prefix ? super_prefix : "",
 			 name);
+	argv_array_pushl(&cp.args, "-C", gs->path, NULL);
 	argv_array_push(&cp.args, "grep");
 
 	/*
@@ -586,7 +587,6 @@ static int grep_submodule_launch(struct grep_opt *opt,
 	}
 
 	cp.git_cmd = 1;
-	cp.dir = gs->path;
 
 	/*
 	 * Capture output to output buffer and check the return code from the
diff --git a/run-command.c b/run-command.c
index 5a4dbb66d..d040f4f77 100644
--- a/run-command.c
+++ b/run-command.c
@@ -393,6 +393,8 @@ int start_command(struct child_process *cmd)
 			close(cmd->out);
 		}
 
+		if (cmd->dir && git_env_bool("GIT_NO_CHDIR", 0))
+			die("temporarily disallowing chdir");
 		if (cmd->dir && chdir(cmd->dir))
 			die_errno("exec '%s': cd to '%s' failed", cmd->argv[0],
 			    cmd->dir);
diff --git a/t/t7814-grep-recurse-submodules.sh b/t/t7814-grep-recurse-submodules.sh
index 050777186..591ff74ed 100755
--- a/t/t7814-grep-recurse-submodules.sh
+++ b/t/t7814-grep-recurse-submodules.sh
@@ -175,7 +175,7 @@ test_expect_success 'grep recurse submodule colon in name' '
 	fi:le:foobar
 	su:b/fi:le:foobar
 	EOF
-	git -C parent grep -e "foobar" --recurse-submodules >actual &&
+	GIT_NO_CHDIR=1 strace -o foo.out -f git -C parent grep -e "foobar" --recurse-submodules >actual &&
 	test_cmp expect actual &&
 
 	cat >expect <<-\EOF &&


So I think there is some other chdir(). I'm not sure if there is an easy
way to get a backtrace on every call to chdir() in every thread. I'm
sure somebody more clever than me could figure out how to make gdb do it
automatically, but it might be workable manually. I think the chdir was
in the main thread.

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Wed, Nov 30, 2016 at 06:59:52PM -0500, Jeff King wrote:

> So I think there is some other chdir(). I'm not sure if there is an easy
> way to get a backtrace on every call to chdir() in every thread. I'm
> sure somebody more clever than me could figure out how to make gdb do it
> automatically, but it might be workable manually. I think the chdir was
> in the main thread.

Ah, that turned out to be quite easy. The culprit is probably:

(gdb) bt
#0  chdir () at ../sysdeps/unix/syscall-template.S:84
#1  0x00005555555fe259 in real_path_internal (path=0x5555559f6b30 "su:b/../.git/modules/su:b", die_on_error=1)
    at abspath.c:84
#2  0x00005555555fe48a in real_path (path=0x5555559f6b30 "su:b/../.git/modules/su:b") at abspath.c:135
#3  0x00005555556d09e6 in read_gitfile_gently (path=0x5555559f6ac0 "su:b/.git", return_error_code=0x0)
    at setup.c:555
#4  0x00005555556d19cf in resolve_gitdir (suspect=0x5555559f6ac0 "su:b/.git") at setup.c:1021
#5  0x00005555556e7e34 in is_submodule_populated (path=0x5555559f5ec8 "su:b") at submodule.c:244
#6  0x00005555555a0f05 in grep_submodule (opt=0x7fffffffd8b0, sha1=0x0, filename=0x5555559f5ec8 "su:b", 
    path=0x5555559f5ec8 "su:b") at builtin/grep.c:619
#7  0x00005555555a12ac in grep_cache (opt=0x7fffffffd8b0, pathspec=0x7fffffffd880, cached=0) at builtin/grep.c:700
#8  0x00005555555a36cb in cmd_grep (argc=0, argv=0x7fffffffdf40, prefix=0x0) at builtin/grep.c:1257
#9  0x000055555556603b in run_builtin (p=0x5555559b3ad8 <commands+984>, argc=4, argv=0x7fffffffdf40) at git.c:373
#10 0x00005555555662bc in handle_builtin (argc=4, argv=0x7fffffffdf40) at git.c:572
#11 0x000055555556641a in run_argv (argcp=0x7fffffffddfc, argv=0x7fffffffddf0) at git.c:630
#12 0x00005555555665a8 in cmd_main (argc=4, argv=0x7fffffffdf40) at git.c:702
#13 0x00005555555fde47 in main (argc=7, argv=0x7fffffffdf28) at common-main.c:40

So is_submodule_populated() needs to take a lock. But what's really
gross is that the _other_ threads need to lock just to call lstat().
Presumably it could be done as a reader/writer type of lock where many
"reader" threads can take the "I need to lstat()" lock simultaneously,
but block when an "I'm going to chdir()" writer holds it.

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/30, Jeff King wrote:
> So I think there is some other chdir(). I'm not sure if there is an easy
> way to get a backtrace on every call to chdir() in every thread. I'm
> sure somebody more clever than me could figure out how to make gdb do it
> automatically, but it might be workable manually. I think the chdir was
> in the main thread.
> 
> -Peff

Yeah maybe we're missing something else...

How did you run strace with your stress script?

-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/30, Jeff King wrote:
> On Wed, Nov 30, 2016 at 06:59:52PM -0500, Jeff King wrote:
> 
> > So I think there is some other chdir(). I'm not sure if there is an easy
> > way to get a backtrace on every call to chdir() in every thread. I'm
> > sure somebody more clever than me could figure out how to make gdb do it
> > automatically, but it might be workable manually. I think the chdir was
> > in the main thread.
> 
> Ah, that turned out to be quite easy. The culprit is probably:
> 
> (gdb) bt
> #0  chdir () at ../sysdeps/unix/syscall-template.S:84
> #1  0x00005555555fe259 in real_path_internal (path=0x5555559f6b30 "su:b/../.git/modules/su:b", die_on_error=1)
>     at abspath.c:84
> #2  0x00005555555fe48a in real_path (path=0x5555559f6b30 "su:b/../.git/modules/su:b") at abspath.c:135
> #3  0x00005555556d09e6 in read_gitfile_gently (path=0x5555559f6ac0 "su:b/.git", return_error_code=0x0)
>     at setup.c:555
> #4  0x00005555556d19cf in resolve_gitdir (suspect=0x5555559f6ac0 "su:b/.git") at setup.c:1021
> #5  0x00005555556e7e34 in is_submodule_populated (path=0x5555559f5ec8 "su:b") at submodule.c:244
> #6  0x00005555555a0f05 in grep_submodule (opt=0x7fffffffd8b0, sha1=0x0, filename=0x5555559f5ec8 "su:b", 
>     path=0x5555559f5ec8 "su:b") at builtin/grep.c:619
> #7  0x00005555555a12ac in grep_cache (opt=0x7fffffffd8b0, pathspec=0x7fffffffd880, cached=0) at builtin/grep.c:700
> #8  0x00005555555a36cb in cmd_grep (argc=0, argv=0x7fffffffdf40, prefix=0x0) at builtin/grep.c:1257
> #9  0x000055555556603b in run_builtin (p=0x5555559b3ad8 <commands+984>, argc=4, argv=0x7fffffffdf40) at git.c:373
> #10 0x00005555555662bc in handle_builtin (argc=4, argv=0x7fffffffdf40) at git.c:572
> #11 0x000055555556641a in run_argv (argcp=0x7fffffffddfc, argv=0x7fffffffddf0) at git.c:630
> #12 0x00005555555665a8 in cmd_main (argc=4, argv=0x7fffffffdf40) at git.c:702
> #13 0x00005555555fde47 in main (argc=7, argv=0x7fffffffdf28) at common-main.c:40
> 
> So is_submodule_populated() needs to take a lock. But what's really
> gross is that the _other_ threads need to lock just to call lstat().
> Presumably it could be done as a reader/writer type of lock where many
> "reader" threads can take the "I need to lstat()" lock simultaneously,
> but block when an "I'm going to chdir()" writer holds it.
> 
> -Peff

Oh interesting, I wonder if there is a way to not have to perform a
chdir since taking a lock to lstat wouldn't be ideal.

Thanks for helping out with this!

-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Stefan Beller]

> Oh interesting, I wonder if there is a way to not have to perform a
> chdir since taking a lock to lstat wouldn't be ideal.

I think we could rewrite is_submodule_populated to be

int is_submodule_populated_cheap_with_no_chdir(char *path)
{
    return stat(path + ".git")
}

i.e. just take the presence of the .git file/dir as a hint to run
the child process?

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Wed, Nov 30, 2016 at 04:06:05PM -0800, Brandon Williams wrote:

> On 11/30, Jeff King wrote:
> > So I think there is some other chdir(). I'm not sure if there is an easy
> > way to get a backtrace on every call to chdir() in every thread. I'm
> > sure somebody more clever than me could figure out how to make gdb do it
> > automatically, but it might be workable manually. I think the chdir was
> > in the main thread.
> > 
> > -Peff
> 
> Yeah maybe we're missing something else...
> 
> How did you run strace with your stress script?

It's hidden in the patch I sent a moment ago, but basically just "strace
-o foo.out" will dump the trace in the trash directory. After the stress
script runs, you can "cat fail/trash*/foo.out".

> > (gdb) bt
> > #0  chdir () at ../sysdeps/unix/syscall-template.S:84
> > #1  0x00005555555fe259 in real_path_internal (path=0x5555559f6b30 "su:b/../.git/modules/su:b", die_on_error=1)
> >     at abspath.c:84
> > #2  0x00005555555fe48a in real_path (path=0x5555559f6b30 "su:b/../.git/modules/su:b") at abspath.c:135
> > #3  0x00005555556d09e6 in read_gitfile_gently (path=0x5555559f6ac0 "su:b/.git", return_error_code=0x0)
> >     at setup.c:555
> > #4  0x00005555556d19cf in resolve_gitdir (suspect=0x5555559f6ac0 "su:b/.git") at setup.c:1021
> > #5  0x00005555556e7e34 in is_submodule_populated (path=0x5555559f5ec8 "su:b") at submodule.c:244
> > #6  0x00005555555a0f05 in grep_submodule (opt=0x7fffffffd8b0, sha1=0x0, filename=0x5555559f5ec8 "su:b", 
> >     path=0x5555559f5ec8 "su:b") at builtin/grep.c:619
> > #7  0x00005555555a12ac in grep_cache (opt=0x7fffffffd8b0, pathspec=0x7fffffffd880, cached=0) at builtin/grep.c:700
> > #8  0x00005555555a36cb in cmd_grep (argc=0, argv=0x7fffffffdf40, prefix=0x0) at builtin/grep.c:1257
> > #9  0x000055555556603b in run_builtin (p=0x5555559b3ad8 <commands+984>, argc=4, argv=0x7fffffffdf40) at git.c:373
> > #10 0x00005555555662bc in handle_builtin (argc=4, argv=0x7fffffffdf40) at git.c:572
> > #11 0x000055555556641a in run_argv (argcp=0x7fffffffddfc, argv=0x7fffffffddf0) at git.c:630
> > #12 0x00005555555665a8 in cmd_main (argc=4, argv=0x7fffffffdf40) at git.c:702
> > #13 0x00005555555fde47 in main (argc=7, argv=0x7fffffffdf28) at common-main.c:40
> > 
> > So is_submodule_populated() needs to take a lock. But what's really
> > gross is that the _other_ threads need to lock just to call lstat().
> > Presumably it could be done as a reader/writer type of lock where many
> > "reader" threads can take the "I need to lstat()" lock simultaneously,
> > but block when an "I'm going to chdir()" writer holds it.
> 
> Oh interesting, I wonder if there is a way to not have to perform a
> chdir since taking a lock to lstat wouldn't be ideal.

I don't think so.  It comes from real_path(), which needs to either
chdir(), or start interpreting symbolic links itself (and madness that
way lies).

I think with a reader/writer lock as I described it wouldn't be too bad.
The common case would pay only the locking cost and not ever block,
since submodules are rare (and they're super-heavyweight to descend into
anyway).

I think putting it at the individual lstat() would be way too low, but
probably you could do it right before calling grep_source(). It may even
be possible to do some of the submodule work ahead of time while holding
grep_lock().

> Thanks for helping out with this!

I wasn't planning on it, but this turned into an intriguing puzzle. ;)

-Peff

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Brandon Williams]

On 11/30, Stefan Beller wrote:
> > Oh interesting, I wonder if there is a way to not have to perform a
> > chdir since taking a lock to lstat wouldn't be ideal.
> 
> I think we could rewrite is_submodule_populated to be
> 
> int is_submodule_populated_cheap_with_no_chdir(char *path)
> {
>     return stat(path + ".git")
> }
> 
> i.e. just take the presence of the .git file/dir as a hint to run
> the child process?

I like this approach, its a quick (thread-safe) check to see if the
submodule is interesting.  If there happens to be an error with the
submodule's .git file/directory then the child process will fail out.

-- 
Brandon Williams

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Johannes Sixt]

Am 01.12.2016 um 00:40 schrieb Jeff King:
> 20813 access("su:b/../.git/modules/su:b/refs", X_OK) = 0

Side note: where does this weirdness "su:b" come from? (I haven't looked 
at any of the patches, yet, let alone tested.) Colons in file or 
directory names won't work on Windows (in the way one would expect).

-- Hannes

Re: What's cooking in git.git (Nov 2016, #06; Mon, 28)

[Jeff King]

On Thu, Dec 01, 2016 at 08:09:16AM +0100, Johannes Sixt wrote:

> Am 01.12.2016 um 00:40 schrieb Jeff King:
> > 20813 access("su:b/../.git/modules/su:b/refs", X_OK) = 0
> 
> Side note: where does this weirdness "su:b" come from? (I haven't looked at
> any of the patches, yet, let alone tested.) Colons in file or directory
> names won't work on Windows (in the way one would expect).

It's explicitly used in the test, I assume to check that the recursive
grep is not confused into treating the name as a tree-ish.

I think it would have to either be marked with a prereq on Windows, or
modified to do the whole thing in-index (and use "grep --cached").

-Peff

bw/transport-protocol-policy

[Jeff King]

On Mon, Nov 28, 2016 at 04:15:08PM -0800, Junio C Hamano wrote:

> * bw/transport-protocol-policy (2016-11-09) 2 commits
>   (merged to 'next' on 2016-11-16 at 1391d3eeed)
>  + transport: add protocol policy config option
>  + lib-proto-disable: variable name fix
> 
>  Finer-grained control of what protocols are allowed for transports
>  during clone/fetch/push have been enabled via a new configuration
>  mechanism.
> 
>  Will cook in 'next'.

I was looking at the way the http code feeds protocol restrictions to
CURLOPT_REDIR_PROTOCOLS, and I think this topic is missing two elements:

  1. The new policy config lets you say "only allow this protocol when
     the user specifies it". But when http.c calls is_transport_allowed(),
     the latter has no idea that we are asking it about potential
     redirects (which obviously do _not_ come from the user), and would
     erroneously allow them.

     I think this needs fixed before the topic is merged. It's not a
     regression, as it only comes into play if you use the new policy
     config. But it is a minor security hole in the new feature.

  2. If your curl is too old to support CURLOPT_REDIR_PROTOCOLS, we will
     warn if there is a protocol whitelist in effect. But that check
     only covers the environment whitelist, and we do not warn if you
     restrict other protocols.

     I actually think this should probably just warn indiscriminately.
     Even without a Git protocol whitelist specified, the code serves to
     prevent curl from redirecting to bizarre protocols like smtp. The
     affected curl versions are from 2009 and prior, so I kind of doubt
     it matters much either way (I'm actually tempted to suggest we bump
     the minimum curl version there; there's a ton of #ifdef cruft going
     back to 2002-era versions of libcurl).

-Peff

Re: bw/transport-protocol-policy

[Brandon Williams]

On 12/01, Jeff King wrote:
> On Mon, Nov 28, 2016 at 04:15:08PM -0800, Junio C Hamano wrote:
> 
> > * bw/transport-protocol-policy (2016-11-09) 2 commits
> >   (merged to 'next' on 2016-11-16 at 1391d3eeed)
> >  + transport: add protocol policy config option
> >  + lib-proto-disable: variable name fix
> > 
> >  Finer-grained control of what protocols are allowed for transports
> >  during clone/fetch/push have been enabled via a new configuration
> >  mechanism.
> > 
> >  Will cook in 'next'.
> 
> I was looking at the way the http code feeds protocol restrictions to
> CURLOPT_REDIR_PROTOCOLS, and I think this topic is missing two elements:
> 
>   1. The new policy config lets you say "only allow this protocol when
>      the user specifies it". But when http.c calls is_transport_allowed(),
>      the latter has no idea that we are asking it about potential
>      redirects (which obviously do _not_ come from the user), and would
>      erroneously allow them.
> 
>      I think this needs fixed before the topic is merged. It's not a
>      regression, as it only comes into play if you use the new policy
>      config. But it is a minor security hole in the new feature.

I agree and it should be an easy fix.  We can just add a parameter like
so:

diff --git a/transport.c b/transport.c
index 2c0ec76..d38d50f 100644
--- a/transport.c
+++ b/transport.c
@@ -723,7 +723,7 @@ static enum protocol_allow_config get_protocol_config(const char *type)
 	return PROTOCOL_ALLOW_USER_ONLY;
 }
 
-int is_transport_allowed(const char *type)
+int is_transport_allowed(const char *type, int redirect)
 {
 	const struct string_list *whitelist = protocol_whitelist();
 	if (whitelist)
@@ -735,7 +735,7 @@ int is_transport_allowed(const char *type)
 	case PROTOCOL_ALLOW_NEVER:
 		return 0;
 	case PROTOCOL_ALLOW_USER_ONLY:
-		return git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
+		return git_env_bool("GIT_PROTOCOL_FROM_USER", !redirect);
 	}
 
 	die("BUG: invalid protocol_allow_config type");

That way the libcurl code can say it is asking if it is ok to redirect
to that protocol.

> 
>   2. If your curl is too old to support CURLOPT_REDIR_PROTOCOLS, we will
>      warn if there is a protocol whitelist in effect. But that check
>      only covers the environment whitelist, and we do not warn if you
>      restrict other protocols.
> 
>      I actually think this should probably just warn indiscriminately.
>      Even without a Git protocol whitelist specified, the code serves to
>      prevent curl from redirecting to bizarre protocols like smtp. The
>      affected curl versions are from 2009 and prior, so I kind of doubt
>      it matters much either way (I'm actually tempted to suggest we bump
>      the minimum curl version there; there's a ton of #ifdef cruft going
>      back to 2002-era versions of libcurl).

We should switch to warning all the time since this series adds in
default whitelisted/blacklisted protocols anyways.

-- 
Brandon Williams

Re: bw/transport-protocol-policy

[Jeff King]

On Thu, Dec 01, 2016 at 10:14:15AM -0800, Brandon Williams wrote:

> >   1. The new policy config lets you say "only allow this protocol when
> >      the user specifies it". But when http.c calls is_transport_allowed(),
> >      the latter has no idea that we are asking it about potential
> >      redirects (which obviously do _not_ come from the user), and would
> >      erroneously allow them.
> > 
> >      I think this needs fixed before the topic is merged. It's not a
> >      regression, as it only comes into play if you use the new policy
> >      config. But it is a minor security hole in the new feature.
> 
> I agree and it should be an easy fix.  We can just add a parameter like
> so:
> 
> diff --git a/transport.c b/transport.c
> index 2c0ec76..d38d50f 100644
> --- a/transport.c
> +++ b/transport.c
> @@ -723,7 +723,7 @@ static enum protocol_allow_config get_protocol_config(const char *type)
>  	return PROTOCOL_ALLOW_USER_ONLY;
>  }
>  
> -int is_transport_allowed(const char *type)
> +int is_transport_allowed(const char *type, int redirect)
>  {
>  	const struct string_list *whitelist = protocol_whitelist();
>  	if (whitelist)
> @@ -735,7 +735,7 @@ int is_transport_allowed(const char *type)
>  	case PROTOCOL_ALLOW_NEVER:
>  		return 0;
>  	case PROTOCOL_ALLOW_USER_ONLY:
> -		return git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
> +		return git_env_bool("GIT_PROTOCOL_FROM_USER", !redirect);
>  	}
>  
>  	die("BUG: invalid protocol_allow_config type");
> 
> That way the libcurl code can say it is asking if it is ok to redirect
> to that protocol.

I wouldn't expect anyone to ever set GIT_PROTOCOL_FROM_USER=1, but it
does behave in a funny way here, overriding the "redirect" flag. I think
we'd want something more like:

  if (redirect < 0)
	redirect = git_env_bool("GIT_PROTOCOL_FROM_USER", 1);

and then pass in "-1" from transport_check_allowed().

I think that's sufficient to fix the topic as-is. However, the http
redirect series adds an extra complication, because with http-alternates
we resolve some of the redirects ourselves. So in those cases we'd want
to restrict CURLOPT_PROTOCOLS as if they were redirects. We may need to
set up two CURLOPT values: ones from the user and ones from redirects,
and then feed them to CURLOPT_PROTOCOLS and CURLOPT_REDIR_PROTOCOLS as
appropriate depending on the request context.

> We should switch to warning all the time since this series adds in
> default whitelisted/blacklisted protocols anyways.

Yeah, good point. As a bonus it makes the code simpler.

-Peff

Re: bw/transport-protocol-policy

[Brandon Williams]

On 12/01, Jeff King wrote:
> On Thu, Dec 01, 2016 at 10:14:15AM -0800, Brandon Williams wrote:
> 
> > >   1. The new policy config lets you say "only allow this protocol when
> > >      the user specifies it". But when http.c calls is_transport_allowed(),
> > >      the latter has no idea that we are asking it about potential
> > >      redirects (which obviously do _not_ come from the user), and would
> > >      erroneously allow them.
> > > 
> > >      I think this needs fixed before the topic is merged. It's not a
> > >      regression, as it only comes into play if you use the new policy
> > >      config. But it is a minor security hole in the new feature.
> > 
> > I agree and it should be an easy fix.  We can just add a parameter like
> > so:
> > 
> > diff --git a/transport.c b/transport.c
> > index 2c0ec76..d38d50f 100644
> > --- a/transport.c
> > +++ b/transport.c
> > @@ -723,7 +723,7 @@ static enum protocol_allow_config get_protocol_config(const char *type)
> >  	return PROTOCOL_ALLOW_USER_ONLY;
> >  }
> >  
> > -int is_transport_allowed(const char *type)
> > +int is_transport_allowed(const char *type, int redirect)
> >  {
> >  	const struct string_list *whitelist = protocol_whitelist();
> >  	if (whitelist)
> > @@ -735,7 +735,7 @@ int is_transport_allowed(const char *type)
> >  	case PROTOCOL_ALLOW_NEVER:
> >  		return 0;
> >  	case PROTOCOL_ALLOW_USER_ONLY:
> > -		return git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
> > +		return git_env_bool("GIT_PROTOCOL_FROM_USER", !redirect);
> >  	}
> >  
> >  	die("BUG: invalid protocol_allow_config type");
> > 
> > That way the libcurl code can say it is asking if it is ok to redirect
> > to that protocol.
> 
> I wouldn't expect anyone to ever set GIT_PROTOCOL_FROM_USER=1, but it
> does behave in a funny way here, overriding the "redirect" flag. I think
> we'd want something more like:
> 
>   if (redirect < 0)
> 	redirect = git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
> 
> and then pass in "-1" from transport_check_allowed().

I don't think I quite follow your solution but I came up with this:

  case PROTOCOL_ALLOW_USER_ONLY:
    return redirect ? 0 : git_env_bool("GIT_PROTOCOL_FROM_USER", 1);

Which should address the same issue.

-- 
Brandon Williams

Re: bw/transport-protocol-policy

[Jeff King]

On Thu, Dec 01, 2016 at 11:35:24AM -0800, Brandon Williams wrote:

> > I wouldn't expect anyone to ever set GIT_PROTOCOL_FROM_USER=1, but it
> > does behave in a funny way here, overriding the "redirect" flag. I think
> > we'd want something more like:
> > 
> >   if (redirect < 0)
> > 	redirect = git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
> > 
> > and then pass in "-1" from transport_check_allowed().
> 
> I don't think I quite follow your solution but I came up with this:
> 
>   case PROTOCOL_ALLOW_USER_ONLY:
>     return redirect ? 0 : git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
> 
> Which should address the same issue.

I think mine was confused a bit by using the word "redirect". It was
really meant to be "from_user", and could take three values: definitely
yes, definitely no, and unknown (-1). And in the unknown case, we pull
the value from the environment.

Yours combines "definitely no" and "unknown" into a single value ("1" in
your case, but that is because "redirect" and "from_user" have inverted
logic from each other).

I think that is OK, as there isn't any case where a caller would want to
say "definitely no". The most they would say is "_I_ am not doing
anything to make you think this value is not from the user", but we
would still want to check the environment to see that nobody _else_ had
put in such a restriction.

-Peff

Re: bw/transport-protocol-policy

[Brandon Williams]

On 12/01, Jeff King wrote:
> On Thu, Dec 01, 2016 at 11:35:24AM -0800, Brandon Williams wrote:
> 
> > > I wouldn't expect anyone to ever set GIT_PROTOCOL_FROM_USER=1, but it
> > > does behave in a funny way here, overriding the "redirect" flag. I think
> > > we'd want something more like:
> > > 
> > >   if (redirect < 0)
> > > 	redirect = git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
> > > 
> > > and then pass in "-1" from transport_check_allowed().
> > 
> > I don't think I quite follow your solution but I came up with this:
> > 
> >   case PROTOCOL_ALLOW_USER_ONLY:
> >     return redirect ? 0 : git_env_bool("GIT_PROTOCOL_FROM_USER", 1);
> > 
> > Which should address the same issue.
> 
> I think mine was confused a bit by using the word "redirect". It was
> really meant to be "from_user", and could take three values: definitely
> yes, definitely no, and unknown (-1). And in the unknown case, we pull
> the value from the environment.
> 
> Yours combines "definitely no" and "unknown" into a single value ("1" in
> your case, but that is because "redirect" and "from_user" have inverted
> logic from each other).
> 
> I think that is OK, as there isn't any case where a caller would want to
> say "definitely no". The most they would say is "_I_ am not doing
> anything to make you think this value is not from the user", but we
> would still want to check the environment to see that nobody _else_ had
> put in such a restriction.
> 
> -Peff

Oh ok, I see what you were going for now.  That may be a better
(clearer) solution then what I just sent out.  Mostly because we can
maintain the same logic polarity and use the same vocabulary.

-- 
Brandon Williams