[PATCH] specify encoding for sed command

[PATCH] specify encoding for sed command

[Stephon Harris]

Fixes issue with seeing `sed: RE error: illegal byte sequence` when running git-completion.bash
---
 contrib/completion/git-completion.bash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index b09c8a23626b4..52a4ab5e2165a 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -282,7 +282,7 @@ __gitcomp ()
 
 # Clear the variables caching builtins' options when (re-)sourcing
 # the completion script.
-unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
+unset $(set |LANG=C sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
 
 # This function is equivalent to
 #

--
https://github.com/git/git/pull/481

Re: [PATCH] specify encoding for sed command

[Ævar Arnfjörð Bjarmason]

On Thu, Apr 05 2018, Stephon Harris wrote:

> Fixes issue with seeing `sed: RE error: illegal byte sequence` when running git-completion.bash
> ---
>  contrib/completion/git-completion.bash | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
> index b09c8a23626b4..52a4ab5e2165a 100644
> --- a/contrib/completion/git-completion.bash
> +++ b/contrib/completion/git-completion.bash
> @@ -282,7 +282,7 @@ __gitcomp ()
>
>  # Clear the variables caching builtins' options when (re-)sourcing
>  # the completion script.
> -unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
> +unset $(set |LANG=C sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null

This is getting closer to the issue than your previous patch, but
there's still some open questions:

1) What platform OS / version / sed version is this on?

2) What's the output from "set" that's causing this error? Do we have an
   isolated test case for that?

3) There's other invocations of "sed" in the file, aren't those affected
   as well?

4) Any reason we wouldn't just set LC_AlL=C for the whole file? I see we
   already do it for our invocation to "git merge".

Re: [PATCH] specify encoding for sed command

[Eric Sunshine]

On Thu, Apr 5, 2018 at 2:53 AM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Thu, Apr 05 2018, Stephon Harris wrote:
>> Fixes issue with seeing `sed: RE error: illegal byte sequence` when running git-completion.bash
>
> This is getting closer to the issue than your previous patch, but
> there's still some open questions:
>
> 1) What platform OS / version / sed version is this on?
>
> 2) What's the output from "set" that's causing this error? Do we have an
>    isolated test case for that?
>
> 3) There's other invocations of "sed" in the file, aren't those affected
>    as well?
>
> 4) Any reason we wouldn't just set LC_AlL=C for the whole file? I see we
>    already do it for our invocation to "git merge".

Also, missing Signed-off-by: (see Documentation/SubmittingPatches).

Thanks.

Re: [PATCH] specify encoding for sed command

[SZEDER Gábor]

On Thu, Apr 5, 2018 at 8:53 AM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> On Thu, Apr 05 2018, Stephon Harris wrote:
>
>> Fixes issue with seeing `sed: RE error: illegal byte sequence` when running git-completion.bash

Please:
  - prefix the subject line with "completion:".
  - nit: replace "running" with "sourcing".
  - wrap the body of the commit message around 70 characters.
  - sign off your patch; see Documentation/SubmittingPatches and 'git
    commit -s'.

>> ---
>>  contrib/completion/git-completion.bash | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
>> index b09c8a23626b4..52a4ab5e2165a 100644
>> --- a/contrib/completion/git-completion.bash
>> +++ b/contrib/completion/git-completion.bash
>> @@ -282,7 +282,7 @@ __gitcomp ()
>>
>>  # Clear the variables caching builtins' options when (re-)sourcing
>>  # the completion script.
>> -unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
>> +unset $(set |LANG=C sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null

I was wondering how could you see an error message from 'sed', when the
stderr of the whole thing is redirected to /dev/null.  It turns out that
such a redirection doesn't affect the stderr of the commands executed
inside the command substitution:

  $ echo $(echo foo ; echo >&2 error) 2>/dev/null
  error
  foo

Interesting, I didn't know that.

> This is getting closer to the issue than your previous patch, but
> there's still some open questions:
>
> 1) What platform OS / version / sed version is this on?
>
> 2) What's the output from "set" that's causing this error? Do we have an
>    isolated test case for that?

A quick internet search suggests that this error message tends to appear
on Macs, when its 'sed' encounters an invalid UTF-8 character in its
input.

> 3) There's other invocations of "sed" in the file, aren't those affected
>    as well?

The two 'sed' invocations in _git_stash() are suspect, as they process
the output of 'git stash list', which includes the stashes'
descriptions, which can contain whatever the users wished to store there
(though they would probably get a "Warning: commit message did not
conform to UTF-8" message from 9e83266525 (commit-tree: encourage UTF-8
commit messages., 2006-12-22) when doing so).

The 'sed' invocation in __git_complete_revlist_file() is suspect as
well, as it processes the output of 'git ls-tree'.  Pathnames can
contain anything, and while 'git ls-tree' quotes pathnames with
"unusual" characters, I don't know whether it quotes all characters that
can possibly upset Stephon's 'sed'.

What about 'awk' on Stephon's platform?  We already have one 'awk'
invocation in __git_match_ctag(), which we use for 'git grep <TAB>' and
'git log -L:<TAB>'.  That 'awk' script uses a regexp to match the symbol
name in a ctags file; does any programming language allow such unusual
characters in symbol names?

What about 'sed' and 'awk' scripts that don't use regexps at all?  (I
intend to add such an 'awk' script in a day or two...)

> 4) Any reason we wouldn't just set LC_AlL=C for the whole file? I see we
>    already do it for our invocation to "git merge".

That would perhaps be the safest thing to do...
But how can we set it for a whole file, when the file is not executed as
a script but sourced into the user's shell environment?

Re: [PATCH] specify encoding for sed command

[Matt Coleman]

> 1) What platform OS / version / sed version is this on?
I'm experiencing this on macOS Sierra (10.12.6). The issue occurs with the OS's native sed, which is FreeBSD sed so the version number is kind of ambiguous.

The error goes away if I set LANG=C or LC_ALL=C or change it to use GNU sed (installed via homebrew as gsed).

> 2) What's the output from "set" that's causing this error? Do we have an
>   isolated test case for that?
On my system, it's caused by whatever powerline is doing to Bash's PS1 variable:
$ set | grep ^PS1= | sed 's/foo/bar/'
sed: RE error: illegal byte sequence

I'm not sure exactly which character is bad here:
$ xxd <<<$PS1
00000000: 5c5b 1b5b 303b 3338 3b32 3b32 3535 3b32  \[.[0;38;2;255;2
00000010: 3535 3b32 3535 3b34 383b 323b 303b 3133  55;255;48;2;0;13
00000020: 353b 3137 353b 316d 5c5d c2a0 6d61 7474  5;175;1m\]..matt
00000030: c2a0 5c5b 1b5b 303b 3338 3b32 3b30 3b31  ..\[.[0;38;2;0;1
00000040: 3335 3b31 3735 3b34 383b 323b 3838 3b38  35;175;48;2;88;8
00000050: 383b 3838 3b32 326d 5c5d ee82 b0c2 a05c  8;88;22m\].....\
00000060: 5b1b 5b30 3b33 383b 323b 3230 383b 3230  [.[0;38;2;208;20
00000070: 383b 3230 383b 3438 3b32 3b38 383b 3838  8;208;48;2;88;88
00000080: 3b38 383b 316d 5c5d 7ec2 a05c 5b1b 5b30  ;88;1m\]~..\[.[0
00000090: 3b33 383b 323b 3133 383b 3133 383b 3133  ;38;2;138;138;13
000000a0: 383b 3438 3b32 3b38 383b 3838 3b38 383b  8;48;2;88;88;88;
000000b0: 3232 6d5c 5dee 82b1 c2a0 5c5b 1b5b 303b  22m\].....\[.[0;
000000c0: 3338 3b32 3b32 3038 3b32 3038 3b32 3038  38;2;208;208;208
000000d0: 3b34 383b 323b 3838 3b38 383b 3838 3b31  ;48;2;88;88;88;1
000000e0: 6d5c 5dc2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2  m\].............
000000f0: a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2  ................
00000100: a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2  ................
00000110: a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2  ................
00000120: a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2  ................
00000130: a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a0c2 a05c  ...............\
00000140: 5b1b 5b30 3b33 383b 323b 3438 3b34 383b  [.[0;38;2;48;48;
00000150: 3438 3b34 383b 323b 3838 3b38 383b 3838  48;48;2;88;88;88
00000160: 3b32 326d 5c5d c2a0 ee82 b25c 5b1b 5b30  ;22m\].....\[.[0
00000170: 3b33 383b 323b 3135 383b 3135 383b 3135  ;38;2;158;158;15
00000180: 383b 3438 3b32 3b34 383b 3438 3b34 386d  8;48;2;48;48;48m
00000190: 5c5d c2a0 3230 3138 2d30 342d 3130 5c5b  \]..2018-04-10\[
000001a0: 1b5b 303b 3338 3b32 3b39 383b 3938 3b39  .[0;38;2;98;98;9
000001b0: 383b 3438 3b32 3b34 383b 3438 3b34 383b  8;48;2;48;48;48;
000001c0: 3232 6d5c 5dc2 a0ee 82b3 5c5b 1b5b 303b  22m\].....\[.[0;
000001d0: 3338 3b32 3b32 3038 3b32 3038 3b32 3038  38;2;208;208;208
000001e0: 3b34 383b 323b 3438 3b34 383b 3438 3b31  ;48;2;48;48;48;1
000001f0: 6d5c 5dc2 a030 333a 3136 c2a0 5c5b 1b5b  m\]..03:16..\[.[
00000200: 306d 5c5d 205c 5b1b 5b30 3b33 383b 323b  0m\] \[.[0;38;2;
00000210: 3135 383b 3135 383b 3135 383b 3438 3b32  158;158;158;48;2
00000220: 3b34 383b 3438 3b34 386d 5c5d c2a0 c2a0  ;48;48;48m\]....
00000230: 5c5b 1b5b 303b 3338 3b32 3b34 383b 3438  \[.[0;38;2;48;48
00000240: 3b34 383b 3439 3b32 326d 5c5d ee82 b0c2  ;48;49;22m\]....
00000250: a05c 5b1b 5b30 6d5c 5d0a                 .\[.[0m\].

Re: [PATCH] specify encoding for sed command

[Matt Coleman]

I found another (possibly better) way to fix this:

> On Apr 10, 2018, at 3:18 AM, Matt Coleman <cat.moleman@gmail.com> wrote:
> 
>> 1) What platform OS / version / sed version is this on?
> I'm experiencing this on macOS Sierra (10.12.6). The issue occurs with the OS's native sed, which is FreeBSD sed so the version number is kind of ambiguous.
> 
> The error goes away if I set LANG=C or LC_ALL=C or change it to use GNU sed (installed via homebrew as gsed).

If I change it to use awk instead of sed, it works with mawk, gawk, and macOS awk:
unset $(set | awk -F '=' '/^__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*=/ {print $1}') 2>/dev/null

I compared sed vs. awk on Linux and Mac and they all take about the same amount of time to run (within 0.002ms).

Re: [PATCH] specify encoding for sed command

[Matthew Coleman]

I did a little more digging into this issue today.

> On Apr 11, 2018, at 4:42 PM, Matt Coleman <cat.moleman@gmail.com> wrote:
> 
> I found another (possibly better) way to fix this:
> 
>> On Apr 10, 2018, at 3:18 AM, Matt Coleman <cat.moleman@gmail.com> wrote:
>> 
>>> 1) What platform OS / version / sed version is this on?
>> I'm experiencing this on macOS Sierra (10.12.6). The issue occurs with the OS's native sed, which is FreeBSD sed so the version number is kind of ambiguous.
>> 
>> The error goes away if I set LANG=C or LC_ALL=C or change it to use GNU sed (installed via homebrew as gsed).
> 
> If I change it to use awk instead of sed, it works with mawk, gawk, and macOS awk:
> unset $(set | awk -F '=' '/^__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*=/ {print $1}') 2>/dev/null
> 
> I compared sed vs. awk on Linux and Mac and they all take about the same amount of time to run (within 0.002ms).

The issue isn't actually caused by `sed`: it's caused by the way that the `set` builtin outputs characters in the Unicode Private Use Area (PUA) in the build of Bash 3.2.57 that macOS Sierra ships with.

Powerline uses several PUA code points to make some of its pretty text UI elements:

Code point (hex value): description
U+E0A0 (0xEE82A0): Version control branch
U+E0A1 (0xEE82A1): LN (line) symbol
U+E0A2 (0xEE82A2): Closed padlock
U+E0B0 (0xEE82B0): Rightwards black arrowhead
U+E0B1 (0xEE82B1): Rightwards arrowhead
U+E0B2 (0xEE82B2): Leftwards black arrowhead
U+E0B3 (0xEE82B3): Leftwards arrowhead

macOS Bash 3.2.57's `set` builtin has garbled output where Powerline's special symbols should be in the PS1 variable, but Bash 4.4.19 (installed on macOS via homebrew) and Bash 4.3.38 (Ubuntu 16.04) both display it correctly in the output of `set`. `echo $PS1` does display the symbols correctly on these versions of Bash.

So...

> 3) There's other invocations of "sed" in the file, aren't those affected as well?
The short answer: no. Slightly longer: not by the same thing that's affecting the line in the patch, at least.

Long: As described above, the problem isn't actually `sed`: it's the `set` builtin in macOS's build of Bash. The other invocations of `sed` should be safe, because `sed` properly handles the PUA code points on its own:

$ echo $'\xee\x82\xb0' | sed 's/./@/'
@

The way that `set` is displaying the PS1 variable seems to be sending them to `sed` individually or somehow split up:

$ for character in $'\xee' $'\x82' $'\xb0' $'\xee\x82' $'\x82\xb0'; do echo $character | sed 's/./@/'; done
sed: RE error: illegal byte sequence
sed: RE error: illegal byte sequence
sed: RE error: illegal byte sequence
sed: RE error: illegal byte sequence
sed: RE error: illegal byte sequence

Interestingly, Bash 3.2.25's `set` builtin on CentOS 5 correctly displays the octal values for the symbols (prompt edited to keep this email ASCII):

$ PS1=$'\xee\x82\xb0 '
>  set | grep PS1
PS1=$'\356\202\260 '

I haven't started digging through Bash's codebase, but it could either be a bug that was introduced between 3.2.25 and 3.2.57 or Apple used some silly flags when compiling Bash. Ideally, this should be fixed in Bash, but since Apple's using such an old version of Bash for license reasons, I think it's unlikely that they'll fix the issue.

I think the best way to move forward with this is a new patch that uses `awk` instead of `sed`: I tested several `awk` variants and the command was portable without requiring any changes to LANG or LC_ALL.

Does that sound like a good plan?

Re: [PATCH] specify encoding for sed command

[SZEDER Gábor]

> I did a little more digging into this issue today.

> The issue isn't actually caused by `sed`: it's caused by the way that
> the `set` builtin outputs characters in the Unicode Private Use Area
> (PUA) in the build of Bash 3.2.57 that macOS Sierra ships with.
> 
> Powerline uses several PUA code points to make some of its pretty text
> UI elements:
> 
> Code point (hex value): description
> U+E0A0 (0xEE82A0): Version control branch
> U+E0A1 (0xEE82A1): LN (line) symbol
> U+E0A2 (0xEE82A2): Closed padlock
> U+E0B0 (0xEE82B0): Rightwards black arrowhead
> U+E0B1 (0xEE82B1): Rightwards arrowhead
> U+E0B2 (0xEE82B2): Leftwards black arrowhead
> U+E0B3 (0xEE82B3): Leftwards arrowhead
> 
> macOS Bash 3.2.57's `set` builtin has garbled output where Powerline's
> special symbols should be in the PS1 variable, but Bash 4.4.19
> (installed on macOS via homebrew) and Bash 4.3.38 (Ubuntu 16.04) both
> display it correctly in the output of `set`. `echo $PS1` does display
> the symbols correctly on these versions of Bash.

Thanks for digging.

> I think the best way to move forward with this is a new patch that uses
> `awk` instead of `sed`: I tested several `awk` variants and the command
> was portable without requiring any changes to LANG or LC_ALL.
> 
> Does that sound like a good plan?

No ;)
Could you please give the patch below a try?

I intended it as a fix for a minor performance regression introduced in
the same commit that triggered this issue, but if it can work around
this issue, too, all the better.

I'm just not sure whether we should burden this otherwise nice and short
commit message with the details of this Powerline-macOS-Bash-sed issue...


  -- >8 --

Subject: [PATCH] completion: reduce overhead of clearing cached --options

To get the names of all '$__git_builtin_*' variables caching --options
of builtin commands in order to unset them, 8b0eaa41f2 (completion:
clear cached --options when sourcing the completion script,
2018-03-22) runs a 'set |sed s///' pipeline.  This works both in Bash
and in ZSH, but has a higher than necessasry overhead with the extra
processes.

In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
builtin command, which lists the same variables, but without a
pipeline and 'sed' it can do so with lower overhead.

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>

TODO: as an unexpected bonus, this might work around the issue with
'sed' and invalid UTF-8 characters in the environment as well, at least
in Bash.

  https://public-inbox.org/git/0102016293c8dca7-6626fcde-548d-476e-b61f-c83ecdeedfe1-000000@eu-west-1.amazonses.com/
---
 contrib/completion/git-completion.bash | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index b09c8a2362..4ef59a51be 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -282,7 +282,11 @@ __gitcomp ()
 
 # Clear the variables caching builtins' options when (re-)sourcing
 # the completion script.
-unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
+if [[ -n ${ZSH_VERSION-} ]]; then
+	unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
+else
+	unset $(compgen -v __gitcomp_builtin_)
+fi
 
 # This function is equivalent to
 #
-- 
2.17.0.366.gbe216a3084

Re: [PATCH] specify encoding for sed command

[Matthew Coleman]

>> I think the best way to move forward with this is a new patch that uses
>> `awk` instead of `sed`: I tested several `awk` variants and the command
>> was portable without requiring any changes to LANG or LC_ALL.
>> 
>> Does that sound like a good plan?
> 
> No ;)
> Could you please give the patch below a try?
Your patch works great and is clearly a way better approach than parsing the output of `set`.

> I'm just not sure whether we should burden this otherwise nice and short
> commit message with the details of this Powerline-macOS-Bash-sed issue...
It might be worth mentioning it briefly, in order to reduce the chances of a regression in the future. Maybe something like...
"In Bash we can do better: run the 'compgen -v __gitcomp_builtin_' builtin command, which lists the same variables, but without the risk of errors caused by poor Unicode handling in some shells' 'set' builtin command or the overhead of executing 'sed'."

[PATCH] completion: reduce overhead of clearing cached --options

[SZEDER Gábor]

To get the names of all '$__git_builtin_*' variables caching --options
of builtin commands in order to unset them, 8b0eaa41f2 (completion:
clear cached --options when sourcing the completion script,
2018-03-22) runs a 'set |sed s///' pipeline.  This works both in Bash
and in ZSH, but has a higher than necessasry overhead with the extra
processes.

In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
builtin command, which lists the same variables, but without a
pipeline and 'sed' it can do so with lower overhead.

This change also happens to work around an issue reported by users of
the Powerline shell prompt on macOS, which was triggered by the same
commit 8b0eaa41f2 as well.  Powerline uses several Unicode Private Use
Area code points to represent some of its pretty text UI elements
(arrows and what not), and these are stored in the $PS1 variable.
Apparently the 'set' builtin command of the default Bash version shipped
in macOS (3.2.57) has issues with these code points, and produces
garbled output where Powerline's special symbols should be in the $PS1
variable.  This, in turn, triggers the following error message in the
downstream 'sed' process:

  sed: RE error: illegal byte sequence

Other Bash versions, notably 4.4.19 on macOS (via homebrew) and 3.2.25
on CentOS don't seem to be affected.

With this patch neither the 'set' builtin is invoked to print garbage,
nor 'sed' to choke on it.

Issue-on-macOS-reported-by: Stephon Harris <theonestep4@gmail.com>
Issue-on-macOS-explained-by: Matthew Coleman <matt@1eanda.com>
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 contrib/completion/git-completion.bash | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index b09c8a2362..4ef59a51be 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -282,7 +282,11 @@ __gitcomp ()
 
 # Clear the variables caching builtins' options when (re-)sourcing
 # the completion script.
-unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
+if [[ -n ${ZSH_VERSION-} ]]; then
+	unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
+else
+	unset $(compgen -v __gitcomp_builtin_)
+fi
 
 # This function is equivalent to
 #
-- 
2.17.0.366.gbe216a3084

Re: [PATCH] completion: reduce overhead of clearing cached --options

[Jakub Narebski]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> To get the names of all '$__git_builtin_*' variables caching --options
> of builtin commands in order to unset them, 8b0eaa41f2 (completion:
> clear cached --options when sourcing the completion script,
> 2018-03-22) runs a 'set |sed s///' pipeline.  This works both in Bash
> and in ZSH, but has a higher than necessasry overhead with the extra
> processes.

Typo: necessasry -> necessary

>
> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
> builtin command, which lists the same variables, but without a
> pipeline and 'sed' it can do so with lower overhead.

What about ZSH?

-- 
Jakub Narębski

Re: [PATCH] completion: reduce overhead of clearing cached --options

[SZEDER Gábor]

On Fri, Apr 13, 2018 at 11:44 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> SZEDER Gábor <szeder.dev@gmail.com> writes:
>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
>> builtin command, which lists the same variables, but without a
>> pipeline and 'sed' it can do so with lower overhead.
>
> What about ZSH?

Nothing, ZSH is unaffected by this patch.

Re: [PATCH] completion: reduce overhead of clearing cached --options

[Jakub Narebski]

SZEDER Gábor <szeder.dev@gmail.com> writes:
> On Fri, Apr 13, 2018 at 11:44 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>> SZEDER Gábor <szeder.dev@gmail.com> writes:
>>>
>>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
>>> builtin command, which lists the same variables, but without a
>>> pipeline and 'sed' it can do so with lower overhead.
>>
>> What about ZSH?
>
> Nothing, ZSH is unaffected by this patch.

All right, so for ZSH we would need LC_ALL=C trick, or come with some
equivalent of 'compgen -v __gitcomp_builtin_' for this shell.

Good patch, though it does not solve whole of the problem.

Best,
-- 
Jakub Narębski

Re: [PATCH] completion: reduce overhead of clearing cached --options

[Jacob Keller]

On Sat, Apr 14, 2018 at 6:27 AM, Jakub Narebski <jnareb@gmail.com> wrote:
> SZEDER Gábor <szeder.dev@gmail.com> writes:
>> On Fri, Apr 13, 2018 at 11:44 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>>> SZEDER Gábor <szeder.dev@gmail.com> writes:
>>>>
>>>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
>>>> builtin command, which lists the same variables, but without a
>>>> pipeline and 'sed' it can do so with lower overhead.
>>>
>>> What about ZSH?
>>
>> Nothing, ZSH is unaffected by this patch.
>
> All right, so for ZSH we would need LC_ALL=C trick, or come with some
> equivalent of 'compgen -v __gitcomp_builtin_' for this shell.
>
> Good patch, though it does not solve whole of the problem.
>
> Best,
> --
> Jakub Narębski

Is ZSH actually affected by the broken set behavior, though?

Thanks,
Jake

Re: [PATCH] completion: reduce overhead of clearing cached --options

[Matthew Coleman]

Disclaimer: I'm not a zsh user, so please correct anything I might have gotten wrong.

I created a .zshrc with the following contents:
autoload -Uz compinit
compinit
source /usr/local/lib/python3.6/site-packages/powerline/bindings/zsh/powerline.zsh

zsh doesn't have broken Unicode output in its `set` builtin, so it was not affected by the original issue. Applying the patch does not cause any change in behavior.

Since the commit only changes a file with "bash" in its name, but the conditional references zsh variables, I think it's worth mentioning something about it in the commit message.

I think the bash side of things is all set for this commit, but can a similar improvement be made (using a builtin instead of parsing set|sed) for zsh? Again, I'm not a zsh user, so some input from someone who's written zsh completion rules would be very helpful. Can any of the builtins mentioned in this zsh documentation be used instead of set|sed?
http://zsh.sourceforge.net/Doc/Release/Zsh-Modules.html#The-zsh_002fcomputil-Module

> On Apr 16, 2018, at 2:23 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
> 
> On Sat, Apr 14, 2018 at 6:27 AM, Jakub Narebski <jnareb@gmail.com> wrote:
>> SZEDER Gábor <szeder.dev@gmail.com> writes:
>>> On Fri, Apr 13, 2018 at 11:44 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>>>> SZEDER Gábor <szeder.dev@gmail.com> writes:
>>>>> 
>>>>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
>>>>> builtin command, which lists the same variables, but without a
>>>>> pipeline and 'sed' it can do so with lower overhead.
>>>> 
>>>> What about ZSH?
>>> 
>>> Nothing, ZSH is unaffected by this patch.
>> 
>> All right, so for ZSH we would need LC_ALL=C trick, or come with some
>> equivalent of 'compgen -v __gitcomp_builtin_' for this shell.
>> 
>> Good patch, though it does not solve whole of the problem.
>> 
>> Best,
>> --
>> Jakub Narębski
> 
> Is ZSH actually affected by the broken set behavior, though?
> 
> Thanks,
> Jake

Re: [PATCH] completion: reduce overhead of clearing cached --options

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> On Fri, Apr 13, 2018 at 11:44 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>> SZEDER Gábor <szeder.dev@gmail.com> writes:
>>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
>>> builtin command, which lists the same variables, but without a
>>> pipeline and 'sed' it can do so with lower overhead.
>>
>> What about ZSH?
>
> Nothing, ZSH is unaffected by this patch.

OK, do we want to follow it up with [PATCH 2/1] to add the LC_ALL=C
thing to the ZSH side to help that "sed", then?

Re: [PATCH] completion: reduce overhead of clearing cached --options

[SZEDER Gábor]

On Mon, Apr 16, 2018 at 7:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
> SZEDER Gábor <szeder.dev@gmail.com> writes:
>
>> On Fri, Apr 13, 2018 at 11:44 PM, Jakub Narebski <jnareb@gmail.com> wrote:
>>> SZEDER Gábor <szeder.dev@gmail.com> writes:
>>>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
>>>> builtin command, which lists the same variables, but without a
>>>> pipeline and 'sed' it can do so with lower overhead.
>>>
>>> What about ZSH?
>>
>> Nothing, ZSH is unaffected by this patch.
>
> OK, do we want to follow it up with [PATCH 2/1] to add the LC_ALL=C
> thing to the ZSH side to help that "sed", then?

No.  'sed' would only need need help when its input comes from a buggy
'set' builtin of a particular version of Bash from a particular vendor.

As far as I can test this in Travis CI's OSX builds, ZSH doesn't seem to
be affected, neither the version Apple ships by default nor the version
installed via homebrew.

Re: [PATCH] completion: reduce overhead of clearing cached --options

[Jakub Narębski]

On 16 April 2018 at 15:15, SZEDER Gábor <szeder.dev@gmail.com> wrote:
> On Mon, Apr 16, 2018 at 7:10 AM, Junio C Hamano <gitster@pobox.com> wrote:
> > SZEDER Gábor <szeder.dev@gmail.com> writes:
> >> On Fri, Apr 13, 2018 at 11:44 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> >>> SZEDER Gábor <szeder.dev@gmail.com> writes:
> >>>>
> >>>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
> >>>> builtin command, which lists the same variables, but without a
> >>>> pipeline and 'sed' it can do so with lower overhead.
> >>>
> >>> What about ZSH?
> >>
> >> Nothing, ZSH is unaffected by this patch.
> >
> > OK, do we want to follow it up with [PATCH 2/1] to add the LC_ALL=C
> > thing to the ZSH side to help that "sed", then?
>
> No.  'sed' would only need need help when its input comes from a buggy
> 'set' builtin of a particular version of Bash from a particular vendor.
>
> As far as I can test this in Travis CI's OSX builds, ZSH doesn't seem to
> be affected, neither the version Apple ships by default nor the version
> installed via homebrew.

That's nice - this means that the patch fixes all of the issue.
The above information should be, in my opinion, included
in the commit message, though.

Best,
-- 
Jakub Narębski

Re: [PATCH] completion: reduce overhead of clearing cached --options

[Junio C Hamano]

Jakub Narębski <jnareb@gmail.com> writes:

> On 16 April 2018 at 15:15, SZEDER Gábor <szeder.dev@gmail.com> wrote:
>> No.  'sed' would only need need help when its input comes from a buggy
>> 'set' builtin of a particular version of Bash from a particular vendor.
>>
>> As far as I can test this in Travis CI's OSX builds, ZSH doesn't seem to
>> be affected, neither the version Apple ships by default nor the version
>> installed via homebrew.
>
> That's nice - this means that the patch fixes all of the issue.
> The above information should be, in my opinion, included
> in the commit message, though.

Yeah, I tend to agree.  Thanks.

[PATCH v2] completion: reduce overhead of clearing cached --options

[SZEDER Gábor]

To get the names of all '$__git_builtin_*' variables caching --options
of builtin commands in order to unset them, 8b0eaa41f2 (completion:
clear cached --options when sourcing the completion script,
2018-03-22) runs a 'set |sed s///' pipeline.  This works both in Bash
and in ZSH, but has a higher than necessary overhead with the extra
processes.

In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
builtin command, which lists the same variables, but without a
pipeline and 'sed' it can do so with lower overhead.
ZSH will still continue to run that pipeline.

This change also happens to work around an issue in the default Bash
version shipped in macOS (3.2.57), reported by users of the Powerline
shell prompt, which was triggered by the same commit 8b0eaa41f2 as
well.  Powerline uses several Unicode Private Use Area code points to
represent some of its pretty text UI elements (arrows and what not),
and these are stored in the $PS1 variable.  Apparently the 'set'
builtin of said Bash version on macOS has issues with these code
points, and produces garbled output where Powerline's special symbols
should be in the $PS1 variable.  This, in turn, triggers the following
error message in the downstream 'sed' process:

  sed: RE error: illegal byte sequence

Other Bash versions, notably 4.4.19 on macOS via homebrew (i.e. a
newer version on the same platform) and 3.2.25 on CentOS (i.e. a
slightly earlier version, though on a different platform) are not
affected.  ZSH in macOS (the versions shipped by default or installed
via homebrew) or on other platforms isn't affected either.

With this patch neither the 'set' builtin is invoked to print garbage,
nor 'sed' to choke on it.

Issue-on-macOS-reported-by: Stephon Harris <theonestep4@gmail.com>
Issue-on-macOS-explained-by: Matthew Coleman <matt@1eanda.com>
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---

Updated the commit message to explicitly mention that ZSH is
unaffected.  The patch is the same.

 contrib/completion/git-completion.bash | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index b09c8a2362..4ef59a51be 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -282,7 +282,11 @@ __gitcomp ()
 
 # Clear the variables caching builtins' options when (re-)sourcing
 # the completion script.
-unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
+if [[ -n ${ZSH_VERSION-} ]]; then
+	unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
+else
+	unset $(compgen -v __gitcomp_builtin_)
+fi
 
 # This function is equivalent to
 #
-- 
2.17.0.366.gbe216a3084

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Junio C Hamano]

SZEDER Gábor <szeder.dev@gmail.com> writes:

> To get the names of all '$__git_builtin_*' variables caching --options
> of builtin commands in order to unset them, 8b0eaa41f2 (completion:
> clear cached --options when sourcing the completion script,
> 2018-03-22) runs a 'set |sed s///' pipeline.  This works both in Bash
> and in ZSH, but has a higher than necessary overhead with the extra
> processes.
>
> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
> builtin command, which lists the same variables, but without a
> pipeline and 'sed' it can do so with lower overhead.
> ZSH will still continue to run that pipeline.
> 
> This change also happens to work around an issue in the default Bash
> ...
> Updated the commit message to explicitly mention that ZSH is
> unaffected.  The patch is the same.

Thanks.

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Matthew Coleman]

I haven't seen any discussion about this recently. What are the chances of getting it merged? I'd like to see this included in 2.18.

>> To get the names of all '$__git_builtin_*' variables caching --options
>> of builtin commands in order to unset them, 8b0eaa41f2 (completion:
>> clear cached --options when sourcing the completion script,
>> 2018-03-22) runs a 'set |sed s///' pipeline.  This works both in Bash
>> and in ZSH, but has a higher than necessary overhead with the extra
>> processes.
>> 
>> In Bash we can do better: run the 'compgen -v __gitcomp_builtin_'
>> builtin command, which lists the same variables, but without a
>> pipeline and 'sed' it can do so with lower overhead.
>> ZSH will still continue to run that pipeline.
>> 
>> This change also happens to work around an issue in the default Bash
>> ...
>> Updated the commit message to explicitly mention that ZSH is
>> unaffected.  The patch is the same.
> 
> Thanks.

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Todd Zullinger]

Hi Matthew,

Matthew Coleman wrote:
> I haven't seen any discussion about this recently. What
> are the chances of getting it merged? I'd like to see this
> included in 2.18.

Junio's last few "What's cooking" updates have mentioned it:

> * sg/completion-clear-cached (2018-04-18) 1 commit
>   (merged to 'next' on 2018-04-25 at 9178da6c3d)
>  + completion: reduce overhead of clearing cached --options
> 
>  The completion script (in contrib/) learned to clear cached list of
>  command line options upon dot-sourcing it again in a more efficient
>  way.
> 
>  Will merge to 'master'.

I imagine it will show up on master sometime soon, in time
for 2.18.0.

-- 
Todd
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I expected times like this -- but never thought they'd be so bad, so
long, and so frequent.
    -- Demotivators (www.despair.com)

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Junio C Hamano]

Todd Zullinger <tmz@pobox.com> writes:

> Hi Matthew,
>
> Matthew Coleman wrote:
>> I haven't seen any discussion about this recently. What
>> are the chances of getting it merged? I'd like to see this
>> included in 2.18.
>
> Junio's last few "What's cooking" updates have mentioned it:
>
>> * sg/completion-clear-cached (2018-04-18) 1 commit
>>   (merged to 'next' on 2018-04-25 at 9178da6c3d)
>>  + completion: reduce overhead of clearing cached --options
>> 
>>  The completion script (in contrib/) learned to clear cached list of
>>  command line options upon dot-sourcing it again in a more efficient
>>  way.
>> 
>>  Will merge to 'master'.
>
> I imagine it will show up on master sometime soon, in time
> for 2.18.0.

Yup, I do not foresee at this moment why that shouldn't be the case.

Thanks.

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Jonathan Nieder]

Hi,

SZEDER Gábor wrote:

> Other Bash versions, notably 4.4.19 on macOS via homebrew (i.e. a
> newer version on the same platform) and 3.2.25 on CentOS (i.e. a
> slightly earlier version, though on a different platform) are not
> affected.  ZSH in macOS (the versions shipped by default or installed
> via homebrew) or on other platforms isn't affected either.
[...]
> --- a/contrib/completion/git-completion.bash
> +++ b/contrib/completion/git-completion.bash
> @@ -282,7 +282,11 @@ __gitcomp ()
>  
>  # Clear the variables caching builtins' options when (re-)sourcing
>  # the completion script.
> -unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
> +if [[ -n ${ZSH_VERSION-} ]]; then
> +	unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
> +else
> +	unset $(compgen -v __gitcomp_builtin_)
> +fi

As discussed at [1], this fails catastrophically when sourced by
git-completion.zsh, which explicitly clears ZSH_VERSION.  By
catastrophically, I mean that Rick van Hattem and Dave Borowitz report
that it segfaults.

I can't reproduce it since I am having trouble following the
instructions at the top of git-completion.zsh to get the recommended
zsh completion script loading mechanism working at all.  (I've
succeeded in using git's bash completion on zsh before, but that was
many years ago.)  First attempt:

 1. rm -fr ~/.zsh ~/.zshrc
    # you can move them out of the way instead for a less destructive
    # reproduction recipe
 2. zsh
 3. hit "q"
 4. zstyle ':completion:*:*:git:*' script \
      $(pwd)/contrib/completion/git-completion.zsh
 5. git checkout <TAB>

Expected result: segfault

Actual result: succeeds, using zsh's standard completion script (not
git's).

I also tried

 1. rm -fr ~/.zsh ~/.zshrc
    # you can move them out of the way instead for a less destructive
    # reproduction recipe
 2. zsh
 3. hit "2"
 4. mkdir ~/.zsh; cp contrib/completion/git-completion.zsh ~/.zsh/_git
 5. echo 'fpath=(~/.zsh $fpath)' >>~/.zshrc
 6. ^D; zsh
 7. git checkout <TAB>

and a similar process with fpath earlier in the zshrc file.  Whatever
I try, I end up with one of two results: either it uses zsh's standard
completion, or it spews a stream of errors about missing functions
like compgen.  What am I doing wrong?

Related question: what would it take to add a zsh completion sanity
check to t/t9902-completion.sh?

When running with "set -x", here is what Dave Borowitz gets before the
segfault.  I don't have a stacktrace because, as mentioned above, I
haven't been able to reproduce it.

	str=git
	[[ -n git ]]
	service=git
	i=
	_compskip=default
	eval ''
	ret=0
	[[ default = *patterns* ]]
	[[ default = all ]]
	str=/usr/bin/git
	[[ -n /usr/bin/git ]]
	service=_dispatch:70: bad math expression: operand expected at `/usr/bin/g...'

	zsh: segmentation fault  zsh

Here's a minimal fix, untested.  As a followup, as Gábor suggests at [2],
it would presumably make sense to stop overriding ZSH_VERSION, using
this GIT_ scoped variable everywhere instead.

Thoughts?

Reported-by: Rick van Hattem <wolph@wol.ph>
Reported-by: Dave Borowitz <dborowitz@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

[1] https://public-inbox.org/git/01020163c683e753-04629405-15f8-4a30-9dc3-e4e3f2a5aa26-000000@eu-west-1.amazonses.com/
[2] https://public-inbox.org/git/20180606114147.7753-1-szeder.dev@gmail.com/

diff --git i/contrib/completion/git-completion.bash w/contrib/completion/git-completion.bash
index 12814e9bbf..e4bcc435ea 100644
--- i/contrib/completion/git-completion.bash
+++ w/contrib/completion/git-completion.bash
@@ -348,7 +348,7 @@ __gitcomp ()
 
 # Clear the variables caching builtins' options when (re-)sourcing
 # the completion script.
-if [[ -n ${ZSH_VERSION-} ]]; then
+if [[ -n ${ZSH_VERSION-} || -n ${GIT_SOURCING_ZSH_COMPLETION-} ]]; then
 	unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
 else
 	unset $(compgen -v __gitcomp_builtin_)
diff --git i/contrib/completion/git-completion.zsh w/contrib/completion/git-completion.zsh
index 53cb0f934f..c7be9fd4dc 100644
--- i/contrib/completion/git-completion.zsh
+++ w/contrib/completion/git-completion.zsh
@@ -39,7 +39,7 @@ if [ -z "$script" ]; then
 		test -f $e && script="$e" && break
 	done
 fi
-ZSH_VERSION='' . "$script"
+GIT_SOURCING_ZSH_COMPLETION=1 ZSH_VERSION='' . "$script"
 
 __gitcomp ()
 {

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Dave Borowitz]

On Thu, Jun 7, 2018 at 1:48 AM Jonathan Nieder <jrnieder@gmail.com> wrote:
> Whatever
> I try, I end up with one of two results: either it uses zsh's standard
> completion, or it spews a stream of errors about missing functions
> like compgen.  What am I doing wrong?

Try adding to the top of your .zshrc:
autoload -U compinit; compinit
autoload -U bashcompinit; bashcompinit

Those are both in my .zshrc, and I think they are sufficient magic to
define compgen.

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Rick van Hattem]

The zsh script expects the bash completion script to be available so
that might be the issue here.

To reproduce this is what I do. First, a sparse checkout:
# mkdir ~/git
# cd ~/git
# git init
# git remote add origin git@github.com:git/git.git
# git config core.sparseCheckout true
# mkdir .git/info
# echo contrib/completion >> .git/info/sparse-checkout
# git pull origin master --depth 1

After that I simply link the zsh script to _git:
# cd git/contrib/completion
# ln git-completion.zsh _git

And add the following to a new .zshrc file:
autoload -U compinit; compinit
autoload -U bashcompinit; bashcompinit
fpath=(~/git/contrib/completion $fpath)

And that appears to be enough to reproduce:
# git<tab>
git/contrib/completion/git-completion.bash:354: read-only variable: QISUFFIX
zsh:12: command not found: ___main
zsh:15: _default: function definition file not found
_dispatch:70: bad math expression: operand expected at `/usr/bin/g...'
zsh: segmentation fault  zsh

~rick

On 7 June 2018 at 07:48, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> SZEDER Gábor wrote:
>
>> Other Bash versions, notably 4.4.19 on macOS via homebrew (i.e. a
>> newer version on the same platform) and 3.2.25 on CentOS (i.e. a
>> slightly earlier version, though on a different platform) are not
>> affected.  ZSH in macOS (the versions shipped by default or installed
>> via homebrew) or on other platforms isn't affected either.
> [...]
>> --- a/contrib/completion/git-completion.bash
>> +++ b/contrib/completion/git-completion.bash
>> @@ -282,7 +282,11 @@ __gitcomp ()
>>
>>  # Clear the variables caching builtins' options when (re-)sourcing
>>  # the completion script.
>> -unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
>> +if [[ -n ${ZSH_VERSION-} ]]; then
>> +     unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
>> +else
>> +     unset $(compgen -v __gitcomp_builtin_)
>> +fi
>
> As discussed at [1], this fails catastrophically when sourced by
> git-completion.zsh, which explicitly clears ZSH_VERSION.  By
> catastrophically, I mean that Rick van Hattem and Dave Borowitz report
> that it segfaults.
>
> I can't reproduce it since I am having trouble following the
> instructions at the top of git-completion.zsh to get the recommended
> zsh completion script loading mechanism working at all.  (I've
> succeeded in using git's bash completion on zsh before, but that was
> many years ago.)  First attempt:
>
>  1. rm -fr ~/.zsh ~/.zshrc
>     # you can move them out of the way instead for a less destructive
>     # reproduction recipe
>  2. zsh
>  3. hit "q"
>  4. zstyle ':completion:*:*:git:*' script \
>       $(pwd)/contrib/completion/git-completion.zsh
>  5. git checkout <TAB>
>
> Expected result: segfault
>
> Actual result: succeeds, using zsh's standard completion script (not
> git's).
>
> I also tried
>
>  1. rm -fr ~/.zsh ~/.zshrc
>     # you can move them out of the way instead for a less destructive
>     # reproduction recipe
>  2. zsh
>  3. hit "2"
>  4. mkdir ~/.zsh; cp contrib/completion/git-completion.zsh ~/.zsh/_git
>  5. echo 'fpath=(~/.zsh $fpath)' >>~/.zshrc
>  6. ^D; zsh
>  7. git checkout <TAB>
>
> and a similar process with fpath earlier in the zshrc file.  Whatever
> I try, I end up with one of two results: either it uses zsh's standard
> completion, or it spews a stream of errors about missing functions
> like compgen.  What am I doing wrong?
>
> Related question: what would it take to add a zsh completion sanity
> check to t/t9902-completion.sh?
>
> When running with "set -x", here is what Dave Borowitz gets before the
> segfault.  I don't have a stacktrace because, as mentioned above, I
> haven't been able to reproduce it.
>
>         str=git
>         [[ -n git ]]
>         service=git
>         i=
>         _compskip=default
>         eval ''
>         ret=0
>         [[ default = *patterns* ]]
>         [[ default = all ]]
>         str=/usr/bin/git
>         [[ -n /usr/bin/git ]]
>         service=_dispatch:70: bad math expression: operand expected at `/usr/bin/g...'
>
>         zsh: segmentation fault  zsh
>
> Here's a minimal fix, untested.  As a followup, as Gábor suggests at [2],
> it would presumably make sense to stop overriding ZSH_VERSION, using
> this GIT_ scoped variable everywhere instead.
>
> Thoughts?
>
> Reported-by: Rick van Hattem <wolph@wol.ph>
> Reported-by: Dave Borowitz <dborowitz@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
>
> [1] https://public-inbox.org/git/01020163c683e753-04629405-15f8-4a30-9dc3-e4e3f2a5aa26-000000@eu-west-1.amazonses.com/
> [2] https://public-inbox.org/git/20180606114147.7753-1-szeder.dev@gmail.com/
>
> diff --git i/contrib/completion/git-completion.bash w/contrib/completion/git-completion.bash
> index 12814e9bbf..e4bcc435ea 100644
> --- i/contrib/completion/git-completion.bash
> +++ w/contrib/completion/git-completion.bash
> @@ -348,7 +348,7 @@ __gitcomp ()
>
>  # Clear the variables caching builtins' options when (re-)sourcing
>  # the completion script.
> -if [[ -n ${ZSH_VERSION-} ]]; then
> +if [[ -n ${ZSH_VERSION-} || -n ${GIT_SOURCING_ZSH_COMPLETION-} ]]; then
>         unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
>  else
>         unset $(compgen -v __gitcomp_builtin_)
> diff --git i/contrib/completion/git-completion.zsh w/contrib/completion/git-completion.zsh
> index 53cb0f934f..c7be9fd4dc 100644
> --- i/contrib/completion/git-completion.zsh
> +++ w/contrib/completion/git-completion.zsh
> @@ -39,7 +39,7 @@ if [ -z "$script" ]; then
>                 test -f $e && script="$e" && break
>         done
>  fi
> -ZSH_VERSION='' . "$script"
> +GIT_SOURCING_ZSH_COMPLETION=1 ZSH_VERSION='' . "$script"
>
>  __gitcomp ()
>  {

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[SZEDER Gábor]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=UTF-8, Size: 3148 bytes --]


> Related question: what would it take to add a zsh completion sanity
> check to t/t9902-completion.sh?

I don't know.  What I do know is that we can't just run our tests with
ZSH, e.g. running 'zsh ./t0000-basic.sh' shows mostly failures.  So it
won't be as simple as modifying 't/lib-bash.sh' to somehow support ZSH
as well.

> Here's a minimal fix, untested.  As a followup, as Gábor suggests at [2],
> it would presumably make sense to stop overriding ZSH_VERSION, using
> this GIT_ scoped variable everywhere instead.
> 
> Thoughts?
> 
> Reported-by: Rick van Hattem <wolph@wol.ph>
> Reported-by: Dave Borowitz <dborowitz@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> 
> [1] https://public-inbox.org/git/01020163c683e753-04629405-15f8-4a30-9dc3-e4e3f2a5aa26-000000@eu-west-1.amazonses.com/
> [2] https://public-inbox.org/git/20180606114147.7753-1-szeder.dev@gmail.com/
> 
> diff --git i/contrib/completion/git-completion.bash w/contrib/completion/git-completion.bash
> index 12814e9bbf..e4bcc435ea 100644
> --- i/contrib/completion/git-completion.bash
> +++ w/contrib/completion/git-completion.bash
> @@ -348,7 +348,7 @@ __gitcomp ()
>  
>  # Clear the variables caching builtins' options when (re-)sourcing
>  # the completion script.
> -if [[ -n ${ZSH_VERSION-} ]]; then
> +if [[ -n ${ZSH_VERSION-} || -n ${GIT_SOURCING_ZSH_COMPLETION-} ]]; then
>  	unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
>  else
>  	unset $(compgen -v __gitcomp_builtin_)
> diff --git i/contrib/completion/git-completion.zsh w/contrib/completion/git-completion.zsh
> index 53cb0f934f..c7be9fd4dc 100644
> --- i/contrib/completion/git-completion.zsh
> +++ w/contrib/completion/git-completion.zsh
> @@ -39,7 +39,7 @@ if [ -z "$script" ]; then
>  		test -f $e && script="$e" && break
>  	done
>  fi
> -ZSH_VERSION='' . "$script"
> +GIT_SOURCING_ZSH_COMPLETION=1 ZSH_VERSION='' . "$script"
>  
>  __gitcomp ()
>  {
> 

Being in RC phase, I'm all for aiming for a minimal solution.
However, I don't think that the better fix would be erm.. any "less
minimal":

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index f2aa484758..7aeb575cd1 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -3244,7 +3244,10 @@ __gitk_main ()
 	__git_complete_revlist
 }
 
-if [[ -n ${ZSH_VERSION-} ]]; then
+if [[ -n ${ZSH_VERSION-} ]] &&
+   # Don't define these functions when sourced from 'git-completion.zsh',
+   # it has its own implementations.
+   [[ -z "${GIT_SOURCING_ZSH_COMPLETION}" ]] ; then
 	echo "WARNING: this script is deprecated, please see git-completion.zsh" 1>&2
 
 	autoload -U +X compinit && compinit
diff --git a/contrib/completion/git-completion.zsh b/contrib/completion/git-completion.zsh
index 53cb0f934f..049d6b80f6 100644
--- a/contrib/completion/git-completion.zsh
+++ b/contrib/completion/git-completion.zsh
@@ -39,7 +39,7 @@ if [ -z "$script" ]; then
 		test -f $e && script="$e" && break
 	done
 fi
-ZSH_VERSION='' . "$script"
+GIT_SOURCING_ZSH_COMPLETION=y . "$script"
 
 __gitcomp ()
 {

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Jonathan Nieder]

Hi,

SZEDER Gábor wrote:

> Being in RC phase, I'm all for aiming for a minimal solution.
> However, I don't think that the better fix would be erm.. any "less
> minimal":
>
> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
> index f2aa484758..7aeb575cd1 100644
> --- a/contrib/completion/git-completion.bash
> +++ b/contrib/completion/git-completion.bash
> @@ -3244,7 +3244,10 @@ __gitk_main ()
>  	__git_complete_revlist
>  }
>  
> -if [[ -n ${ZSH_VERSION-} ]]; then
> +if [[ -n ${ZSH_VERSION-} ]] &&
> +   # Don't define these functions when sourced from 'git-completion.zsh',
> +   # it has its own implementations.
> +   [[ -z "${GIT_SOURCING_ZSH_COMPLETION}" ]] ; then

Needs a - before the } to avoid errors in a shell where the user has
chosen to use "set -u".  See v1.7.4-rc0~159 (completion: fix zsh check
under bash with 'set -u', 2010-10-27) for more details.

>  	echo "WARNING: this script is deprecated, please see git-completion.zsh" 1>&2
>  
>  	autoload -U +X compinit && compinit
> diff --git a/contrib/completion/git-completion.zsh b/contrib/completion/git-completion.zsh
> index 53cb0f934f..049d6b80f6 100644
> --- a/contrib/completion/git-completion.zsh
> +++ b/contrib/completion/git-completion.zsh
> @@ -39,7 +39,7 @@ if [ -z "$script" ]; then
>  		test -f $e && script="$e" && break
>  	done
>  fi
> -ZSH_VERSION='' . "$script"
> +GIT_SOURCING_ZSH_COMPLETION=y . "$script"
>  
>  __gitcomp ()
>  {

Except for that tweak,
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Thanks.

Now it just needs a commit message. :)

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[SZEDER Gábor]

On Fri, Jun 8, 2018 at 11:23 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> SZEDER Gábor wrote:
>
>> Being in RC phase, I'm all for aiming for a minimal solution.
>> However, I don't think that the better fix would be erm.. any "less
>> minimal":
>>
>> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
>> index f2aa484758..7aeb575cd1 100644
>> --- a/contrib/completion/git-completion.bash
>> +++ b/contrib/completion/git-completion.bash
>> @@ -3244,7 +3244,10 @@ __gitk_main ()
>>       __git_complete_revlist
>>  }
>>
>> -if [[ -n ${ZSH_VERSION-} ]]; then
>> +if [[ -n ${ZSH_VERSION-} ]] &&
>> +   # Don't define these functions when sourced from 'git-completion.zsh',
>> +   # it has its own implementations.
>> +   [[ -z "${GIT_SOURCING_ZSH_COMPLETION}" ]] ; then
>
> Needs a - before the } to avoid errors in a shell where the user has
> chosen to use "set -u".  See v1.7.4-rc0~159 (completion: fix zsh check
> under bash with 'set -u', 2010-10-27) for more details.

Right...  I did remember that, but by the time I finished typing out
that long variable name I forgot about it... :)

However, I'm not sure it's worth caring about, because the
bash-competion scripts don't work with 'set -u' anyway...

>>       echo "WARNING: this script is deprecated, please see git-completion.zsh" 1>&2
>>
>>       autoload -U +X compinit && compinit
>> diff --git a/contrib/completion/git-completion.zsh b/contrib/completion/git-completion.zsh
>> index 53cb0f934f..049d6b80f6 100644
>> --- a/contrib/completion/git-completion.zsh
>> +++ b/contrib/completion/git-completion.zsh
>> @@ -39,7 +39,7 @@ if [ -z "$script" ]; then
>>               test -f $e && script="$e" && break
>>       done
>>  fi
>> -ZSH_VERSION='' . "$script"
>> +GIT_SOURCING_ZSH_COMPLETION=y . "$script"
>>
>>  __gitcomp ()
>>  {
>
> Except for that tweak,
> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
> Thanks.
>
> Now it just needs a commit message. :)

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Jonathan Nieder]

SZEDER Gábor wrote:
> On Fri, Jun 8, 2018 at 11:23 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

>>> +   [[ -z "${GIT_SOURCING_ZSH_COMPLETION}" ]] ; then
>>
>> Needs a - before the } to avoid errors in a shell where the user has
>> chosen to use "set -u".  See v1.7.4-rc0~159 (completion: fix zsh check
>> under bash with 'set -u', 2010-10-27) for more details.
>
> Right...  I did remember that, but by the time I finished typing out
> that long variable name I forgot about it... :)
>
> However, I'm not sure it's worth caring about, because the
> bash-competion scripts don't work with 'set -u' anyway...

I'd be happy to see these '-'s go away (it would make the code both
easier to read and easier to maintain), but as a post-2.18 topic,
please.

Alternatively, if there's a real need for them, I'd be happy to see
the completion tests start using 'set -u'.

Thanks,
Jonathan

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[SZEDER Gábor]

On Fri, Jun 8, 2018 at 11:52 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> SZEDER Gábor wrote:
>> On Fri, Jun 8, 2018 at 11:23 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>
>>>> +   [[ -z "${GIT_SOURCING_ZSH_COMPLETION}" ]] ; then
>>>
>>> Needs a - before the } to avoid errors in a shell where the user has
>>> chosen to use "set -u".  See v1.7.4-rc0~159 (completion: fix zsh check
>>> under bash with 'set -u', 2010-10-27) for more details.
>>
>> Right...  I did remember that, but by the time I finished typing out
>> that long variable name I forgot about it... :)
>>
>> However, I'm not sure it's worth caring about, because the
>> bash-competion scripts don't work with 'set -u' anyway...
>
> I'd be happy to see these '-'s go away (it would make the code both
> easier to read and easier to maintain), but as a post-2.18 topic,
> please.

Of course.

> Alternatively, if there's a real need for them, I'd be happy to see
> the completion tests start using 'set -u'.

Our test harness doesn't work with 'set -u' at all...

Re: [PATCH v2] completion: reduce overhead of clearing cached --options

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> Hi,
>
> SZEDER Gábor wrote:
>
>> Being in RC phase, I'm all for aiming for a minimal solution.
>> However, I don't think that the better fix would be erm.. any "less
>> minimal":
>>
>> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
>> index f2aa484758..7aeb575cd1 100644
>> --- a/contrib/completion/git-completion.bash
>> +++ b/contrib/completion/git-completion.bash
>> @@ -3244,7 +3244,10 @@ __gitk_main ()
>>  	__git_complete_revlist
>>  }
>>  
>> -if [[ -n ${ZSH_VERSION-} ]]; then
>> +if [[ -n ${ZSH_VERSION-} ]] &&
>> +   # Don't define these functions when sourced from 'git-completion.zsh',
>> +   # it has its own implementations.
>> +   [[ -z "${GIT_SOURCING_ZSH_COMPLETION}" ]] ; then
>
> Needs a - before the } to avoid errors in a shell where the user has
> chosen to use "set -u".  See v1.7.4-rc0~159 (completion: fix zsh check
> under bash with 'set -u', 2010-10-27) for more details.
>
>>  	echo "WARNING: this script is deprecated, please see git-completion.zsh" 1>&2
>>  
>>  	autoload -U +X compinit && compinit
>> diff --git a/contrib/completion/git-completion.zsh b/contrib/completion/git-completion.zsh
>> index 53cb0f934f..049d6b80f6 100644
>> --- a/contrib/completion/git-completion.zsh
>> +++ b/contrib/completion/git-completion.zsh
>> @@ -39,7 +39,7 @@ if [ -z "$script" ]; then
>>  		test -f $e && script="$e" && break
>>  	done
>>  fi
>> -ZSH_VERSION='' . "$script"
>> +GIT_SOURCING_ZSH_COMPLETION=y . "$script"
>>  
>>  __gitcomp ()
>>  {
>
> Except for that tweak,
> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
> Thanks.
>
> Now it just needs a commit message. :)

OK, can we see the final version of this patch so that we can work
it around before 2.18.0-rc2?  I am assuming that we are treating
this as a Git 2.18-rc regression that needs to be fixed before the
final?

Thanks.

[PATCH] completion: correct zsh detection when run from git-completion.zsh (Re: [PATCH v2] completion: reduce overhead of clearing cached --options)

[Jonathan Nieder]

From: SZEDER Gábor <szeder.dev@gmail.com>
Subject: completion: correct zsh detection when run from git-completion.zsh

v2.18.0-rc0~90^2 (completion: reduce overhead of clearing cached
--options, 2018-04-18) worked around a bug in bash's "set" builtin on
MacOS by using compgen instead.  It was careful to avoid breaking zsh
by guarding this workaround with

	if [[ -n ${ZSH_VERSION-}} ]]

Alas, this interacts poorly with git-completion.zsh's bash emulation:

	ZSH_VERSION='' . "$script"

Correct it by instead using a new GIT_SOURCING_ZSH_COMPLETION shell
variable to detect whether git-completion.bash is being sourced from
git-completion.zsh.  This way, the zsh variant is used both when run
from zsh directly and when run via git-completion.zsh.

Reproduction recipe:

 1. cd git/contrib/completion && cp git-completion.zsh _git
 2. Put the following in a new ~/.zshrc file:

 	autoload -U compinit; compinit
	autoload -U bashcompinit; bashcompinit
	fpath=(~/src/git/contrib/completion $fpath)

 3. Open zsh and "git <TAB>".

With this patch:
Triggers nice git-completion.bash based tab completion

Without:
 contrib/completion/git-completion.bash:354: read-only variable: QISUFFIX
 zsh:12: command not found: ___main
 zsh:15: _default: function definition file not found
 _dispatch:70: bad math expression: operand expected at `/usr/bin/g...'
 Segmentation fault

Reported-by: Rick van Hattem <wolph@wol.ph>
Reported-by: Dave Borowitz <dborowitz@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
SZEDER Gábor wrote:

> Being in RC phase, I'm all for aiming for a minimal solution.
> However, I don't think that the better fix would be erm.. any "less
> minimal":

Thanks again. May we have your sign-off?

 contrib/completion/git-completion.bash | 5 ++++-
 contrib/completion/git-completion.zsh  | 2 +-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
index 12814e9bbf..f4a2e6774b 100644
--- a/contrib/completion/git-completion.bash
+++ b/contrib/completion/git-completion.bash
@@ -3223,7 +3223,10 @@ __gitk_main ()
 	__git_complete_revlist
 }
 
-if [[ -n ${ZSH_VERSION-} ]]; then
+if [[ -n ${ZSH_VERSION-} ]] &&
+   # Don't define these functions when sourced from 'git-completion.zsh',
+   # it has its own implementations.
+   [[ -z ${GIT_SOURCING_ZSH_COMPLETION-} ]]; then
 	echo "WARNING: this script is deprecated, please see git-completion.zsh" 1>&2
 
 	autoload -U +X compinit && compinit
diff --git a/contrib/completion/git-completion.zsh b/contrib/completion/git-completion.zsh
index 53cb0f934f..049d6b80f6 100644
--- a/contrib/completion/git-completion.zsh
+++ b/contrib/completion/git-completion.zsh
@@ -39,7 +39,7 @@ if [ -z "$script" ]; then
 		test -f $e && script="$e" && break
 	done
 fi
-ZSH_VERSION='' . "$script"
+GIT_SOURCING_ZSH_COMPLETION=y . "$script"
 
 __gitcomp ()
 {
-- 
2.18.0.rc1.242.g61856ae69a

Re: [PATCH] completion: correct zsh detection when run from git-completion.zsh (Re: [PATCH v2] completion: reduce overhead of clearing cached --options)

[SZEDER Gábor]

On Mon, Jun 11, 2018 at 8:20 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> From: SZEDER Gábor <szeder.dev@gmail.com>
> Subject: completion: correct zsh detection when run from git-completion.zsh
>
> v2.18.0-rc0~90^2 (completion: reduce overhead of clearing cached
> --options, 2018-04-18) worked around a bug in bash's "set" builtin on
> MacOS by using compgen instead.  It was careful to avoid breaking zsh
> by guarding this workaround with
>
>         if [[ -n ${ZSH_VERSION-}} ]]
>
> Alas, this interacts poorly with git-completion.zsh's bash emulation:
>
>         ZSH_VERSION='' . "$script"
>
> Correct it by instead using a new GIT_SOURCING_ZSH_COMPLETION shell
> variable to detect whether git-completion.bash is being sourced from
> git-completion.zsh.  This way, the zsh variant is used both when run
> from zsh directly and when run via git-completion.zsh.
>
> Reproduction recipe:
>
>  1. cd git/contrib/completion && cp git-completion.zsh _git
>  2. Put the following in a new ~/.zshrc file:
>
>         autoload -U compinit; compinit
>         autoload -U bashcompinit; bashcompinit
>         fpath=(~/src/git/contrib/completion $fpath)
>
>  3. Open zsh and "git <TAB>".
>
> With this patch:
> Triggers nice git-completion.bash based tab completion
>
> Without:
>  contrib/completion/git-completion.bash:354: read-only variable: QISUFFIX
>  zsh:12: command not found: ___main
>  zsh:15: _default: function definition file not found
>  _dispatch:70: bad math expression: operand expected at `/usr/bin/g...'
>  Segmentation fault
>
> Reported-by: Rick van Hattem <wolph@wol.ph>
> Reported-by: Dave Borowitz <dborowitz@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
> SZEDER Gábor wrote:
>
>> Being in RC phase, I'm all for aiming for a minimal solution.
>> However, I don't think that the better fix would be erm.. any "less
>> minimal":
>
> Thanks again. May we have your sign-off?

Sure:

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>

Thank you for the commit message; had a family visit the last couple
of days and could not get around to it.


>  contrib/completion/git-completion.bash | 5 ++++-
>  contrib/completion/git-completion.zsh  | 2 +-
>  2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
> index 12814e9bbf..f4a2e6774b 100644
> --- a/contrib/completion/git-completion.bash
> +++ b/contrib/completion/git-completion.bash
> @@ -3223,7 +3223,10 @@ __gitk_main ()
>         __git_complete_revlist
>  }
>
> -if [[ -n ${ZSH_VERSION-} ]]; then
> +if [[ -n ${ZSH_VERSION-} ]] &&
> +   # Don't define these functions when sourced from 'git-completion.zsh',
> +   # it has its own implementations.
> +   [[ -z ${GIT_SOURCING_ZSH_COMPLETION-} ]]; then
>         echo "WARNING: this script is deprecated, please see git-completion.zsh" 1>&2
>
>         autoload -U +X compinit && compinit
> diff --git a/contrib/completion/git-completion.zsh b/contrib/completion/git-completion.zsh
> index 53cb0f934f..049d6b80f6 100644
> --- a/contrib/completion/git-completion.zsh
> +++ b/contrib/completion/git-completion.zsh
> @@ -39,7 +39,7 @@ if [ -z "$script" ]; then
>                 test -f $e && script="$e" && break
>         done
>  fi
> -ZSH_VERSION='' . "$script"
> +GIT_SOURCING_ZSH_COMPLETION=y . "$script"
>
>  __gitcomp ()
>  {
> --
> 2.18.0.rc1.242.g61856ae69a
>

Re: [PATCH] completion: correct zsh detection when run from git-completion.zsh (Re: [PATCH v2] completion: reduce overhead of clearing cached --options)

[SZEDER Gábor]

On Mon, Jun 11, 2018 at 8:20 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> From: SZEDER Gábor <szeder.dev@gmail.com>
> Subject: completion: correct zsh detection when run from git-completion.zsh
>
> v2.18.0-rc0~90^2 (completion: reduce overhead of clearing cached
> --options, 2018-04-18) worked around a bug in bash's "set" builtin on
> MacOS by using compgen instead.  It was careful to avoid breaking zsh
> by guarding this workaround with
>
>         if [[ -n ${ZSH_VERSION-}} ]]
>
> Alas, this interacts poorly with git-completion.zsh's bash emulation:
>
>         ZSH_VERSION='' . "$script"
>
> Correct it by instead using a new GIT_SOURCING_ZSH_COMPLETION shell
> variable to detect whether git-completion.bash is being sourced from
> git-completion.zsh.  This way, the zsh variant is used both when run
> from zsh directly and when run via git-completion.zsh.
>
> Reproduction recipe:
>
>  1. cd git/contrib/completion && cp git-completion.zsh _git
>  2. Put the following in a new ~/.zshrc file:
>
>         autoload -U compinit; compinit
>         autoload -U bashcompinit; bashcompinit
>         fpath=(~/src/git/contrib/completion $fpath)
>
>  3. Open zsh and "git <TAB>".
>
> With this patch:
> Triggers nice git-completion.bash based tab completion
>
> Without:
>  contrib/completion/git-completion.bash:354: read-only variable: QISUFFIX
>  zsh:12: command not found: ___main
>  zsh:15: _default: function definition file not found
>  _dispatch:70: bad math expression: operand expected at `/usr/bin/g...'
>  Segmentation fault

Btw, even if ZSH were not to segfault, the resulting shell would be
next to useless.  The thing is that 'compgen -v foo' is supposed to
list only variable names starting with 'foo', at least that's what it
does under Bash.  In ZSH's Bash emulation, however, it lists _all_
variable names, including such fundamental env vars like PATH, HOME,
and IFS.  So when ZSH took the wrong if branch and run that

  unset $(compgen -v __gitcomp_builtin_)

then it would unset PATH and everything else as well.

Re: [PATCH] completion: correct zsh detection when run from git-completion.zsh (Re: [PATCH v2] completion: reduce overhead of clearing cached --options)

[Rick van Hattem]

On 11 June 2018 at 20:20, Jonathan Nieder <jrnieder@gmail.com> wrote:
> SZEDER Gábor wrote:
>
>> Being in RC phase, I'm all for aiming for a minimal solution.
>> However, I don't think that the better fix would be erm.. any "less
>> minimal":
>
> Thanks again. May we have your sign-off?
>
>  contrib/completion/git-completion.bash | 5 ++++-
>  contrib/completion/git-completion.zsh  | 2 +-
>  2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
> index 12814e9bbf..f4a2e6774b 100644
> --- a/contrib/completion/git-completion.bash
> +++ b/contrib/completion/git-completion.bash
> @@ -3223,7 +3223,10 @@ __gitk_main ()
>         __git_complete_revlist
>  }
>
> -if [[ -n ${ZSH_VERSION-} ]]; then
> +if [[ -n ${ZSH_VERSION-} ]] &&
> +   # Don't define these functions when sourced from 'git-completion.zsh',
> +   # it has its own implementations.
> +   [[ -z ${GIT_SOURCING_ZSH_COMPLETION-} ]]; then
>         echo "WARNING: this script is deprecated, please see git-completion.zsh" 1>&2
>
>         autoload -U +X compinit && compinit
> diff --git a/contrib/completion/git-completion.zsh b/contrib/completion/git-completion.zsh
> index 53cb0f934f..049d6b80f6 100644
> --- a/contrib/completion/git-completion.zsh
> +++ b/contrib/completion/git-completion.zsh
> @@ -39,7 +39,7 @@ if [ -z "$script" ]; then
>                 test -f $e && script="$e" && break
>         done
>  fi
> -ZSH_VERSION='' . "$script"
> +GIT_SOURCING_ZSH_COMPLETION=y . "$script"
>
>  __gitcomp ()
>  {
> --
> 2.18.0.rc1.242.g61856ae69a

The change looks good to me :)

Re: [PATCH] specify encoding for sed command

[Junio C Hamano]

Stephon Harris <theonestep4@gmail.com> writes:

> Fixes issue with seeing `sed: RE error: illegal byte sequence` when running git-completion.bash
> ---
>  contrib/completion/git-completion.bash | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash
> index b09c8a23626b4..52a4ab5e2165a 100644
> --- a/contrib/completion/git-completion.bash
> +++ b/contrib/completion/git-completion.bash
> @@ -282,7 +282,7 @@ __gitcomp ()
>  
>  # Clear the variables caching builtins' options when (re-)sourcing
>  # the completion script.
> -unset $(set |sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null
> +unset $(set |LANG=C sed -ne 's/^\(__gitcomp_builtin_[a-zA-Z0-9_][a-zA-Z0-9_]*\)=.*/\1/p') 2>/dev/null

Shouldn't LC_ALL and LANG both set and exported to C, as

 (1) ancient systems understand only LANG but not LC_*; and

 (2) modern ones that understand both give precedence to LC_ALL over
     LANG?

If we were to set only one, it is probably more sensible to set
LC_ALL, I suspect, but it may be better to set both, which sends a
sign to the readers that we know what we are doing ;-)