Git bisect run should check for the existence of the script

Git bisect run should check for the existence of the script

[Jean Abou Samra]

Hi,

When using 'git bisect run', if the name of the script is misspelled or 
if the script was not made executable, 'git bisect' considers it to be a 
failure and stops at the first revision after the one claimed good. It 
would be better in my eyes to error out.

$ git bisect start
$ git bisect bad HEAD
$ git bisect good HEAD~10
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[344dce312a0cf86d5a5772d54843cc179acaf6e3] bpo-41228: Fix /a/are/ in 
monthcalendar() descripton (GH-21372)
$ git bisect run ./non-existent.sh
running ./non-existent.sh
/usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
Bisecting: 2 revisions left to test after this (roughly 1 step)
[6fc732a2116e2c42b0431bb7e2a21719351af755] Fix typo in docs: 'created by 
th' -> 'created by the' (GH-21384)
running ./non-existent.sh
/usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1] bpo-39573: Use the Py_TYPE() 
macro (GH-21433)
running ./non-existent.sh
/usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1 is the first bad commit
commit 8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1
Author: Victor Stinner <vstinner@python.org>
Date:   Fri Jul 10 12:40:38 2020 +0200

     bpo-39573: Use the Py_TYPE() macro (GH-21433)

     Replace obj->ob_type with Py_TYPE(obj).

  Modules/_elementtree.c       | 2 +-
  Objects/abstract.c           | 4 ++--
  Objects/genericaliasobject.c | 2 +-
  Objects/unicodeobject.c      | 4 ++--
  PC/_msi.c                    | 6 +++---
  PC/winreg.c                  | 4 ++--
  Tools/scripts/combinerefs.py | 2 +-
  7 files changed, 12 insertions(+), 12 deletions(-)
bisect run success


Best regards,
Jean Abou Samra

Re: Git bisect run should check for the existence of the script

[Junio C Hamano]

Jean Abou Samra <jean@abou-samra.fr> writes:

> $ git bisect run ./non-existent.sh
> running ./non-existent.sh
> /usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
> Bisecting: 2 revisions left to test after this (roughly 1 step)
> [6fc732a2116e2c42b0431bb7e2a21719351af755] Fix typo in docs: 'created
> by th' -> 'created by the' (GH-21384)
> running ./non-existent.sh
> /usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
> Bisecting: 0 revisions left to test after this (roughly 0 steps)

Yes, it would be nice if "git bisect run" can reliably tell that it
got a "not found" error and not a "test performed by the script did
not pass" and stop at the first failure.

On the other hand, the "./non-existent.sh" script could be part of
the tracked contents (i.e. some revisions have it and the working
tree has it when they get checked out, some revisions don't and the
working tree does not have it), and the user is trying to find the
first revision that stopped having a working script in its tree.  In
such a case, the script that does not exist and the script that fails
need to be treated the same way by "git bisect run" as failures.  

So... I dunno.

Re: Git bisect run should check for the existence of the script

[Jean Abou Samra]

Le 15/07/2020 à 16:55, Junio C Hamano a écrit :
> Jean Abou Samra <jean@abou-samra.fr> writes:
>
>> $ git bisect run ./non-existent.sh
>> running ./non-existent.sh
>> /usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
>> Bisecting: 2 revisions left to test after this (roughly 1 step)
>> [6fc732a2116e2c42b0431bb7e2a21719351af755] Fix typo in docs: 'created
>> by th' -> 'created by the' (GH-21384)
>> running ./non-existent.sh
>> /usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
>> Bisecting: 0 revisions left to test after this (roughly 0 steps)
> Yes, it would be nice if "git bisect run" can reliably tell that it
> got a "not found" error and not a "test performed by the script did
> not pass" and stop at the first failure.
>
> On the other hand, the "./non-existent.sh" script could be part of
> the tracked contents (i.e. some revisions have it and the working
> tree has it when they get checked out, some revisions don't and the
> working tree does not have it), and the user is trying to find the
> first revision that stopped having a working script in its tree.  In
> such a case, the script that does not exist and the script that fails
> need to be treated the same way by "git bisect run" as failures.
>
> So... I dunno.

Perhapsa --not-found-as-failure option could help?

Best,
Jean Abou Samra

Re: Git bisect run should check for the existence of the script

[Junio C Hamano]

Jean Abou Samra <jean@abou-samra.fr> writes:

> Le 15/07/2020 à 16:55, Junio C Hamano a écrit :
>> Jean Abou Samra <jean@abou-samra.fr> writes:
>>
>>> $ git bisect run ./non-existent.sh
>>> running ./non-existent.sh
>>> /usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
> ...
>> On the other hand, the "./non-existent.sh" script could be part of
>> the tracked contents (i.e. some revisions have it and the working
>> tree has it when they get checked out, some revisions don't and the
>> working tree does not have it), and the user is trying to find the
>> first revision that stopped having a working script in its tree.  In
>> such a case, the script that does not exist and the script that fails
>> need to be treated the same way by "git bisect run" as failures.
>>
>> So... I dunno.
>
> Perhapsa --not-found-as-failure option could help?

Do you mean a new option must be passed if the end-user expects the
script to always exist across revisions, or the script is not
tracked to begin with?  It feels somewhat backwards and the effort
by the end-user to always type the option is better spent to make
sure there is no typo on the command line.

Besides, we need to take into account that "bisect run" takes an
arbitrary shell snippet, e.g. the user may try to switch between two
scripts, test-1 and test-2, based on some condition, and wrote this:

  $ git bisect run "if some-condition; then ./test-1; else ./test-3; fi"

but made a typo in the name of test-2.  It would be noticed after
attempting to run the "if ... fi" as a scriptlet by the returned
status from it being 127 (command not found), which is not treated
any specially by "git bisect run".  So a typo in the script name in
the above example cannot be distinguished from this error where the
user wanted to run test-1 under some condition, but wanted to
declare that the revision is bad if that condition did not hold,
i.e.

  $ git bisect run "if some-condition; then ./test-1; else (exit 127); fi"

Even if we somehow could tell these two apart, we cannot pick the
first scriptlet apart and guess which substring in it were meant as
a filename with a typo.

It may be possible to forbid the general use of exit code 127 and
instead reserve it to signal "we know that the test script is so
broken (this includes the case where it does not even exist) that
using it with 'bisect run' is meaningless; please stop immediately."
and nothing else, just like exit code 125 is reserved for "this
revision is not testable and cannot say good or bad", but making
such a change retroactively is not to be done very lightly.

So, again, I dunno.

 git-bisect.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-bisect.sh b/git-bisect.sh
index 7a8f796251..5c03bf8533 100755
--- a/git-bisect.sh
+++ b/git-bisect.sh
@@ -122,7 +122,7 @@ bisect_run () {
 		res=$?
 
 		# Check for really bad run error.
-		if [ $res -lt 0 -o $res -ge 128 ]
+		if [ $res -lt 0 -o $res -ge 128 -o $res -eq 127 ]
 		then
 			eval_gettextln "bisect run failed:
 exit code \$res from '\$command' is < 0 or >= 128" >&2

Re: Git bisect run should check for the existence of the script

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Do you mean a new option must be passed if the end-user expects the
> script to always exist across revisions, or the script is not
> tracked to begin with?  It feels somewhat backwards and the effort
> by the end-user to always type the option is better spent to make
> sure there is no typo on the command line.

I forgot to follow up on this part.

It does not change the conclusion that it needs to be done carefully
if we chose to retroactivelyreserve return code 127 for our own use,
but such a backward incompatible change can easily be worked around
if users relied on the current behaviour that a missing (tracked)
script would mark the revision "bad" without being a fatal error in
"git bisect run".  Instead of

    git bisect run "./tracked-script"

they can just do

    git bisect run "test -f ./tracked-script && ./tracked-script"

and the problem is solved ;-)