Re: [PATCH 1/3] [RFC] tests: add test_todo() to mark known breakages

[Phillip Wood <phillip.wood123@gmail.com>]

Hi Junio

On 09/12/2022 01:09, Junio C Hamano wrote:
> Phillip Wood <phillip.wood123@gmail.com> writes:
> 
>> I added the restrictions to 'test_todo' to try and stop it being
>> misused but I'm happy to relax them if needed. I'm keen that test_todo
>> is able to distinguish between an expected failure and a failure due
>> to the wrapped command being misused e.g. 'test_todo grep
>> --invalid-option' should report an error.
> 
> Hmm, but it is not useful if the failure is from "you cannot use the
> system command grep with test_todo", as the (implicitly) encouraged
> "fix" the developer who wrote the test would pick would be to use
> "! grep --invalid-option" which would still fail for a wrong reason.
> 
> If a "git" command is expected to run to a completion but is
> currently broken and produces a wrong output, it would be very
> useful to be able to write
> 
> 	git command --args >actual &&
> 	test_todo grep -e "$expected_token" actual
> 
> to say "when 'git command' is fixed, the output should contains
> this, but we know it currently is broken".

You can do that now, the next patch adds support for "test_todo [!] 
grep" but you currently cannot pass arbitrary commands/helper functions.

Best Wishes

Phillip

>> I think you've convinced be to remove the restrictions on what can be
>> wrapped by 'test_todo' when I re-roll.
>>
>> Thanks for your thoughtful comments
> 
> Thanks.