[PATCH] userdiff: add support for Emacs Lisp

[PATCH] userdiff: add support for Emacs Lisp

[Adam Spiers]

Add a diff driver which recognises Elisp top-level forms and outline
headings for display in hunk headers, and which correctly renders word
diffs.

This approach was previously discussed on the emacs-devel mailing list:

   https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg00705.html

* userdiff.c (builtin_drivers): Provide regexen for Elisp top level
  forms and outline headings, and for word diffs.
* t/t4018-diff-funcname.sh (diffpatterns): Add test for elisp driver
* t/t4018/elisp-outline-heading: Test fixture for outline headings
* t/t4018/elisp-top-level-form: Test fixture for top level forms
* t/t4034-diff-words.sh: Add test for elisp driver
* t/t4034/elisp/*: Test fixtures for word diffs

Signed-off-by: Protesilaos Stavrou <info@protesilaos.com>
Signed-off-by: Adam Spiers <git@adamspiers.org>
---
 Documentation/gitattributes.txt | 2 ++
 t/t4018-diff-funcname.sh        | 1 +
 t/t4018/elisp-outline-heading   | 6 ++++++
 t/t4018/elisp-top-level-form    | 7 +++++++
 t/t4034-diff-words.sh           | 1 +
 t/t4034/elisp/expect            | 9 +++++++++
 t/t4034/elisp/post              | 4 ++++
 t/t4034/elisp/pre               | 4 ++++
 userdiff.c                      | 9 +++++++++
 9 files changed, 43 insertions(+)
 create mode 100644 t/t4018/elisp-outline-heading
 create mode 100644 t/t4018/elisp-top-level-form
 create mode 100644 t/t4034/elisp/expect
 create mode 100644 t/t4034/elisp/post
 create mode 100644 t/t4034/elisp/pre

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index e84e104f93..0026055f99 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -815,6 +815,8 @@ patterns are available:
 
 - `dts` suitable for devicetree (DTS) files.
 
+- `elisp` suitable for source code in the Emacs Lisp language.
+
 - `elixir` suitable for source code in the Elixir language.
 
 - `fortran` suitable for source code in the Fortran language.
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 9675bc17db..a9ea2d3cd5 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -33,6 +33,7 @@ diffpatterns="
 	csharp
 	css
 	dts
+	elisp
 	elixir
 	fortran
 	fountain
diff --git a/t/t4018/elisp-outline-heading b/t/t4018/elisp-outline-heading
new file mode 100644
index 0000000000..c13bdafafe
--- /dev/null
+++ b/t/t4018/elisp-outline-heading
@@ -0,0 +1,6 @@
+;;; A top-level outline heading
+;;;; A second-level outline heading RIGHT
+
+;; This is a ChangeMe comment outside top-level forms
+(defun foo ()
+  (bar 1 2 3)
diff --git a/t/t4018/elisp-top-level-form b/t/t4018/elisp-top-level-form
new file mode 100644
index 0000000000..683f7ffcf1
--- /dev/null
+++ b/t/t4018/elisp-top-level-form
@@ -0,0 +1,7 @@
+;;; Outline heading
+
+;; This is a comment
+(RIGHT
+  (list 1 2 3)
+  ChangeMe
+  (list a b c))
diff --git a/t/t4034-diff-words.sh b/t/t4034-diff-words.sh
index 0c8fb39ced..a546ee831a 100755
--- a/t/t4034-diff-words.sh
+++ b/t/t4034-diff-words.sh
@@ -315,6 +315,7 @@ test_language_driver cpp
 test_language_driver csharp
 test_language_driver css
 test_language_driver dts
+test_language_driver elisp
 test_language_driver fortran
 test_language_driver html
 test_language_driver java
diff --git a/t/t4034/elisp/expect b/t/t4034/elisp/expect
new file mode 100644
index 0000000000..29a6ef2520
--- /dev/null
+++ b/t/t4034/elisp/expect
@@ -0,0 +1,9 @@
+<BOLD>diff --git a/pre b/post<RESET>
+<BOLD>index 4a39df8..6619e96 100644<RESET>
+<BOLD>--- a/pre<RESET>
+<BOLD>+++ b/post<RESET>
+<CYAN>@@ -1,4 +1,4 @@<RESET>
+(defun <RED>myfunc<RESET><GREEN>my-func<RESET> (<RED>a b<RESET><GREEN>first second<RESET>)
+  "This is a <RED>really<RESET><GREEN>(moderately)<RESET> cool function."
+  (let ((c (<RED>+ a b<RESET><GREEN>1+ first<RESET>)))
+    (format "one more than the total is %d" (<RED>1+<RESET><GREEN>+<RESET> c <GREEN>second<RESET>))))
diff --git a/t/t4034/elisp/post b/t/t4034/elisp/post
new file mode 100644
index 0000000000..6619e96657
--- /dev/null
+++ b/t/t4034/elisp/post
@@ -0,0 +1,4 @@
+(defun my-func (first second)
+  "This is a (moderately) cool function."
+  (let ((c (1+ first)))
+    (format "one more than the total is %d" (+ c second))))
diff --git a/t/t4034/elisp/pre b/t/t4034/elisp/pre
new file mode 100644
index 0000000000..4a39df8ffb
--- /dev/null
+++ b/t/t4034/elisp/pre
@@ -0,0 +1,4 @@
+(defun myfunc (a b)
+  "This is a really cool function."
+  (let ((c (+ a b)))
+    (format "one more than the total is %d" (1+ c))))
diff --git a/userdiff.c b/userdiff.c
index 3f81a2261c..292e51674a 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -53,6 +53,15 @@ PATTERNS("dts",
 	 /* Property names and math operators */
 	 "[a-zA-Z0-9,._+?#-]+"
 	 "|[-+*/%&^|!~]|>>|<<|&&|\\|\\|"),
+PATTERNS("elisp",
+	 /* Top level forms and outline headings */
+	 "^((\\(|;;;+ +).+)",
+	 /*
+	  * Emacs Lisp allows symbol names containing any characters.
+	  * However spaces within the symbol must be escaped.
+	  */
+	 "(\\.|[^ ()])+"
+	 ),
 PATTERNS("elixir",
 	 "^[ \t]*((def(macro|module|impl|protocol|p)?|test)[ \t].*)$",
 	 /* -- */
-- 
2.30.1.490.ge54fde04c8.dirty

Re: [PATCH] userdiff: add support for Emacs Lisp

[Ævar Arnfjörð Bjarmason]

On Sat, Feb 13 2021, Adam Spiers wrote:

> Add a diff driver which recognises Elisp top-level forms and outline
> headings for display in hunk headers, and which correctly renders word
> diffs.

Neat, I for one would find this useful.

> This approach was previously discussed on the emacs-devel mailing list:
>
>    https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg00705.html
>
> * userdiff.c (builtin_drivers): Provide regexen for Elisp top level
>   forms and outline headings, and for word diffs.
> * t/t4018-diff-funcname.sh (diffpatterns): Add test for elisp driver
> * t/t4018/elisp-outline-heading: Test fixture for outline headings
> * t/t4018/elisp-top-level-form: Test fixture for top level forms
> * t/t4034-diff-words.sh: Add test for elisp driver
> * t/t4034/elisp/*: Test fixtures for word diffs

Please no on the overly verbose emacs.git convention of per-file
changelogs in commit messages :)

> diff --git a/t/t4018/elisp-outline-heading b/t/t4018/elisp-outline-heading
> new file mode 100644
> index 0000000000..c13bdafafe
> --- /dev/null
> +++ b/t/t4018/elisp-outline-heading
> @@ -0,0 +1,6 @@
> +;;; A top-level outline heading
> +;;;; A second-level outline heading RIGHT
> +
> +;; This is a ChangeMe comment outside top-level forms
> +(defun foo ()
> +  (bar 1 2 3)
> diff --git a/t/t4018/elisp-top-level-form b/t/t4018/elisp-top-level-form
> new file mode 100644
> index 0000000000..683f7ffcf1
> --- /dev/null
> +++ b/t/t4018/elisp-top-level-form
> @@ -0,0 +1,7 @@
> +;;; Outline heading
> +
> +;; This is a comment
> +(RIGHT
> +  (list 1 2 3)
> +  ChangeMe
> +  (list a b c))
> diff --git a/t/t4034-diff-words.sh b/t/t4034-diff-words.sh
> index 0c8fb39ced..a546ee831a 100755
> --- a/t/t4034-diff-words.sh
> +++ b/t/t4034-diff-words.sh
> @@ -315,6 +315,7 @@ test_language_driver cpp
>  test_language_driver csharp
>  test_language_driver css
>  test_language_driver dts
> +test_language_driver elisp
>  test_language_driver fortran
>  test_language_driver html
>  test_language_driver java
> diff --git a/t/t4034/elisp/expect b/t/t4034/elisp/expect
> new file mode 100644
> index 0000000000..29a6ef2520
> --- /dev/null
> +++ b/t/t4034/elisp/expect
> @@ -0,0 +1,9 @@
> +<BOLD>diff --git a/pre b/post<RESET>
> +<BOLD>index 4a39df8..6619e96 100644<RESET>
> +<BOLD>--- a/pre<RESET>
> +<BOLD>+++ b/post<RESET>
> +<CYAN>@@ -1,4 +1,4 @@<RESET>
> +(defun <RED>myfunc<RESET><GREEN>my-func<RESET> (<RED>a b<RESET><GREEN>first second<RESET>)
> +  "This is a <RED>really<RESET><GREEN>(moderately)<RESET> cool function."
> +  (let ((c (<RED>+ a b<RESET><GREEN>1+ first<RESET>)))
> +    (format "one more than the total is %d" (<RED>1+<RESET><GREEN>+<RESET> c <GREEN>second<RESET>))))
> diff --git a/t/t4034/elisp/post b/t/t4034/elisp/post
> new file mode 100644
> index 0000000000..6619e96657
> --- /dev/null
> +++ b/t/t4034/elisp/post
> @@ -0,0 +1,4 @@
> +(defun my-func (first second)
> +  "This is a (moderately) cool function."
> +  (let ((c (1+ first)))
> +    (format "one more than the total is %d" (+ c second))))
> diff --git a/t/t4034/elisp/pre b/t/t4034/elisp/pre
> new file mode 100644
> index 0000000000..4a39df8ffb
> --- /dev/null
> +++ b/t/t4034/elisp/pre
> @@ -0,0 +1,4 @@
> +(defun myfunc (a b)
> +  "This is a really cool function."
> +  (let ((c (+ a b)))
> +    (format "one more than the total is %d" (1+ c))))
> diff --git a/userdiff.c b/userdiff.c
> index 3f81a2261c..292e51674a 100644
> --- a/userdiff.c
> +++ b/userdiff.c
> @@ -53,6 +53,15 @@ PATTERNS("dts",
>  	 /* Property names and math operators */
>  	 "[a-zA-Z0-9,._+?#-]+"
>  	 "|[-+*/%&^|!~]|>>|<<|&&|\\|\\|"),
> +PATTERNS("elisp",
> +	 /* Top level forms and outline headings */
> +	 "^((\\(|;;;+ +).+)",
> +	 /*
> +	  * Emacs Lisp allows symbol names containing any characters.
> +	  * However spaces within the symbol must be escaped.
> +	  */
> +	 "(\\.|[^ ()])+"
> +	 ),
>  PATTERNS("elixir",
>  	 "^[ \t]*((def(macro|module|impl|protocol|p)?|test)[ \t].*)$",
>  	 /* -- */

I think this patch would benefit from first being dogfooded in the
emacs-devel community. There's an existing regex in emacs.git's
autoconf.sh which anyone developing emacs.git is using.

If you run:

    diff -u <(git -c diff.elisp.xfuncname='^\(def[^[:space:]]+[[:space:]]+([^()[:space:]]+)' log -p -- lisp | grep -m 100 @@) \
	    <(git -c diff.elisp.xfuncname='^((\(|;;;+ +).*)$' log -p -- lisp | grep -m 100 @@) \
    | less

You can see how it differs from yours, and that also neatly answers the
question[1] you had in the linked thread about why these patterns tend
to be explicitly scoped to how you define a function/package/whatever in
the language in question.

Just a cursory "git log -p -- lisp" in emacs.git with your patch shows
e.g. lisp/thingatpt.el where forms in a "defun" aren't indented (before
it selects the "defun", after with yours it's a "put" in the middle of
the function).

Yours also changes it from e.g.:

    @@ -61,7 +61,7 @@ forward-thing

to:

    @@ -61,7 +61,7 @@ (defun forward-thing (thing &optional n)

Is this really desired in elisp? I also note how our tests in
t4018-diff-funcname.sh are really bad in not testing for this at
all. I.e. we just test that we match the right line, not how we extract
a match from it.

1. https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg00739.html

Re: [PATCH] userdiff: add support for Emacs Lisp

[Johannes Sixt]

Am 14.02.21 um 02:41 schrieb Ævar Arnfjörð Bjarmason:
> Just a cursory "git log -p -- lisp" in emacs.git with your patch shows
> e.g. lisp/thingatpt.el where forms in a "defun" aren't indented (before
> it selects the "defun", after with yours it's a "put" in the middle of
> the function).

Note that negative matches can be specified. We use the feature in the 
cpp case to exclude public:/protected:/private: and label: that happen 
to be not indented. Perhaps that can be useful here?

Oh, and BTW, what the patterns treat as "function" must not match what 
the language treats as function. The purpose of the hunk header is to 
spot a place in the source file easily. So, it does not hurt if 
eval-and-compile forms are captured (as was mentioned in the linked 
thread) if desired.

> 
> Yours also changes it from e.g.:
> 
>      @@ -61,7 +61,7 @@ forward-thing
> 
> to:
> 
>      @@ -61,7 +61,7 @@ (defun forward-thing (thing &optional n)
> 
> Is this really desired in elisp?

It's common practice to extract the entire line (sans indentation if 
applicable), not just the function name. Why would somebody want the 
latter? It doesn't carry as much information as could be possible.

> I also note how our tests in
> t4018-diff-funcname.sh are really bad in not testing for this at
> all. I.e. we just test that we match the right line, not how we extract
> a match from it.

Oh, well. These are "semi-automated" tests cases. If you have an idea 
how to achieve the goal without burdening test writers with lots of 
subtleties, be my guest. ;)

> 
> 1. https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg00739.html
> 

-- Hannes

Re: [PATCH] userdiff: add support for Emacs Lisp

[Johannes Sixt]

Am 14.02.21 um 09:12 schrieb Johannes Sixt:
> Am 14.02.21 um 02:41 schrieb Ævar Arnfjörð Bjarmason:
>> Just a cursory "git log -p -- lisp" in emacs.git with your patch shows
>> e.g. lisp/thingatpt.el where forms in a "defun" aren't indented (before
>> it selects the "defun", after with yours it's a "put" in the middle of
>> the function).
> 
> Note that negative matches can be specified. We use the feature in the 
> cpp case to exclude public:/protected:/private: and label: that happen 
> to be not indented. Perhaps that can be useful here?
> 
> Oh, and BTW, what the patterns treat as "function" must not match what 

I meant to say "need not match".

> the language treats as function. The purpose of the hunk header is to 
> spot a place in the source file easily. So, it does not hurt if 
> eval-and-compile forms are captured (as was mentioned in the linked 
> thread) if desired.
> 
>>
>> Yours also changes it from e.g.:
>>
>>      @@ -61,7 +61,7 @@ forward-thing
>>
>> to:
>>
>>      @@ -61,7 +61,7 @@ (defun forward-thing (thing &optional n)
>>
>> Is this really desired in elisp?
> 
> It's common practice to extract the entire line (sans indentation if 
> applicable), not just the function name. Why would somebody want the 
> latter? It doesn't carry as much information as could be possible.
> 
>> I also note how our tests in
>> t4018-diff-funcname.sh are really bad in not testing for this at
>> all. I.e. we just test that we match the right line, not how we extract
>> a match from it.
> 
> Oh, well. These are "semi-automated" tests cases. If you have an idea 
> how to achieve the goal without burdening test writers with lots of 
> subtleties, be my guest. ;)
> 
>>
>> 1. https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg00739.html
>>
> 
> -- Hannes

Re: [PATCH] userdiff: add support for Emacs Lisp

[Ævar Arnfjörð Bjarmason]

On Sun, Feb 14 2021, Johannes Sixt wrote:

> Am 14.02.21 um 02:41 schrieb Ævar Arnfjörð Bjarmason:
>> Just a cursory "git log -p -- lisp" in emacs.git with your patch shows
>> e.g. lisp/thingatpt.el where forms in a "defun" aren't indented (before
>> it selects the "defun", after with yours it's a "put" in the middle of
>> the function).
>
> Note that negative matches can be specified. We use the feature in the
> cpp case to exclude public:/protected:/private: and label: that happen 
> to be not indented. Perhaps that can be useful here?
>
> Oh, and BTW, what the patterns treat as "function" must not match what
> the language treats as function. The purpose of the hunk header is to 
> spot a place in the source file easily. So, it does not hurt if
> eval-and-compile forms are captured (as was mentioned in the linked 
> thread) if desired.

Right, so having lots of test-case is helpful, e.g. for elisp maybe you
have a top-level defun, maybe not, maybe the top-level is a "(progn" so
you'd like a second-level more meaningful context, or maybe not...

Obviously these userdiff patterns aren't a general parser and will
always be hit-and-miss, it's just useful to at least eyeball them
against in-the-wild test data to check their sanity & add some tests.

My cursory glance at the emacs.git version v.s. what's being submitted
here is that this one does a worse job in *very common* cases.

>> Yours also changes it from e.g.:
>>      @@ -61,7 +61,7 @@ forward-thing
>> to:
>>      @@ -61,7 +61,7 @@ (defun forward-thing (thing &optional n)
>> Is this really desired in elisp?
>
> It's common practice to extract the entire line (sans indentation if
> applicable), not just the function name. Why would somebody want the 
> latter? It doesn't carry as much information as could be possible.

Because I'm familiar with the codebase I'm editing and I just need to
know that the diff I'm viewing is on the "main" function, not that it's
"main()", "int main(int argv, char **argv)", "int main(const int argv,
const char *argv[])", or to either have a " {" or not at the end
depending on the project's coding style.

I know our own userdiff builtin patterns don't do this, but it would be
very useful to retrofit this capability / maybe make it a configurable
feature, i.e. have them capture the meaningful part of the line, and you
could either print it all, or just that part.

>> I also note how our tests in
>> t4018-diff-funcname.sh are really bad in not testing for this at
>> all. I.e. we just test that we match the right line, not how we extract
>> a match from it.
>
> Oh, well. These are "semi-automated" tests cases. If you have an idea
> how to achieve the goal without burdening test writers with lots of 
> subtleties, be my guest. ;)

I'll do that soon.

>> 1. https://lists.gnu.org/archive/html/emacs-devel/2021-02/msg00739.html
>> 
>
> -- Hannes

[PATCH 00/20] userdiff: refactor + test + doc + misc improvements

[Ævar Arnfjörð Bjarmason]

This started out as a version of 07/20 since I noticed that we could
easily break userdiff patterns in subtle ways and and not notice,
since nothing tested the full @@ line (just whether we had a "RIGHT"
token).

But it grew into fixing warts in the code, tests docs, and I even
managed to improve the golang userdiff driver while I was at it.

Hopefully this isn't too hard to review, the diffstat is scary, but
it's mostly moving things around. I went out of my way to make 08/20
as easy as possible to review with the color-move detection.

Ævar Arnfjörð Bjarmason (20):
  userdiff: refactor away the parse_bool() function
  userdiff: re-order builtin drivers in alphabetical order
  userdiff: add and use for_each_userdiff_driver()
  userdiff tests: explicitly test "default" pattern
  userdiff tests: list builtin drivers via test-tool
  userdiff: remove support for "broken" tests
  userdiff tests: match full hunk headers
  userdiff tests: rewrite hunk header test infrastructure
  blame tests: don't rely on t/t4018/ directory
  userdiff tests: move custom patterns into one test file
  userdiff tests: remove hack for "RIGHT" token
  userdiff: match "package" in diff=golang
  userdiff tests + docs: document & test "diff.<driver>.x?funcname"
  gitattributes doc: reword discussion of built-in userdiff patterns
  gitattributes doc: document multi-line userdiff patterns
  userdiff tests: remove "funcname" from custom3 test
  userdiff tests: factor out test_diff_funcname() logic
  userdiff tests: test hunk headers on accumulated files
  userdiff tests: test hunk header selection with -U0
  userdiff tests: assert empty hunk header context on -U<large>

 Documentation/config/diff.txt              |  12 ++
 Documentation/gitattributes.txt            |  33 ++-
 Makefile                                   |   1 +
 t/annotate-tests.sh                        |  16 +-
 t/helper/test-tool.c                       |   1 +
 t/helper/test-tool.h                       |   1 +
 t/helper/test-userdiff.c                   |  32 +++
 t/t4018-diff-funcname.sh                   | 154 +++++++------
 t/t4018/README                             |  18 --
 t/t4018/bash-arithmetic-function           |   4 -
 t/t4018/bash-bashism-style-compact         |   6 -
 t/t4018/bash-bashism-style-function        |   4 -
 t/t4018/bash-bashism-style-whitespace      |   4 -
 t/t4018/bash-conditional-function          |   4 -
 t/t4018/bash-missing-parentheses           |   6 -
 t/t4018/bash-mixed-style-compact           |   4 -
 t/t4018/bash-mixed-style-function          |   4 -
 t/t4018/bash-nested-functions              |   6 -
 t/t4018/bash-other-characters              |   4 -
 t/t4018/bash-posix-style-compact           |   4 -
 t/t4018/bash-posix-style-function          |   4 -
 t/t4018/bash-posix-style-whitespace        |   4 -
 t/t4018/bash-subshell-function             |   4 -
 t/t4018/bash-trailing-comment              |   4 -
 t/t4018/bash.sh                            | 160 ++++++++++++++
 t/t4018/cpp-c++-function                   |   4 -
 t/t4018/cpp-class-constructor              |   4 -
 t/t4018/cpp-class-constructor-mem-init     |   5 -
 t/t4018/cpp-class-definition               |   4 -
 t/t4018/cpp-class-definition-derived       |   5 -
 t/t4018/cpp-class-destructor               |   4 -
 t/t4018/cpp-function-returning-global-type |   4 -
 t/t4018/cpp-function-returning-nested      |   5 -
 t/t4018/cpp-function-returning-pointer     |   4 -
 t/t4018/cpp-function-returning-reference   |   4 -
 t/t4018/cpp-gnu-style-function             |   5 -
 t/t4018/cpp-namespace-definition           |   4 -
 t/t4018/cpp-operator-definition            |   4 -
 t/t4018/cpp-skip-access-specifiers         |   8 -
 t/t4018/cpp-skip-comment-block             |   9 -
 t/t4018/cpp-skip-labels                    |   8 -
 t/t4018/cpp-struct-definition              |   9 -
 t/t4018/cpp-struct-single-line             |   7 -
 t/t4018/cpp-template-function-definition   |   4 -
 t/t4018/cpp-union-definition               |   4 -
 t/t4018/cpp-void-c-function                |   4 -
 t/t4018/cpp.sh                             | 240 +++++++++++++++++++++
 t/t4018/css-attribute-value-selector       |   4 -
 t/t4018/css-block-level-@-statements       |  10 -
 t/t4018/css-brace-in-col-1                 |   5 -
 t/t4018/css-class-selector                 |   4 -
 t/t4018/css-colon-eol                      |   4 -
 t/t4018/css-colon-selector                 |   5 -
 t/t4018/css-common                         |   4 -
 t/t4018/css-id-selector                    |   4 -
 t/t4018/css-long-selector-list             |   6 -
 t/t4018/css-prop-sans-indent               |   5 -
 t/t4018/css-root-selector                  |   4 -
 t/t4018/css-short-selector-list            |   4 -
 t/t4018/css-trailing-space                 |   5 -
 t/t4018/css.sh                             | 146 +++++++++++++
 t/t4018/custom.sh                          | 160 ++++++++++++++
 t/t4018/custom1-pattern                    |  17 --
 t/t4018/custom2-match-to-end-of-line       |   8 -
 t/t4018/custom3-alternation-in-pattern     |  17 --
 t/t4018/dts-labels                         |   9 -
 t/t4018/dts-node-unitless                  |   8 -
 t/t4018/dts-nodes                          |   8 -
 t/t4018/dts-nodes-boolean-prop             |   9 -
 t/t4018/dts-nodes-comment1                 |   8 -
 t/t4018/dts-nodes-comment2                 |   8 -
 t/t4018/dts-nodes-multiline-prop           |  13 --
 t/t4018/dts-reference                      |   9 -
 t/t4018/dts-root                           |   5 -
 t/t4018/dts-root-comment                   |   8 -
 t/t4018/dts.sh                             | 149 +++++++++++++
 t/t4018/elixir-do-not-pick-end             |   5 -
 t/t4018/elixir-ex-unit-test                |   6 -
 t/t4018/elixir-function                    |   5 -
 t/t4018/elixir-macro                       |   5 -
 t/t4018/elixir-module                      |   9 -
 t/t4018/elixir-module-func                 |   8 -
 t/t4018/elixir-nested-module               |   9 -
 t/t4018/elixir-private-function            |   5 -
 t/t4018/elixir-protocol                    |   6 -
 t/t4018/elixir-protocol-implementation     |   5 -
 t/t4018/elixir.sh                          | 127 +++++++++++
 t/t4018/fortran-block-data                 |   5 -
 t/t4018/fortran-comment                    |  13 --
 t/t4018/fortran-comment-keyword            |  14 --
 t/t4018/fortran-comment-legacy             |  13 --
 t/t4018/fortran-comment-legacy-star        |  13 --
 t/t4018/fortran-external-function          |   9 -
 t/t4018/fortran-external-subroutine        |   5 -
 t/t4018/fortran-module                     |   5 -
 t/t4018/fortran-module-procedure           |  13 --
 t/t4018/fortran-program                    |   5 -
 t/t4018/fortran.sh                         | 159 ++++++++++++++
 t/t4018/fountain-scene                     |   4 -
 t/t4018/fountain.sh                        |  14 ++
 t/t4018/golang-complex-function            |   8 -
 t/t4018/golang-func                        |   4 -
 t/t4018/golang-interface                   |   4 -
 t/t4018/golang-long-func                   |   5 -
 t/t4018/golang-struct                      |   4 -
 t/t4018/golang.sh                          |  69 ++++++
 t/t4018/java-class-member-function         |   8 -
 t/t4018/java.sh                            |  18 ++
 t/t4018/markdown-heading-indented          |   6 -
 t/t4018/markdown-heading-non-headings      |  17 --
 t/t4018/markdown.sh                        |  39 ++++
 t/t4018/matlab-class-definition            |   5 -
 t/t4018/matlab-function                    |   4 -
 t/t4018/matlab-octave-section-1            |   3 -
 t/t4018/matlab-octave-section-2            |   3 -
 t/t4018/matlab-section                     |   3 -
 t/t4018/matlab.sh                          |  55 +++++
 t/t4018/perl-skip-end-of-heredoc           |   8 -
 t/t4018/perl-skip-forward-decl             |  10 -
 t/t4018/perl-skip-sub-in-pod               |  18 --
 t/t4018/perl-sub-definition                |   4 -
 t/t4018/perl-sub-definition-kr-brace       |   4 -
 t/t4018/perl.sh                            |  78 +++++++
 t/t4018/php-abstract-class                 |   4 -
 t/t4018/php-abstract-method                |   7 -
 t/t4018/php-class                          |   4 -
 t/t4018/php-final-class                    |   4 -
 t/t4018/php-final-method                   |   7 -
 t/t4018/php-function                       |   4 -
 t/t4018/php-interface                      |   4 -
 t/t4018/php-method                         |   7 -
 t/t4018/php-trait                          |   7 -
 t/t4018/php.sh                             | 106 +++++++++
 t/t4018/python-async-def                   |   4 -
 t/t4018/python-class                       |   4 -
 t/t4018/python-def                         |   4 -
 t/t4018/python-indented-async-def          |   7 -
 t/t4018/python-indented-class              |   5 -
 t/t4018/python-indented-def                |   7 -
 t/t4018/python.sh                          |  71 ++++++
 t/t4018/rust-fn                            |   5 -
 t/t4018/rust-impl                          |   5 -
 t/t4018/rust-macro-rules                   |   6 -
 t/t4018/rust-struct                        |   5 -
 t/t4018/rust-trait                         |   5 -
 t/t4018/rust.sh                            |  60 ++++++
 userdiff.c                                 | 157 ++++++++------
 userdiff.h                                 |  15 ++
 148 files changed, 1937 insertions(+), 905 deletions(-)
 create mode 100644 t/helper/test-userdiff.c
 delete mode 100644 t/t4018/README
 delete mode 100644 t/t4018/bash-arithmetic-function
 delete mode 100644 t/t4018/bash-bashism-style-compact
 delete mode 100644 t/t4018/bash-bashism-style-function
 delete mode 100644 t/t4018/bash-bashism-style-whitespace
 delete mode 100644 t/t4018/bash-conditional-function
 delete mode 100644 t/t4018/bash-missing-parentheses
 delete mode 100644 t/t4018/bash-mixed-style-compact
 delete mode 100644 t/t4018/bash-mixed-style-function
 delete mode 100644 t/t4018/bash-nested-functions
 delete mode 100644 t/t4018/bash-other-characters
 delete mode 100644 t/t4018/bash-posix-style-compact
 delete mode 100644 t/t4018/bash-posix-style-function
 delete mode 100644 t/t4018/bash-posix-style-whitespace
 delete mode 100644 t/t4018/bash-subshell-function
 delete mode 100644 t/t4018/bash-trailing-comment
 create mode 100755 t/t4018/bash.sh
 delete mode 100644 t/t4018/cpp-c++-function
 delete mode 100644 t/t4018/cpp-class-constructor
 delete mode 100644 t/t4018/cpp-class-constructor-mem-init
 delete mode 100644 t/t4018/cpp-class-definition
 delete mode 100644 t/t4018/cpp-class-definition-derived
 delete mode 100644 t/t4018/cpp-class-destructor
 delete mode 100644 t/t4018/cpp-function-returning-global-type
 delete mode 100644 t/t4018/cpp-function-returning-nested
 delete mode 100644 t/t4018/cpp-function-returning-pointer
 delete mode 100644 t/t4018/cpp-function-returning-reference
 delete mode 100644 t/t4018/cpp-gnu-style-function
 delete mode 100644 t/t4018/cpp-namespace-definition
 delete mode 100644 t/t4018/cpp-operator-definition
 delete mode 100644 t/t4018/cpp-skip-access-specifiers
 delete mode 100644 t/t4018/cpp-skip-comment-block
 delete mode 100644 t/t4018/cpp-skip-labels
 delete mode 100644 t/t4018/cpp-struct-definition
 delete mode 100644 t/t4018/cpp-struct-single-line
 delete mode 100644 t/t4018/cpp-template-function-definition
 delete mode 100644 t/t4018/cpp-union-definition
 delete mode 100644 t/t4018/cpp-void-c-function
 create mode 100755 t/t4018/cpp.sh
 delete mode 100644 t/t4018/css-attribute-value-selector
 delete mode 100644 t/t4018/css-block-level-@-statements
 delete mode 100644 t/t4018/css-brace-in-col-1
 delete mode 100644 t/t4018/css-class-selector
 delete mode 100644 t/t4018/css-colon-eol
 delete mode 100644 t/t4018/css-colon-selector
 delete mode 100644 t/t4018/css-common
 delete mode 100644 t/t4018/css-id-selector
 delete mode 100644 t/t4018/css-long-selector-list
 delete mode 100644 t/t4018/css-prop-sans-indent
 delete mode 100644 t/t4018/css-root-selector
 delete mode 100644 t/t4018/css-short-selector-list
 delete mode 100644 t/t4018/css-trailing-space
 create mode 100755 t/t4018/css.sh
 create mode 100755 t/t4018/custom.sh
 delete mode 100644 t/t4018/custom1-pattern
 delete mode 100644 t/t4018/custom2-match-to-end-of-line
 delete mode 100644 t/t4018/custom3-alternation-in-pattern
 delete mode 100644 t/t4018/dts-labels
 delete mode 100644 t/t4018/dts-node-unitless
 delete mode 100644 t/t4018/dts-nodes
 delete mode 100644 t/t4018/dts-nodes-boolean-prop
 delete mode 100644 t/t4018/dts-nodes-comment1
 delete mode 100644 t/t4018/dts-nodes-comment2
 delete mode 100644 t/t4018/dts-nodes-multiline-prop
 delete mode 100644 t/t4018/dts-reference
 delete mode 100644 t/t4018/dts-root
 delete mode 100644 t/t4018/dts-root-comment
 create mode 100755 t/t4018/dts.sh
 delete mode 100644 t/t4018/elixir-do-not-pick-end
 delete mode 100644 t/t4018/elixir-ex-unit-test
 delete mode 100644 t/t4018/elixir-function
 delete mode 100644 t/t4018/elixir-macro
 delete mode 100644 t/t4018/elixir-module
 delete mode 100644 t/t4018/elixir-module-func
 delete mode 100644 t/t4018/elixir-nested-module
 delete mode 100644 t/t4018/elixir-private-function
 delete mode 100644 t/t4018/elixir-protocol
 delete mode 100644 t/t4018/elixir-protocol-implementation
 create mode 100755 t/t4018/elixir.sh
 delete mode 100644 t/t4018/fortran-block-data
 delete mode 100644 t/t4018/fortran-comment
 delete mode 100644 t/t4018/fortran-comment-keyword
 delete mode 100644 t/t4018/fortran-comment-legacy
 delete mode 100644 t/t4018/fortran-comment-legacy-star
 delete mode 100644 t/t4018/fortran-external-function
 delete mode 100644 t/t4018/fortran-external-subroutine
 delete mode 100644 t/t4018/fortran-module
 delete mode 100644 t/t4018/fortran-module-procedure
 delete mode 100644 t/t4018/fortran-program
 create mode 100755 t/t4018/fortran.sh
 delete mode 100644 t/t4018/fountain-scene
 create mode 100755 t/t4018/fountain.sh
 delete mode 100644 t/t4018/golang-complex-function
 delete mode 100644 t/t4018/golang-func
 delete mode 100644 t/t4018/golang-interface
 delete mode 100644 t/t4018/golang-long-func
 delete mode 100644 t/t4018/golang-struct
 create mode 100755 t/t4018/golang.sh
 delete mode 100644 t/t4018/java-class-member-function
 create mode 100755 t/t4018/java.sh
 delete mode 100644 t/t4018/markdown-heading-indented
 delete mode 100644 t/t4018/markdown-heading-non-headings
 create mode 100755 t/t4018/markdown.sh
 delete mode 100644 t/t4018/matlab-class-definition
 delete mode 100644 t/t4018/matlab-function
 delete mode 100644 t/t4018/matlab-octave-section-1
 delete mode 100644 t/t4018/matlab-octave-section-2
 delete mode 100644 t/t4018/matlab-section
 create mode 100755 t/t4018/matlab.sh
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc
 delete mode 100644 t/t4018/perl-skip-forward-decl
 delete mode 100644 t/t4018/perl-skip-sub-in-pod
 delete mode 100644 t/t4018/perl-sub-definition
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace
 create mode 100755 t/t4018/perl.sh
 delete mode 100644 t/t4018/php-abstract-class
 delete mode 100644 t/t4018/php-abstract-method
 delete mode 100644 t/t4018/php-class
 delete mode 100644 t/t4018/php-final-class
 delete mode 100644 t/t4018/php-final-method
 delete mode 100644 t/t4018/php-function
 delete mode 100644 t/t4018/php-interface
 delete mode 100644 t/t4018/php-method
 delete mode 100644 t/t4018/php-trait
 create mode 100755 t/t4018/php.sh
 delete mode 100644 t/t4018/python-async-def
 delete mode 100644 t/t4018/python-class
 delete mode 100644 t/t4018/python-def
 delete mode 100644 t/t4018/python-indented-async-def
 delete mode 100644 t/t4018/python-indented-class
 delete mode 100644 t/t4018/python-indented-def
 create mode 100755 t/t4018/python.sh
 delete mode 100644 t/t4018/rust-fn
 delete mode 100644 t/t4018/rust-impl
 delete mode 100644 t/t4018/rust-macro-rules
 delete mode 100644 t/t4018/rust-struct
 delete mode 100644 t/t4018/rust-trait
 create mode 100755 t/t4018/rust.sh

-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 00/27] userdiff: refactor + test + doc + misc improvements

[Ævar Arnfjörð Bjarmason]

Incorporates all the feedback on v2 and more, see the range-diff
below.

Ævar Arnfjörð Bjarmason (27):
  userdiff: refactor away the parse_bool() function
  userdiff style: re-order drivers in alphabetical order
  userdiff style: declare patterns with consistent style
  userdiff style: normalize pascal regex declaration
  userdiff: add and use for_each_userdiff_driver()
  userdiff tests: explicitly test "default" pattern
  userdiff tests: list builtin drivers via test-tool
  userdiff: remove support for "broken" tests
  userdiff tests: match full hunk headers
  blame tests: don't rely on t/t4018/ directory
  blame tests: simplify userdiff driver test
  userdiff tests: rewrite hunk header test infrastructure
  userdiff tests: do config teardown in test_diff_funcname()
  userdiff tests: move custom patterns into one test file
  userdiff tests: remove hack for "RIGHT" token
  userdiff tests: do not do compile tests on "custom" pattern
  userdiff tests + docs: document & test "diff.<driver>.x?funcname"
  gitattributes doc: reword discussion of built-in userdiff patterns
  gitattributes doc: document multi-line userdiff patterns
  userdiff tests: remove "funcname" from custom3 test
  userdiff tests: factor out test_diff_funcname() logic
  userdiff tests: test hunk headers on accumulated files
  userdiff tests: test hunk header selection with -U0
  userdiff tests: assert empty hunk header context on -U<large>
  userdiff: match "package" in diff=golang
  userdiff tests: add basic test for ada
  userdiff tests: add basic test for ruby

 Documentation/config/diff.txt              |  11 +
 Documentation/gitattributes.txt            |  41 +++-
 Makefile                                   |   1 +
 t/annotate-tests.sh                        |  34 +--
 t/helper/test-tool.c                       |   1 +
 t/helper/test-tool.h                       |   1 +
 t/helper/test-userdiff.c                   |  30 +++
 t/t4018-diff-funcname.sh                   | 163 ++++++++------
 t/t4018/README                             |  18 --
 t/t4018/ada.sh                             |  37 ++++
 t/t4018/bash-arithmetic-function           |   4 -
 t/t4018/bash-bashism-style-compact         |   6 -
 t/t4018/bash-bashism-style-function        |   4 -
 t/t4018/bash-bashism-style-whitespace      |   4 -
 t/t4018/bash-conditional-function          |   4 -
 t/t4018/bash-missing-parentheses           |   6 -
 t/t4018/bash-mixed-style-compact           |   4 -
 t/t4018/bash-mixed-style-function          |   4 -
 t/t4018/bash-nested-functions              |   6 -
 t/t4018/bash-other-characters              |   4 -
 t/t4018/bash-posix-style-compact           |   4 -
 t/t4018/bash-posix-style-function          |   4 -
 t/t4018/bash-posix-style-whitespace        |   4 -
 t/t4018/bash-subshell-function             |   4 -
 t/t4018/bash-trailing-comment              |   4 -
 t/t4018/bash.sh                            | 160 ++++++++++++++
 t/t4018/cpp-c++-function                   |   4 -
 t/t4018/cpp-class-constructor              |   4 -
 t/t4018/cpp-class-constructor-mem-init     |   5 -
 t/t4018/cpp-class-definition               |   4 -
 t/t4018/cpp-class-definition-derived       |   5 -
 t/t4018/cpp-class-destructor               |   4 -
 t/t4018/cpp-function-returning-global-type |   4 -
 t/t4018/cpp-function-returning-nested      |   5 -
 t/t4018/cpp-function-returning-pointer     |   4 -
 t/t4018/cpp-function-returning-reference   |   4 -
 t/t4018/cpp-gnu-style-function             |   5 -
 t/t4018/cpp-namespace-definition           |   4 -
 t/t4018/cpp-operator-definition            |   4 -
 t/t4018/cpp-skip-access-specifiers         |   8 -
 t/t4018/cpp-skip-comment-block             |   9 -
 t/t4018/cpp-skip-labels                    |   8 -
 t/t4018/cpp-struct-definition              |   9 -
 t/t4018/cpp-struct-single-line             |   7 -
 t/t4018/cpp-template-function-definition   |   4 -
 t/t4018/cpp-union-definition               |   4 -
 t/t4018/cpp-void-c-function                |   4 -
 t/t4018/cpp.sh                             | 240 +++++++++++++++++++++
 t/t4018/css-attribute-value-selector       |   4 -
 t/t4018/css-block-level-@-statements       |  10 -
 t/t4018/css-brace-in-col-1                 |   5 -
 t/t4018/css-class-selector                 |   4 -
 t/t4018/css-colon-eol                      |   4 -
 t/t4018/css-colon-selector                 |   5 -
 t/t4018/css-common                         |   4 -
 t/t4018/css-id-selector                    |   4 -
 t/t4018/css-long-selector-list             |   6 -
 t/t4018/css-prop-sans-indent               |   5 -
 t/t4018/css-root-selector                  |   4 -
 t/t4018/css-short-selector-list            |   4 -
 t/t4018/css-trailing-space                 |   5 -
 t/t4018/css.sh                             | 146 +++++++++++++
 t/t4018/custom.sh                          | 163 ++++++++++++++
 t/t4018/custom1-pattern                    |  17 --
 t/t4018/custom2-match-to-end-of-line       |   8 -
 t/t4018/custom3-alternation-in-pattern     |  17 --
 t/t4018/dts-labels                         |   9 -
 t/t4018/dts-node-unitless                  |   8 -
 t/t4018/dts-nodes                          |   8 -
 t/t4018/dts-nodes-boolean-prop             |   9 -
 t/t4018/dts-nodes-comment1                 |   8 -
 t/t4018/dts-nodes-comment2                 |   8 -
 t/t4018/dts-nodes-multiline-prop           |  13 --
 t/t4018/dts-reference                      |   9 -
 t/t4018/dts-root                           |   5 -
 t/t4018/dts-root-comment                   |   8 -
 t/t4018/dts.sh                             | 149 +++++++++++++
 t/t4018/elixir-do-not-pick-end             |   5 -
 t/t4018/elixir-ex-unit-test                |   6 -
 t/t4018/elixir-function                    |   5 -
 t/t4018/elixir-macro                       |   5 -
 t/t4018/elixir-module                      |   9 -
 t/t4018/elixir-module-func                 |   8 -
 t/t4018/elixir-nested-module               |   9 -
 t/t4018/elixir-private-function            |   5 -
 t/t4018/elixir-protocol                    |   6 -
 t/t4018/elixir-protocol-implementation     |   5 -
 t/t4018/elixir.sh                          | 127 +++++++++++
 t/t4018/fortran-block-data                 |   5 -
 t/t4018/fortran-comment                    |  13 --
 t/t4018/fortran-comment-keyword            |  14 --
 t/t4018/fortran-comment-legacy             |  13 --
 t/t4018/fortran-comment-legacy-star        |  13 --
 t/t4018/fortran-external-function          |   9 -
 t/t4018/fortran-external-subroutine        |   5 -
 t/t4018/fortran-module                     |   5 -
 t/t4018/fortran-module-procedure           |  13 --
 t/t4018/fortran-program                    |   5 -
 t/t4018/fortran.sh                         | 159 ++++++++++++++
 t/t4018/fountain-scene                     |   4 -
 t/t4018/fountain.sh                        |  14 ++
 t/t4018/golang-complex-function            |   8 -
 t/t4018/golang-func                        |   4 -
 t/t4018/golang-interface                   |   4 -
 t/t4018/golang-long-func                   |   5 -
 t/t4018/golang-struct                      |   4 -
 t/t4018/golang.sh                          |  69 ++++++
 t/t4018/java-class-member-function         |   8 -
 t/t4018/java.sh                            |  18 ++
 t/t4018/markdown-heading-indented          |   6 -
 t/t4018/markdown-heading-non-headings      |  17 --
 t/t4018/markdown.sh                        |  39 ++++
 t/t4018/matlab-class-definition            |   5 -
 t/t4018/matlab-function                    |   4 -
 t/t4018/matlab-octave-section-1            |   3 -
 t/t4018/matlab-octave-section-2            |   3 -
 t/t4018/matlab-section                     |   3 -
 t/t4018/matlab.sh                          |  55 +++++
 t/t4018/perl-skip-end-of-heredoc           |   8 -
 t/t4018/perl-skip-forward-decl             |  10 -
 t/t4018/perl-skip-sub-in-pod               |  18 --
 t/t4018/perl-sub-definition                |   4 -
 t/t4018/perl-sub-definition-kr-brace       |   4 -
 t/t4018/perl.sh                            |  94 ++++++++
 t/t4018/php-abstract-class                 |   4 -
 t/t4018/php-abstract-method                |   7 -
 t/t4018/php-class                          |   4 -
 t/t4018/php-final-class                    |   4 -
 t/t4018/php-final-method                   |   7 -
 t/t4018/php-function                       |   4 -
 t/t4018/php-interface                      |   4 -
 t/t4018/php-method                         |   7 -
 t/t4018/php-trait                          |   7 -
 t/t4018/php.sh                             | 106 +++++++++
 t/t4018/python-async-def                   |   4 -
 t/t4018/python-class                       |   4 -
 t/t4018/python-def                         |   4 -
 t/t4018/python-indented-async-def          |   7 -
 t/t4018/python-indented-class              |   5 -
 t/t4018/python-indented-def                |   7 -
 t/t4018/python.sh                          |  71 ++++++
 t/t4018/ruby.sh                            |  58 +++++
 t/t4018/rust-fn                            |   5 -
 t/t4018/rust-impl                          |   5 -
 t/t4018/rust-macro-rules                   |   6 -
 t/t4018/rust-struct                        |   5 -
 t/t4018/rust-trait                         |   5 -
 t/t4018/rust.sh                            |  60 ++++++
 userdiff.c                                 | 180 ++++++++++------
 userdiff.h                                 |  15 ++
 150 files changed, 2083 insertions(+), 928 deletions(-)
 create mode 100644 t/helper/test-userdiff.c
 delete mode 100644 t/t4018/README
 create mode 100755 t/t4018/ada.sh
 delete mode 100644 t/t4018/bash-arithmetic-function
 delete mode 100644 t/t4018/bash-bashism-style-compact
 delete mode 100644 t/t4018/bash-bashism-style-function
 delete mode 100644 t/t4018/bash-bashism-style-whitespace
 delete mode 100644 t/t4018/bash-conditional-function
 delete mode 100644 t/t4018/bash-missing-parentheses
 delete mode 100644 t/t4018/bash-mixed-style-compact
 delete mode 100644 t/t4018/bash-mixed-style-function
 delete mode 100644 t/t4018/bash-nested-functions
 delete mode 100644 t/t4018/bash-other-characters
 delete mode 100644 t/t4018/bash-posix-style-compact
 delete mode 100644 t/t4018/bash-posix-style-function
 delete mode 100644 t/t4018/bash-posix-style-whitespace
 delete mode 100644 t/t4018/bash-subshell-function
 delete mode 100644 t/t4018/bash-trailing-comment
 create mode 100755 t/t4018/bash.sh
 delete mode 100644 t/t4018/cpp-c++-function
 delete mode 100644 t/t4018/cpp-class-constructor
 delete mode 100644 t/t4018/cpp-class-constructor-mem-init
 delete mode 100644 t/t4018/cpp-class-definition
 delete mode 100644 t/t4018/cpp-class-definition-derived
 delete mode 100644 t/t4018/cpp-class-destructor
 delete mode 100644 t/t4018/cpp-function-returning-global-type
 delete mode 100644 t/t4018/cpp-function-returning-nested
 delete mode 100644 t/t4018/cpp-function-returning-pointer
 delete mode 100644 t/t4018/cpp-function-returning-reference
 delete mode 100644 t/t4018/cpp-gnu-style-function
 delete mode 100644 t/t4018/cpp-namespace-definition
 delete mode 100644 t/t4018/cpp-operator-definition
 delete mode 100644 t/t4018/cpp-skip-access-specifiers
 delete mode 100644 t/t4018/cpp-skip-comment-block
 delete mode 100644 t/t4018/cpp-skip-labels
 delete mode 100644 t/t4018/cpp-struct-definition
 delete mode 100644 t/t4018/cpp-struct-single-line
 delete mode 100644 t/t4018/cpp-template-function-definition
 delete mode 100644 t/t4018/cpp-union-definition
 delete mode 100644 t/t4018/cpp-void-c-function
 create mode 100755 t/t4018/cpp.sh
 delete mode 100644 t/t4018/css-attribute-value-selector
 delete mode 100644 t/t4018/css-block-level-@-statements
 delete mode 100644 t/t4018/css-brace-in-col-1
 delete mode 100644 t/t4018/css-class-selector
 delete mode 100644 t/t4018/css-colon-eol
 delete mode 100644 t/t4018/css-colon-selector
 delete mode 100644 t/t4018/css-common
 delete mode 100644 t/t4018/css-id-selector
 delete mode 100644 t/t4018/css-long-selector-list
 delete mode 100644 t/t4018/css-prop-sans-indent
 delete mode 100644 t/t4018/css-root-selector
 delete mode 100644 t/t4018/css-short-selector-list
 delete mode 100644 t/t4018/css-trailing-space
 create mode 100755 t/t4018/css.sh
 create mode 100755 t/t4018/custom.sh
 delete mode 100644 t/t4018/custom1-pattern
 delete mode 100644 t/t4018/custom2-match-to-end-of-line
 delete mode 100644 t/t4018/custom3-alternation-in-pattern
 delete mode 100644 t/t4018/dts-labels
 delete mode 100644 t/t4018/dts-node-unitless
 delete mode 100644 t/t4018/dts-nodes
 delete mode 100644 t/t4018/dts-nodes-boolean-prop
 delete mode 100644 t/t4018/dts-nodes-comment1
 delete mode 100644 t/t4018/dts-nodes-comment2
 delete mode 100644 t/t4018/dts-nodes-multiline-prop
 delete mode 100644 t/t4018/dts-reference
 delete mode 100644 t/t4018/dts-root
 delete mode 100644 t/t4018/dts-root-comment
 create mode 100755 t/t4018/dts.sh
 delete mode 100644 t/t4018/elixir-do-not-pick-end
 delete mode 100644 t/t4018/elixir-ex-unit-test
 delete mode 100644 t/t4018/elixir-function
 delete mode 100644 t/t4018/elixir-macro
 delete mode 100644 t/t4018/elixir-module
 delete mode 100644 t/t4018/elixir-module-func
 delete mode 100644 t/t4018/elixir-nested-module
 delete mode 100644 t/t4018/elixir-private-function
 delete mode 100644 t/t4018/elixir-protocol
 delete mode 100644 t/t4018/elixir-protocol-implementation
 create mode 100755 t/t4018/elixir.sh
 delete mode 100644 t/t4018/fortran-block-data
 delete mode 100644 t/t4018/fortran-comment
 delete mode 100644 t/t4018/fortran-comment-keyword
 delete mode 100644 t/t4018/fortran-comment-legacy
 delete mode 100644 t/t4018/fortran-comment-legacy-star
 delete mode 100644 t/t4018/fortran-external-function
 delete mode 100644 t/t4018/fortran-external-subroutine
 delete mode 100644 t/t4018/fortran-module
 delete mode 100644 t/t4018/fortran-module-procedure
 delete mode 100644 t/t4018/fortran-program
 create mode 100755 t/t4018/fortran.sh
 delete mode 100644 t/t4018/fountain-scene
 create mode 100755 t/t4018/fountain.sh
 delete mode 100644 t/t4018/golang-complex-function
 delete mode 100644 t/t4018/golang-func
 delete mode 100644 t/t4018/golang-interface
 delete mode 100644 t/t4018/golang-long-func
 delete mode 100644 t/t4018/golang-struct
 create mode 100755 t/t4018/golang.sh
 delete mode 100644 t/t4018/java-class-member-function
 create mode 100755 t/t4018/java.sh
 delete mode 100644 t/t4018/markdown-heading-indented
 delete mode 100644 t/t4018/markdown-heading-non-headings
 create mode 100755 t/t4018/markdown.sh
 delete mode 100644 t/t4018/matlab-class-definition
 delete mode 100644 t/t4018/matlab-function
 delete mode 100644 t/t4018/matlab-octave-section-1
 delete mode 100644 t/t4018/matlab-octave-section-2
 delete mode 100644 t/t4018/matlab-section
 create mode 100755 t/t4018/matlab.sh
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc
 delete mode 100644 t/t4018/perl-skip-forward-decl
 delete mode 100644 t/t4018/perl-skip-sub-in-pod
 delete mode 100644 t/t4018/perl-sub-definition
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace
 create mode 100755 t/t4018/perl.sh
 delete mode 100644 t/t4018/php-abstract-class
 delete mode 100644 t/t4018/php-abstract-method
 delete mode 100644 t/t4018/php-class
 delete mode 100644 t/t4018/php-final-class
 delete mode 100644 t/t4018/php-final-method
 delete mode 100644 t/t4018/php-function
 delete mode 100644 t/t4018/php-interface
 delete mode 100644 t/t4018/php-method
 delete mode 100644 t/t4018/php-trait
 create mode 100755 t/t4018/php.sh
 delete mode 100644 t/t4018/python-async-def
 delete mode 100644 t/t4018/python-class
 delete mode 100644 t/t4018/python-def
 delete mode 100644 t/t4018/python-indented-async-def
 delete mode 100644 t/t4018/python-indented-class
 delete mode 100644 t/t4018/python-indented-def
 create mode 100755 t/t4018/python.sh
 create mode 100755 t/t4018/ruby.sh
 delete mode 100644 t/t4018/rust-fn
 delete mode 100644 t/t4018/rust-impl
 delete mode 100644 t/t4018/rust-macro-rules
 delete mode 100644 t/t4018/rust-struct
 delete mode 100644 t/t4018/rust-trait
 create mode 100755 t/t4018/rust.sh

Range-diff:
 -:  ---------- >  1:  305fc646d0 userdiff: refactor away the parse_bool() function
 1:  7dbe7d638e !  2:  989438c46a userdiff: re-order builtin drivers in alphabetical order
    @@ Metadata
     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Commit message ##
    -    userdiff: re-order builtin drivers in alphabetical order
    +    userdiff style: re-order drivers in alphabetical order
     
         Address some old code smell and move around the built-in userdiff
    -    definitions so they're both in alphabetical order, and now in the same
    +    drivers so they're both in alphabetical order, and now in the same
         order they appear in the gitattributes(5) documentation.
     
         The two started drifting in be58e70dba (diff: unify external diff and
 -:  ---------- >  3:  4c48e5532c userdiff style: declare patterns with consistent style
 -:  ---------- >  4:  f41fa5b316 userdiff style: normalize pascal regex declaration
 2:  69532bfb68 =  5:  0875d5205c userdiff: add and use for_each_userdiff_driver()
 3:  6ad7a5f608 =  6:  638247d04d userdiff tests: explicitly test "default" pattern
 4:  48b8c39380 !  7:  219043a488 userdiff tests: list builtin drivers via test-tool
    @@ t/helper/test-userdiff.c (new)
     +#include "cache.h"
     +#include "userdiff.h"
     +
    -+static int userdiff_list_builtin_drivers_cb(struct userdiff_driver *driver,
    -+					    enum userdiff_driver_type type,
    -+					    void *priv)
    ++static int driver_cb(struct userdiff_driver *driver,
    ++		     enum userdiff_driver_type type, void *priv)
     +{
     +	puts(driver->name);
     +	return 0;
    @@ t/helper/test-userdiff.c (new)
     +
     +static int list_what(enum userdiff_driver_type type)
     +{
    -+	return for_each_userdiff_driver(userdiff_list_builtin_drivers_cb, type,
    -+					NULL);
    ++	return for_each_userdiff_driver(driver_cb, type, NULL);
     +}
     +
     +int cmd__userdiff(int argc, const char **argv)
    @@ t/t4018-diff-funcname.sh: test_description='Test custom diff function name patte
      . ./test-lib.sh
      
      test_expect_success 'setup' '
    -+	test-tool userdiff list-builtin-drivers >builtin-drivers &&
    -+	test_file_not_empty builtin-drivers &&
    -+	builtin_drivers=$(cat builtin-drivers) &&
    ++	builtin_drivers=$(test-tool userdiff list-builtin-drivers) &&
    ++	test -n "$builtin_drivers" &&
     +
      	# a non-trivial custom pattern
      	git config diff.custom1.funcname "!static
 5:  3073d07409 =  8:  eb66160aac userdiff: remove support for "broken" tests
 6:  fd93d18351 !  9:  c6c54039e2 userdiff tests: match full hunk headers
    @@ Commit message
         towards even better test coverage. I'm structuring these tests in such
         a way as to benefit from the diff.colorMove detection.
     
    +    The "sed -n -e" here was originally a single 's/^.*@@\( \|$\)//p'
    +    pattern, but the '\( \|$\)' part had portability issues on OSX and
    +    AIX.
    +
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## t/t4018-diff-funcname.sh ##
    @@ t/t4018-diff-funcname.sh: test_expect_success 'setup hunk header tests' '
     -		git diff -U1 $i >actual &&
     -		grep '@@ .* @@.*RIGHT' actual
     +		git diff -U1 $i >diff &&
    -+		sed -n -e 's/^.*@@\( \|$\)//p' <diff >ctx &&
    ++		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
     +		test_cmp $i.ctx ctx
      	"
      done
    @@ t/t4018/README
     -of equal signs.
     +The text that must appear in the hunk header must contains the word
     +"RIGHT" by convention. The "LANG-whatever.ctx" file contains what we
    -+expect to appear in the hunk header. We munged the start of the line
    -+to "@@ [...] @@" for ease of not having to hardcode the line numbers
    -+and offsets.
    ++expect to appear in the hunk header. We munged away the starting "@@
    ++[...] @@" part of the line for ease of not having to hardcode the line
    ++numbers and offsets.
     
      ## t/t4018/README.ctx (new) ##
     @@
 8:  034ab9f85b ! 10:  1c6ddf96f6 blame tests: don't rely on t/t4018/ directory
    @@ Commit message
         with userdiff driver, 2020-11-01) so that the blame tests don't rely
         on stealing the contents of "t/t4018/fortran-external-function".
     
    -    I'm about to refactor that directory, just moving the relevant test
    -    file here inline is the easiest solution, and I think also the most
    -    readable.
    +    I'm about to refactor that directory to delete that file, just moving
    +    the relevant test file here inline is the easiest solution, and I
    +    think also the most readable.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
 -:  ---------- > 11:  8a883d8799 blame tests: simplify userdiff driver test
 7:  4f8df97207 ! 12:  e56a7a6b5f userdiff tests: rewrite hunk header test infrastructure
    @@ t/t4018-diff-funcname.sh: test_expect_success 'last regexp must not be negated'
     +
     +	test_expect_success "$desc" '
     +		git diff -U1 "$what" >diff &&
    -+		sed -n -e "s/^.*@@\( \|$\)//p" <diff >actual &&
    ++		sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <diff >actual &&
     +		test_cmp expected actual
     +	'
     +}
    @@ t/t4018-diff-funcname.sh: test_expect_success 'last regexp must not be negated'
      do
     -	test_expect_success "hunk header: $i" "
     -		git diff -U1 $i >diff &&
    --		sed -n -e 's/^.*@@\( \|$\)//p' <diff >ctx &&
    +-		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
     -		test_cmp $i.ctx ctx
     -	"
     +	test="$TEST_DIRECTORY/t4018/$what.sh"
    @@ t/t4018/README (deleted)
     -
     -The text that must appear in the hunk header must contains the word
     -"RIGHT" by convention. The "LANG-whatever.ctx" file contains what we
    --expect to appear in the hunk header. We munged the start of the line
    --to "@@ [...] @@" for ease of not having to hardcode the line numbers
    --and offsets.
    +-expect to appear in the hunk header. We munged away the starting "@@
    +-[...] @@" part of the line for ease of not having to hardcode the line
    +-numbers and offsets.
     
      ## t/t4018/README.ctx (deleted) ##
     @@
 -:  ---------- > 13:  84d20a7cd0 userdiff tests: do config teardown in test_diff_funcname()
 9:  e48ad2b57f ! 14:  70fc9fa565 userdiff tests: move custom patterns into one test file
    @@ Commit message
     
      ## t/t4018-diff-funcname.sh ##
     @@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
    - 	test_file_not_empty builtin-drivers &&
    - 	builtin_drivers=$(cat builtin-drivers) &&
    + 	builtin_drivers=$(test-tool userdiff list-builtin-drivers) &&
    + 	test -n "$builtin_drivers" &&
      
     -	# a non-trivial custom pattern
     -	git config diff.custom1.funcname "!static
    @@ t/t4018/custom.sh (new)
     +	}
     +}
     +EOF_TEST
    -+
    -+test_expect_success 'custom: teardown' '
    -+	test_unconfig diff.custom.funcname &&
    -+	test_unconfig diff.custom.xfuncname
    -+'
     
      ## t/t4018/custom1.sh (deleted) ##
     @@
10:  6a4224b512 = 15:  8539d6d464 userdiff tests: remove hack for "RIGHT" token
 -:  ---------- > 16:  121e5d6dfa userdiff tests: do not do compile tests on "custom" pattern
12:  9ce7de5698 ! 17:  451b7ae453 userdiff tests + docs: document & test "diff.<driver>.x?funcname"
    @@ Documentation/config/diff.txt: diff.<driver>.command::
      	See linkgit:gitattributes[5] for details.
     ++
     +When provided as `diff.<driver>.funcname` the regular expression is
    -+interpreted as a basic regular expression, with
    ++interpreted as a basic regular expression. With
     +`diff.<driver>.xfuncname` it's interpreted as an extended regular
     +expression.
     ++
    -+
     +The `*.funcname` and `*.xfuncname` variables behave as if though they
     +were one configuration variable for the purposes of what value
     +eventually gets used. Setting `*.funcname` will override an earlier
    @@ Documentation/config/diff.txt: diff.<driver>.command::
     
      ## t/t4018/custom.sh ##
     @@ t/t4018/custom.sh: public class Beer
    + 	}
      }
      EOF_TEST
    - 
    -+test_expect_success 'custom; setup config precedence' '
    ++
    ++test_expect_success 'custom: setup config precedence' '
     +	git config diff.custom.funcname "foo" &&
     +	git config diff.custom.xfuncname "bar"
     +'
    @@ t/t4018/custom.sh: public class Beer
     +baz
     +EOF_TEST
     +
    -+test_expect_success 'custom: teardown' '
    -+	test_unconfig diff.custom.funcname &&
    -+	test_unconfig diff.custom.xfuncname
    -+'
    -+
    -+test_expect_success 'custom; setup config precedence' '
    ++test_expect_success 'custom: setup config precedence' '
     +	git config diff.custom.xfuncname "bar" &&
     +	git config diff.custom.funcname "foo"
    -+
     +'
     +
     +test_diff_funcname 'custom: config precedence' \
    @@ t/t4018/custom.sh: public class Beer
     +
     +baz
     +EOF_TEST
    -+
    - test_expect_success 'custom: teardown' '
    - 	test_unconfig diff.custom.funcname &&
    - 	test_unconfig diff.custom.xfuncname
13:  3a9f64b747 ! 18:  5a402bb9bf gitattributes doc: reword discussion of built-in userdiff patterns
    @@ Commit message
         "diff.tex.xfuncname" example which now immediately precedes it. This
         will make a follow-up commit smaller.
     
    +    Helped-by: Eric Sunshine <sunshine@sunshineco.com>
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Documentation/gitattributes.txt ##
    @@ Documentation/gitattributes.txt: backslashes; the pattern above picks a line tha
     +There are built-in patterns shipped as part of git itself. A more
     +advanced version of the `tex` pattern discussed above is one of them.
     +
    -+For built-in patterns you do not need the "diff.tex.xfuncname"
    -+discussed above in your configuration file, but if present it'll
    -+override the built-in pattern.
    ++For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
    ++configuration file as discussed above, but if present, it will
    ++override a built-in pattern.
     +
    -+You still need to enable built-in patterns with the the attribute
    -+mechanism, via `.gitattributes`).
    ++Nevertheless, you need to enable built-in patterns via .gitattributes`
    ++for the pattern to take effect.
     +
    -+The following built in patterns are available:
    ++The following built-in patterns are available:
      
      - `ada` suitable for source code in the Ada language.
      
14:  a3e8d0bfc1 ! 19:  a3badb1a3e gitattributes doc: document multi-line userdiff patterns
    @@ Documentation/gitattributes.txt: backslashes; the pattern above picks a line tha
      backslash, and zero or more occurrences of `sub` followed by
      `section` followed by open brace, to the end of line.
      
    -+Multiple patterns can be supplied by seperating them with
    -+newlines. They will be matched one at a time and are compiled as
    -+separate patterns, and thus the first capture in each such pattern is
    -+`$1`, see further discussion of captures below.
    +-There are built-in patterns shipped as part of git itself. A more
    +-advanced version of the `tex` pattern discussed above is one of them.
    ++Multiple patterns can be supplied by listing them one per line
    ++separated by `\n`. They will be matched one at a time from left to
    ++right. Do not supply a trailing "\n" for the last pattern. E.g.:
     +
    -+Patterns that begin with "!" are negated (to match a literal "!" at
    -+the start of a line use e.g. "[!]"). A matching negated pattern will
    -+cause the matching line to be skipped. Use it to blacklist otherwise
    -+matching non-negated patterns. The last pattern must not be negated,
    -+we'll error out if that's the case.
    ++------------------------
    ++[diff "perl"]
    ++	xfuncname = "!^=head\n^[^ ]+.*"
    ++------------------------
    ++
    ++Patterns in in a list of multiple that begin with "!" are negated. A
    ++matching negated pattern will cause the matched line to be
    ++skipped. Use it to skip a later pattern that would otherwise match. It
    ++is an error if one or more negated patterns aren't followed by a
    ++non-negated pattern.
    ++
    ++To match a literal "!" at the start of a line, use some other regex
    ++construct that will match a literal "!" without "!" being the first
    ++character on that line, such as "[!]".
     +
     +If the pattern contains a `$1` capture it will be used instead of the
     +entire matching line (`$0`) to display the hunk header. This can be
    @@ Documentation/gitattributes.txt: backslashes; the pattern above picks a line tha
     +only display the function name as part of a longer function
     +definition.
     +
    - There are built-in patterns shipped as part of git itself. A more
    - advanced version of the `tex` pattern discussed above is one of them.
    ++There are built-in patterns shipped as part of git itself, see the
    ++full listing below.
    + 
    + For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
    +-configuration file as discussed above, but if present, it will
    +-override a built-in pattern.
    ++configuration file. If present, it will override a built-in pattern,
    ++as shown in the `diff.perl.xfuncname` example above.
      
    + Nevertheless, you need to enable built-in patterns via .gitattributes`
    + for the pattern to take effect.
     
      ## t/t4018/custom.sh ##
    -@@ t/t4018/custom.sh: test_expect_success 'custom: teardown' '
    - 	test_unconfig diff.custom.funcname &&
    - 	test_unconfig diff.custom.xfuncname
    - '
    +@@ t/t4018/custom.sh: ChangeMe
    + 
    + baz
    + EOF_TEST
     +
    -+test_expect_success 'custom: negation syntax, ! is magic' '
    ++test_expect_success 'custom: setup negation syntax, ! is magic' '
     +	git config diff.custom.xfuncname "!negation
     +line"
     +'
     +
    -+test_diff_funcname 'custom: config precedence' \
    ++test_diff_funcname 'custom: negation syntax, ! is magic' \
     +	8<<\EOF_HUNK 9<<\EOF_TEST
     +line
     +EOF_HUNK
    @@ t/t4018/custom.sh: test_expect_success 'custom: teardown' '
     +baz
     +EOF_TEST
     +
    -+test_expect_success 'custom: negation syntax, use [!] to override ! magic' '
    ++test_expect_success 'custom: setup negation syntax, use [!] to override ! magic' '
     +	git config diff.custom.xfuncname "[!]negation
     +line"
     +'
     +
    -+test_diff_funcname 'custom: config precedence' \
    ++test_diff_funcname 'custom: negation syntax, use [!] to override ! magic' \
     +	8<<\EOF_HUNK 9<<\EOF_TEST
     +!negation
     +EOF_HUNK
    @@ t/t4018/custom.sh: test_expect_success 'custom: teardown' '
     +baz
     +EOF_TEST
     +
    ++test_expect_success 'custom: setup captures in multiple patterns' '
    ++	git config diff.custom.xfuncname "!^=head
    ++^format ([^ ]+)
    ++^sub ([^;]+)"
    ++'
    ++
    ++test_diff_funcname 'custom: captures in multiple patterns' \
    ++	8<<\EOF_HUNK 9<<\EOF_TEST
    ++foo
    ++EOF_HUNK
    ++sub foo;
    ++=head1
    ++ChangeMe
    ++
    ++EOF_TEST
    +
    + ## t/t4018/perl.sh ##
    +@@ t/t4018/perl.sh: sub RIGHT
    + 	print "ChangeMe\n";
    + }
    + EOF_TEST
    ++
    ++
    ++test_expect_success 'custom: setup config overrides built-in patterns' '
    ++	git config diff.perl.xfuncname "!^=head
    ++^[^ ]+.*"
    ++'
    ++
    ++test_diff_funcname 'custom: config overrides built-in patterns' \
    ++	8<<\EOF_HUNK 9<<\EOF_TEST
    ++sub foo;
    ++EOF_HUNK
    ++sub foo;
    ++=head1
    ++ChangeMe
    ++
    ++EOF_TEST
15:  b83cf4da91 ! 20:  1b46726e85 userdiff tests: remove "funcname" from custom3 test
    @@ t/t4018/custom.sh: public class Beer
      	git config diff.custom.xfuncname "^[ 	]*((public|static).*)$"
      '
      
    -@@ t/t4018/custom.sh: test_diff_funcname 'custom: config precedence' \
    - line
    - EOF_HUNK
    - line
    -+
    - !negation
    - 
    - ChangeMe
16:  f286e42540 ! 21:  9a18506aff userdiff tests: factor out test_diff_funcname() logic
    @@ t/t4018-diff-funcname.sh: test_expect_success 'last regexp must not be negated'
     +
     +last_diff_context_line () {
     +	file=$1
    -+	sed -n -e "s/^.*@@\( \|$\)//p" <$file
    ++	sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <$file
     +}
     +
      test_diff_funcname () {
    @@ t/t4018-diff-funcname.sh: test_diff_funcname () {
      
      	test_expect_success "$desc" '
      		git diff -U1 "$what" >diff &&
    --		sed -n -e "s/^.*@@\( \|$\)//p" <diff >actual &&
    +-		sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <diff >actual &&
     +		last_diff_context_line diff >actual &&
      		test_cmp expected actual
    - 	'
    - }
    + 	' &&
    + 
17:  12635fefd6 ! 22:  24548fb680 userdiff tests: test hunk headers on accumulated files
    @@ t/t4018-diff-funcname.sh: test_diff_funcname () {
      		git diff -U1 "$what" >diff &&
      		last_diff_context_line diff >actual &&
      		test_cmp expected actual
    -+	' &&
    -+
    + 	' &&
    + 
     +	test_expect_success "$desc (accumulated)" '
     +		git diff -U1 "$what".acc >diff &&
     +		last_diff_context_line diff >actual.lines &&
     +		tail -n 1 actual.lines >actual &&
     +		test_cmp expected actual
    - 	'
    - }
    - 
    ++	' &&
    ++
    + 	test_expect_success "teardown: $desc" '
    + 		# In case any custom config was set immediately before
    + 		# the test itself in the test file
     @@ t/t4018-diff-funcname.sh: do
      		echo "$what" >arg.what
      	' &&
18:  3eef2896ea = 23:  48f00a59d5 userdiff tests: test hunk header selection with -U0
19:  3061fac48c ! 24:  05a01990c9 userdiff tests: assert empty hunk header context on -U<large>
    @@ Commit message
     
      ## t/t4018-diff-funcname.sh ##
     @@ t/t4018-diff-funcname.sh: test_diff_funcname () {
    - 		tail -n 1 actual.lines >actual &&
      		test_cmp expected actual
    - 	'
    -+
    + 	' &&
    + 
     +	test_expect_success "$desc -U9001 (accumulated)" '
     +		git diff -U9001 "$what".acc >diff &&
     +		last_diff_context_line diff >actual.lines &&
     +		tail -n 1 actual.lines >actual &&
     +		echo >blank &&
     +		test_cmp blank actual
    -+	'
    - }
    - 
    - for what in $diffpatterns
    ++	' &&
    ++
    + 	test_expect_success "teardown: $desc" '
    + 		# In case any custom config was set immediately before
    + 		# the test itself in the test file
11:  0901623546 ! 25:  3d2f42d704 userdiff: match "package" in diff=golang
    @@ t/t4018/golang.sh
      func (t *Test) RIGHT(a Type) (Type, error) {
     
      ## userdiff.c ##
    -@@ userdiff.c: IPATTERN("fortran",
    - IPATTERN("fountain", "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
    +@@ userdiff.c: IPATTERN("fountain",
    + 	 /* -- */
      	 "[^ \t-]+"),
      PATTERNS("golang",
     +	 /* Packages */
 -:  ---------- > 26:  b2e16ade06 userdiff tests: add basic test for ada
 -:  ---------- > 27:  826b6f4d6a userdiff tests: add basic test for ruby
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 0/2] diff: do not display hunk context under -W

[Ævar Arnfjörð Bjarmason]

This goes on top of my
https://lore.kernel.org/git/20210215154427.32693-1-avarab@gmail.com/
because it use its newly setup test infrastructure.

Ævar Arnfjörð Bjarmason (2):
  diff: do not display hunk context under -W
  diff: test and document -W interaction with -U<n>

 Documentation/diff-options.txt | 12 ++++++++++++
 t/t4015-diff-whitespace.sh     |  2 +-
 t/t4018-diff-funcname.sh       | 12 ++++++++++++
 xdiff/xemit.c                  |  4 +++-
 4 files changed, 28 insertions(+), 2 deletions(-)

-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 1/2] diff: do not display hunk context under -W

[Ævar Arnfjörð Bjarmason]

Fix what I believe to be a long-standing bug in how "-W" interacts
with displaying the hunk context on the @@ line: It should not be
displayed at all under -W.

The long-standing semantics of how -W works and interacts with -U<n>
are rather easy to reason about:

 * -W extends the context line up to the start of the function. With
    userdiff this means the language-aware regex rules in userdiff.c,
    or user-supplied rules.

 * -U<n>, which defaults to -U3 shows at least <n> lines of context,
    if that's greater than what we'd extend the context to under -W
    then -U<n> wins.

 * When showing the hunk context we look up from the first line we
   show of the diff, and find whatever looks like useful context above
   that line.

Thus in e.g. the xdiff/xemit.c change being made in this commit we'll
correctly show "xdl_emit_diff()" in the hunk context under default
diff settings.

But if we viewed it with the -W option we'd show "is_empty_rec()",
because we'd first find the "xdl_emit_diff()" context line, extend the
diff to that, and then would go look for context to show again.

I don't think this behavior makes any sense, our context in this case
is what we're guaranteed to show as part of the diff itself.

The user already asked us to find that context line and show it, we
don't need to then start showing the context above that line, which
they didn't ask for.

This new behavior does give us the edge case that if we e.g. view the
diff here with "-U150 -W" we'd previously extend the context to the
middle of the "is_func_rec()" function, and show that function in the
hunk context. Now we'll show nothing.

I think that change also makes sense. We're showing a change in the
"xdl_emit_diff()" function. That's our context for the change. It
doesn't make sense with -W to start fishing around for other
context.

Arguably in that case we could save away the context we found in the
"XDL_EMIT_FUNCCONTEXT" in "xdl_emit_diff()" and show that if we end up
extending the diff past the function, either because of a high -U<n>
value, or because our change was right at the start.

I wouldn't really mind if we did that, perhaps it would be a useful
marker with high -U<n> values to remind the user of what they're
looking at, but I also don't see the usefulness in practice, so let's
punt that for now.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/diff-options.txt | 4 ++++
 t/t4015-diff-whitespace.sh     | 2 +-
 t/t4018-diff-funcname.sh       | 7 +++++++
 xdiff/xemit.c                  | 4 +++-
 4 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index e5733ccb2d..8ca59effa7 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -759,6 +759,10 @@ endif::git-format-patch[]
 	The function names are determined in the same way as
 	`git diff` works out patch hunk headers (see 'Defining a
 	custom hunk-header' in linkgit:gitattributes[5]).
++
+When showing the whole function for context the "@@" context line
+itself will always be empty, since the context that would otherwise be
+shown there will be the first line of the hunk being shown.
 
 ifndef::git-format-patch[]
 ifndef::git-log[]
diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh
index 8c574221b2..0ffc845cdd 100755
--- a/t/t4015-diff-whitespace.sh
+++ b/t/t4015-diff-whitespace.sh
@@ -2133,7 +2133,7 @@ test_expect_success 'combine --ignore-blank-lines with --function-context 2' '
 		--ignore-blank-lines --function-context a b >actual.raw &&
 	sed -n "/@@/,\$p" <actual.raw >actual &&
 	cat <<-\EOF >expect &&
-	@@ -5,11 +6,9 @@ c
+	@@ -5,11 +6,9 @@
 	 function
 	 1
 	 2
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 80f35c5e16..f3374abd98 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -91,6 +91,13 @@ test_diff_funcname () {
 		fi
 	' &&
 
+	test_expect_success "$desc -W" '
+		git diff -U0 -W "$what" >W-U0-diff &&
+		echo >W-U0-expected &&
+		last_diff_context_line W-U0-diff >W-U0-actual &&
+		test_cmp W-U0-expected W-U0-actual
+	' &&
+
 	test_expect_success "$desc (accumulated)" '
 		git diff -U1 "$what".acc >diff &&
 		last_diff_context_line diff >actual.lines &&
diff --git a/xdiff/xemit.c b/xdiff/xemit.c
index 9d7d6c5087..02b5dbcc70 100644
--- a/xdiff/xemit.c
+++ b/xdiff/xemit.c
@@ -274,7 +274,9 @@ int xdl_emit_diff(xdfenv_t *xe, xdchange_t *xscr, xdemitcb_t *ecb,
 		 */
 
 		if (xecfg->flags & XDL_EMIT_FUNCNAMES) {
-			get_func_line(xe, xecfg, &func_line,
+			get_func_line(xe, xecfg,
+				      xecfg->flags & XDL_EMIT_FUNCCONTEXT
+				      ? NULL : &func_line,
 				      s1 - 1, funclineprev);
 			funclineprev = s1 - 1;
 		}
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH 1/2] diff: do not display hunk context under -W

[René Scharfe.]

Am 15.02.21 um 16:50 schrieb Ævar Arnfjörð Bjarmason:
> Fix what I believe to be a long-standing bug in how "-W" interacts
> with displaying the hunk context on the @@ line: It should not be
> displayed at all under -W.
>
> The long-standing semantics of how -W works and interacts with -U<n>
> are rather easy to reason about:
>
>  * -W extends the context line up to the start of the function. With
>     userdiff this means the language-aware regex rules in userdiff.c,
>     or user-supplied rules.
>
>  * -U<n>, which defaults to -U3 shows at least <n> lines of context,
>     if that's greater than what we'd extend the context to under -W
>     then -U<n> wins.
>
>  * When showing the hunk context we look up from the first line we
>    show of the diff, and find whatever looks like useful context above
>    that line.
>
> Thus in e.g. the xdiff/xemit.c change being made in this commit we'll
> correctly show "xdl_emit_diff()" in the hunk context under default
> diff settings.
>
> But if we viewed it with the -W option we'd show "is_empty_rec()",
> because we'd first find the "xdl_emit_diff()" context line, extend the
> diff to that, and then would go look for context to show again.
>
> I don't think this behavior makes any sense, our context in this case
> is what we're guaranteed to show as part of the diff itself.
>
> The user already asked us to find that context line and show it, we
> don't need to then start showing the context above that line, which
> they didn't ask for.

Hmm, that's subtle.

Your reasoning applies to patches generated without -W as well.  If the
precontext contains a function line then the @@ line should not contain
a function comment.  However, e.g. with this:

-- snip --
cat >a <<EOF
func a

func b
1
2
3
EOF
sed 's/3/three/' <a >b
diff -up a b
-- snap --

... I get this:

--- a	2021-02-15 18:30:21.000000000 +0100
+++ b	2021-02-15 18:30:21.000000000 +0100
@@ -3,4 +3,4 @@ func a
 func b
 1
 2
-3
+three

So diff(1) shows the previous function line.  git diff does the same.

The behaviour of diff(1) and git diff does make sense to me: It's easy
to implement and the only downside is that it produces extra output in
some cases.

I can understand that users would rather have a tidy diff without
distractions, though.  So I like the output change you propose.

However, I'm not sure it would be a good idea to clear @@ lines of hunks
generated without -W that have function lines in their precontext, even
though it would be a logical thing to do.

> This new behavior does give us the edge case that if we e.g. view the
> diff here with "-U150 -W" we'd previously extend the context to the
> middle of the "is_func_rec()" function, and show that function in the
> hunk context. Now we'll show nothing.

Well, the 150 lines of context are still shown (as they should be), but
the @@ line contains no function name anymore.

> I think that change also makes sense. We're showing a change in the
> "xdl_emit_diff()" function. That's our context for the change. It
> doesn't make sense with -W to start fishing around for other
> context.

It does make sense in the context of the diff(1) -p implementation, but
your change is consistent with the description of that option: "Show
which C function each change is in."

> Arguably in that case we could save away the context we found in the
> "XDL_EMIT_FUNCCONTEXT" in "xdl_emit_diff()" and show that if we end up
> extending the diff past the function, either because of a high -U<n>
> value, or because our change was right at the start.
>
> I wouldn't really mind if we did that, perhaps it would be a useful
> marker with high -U<n> values to remind the user of what they're
> looking at, but I also don't see the usefulness in practice, so let's
> punt that for now.

It could be confusing for someone who expects the old behaviour, leaving
it empty makes more sense to me.

>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  Documentation/diff-options.txt | 4 ++++
>  t/t4015-diff-whitespace.sh     | 2 +-
>  t/t4018-diff-funcname.sh       | 7 +++++++
>  xdiff/xemit.c                  | 4 +++-
>  4 files changed, 15 insertions(+), 2 deletions(-)
>
> diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
> index e5733ccb2d..8ca59effa7 100644
> --- a/Documentation/diff-options.txt
> +++ b/Documentation/diff-options.txt
> @@ -759,6 +759,10 @@ endif::git-format-patch[]
>  	The function names are determined in the same way as
>  	`git diff` works out patch hunk headers (see 'Defining a
>  	custom hunk-header' in linkgit:gitattributes[5]).
> ++
> +When showing the whole function for context the "@@" context line
> +itself will always be empty, since the context that would otherwise be
> +shown there will be the first line of the hunk being shown.
>
>  ifndef::git-format-patch[]
>  ifndef::git-log[]
> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh
> index 8c574221b2..0ffc845cdd 100755
> --- a/t/t4015-diff-whitespace.sh
> +++ b/t/t4015-diff-whitespace.sh
> @@ -2133,7 +2133,7 @@ test_expect_success 'combine --ignore-blank-lines with --function-context 2' '
>  		--ignore-blank-lines --function-context a b >actual.raw &&
>  	sed -n "/@@/,\$p" <actual.raw >actual &&
>  	cat <<-\EOF >expect &&
> -	@@ -5,11 +6,9 @@ c
> +	@@ -5,11 +6,9 @@
>  	 function
>  	 1
>  	 2
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index 80f35c5e16..f3374abd98 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -91,6 +91,13 @@ test_diff_funcname () {
>  		fi
>  	' &&
>
> +	test_expect_success "$desc -W" '
> +		git diff -U0 -W "$what" >W-U0-diff &&
> +		echo >W-U0-expected &&
> +		last_diff_context_line W-U0-diff >W-U0-actual &&
> +		test_cmp W-U0-expected W-U0-actual
> +	' &&
> +
>  	test_expect_success "$desc (accumulated)" '
>  		git diff -U1 "$what".acc >diff &&
>  		last_diff_context_line diff >actual.lines &&
> diff --git a/xdiff/xemit.c b/xdiff/xemit.c
> index 9d7d6c5087..02b5dbcc70 100644
> --- a/xdiff/xemit.c
> +++ b/xdiff/xemit.c
> @@ -274,7 +274,9 @@ int xdl_emit_diff(xdfenv_t *xe, xdchange_t *xscr, xdemitcb_t *ecb,
>  		 */
>
>  		if (xecfg->flags & XDL_EMIT_FUNCNAMES) {
> -			get_func_line(xe, xecfg, &func_line,
> +			get_func_line(xe, xecfg,
> +				      xecfg->flags & XDL_EMIT_FUNCCONTEXT
> +				      ? NULL : &func_line,

Why still search?  It would be better to turn off XDL_EMIT_FUNCNAMES if
XDL_EMIT_FUNCCONTEXT is enabled -- a one-character change in diff.c.

>  				      s1 - 1, funclineprev);
>  			funclineprev = s1 - 1;
>  		}
>

Re: [PATCH 1/2] diff: do not display hunk context under -W

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 15 2021, René Scharfe. wrote:

> Am 15.02.21 um 16:50 schrieb Ævar Arnfjörð Bjarmason:
>> Fix what I believe to be a long-standing bug in how "-W" interacts
>> with displaying the hunk context on the @@ line: It should not be
>> displayed at all under -W.
>>
>> The long-standing semantics of how -W works and interacts with -U<n>
>> are rather easy to reason about:
>>
>>  * -W extends the context line up to the start of the function. With
>>     userdiff this means the language-aware regex rules in userdiff.c,
>>     or user-supplied rules.
>>
>>  * -U<n>, which defaults to -U3 shows at least <n> lines of context,
>>     if that's greater than what we'd extend the context to under -W
>>     then -U<n> wins.
>>
>>  * When showing the hunk context we look up from the first line we
>>    show of the diff, and find whatever looks like useful context above
>>    that line.
>>
>> Thus in e.g. the xdiff/xemit.c change being made in this commit we'll
>> correctly show "xdl_emit_diff()" in the hunk context under default
>> diff settings.
>>
>> But if we viewed it with the -W option we'd show "is_empty_rec()",
>> because we'd first find the "xdl_emit_diff()" context line, extend the
>> diff to that, and then would go look for context to show again.
>>
>> I don't think this behavior makes any sense, our context in this case
>> is what we're guaranteed to show as part of the diff itself.
>>
>> The user already asked us to find that context line and show it, we
>> don't need to then start showing the context above that line, which
>> they didn't ask for.
>
> Hmm, that's subtle.
>
> Your reasoning applies to patches generated without -W as well.  If the
> precontext contains a function line then the @@ line should not contain
> a function comment.  However, e.g. with this:
>
> -- snip --
> cat >a <<EOF
> func a
>
> func b
> 1
> 2
> 3
> EOF
> sed 's/3/three/' <a >b
> diff -up a b
> -- snap --
>
> ... I get this:
>
> --- a	2021-02-15 18:30:21.000000000 +0100
> +++ b	2021-02-15 18:30:21.000000000 +0100
> @@ -3,4 +3,4 @@ func a
>  func b
>  1
>  2
> -3
> +three
>
> So diff(1) shows the previous function line.  git diff does the same.
>
> The behaviour of diff(1) and git diff does make sense to me: It's easy
> to implement and the only downside is that it produces extra output in
> some cases.
>
> I can understand that users would rather have a tidy diff without
> distractions, though.  So I like the output change you propose.

Does GNU diff have something like git's -W, both "diff -U 0 -F func a b"
and "diff -U 0 -p a b" don't extend the context window as we do.

I don't think the patch I'm submitting here would make sense for GNU
diff, since there it just shows the context without being guaranteed to
show the full set of lines leading up to it under -W, but with Git diff
we do that, so I think it makes sense to omit the context.

> However, I'm not sure it would be a good idea to clear @@ lines of hunks
> generated without -W that have function lines in their precontext, even
> though it would be a logical thing to do.

Yes, I don't think that's a good idea either. I think it only makes
sense under -W where the user explicitly asks "show me the function this
change was in", and we're (before this patch) showing different context
on the basis of emergent behavior.

>> This new behavior does give us the edge case that if we e.g. view the
>> diff here with "-U150 -W" we'd previously extend the context to the
>> middle of the "is_func_rec()" function, and show that function in the
>> hunk context. Now we'll show nothing.
>
> Well, the 150 lines of context are still shown (as they should be), but
> the @@ line contains no function name anymore.

Yes, indeed. I'll reword that to "now we'll show no context in that
case" or something...

>> I think that change also makes sense. We're showing a change in the
>> "xdl_emit_diff()" function. That's our context for the change. It
>> doesn't make sense with -W to start fishing around for other
>> context.
>
> It does make sense in the context of the diff(1) -p implementation, but
> your change is consistent with the description of that option: "Show
> which C function each change is in."

I hadn't spotted that, we just said:

    Show whole function as context lines for each change. The function
    names are determined in the same way as git diff works out patch
    hunk headers

Which I think can more obviously be read as the existing behavior being
desired, and this patch being a change to documented behavior.

(I think it is, I just think it makes sense to change the docs &
behavior int this case)

>> Arguably in that case we could save away the context we found in the
>> "XDL_EMIT_FUNCCONTEXT" in "xdl_emit_diff()" and show that if we end up
>> extending the diff past the function, either because of a high -U<n>
>> value, or because our change was right at the start.
>>
>> I wouldn't really mind if we did that, perhaps it would be a useful
>> marker with high -U<n> values to remind the user of what they're
>> looking at, but I also don't see the usefulness in practice, so let's
>> punt that for now.
>
> It could be confusing for someone who expects the old behaviour, leaving
> it empty makes more sense to me.

FWIW it would be useful for:

    git log -U1000 -W

And then searching for "@@" in the pager to find changes to specific
functions.

>>
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>>  Documentation/diff-options.txt | 4 ++++
>>  t/t4015-diff-whitespace.sh     | 2 +-
>>  t/t4018-diff-funcname.sh       | 7 +++++++
>>  xdiff/xemit.c                  | 4 +++-
>>  4 files changed, 15 insertions(+), 2 deletions(-)
>>
>> diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
>> index e5733ccb2d..8ca59effa7 100644
>> --- a/Documentation/diff-options.txt
>> +++ b/Documentation/diff-options.txt
>> @@ -759,6 +759,10 @@ endif::git-format-patch[]
>>  	The function names are determined in the same way as
>>  	`git diff` works out patch hunk headers (see 'Defining a
>>  	custom hunk-header' in linkgit:gitattributes[5]).
>> ++
>> +When showing the whole function for context the "@@" context line
>> +itself will always be empty, since the context that would otherwise be
>> +shown there will be the first line of the hunk being shown.
>>
>>  ifndef::git-format-patch[]
>>  ifndef::git-log[]
>> diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh
>> index 8c574221b2..0ffc845cdd 100755
>> --- a/t/t4015-diff-whitespace.sh
>> +++ b/t/t4015-diff-whitespace.sh
>> @@ -2133,7 +2133,7 @@ test_expect_success 'combine --ignore-blank-lines with --function-context 2' '
>>  		--ignore-blank-lines --function-context a b >actual.raw &&
>>  	sed -n "/@@/,\$p" <actual.raw >actual &&
>>  	cat <<-\EOF >expect &&
>> -	@@ -5,11 +6,9 @@ c
>> +	@@ -5,11 +6,9 @@
>>  	 function
>>  	 1
>>  	 2
>> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
>> index 80f35c5e16..f3374abd98 100755
>> --- a/t/t4018-diff-funcname.sh
>> +++ b/t/t4018-diff-funcname.sh
>> @@ -91,6 +91,13 @@ test_diff_funcname () {
>>  		fi
>>  	' &&
>>
>> +	test_expect_success "$desc -W" '
>> +		git diff -U0 -W "$what" >W-U0-diff &&
>> +		echo >W-U0-expected &&
>> +		last_diff_context_line W-U0-diff >W-U0-actual &&
>> +		test_cmp W-U0-expected W-U0-actual
>> +	' &&
>> +
>>  	test_expect_success "$desc (accumulated)" '
>>  		git diff -U1 "$what".acc >diff &&
>>  		last_diff_context_line diff >actual.lines &&
>> diff --git a/xdiff/xemit.c b/xdiff/xemit.c
>> index 9d7d6c5087..02b5dbcc70 100644
>> --- a/xdiff/xemit.c
>> +++ b/xdiff/xemit.c
>> @@ -274,7 +274,9 @@ int xdl_emit_diff(xdfenv_t *xe, xdchange_t *xscr, xdemitcb_t *ecb,
>>  		 */
>>
>>  		if (xecfg->flags & XDL_EMIT_FUNCNAMES) {
>> -			get_func_line(xe, xecfg, &func_line,
>> +			get_func_line(xe, xecfg,
>> +				      xecfg->flags & XDL_EMIT_FUNCCONTEXT
>> +				      ? NULL : &func_line,
>
> Why still search?  It would be better to turn off XDL_EMIT_FUNCNAMES if
> XDL_EMIT_FUNCCONTEXT is enabled -- a one-character change in diff.c.

I just didn't read the diff/xdiff code carefully enough. Will fix.

>>  				      s1 - 1, funclineprev);
>>  			funclineprev = s1 - 1;
>>  		}
>>

Re: [PATCH 1/2] diff: do not display hunk context under -W

[René Scharfe.]

Am 15.02.21 um 20:24 schrieb Ævar Arnfjörð Bjarmason:
>
> On Mon, Feb 15 2021, René Scharfe. wrote:
>
>> Your reasoning applies to patches generated without -W as well.  If the
>> precontext contains a function line then the @@ line should not contain
>> a function comment.  However, e.g. with this:
>>
>> -- snip --
>> cat >a <<EOF
>> func a
>>
>> func b
>> 1
>> 2
>> 3
>> EOF
>> sed 's/3/three/' <a >b
>> diff -up a b
>> -- snap --
>>
>> ... I get this:
>>
>> --- a	2021-02-15 18:30:21.000000000 +0100
>> +++ b	2021-02-15 18:30:21.000000000 +0100
>> @@ -3,4 +3,4 @@ func a
>>  func b
>>  1
>>  2
>> -3
>> +three
>>
>> So diff(1) shows the previous function line.  git diff does the same.
>>
>> The behaviour of diff(1) and git diff does make sense to me: It's easy
>> to implement and the only downside is that it produces extra output in
>> some cases.
>>
>> I can understand that users would rather have a tidy diff without
>> distractions, though.  So I like the output change you propose.
>
> Does GNU diff have something like git's -W, both "diff -U 0 -F func a b"
> and "diff -U 0 -p a b" don't extend the context window as we do.

Exactly my point: GNU diff doesn't have something like -W, but still can
show an unchanged function at the @@ line, because it searches upwards
starting just above the first shown line, and the name of the actually
changed function might be in one of the shown lines.

> I don't think the patch I'm submitting here would make sense for GNU
> diff, since there it just shows the context without being guaranteed to
> show the full set of lines leading up to it under -W, but with Git diff
> we do that, so I think it makes sense to omit the context.

Given the goal to show the name of the changed function it *would* make
sense to start searching at the first changed line instead.

>> However, I'm not sure it would be a good idea to clear @@ lines of hunks
>> generated without -W that have function lines in their precontext, even
>> though it would be a logical thing to do.
>
> Yes, I don't think that's a good idea either. I think it only makes
> sense under -W where the user explicitly asks "show me the function this
> change was in", and we're (before this patch) showing different context
> on the basis of emergent behavior.

Right, -W never needs diff(1)'s -p, while the case is less clear for
diffs generated without -W.

René

Re: [PATCH 1/2] diff: do not display hunk context under -W

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> The long-standing semantics of how -W works and interacts with -U<n>
> are rather easy to reason about:
>
>  * -W extends the context line up to the start of the function. With
>     userdiff this means the language-aware regex rules in userdiff.c,
>     or user-supplied rules.

OK.

>  * -U<n>, which defaults to -U3 shows at least <n> lines of context,
>     if that's greater than what we'd extend the context to under -W
>     then -U<n> wins.

Sorry, but I cannot quite guess what you mean here.  Do you mean:

    We first go back <n> lines due to -U<n>.  If we have not found a
    function header line that comes before the first changed line
    within that <n> lines, we keep going back to satisfy -W (i.e. -W
    "wins" if "-U<n>" is small).  On the other hand, after going
    back <n> lines, if we have already seen a function header before
    the first changed line within these <n> lines, there is no need
    to go back further to satisfy -W (i.e. -U<n> "wins" and makes -W
    irrelevant).

If that is the case, I think I understand it.

>  * When showing the hunk context we look up from the first line we
>    show of the diff, and find whatever looks like useful context above
>    that line.

Yes.

> Thus in e.g. the xdiff/xemit.c change being made in this commit we'll
> correctly show "xdl_emit_diff()" in the hunk context under default
> diff settings.
>
> But if we viewed it with the -W option we'd show "is_empty_rec()",
> because we'd first find the "xdl_emit_diff()" context line, extend the
> diff to that, and then would go look for context to show again.

Makes sense.  As long as we apply this when "-W wins", i.e we extend
the precontext so that the hunk begins on the line that match the
function header pattern, there isn't much point in showing only the
name of the function that comes before this hunk.

On the other hand, if "-U<n> won", i.e. the first precontext line
that is shown in the hunk is _before_ the line -W chose to extend
the hunk to (in this case, where xdl_emit_diff begins) because -U<n>
gave a large enough number, I do want to see the name of the fuction
I am looking at the tail of in the precontext on the hunk header
line.  E.g. "git diff -W -U120 xdiff/xemit.c" on this patch should
show is_empty_rec() as the hunk header.

> This new behavior does give us the edge case that if we e.g. view the
> diff here with "-U150 -W" we'd previously extend the context to the
> middle of the "is_func_rec()" function, and show that function in the
> hunk context. Now we'll show nothing.

To me, that sounds like a grave regression.  Why lose the
information?

This may be coming from the difference between us, i.e. I read a lot
more patches written by other people than my own changes written for
my next commit, so every bit of hint helps, and the name of the
function I am seeing its latter half in the precontext is sometimes
a useful thing to see.

Re: [PATCH 1/2] diff: do not display hunk context under -W

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>
>> But if we viewed it with the -W option we'd show "is_empty_rec()",
>> because we'd first find the "xdl_emit_diff()" context line, extend the
>> diff to that, and then would go look for context to show again.
>
> Makes sense.

Ehh, I did not mean "the current behaviour with -W that shows
is_empty_rec makes sense".  The observation you made (which lead to
the conclusion in your next paragraph that it is not a good idea
to show is_empty_rec on the hunk header line) made sense to me.

But I do not think any change from the current behaviour should be
made if -U<n> wins -W (i.e. the first precontext line shown in the
hunk header is not due to -W).  We should hunt for the name of the
function whose latter half we are seeing at the beginning of the
hunk in that case.

Thanks.

Re: [PATCH 1/2] diff: do not display hunk context under -W

[Johannes Sixt]

Am 16.02.21 um 02:30 schrieb Junio C Hamano:
> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>> This new behavior does give us the edge case that if we e.g. view the
>> diff here with "-U150 -W" we'd previously extend the context to the
>> middle of the "is_func_rec()" function, and show that function in the
>> hunk context. Now we'll show nothing.
> 
> To me, that sounds like a grave regression.  Why lose the
> information?
> 
> This may be coming from the difference between us, i.e. I read a lot
> more patches written by other people than my own changes written for
> my next commit, so every bit of hint helps, and the name of the
> function I am seeing its latter half in the precontext is sometimes
> a useful thing to see.

I totally agree with your assessment. I wouldn't even have removed the 
hunk header in the case of "-W wins", either, but that is a case that I 
can live with when others think it makes sense.

-- Hannes

Re: [PATCH 1/2] diff: do not display hunk context under -W

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> Am 16.02.21 um 02:30 schrieb Junio C Hamano:
>> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>>> This new behavior does give us the edge case that if we e.g. view the
>>> diff here with "-U150 -W" we'd previously extend the context to the
>>> middle of the "is_func_rec()" function, and show that function in the
>>> hunk context. Now we'll show nothing.
>> To me, that sounds like a grave regression.  Why lose the
>> information?
>> This may be coming from the difference between us, i.e. I read a lot
>> more patches written by other people than my own changes written for
>> my next commit, so every bit of hint helps, and the name of the
>> function I am seeing its latter half in the precontext is sometimes
>> a useful thing to see.
>
> I totally agree with your assessment. I wouldn't even have removed the
> hunk header in the case of "-W wins", either, but that is a case that
> I can live with when others think it makes sense.

Ditto.

The information on @@ ... @@ line may look misleading especially to
those who are not used to reading patches in the "-W wins" case.  It
is otherwise not hurting anybody and it loses information to remove
it, but it is not as grave as in the "-U<n> wins" case, so I do not
mind losing it, if it helps them.

[PATCH 2/2] diff: test and document -W interaction with -U<n>

[Ævar Arnfjörð Bjarmason]

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/diff-options.txt | 8 ++++++++
 t/t4018-diff-funcname.sh       | 5 +++++
 2 files changed, 13 insertions(+)

diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index 8ca59effa7..3c19c78616 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -88,6 +88,11 @@ endif::git-log[]
 --unified=<n>::
 	Generate diffs with <n> lines of context instead of
 	the usual three.
++
+Under `-W` generates diffs with at least <n> lines of context, if the
+number is lower than the context `-U<n>` would extend the diff to then
+`-U<n>` takes precedence.
+
 ifndef::git-format-patch[]
 	Implies `--patch`.
 endif::git-format-patch[]
@@ -763,6 +768,9 @@ endif::git-format-patch[]
 When showing the whole function for context the "@@" context line
 itself will always be empty, since the context that would otherwise be
 shown there will be the first line of the hunk being shown.
++
+See the documentation for `-U<n>` above for how the two options
+interact.
 
 ifndef::git-format-patch[]
 ifndef::git-log[]
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index f3374abd98..38dc029917 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -98,6 +98,11 @@ test_diff_funcname () {
 		test_cmp W-U0-expected W-U0-actual
 	' &&
 
+	test_expect_success "$desc -W interaction with -U<n>" '
+		git diff -U9001 "$what" >W-U9001-diff &&
+		grep "^@@ -1," W-U9001-diff
+	' &&
+
 	test_expect_success "$desc (accumulated)" '
 		git diff -U1 "$what".acc >diff &&
 		last_diff_context_line diff >actual.lines &&
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH 2/2] diff: test and document -W interaction with -U<n>

[Johannes Sixt]

Am 15.02.21 um 16:50 schrieb Ævar Arnfjörð Bjarmason:
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   Documentation/diff-options.txt | 8 ++++++++
>   t/t4018-diff-funcname.sh       | 5 +++++
>   2 files changed, 13 insertions(+)
> 
> diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
> index 8ca59effa7..3c19c78616 100644
> --- a/Documentation/diff-options.txt
> +++ b/Documentation/diff-options.txt
> @@ -88,6 +88,11 @@ endif::git-log[]
>   --unified=<n>::
>   	Generate diffs with <n> lines of context instead of
>   	the usual three.
> ++
> +Under `-W` generates diffs with at least <n> lines of context, if the
> +number is lower than the context `-U<n>` would extend the diff to then
> +`-U<n>` takes precedence.
> +

How about (not as separate paragraph):

When combined with `--function-context`, this specifies the minimum of 
context lines.

>   ifndef::git-format-patch[]
>   	Implies `--patch`.
>   endif::git-format-patch[]
> @@ -763,6 +768,9 @@ endif::git-format-patch[]
>   When showing the whole function for context the "@@" context line
>   itself will always be empty, since the context that would otherwise be
>   shown there will be the first line of the hunk being shown.
> ++
> +See the documentation for `-U<n>` above for how the two options
> +interact.

How about

Use `-U<n>` to specify a minimum of context (default three lines).

so that readers do not have to search.

-- Hannes

Re: [PATCH v2 00/27] userdiff: refactor + test + doc + misc improvements

[Eric Sunshine]

On Mon, Feb 15, 2021 at 10:44 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> Incorporates all the feedback on v2 and more, see the range-diff
> below.

I hadn't finished reading the previous version of this series...
Nevertheless, see a few issues below which I noticed while scanning
the range-diff...

>     ++expect to appear in the hunk header. We munged away the starting "@@
>     ++[...] @@" part of the line for ease of not having to hardcode the line
>     ++numbers and offsets.

Nit: "munge" is an oddball word to use here. "We strip away the
starting..." would be simpler, but perhaps it doesn't matter too much
as this documentation is aimed at developers, not end users.

>     ++For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
>     ++configuration file as discussed above, but if present, it will
>     ++override a built-in pattern.
>      +
>     -+You still need to enable built-in patterns with the the attribute
>     -+mechanism, via `.gitattributes`).
>     ++Nevertheless, you need to enable built-in patterns via .gitattributes`
>     ++for the pattern to take effect.

Missing opening backtick on `.gitattributes`.

>     ++Patterns in in a list of multiple that begin with "!" are negated. A
>     ++matching negated pattern will cause the matched line to be
>     ++skipped. Use it to skip a later pattern that would otherwise match. It
>     ++is an error if one or more negated patterns aren't followed by a
>     ++non-negated pattern.

s/in in/in/

Also, "of multiple" what?

>     ++To match a literal "!" at the start of a line, use some other regex
>     ++construct that will match a literal "!" without "!" being the first
>     ++character on that line, such as "[!]".

Overall, I find this description easier to read and understand than in
the previous version.

Re: [PATCH v2 00/27] userdiff: refactor + test + doc + misc improvements

[Johannes Sixt]

Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
> Incorporates all the feedback on v2 and more, see the range-diff
> below.

I've read through all patches and left a few comments.

My main critique is that the new way to specify test cases is not an 
improvement because the code is hard to parse. The  desire to improve 
test precision moves the balance too far away from the simplicity to add 
new test cases, IMO.

I appreciate the new test cases for ADA and Ruby as well as the 
discovery and fixes of the many small deficiencies.

-- Hannes

[PATCH v3 00/35] 20210215154427.32693-1-avarab@gmail.com

[Ævar Arnfjörð Bjarmason]

Addresses feedback on v2. Since Junio & Johannes expressed a desire to
keep the existing test scheme in t4018/* it's still there, but it's
also possible to add *.sh tests in that directory to use the more
familiar test framework used elsewhere in the test suite.

The tests added here make use of it to e.g. supply custom -U<n>
arguments, set config before the tests etc.

I also improved that existing test support so you can have N tests in
one file with (mostly) the existing test syntax. See the "userdiff
tests: add a test with multiple tests in a LANG file" patch.

Range-diff below.

Ævar Arnfjörð Bjarmason (35):
  userdiff: refactor away the parse_bool() function
  userdiff style: re-order drivers in alphabetical order
  userdiff style: declare patterns with consistent style
  userdiff style: normalize pascal regex declaration
  userdiff: add and use for_each_userdiff_driver()
  userdiff tests: explicitly test "default" pattern
  userdiff tests: list builtin drivers via test-tool
  userdiff: remove support for "broken" tests
  blame tests: don't rely on t/t4018/ directory
  blame tests: simplify userdiff driver test
  userdiff tests: match full hunk headers
  userdiff tests: change setup loop to individual test setup
  userdiff tests: factor out test_diff_funcname() logic
  userdiff tests: add alternative hunk header test infrastructure
  userdiff tests: add a test with multiple tests in a LANG file
  userdiff tests: do config teardown in test_diff_funcname()
  userdiff tests: move custom patterns into one test file
  userdiff tests: remove hack for "RIGHT" token
  userdiff tests: do not do compile tests on "custom" pattern
  userdiff tests: assert that new built-in drivers have tests
  userdiff tests + docs: document & test "diff.<driver>.x?funcname"
  gitattributes doc: reword discussion of built-in userdiff patterns
  userdiff tests: move perl tests to perl.sh
  userdiff tests: move away from "RIGHT" in perl.sh
  gitattributes doc: document multi-line userdiff patterns
  userdiff tests: switch to -U0 by default
  userdiff tests: remove "funcname" from custom3 test
  userdiff tests: assert empty hunk header context on -U<large>
  userdiff tests: test for a bug in 1dbf0c0ad6c
  userdiff golang: simplify and correct matching regex
  userdiff golang: don't over-match intented constructs
  userdiff golang: add a rule to match "package"
  userdiff golang: match multi-line "const" and "import"
  userdiff tests: add basic test for ada
  userdiff tests: add basic test for ruby

 Documentation/config/diff.txt              |  11 ++
 Documentation/gitattributes.txt            |  46 ++++-
 Makefile                                   |   1 +
 t/annotate-tests.sh                        |  34 ++--
 t/helper/test-tool.c                       |   1 +
 t/helper/test-tool.h                       |   1 +
 t/helper/test-userdiff.c                   |  31 ++++
 t/t4018-diff-funcname.sh                   | 180 +++++++++++++-------
 t/t4018/README                             |  69 ++++++--
 t/t4018/ada.sh                             |  37 ++++
 t/t4018/bash-arithmetic-function           |   1 +
 t/t4018/bash-bashism-style-compact         |   1 +
 t/t4018/bash-bashism-style-function        |   1 +
 t/t4018/bash-bashism-style-whitespace      |   1 +
 t/t4018/bash-conditional-function          |   1 +
 t/t4018/bash-missing-parentheses           |   1 +
 t/t4018/bash-mixed-style-compact           |   1 +
 t/t4018/bash-mixed-style-function          |   1 +
 t/t4018/bash-nested-functions              |   1 +
 t/t4018/bash-other-characters              |   1 +
 t/t4018/bash-posix-style-compact           |   1 +
 t/t4018/bash-posix-style-function          |   1 +
 t/t4018/bash-posix-style-whitespace        |   1 +
 t/t4018/bash-subshell-function             |   1 +
 t/t4018/bash-trailing-comment              |   1 +
 t/t4018/cpp-c++-function                   |   1 +
 t/t4018/cpp-class-constructor              |   1 +
 t/t4018/cpp-class-constructor-mem-init     |   1 +
 t/t4018/cpp-class-definition               |   1 +
 t/t4018/cpp-class-definition-derived       |   1 +
 t/t4018/cpp-class-destructor               |   1 +
 t/t4018/cpp-function-returning-global-type |   1 +
 t/t4018/cpp-function-returning-nested      |   1 +
 t/t4018/cpp-function-returning-pointer     |   1 +
 t/t4018/cpp-function-returning-reference   |   1 +
 t/t4018/cpp-gnu-style-function             |   1 +
 t/t4018/cpp-namespace-definition           |   1 +
 t/t4018/cpp-operator-definition            |   1 +
 t/t4018/cpp-skip-access-specifiers         |   1 +
 t/t4018/cpp-skip-comment-block             |   1 +
 t/t4018/cpp-skip-labels                    |   1 +
 t/t4018/cpp-struct-definition              |   1 +
 t/t4018/cpp-struct-single-line             |   1 +
 t/t4018/cpp-template-function-definition   |   1 +
 t/t4018/cpp-union-definition               |   1 +
 t/t4018/cpp-void-c-function                |   1 +
 t/t4018/css-attribute-value-selector       |   1 +
 t/t4018/css-block-level-@-statements       |   1 +
 t/t4018/css-brace-in-col-1                 |   1 +
 t/t4018/css-class-selector                 |   1 +
 t/t4018/css-colon-eol                      |   1 +
 t/t4018/css-colon-selector                 |   1 +
 t/t4018/css-common                         |   1 +
 t/t4018/css-id-selector                    |   1 +
 t/t4018/css-long-selector-list             |   1 +
 t/t4018/css-prop-sans-indent               |   1 +
 t/t4018/css-root-selector                  |   1 +
 t/t4018/css-short-selector-list            |   1 +
 t/t4018/css-trailing-space                 |   1 +
 t/t4018/custom.sh                          | 183 ++++++++++++++++++++
 t/t4018/custom1-pattern                    |  17 --
 t/t4018/custom2-match-to-end-of-line       |   8 -
 t/t4018/custom3-alternation-in-pattern     |  17 --
 t/t4018/dts-labels                         |   1 +
 t/t4018/dts-node-unitless                  |   1 +
 t/t4018/dts-nodes                          |   1 +
 t/t4018/dts-nodes-boolean-prop             |   1 +
 t/t4018/dts-nodes-comment1                 |   1 +
 t/t4018/dts-nodes-comment2                 |   1 +
 t/t4018/dts-nodes-multiline-prop           |   1 +
 t/t4018/dts-reference                      |   1 +
 t/t4018/dts-root                           |   1 +
 t/t4018/dts-root-comment                   |   1 +
 t/t4018/elixir-do-not-pick-end             |   1 +
 t/t4018/elixir-ex-unit-test                |   1 +
 t/t4018/elixir-function                    |   1 +
 t/t4018/elixir-macro                       |   1 +
 t/t4018/elixir-module                      |   1 +
 t/t4018/elixir-module-func                 |   1 +
 t/t4018/elixir-nested-module               |   1 +
 t/t4018/elixir-private-function            |   1 +
 t/t4018/elixir-protocol                    |   1 +
 t/t4018/elixir-protocol-implementation     |   1 +
 t/t4018/fortran-block-data                 |   1 +
 t/t4018/fortran-comment                    |   1 +
 t/t4018/fortran-comment-keyword            |   1 +
 t/t4018/fortran-comment-legacy             |   1 +
 t/t4018/fortran-comment-legacy-star        |   1 +
 t/t4018/fortran-external-function          |   1 +
 t/t4018/fortran-external-subroutine        |   1 +
 t/t4018/fortran-module                     |   1 +
 t/t4018/fortran-module-procedure           |   1 +
 t/t4018/fortran-program                    |   1 +
 t/t4018/fountain-scene                     |   1 +
 t/t4018/golang                             | 148 ++++++++++++++++
 t/t4018/golang-complex-function            |   8 -
 t/t4018/golang-func                        |   4 -
 t/t4018/golang-interface                   |   4 -
 t/t4018/golang-long-func                   |   5 -
 t/t4018/golang-struct                      |   4 -
 t/t4018/java-class-member-function         |   1 +
 t/t4018/markdown-heading-indented          |   1 +
 t/t4018/markdown-heading-non-headings      |   1 +
 t/t4018/matlab-class-definition            |   1 +
 t/t4018/matlab-function                    |   1 +
 t/t4018/matlab-octave-section-1            |   1 +
 t/t4018/matlab-octave-section-2            |   1 +
 t/t4018/matlab-section                     |   1 +
 t/t4018/perl-skip-end-of-heredoc           |   8 -
 t/t4018/perl-skip-forward-decl             |  10 --
 t/t4018/perl-skip-sub-in-pod               |  18 --
 t/t4018/perl-sub-definition                |   4 -
 t/t4018/perl-sub-definition-kr-brace       |   4 -
 t/t4018/perl.sh                            |  93 +++++++++++
 t/t4018/php-abstract-class                 |   1 +
 t/t4018/php-abstract-method                |   1 +
 t/t4018/php-class                          |   1 +
 t/t4018/php-final-class                    |   1 +
 t/t4018/php-final-method                   |   1 +
 t/t4018/php-function                       |   1 +
 t/t4018/php-interface                      |   1 +
 t/t4018/php-method                         |   1 +
 t/t4018/php-trait                          |   1 +
 t/t4018/python-async-def                   |   1 +
 t/t4018/python-class                       |   1 +
 t/t4018/python-def                         |   1 +
 t/t4018/python-indented-async-def          |   1 +
 t/t4018/python-indented-class              |   1 +
 t/t4018/python-indented-def                |   1 +
 t/t4018/ruby.sh                            |  58 +++++++
 t/t4018/rust-fn                            |   1 +
 t/t4018/rust-impl                          |   1 +
 t/t4018/rust-macro-rules                   |   1 +
 t/t4018/rust-struct                        |   1 +
 t/t4018/rust-trait                         |   1 +
 userdiff.c                                 | 186 +++++++++++++--------
 userdiff.h                                 |  15 ++
 137 files changed, 1036 insertions(+), 277 deletions(-)
 create mode 100644 t/helper/test-userdiff.c
 create mode 100755 t/t4018/ada.sh
 create mode 100755 t/t4018/custom.sh
 delete mode 100644 t/t4018/custom1-pattern
 delete mode 100644 t/t4018/custom2-match-to-end-of-line
 delete mode 100644 t/t4018/custom3-alternation-in-pattern
 create mode 100644 t/t4018/golang
 delete mode 100644 t/t4018/golang-complex-function
 delete mode 100644 t/t4018/golang-func
 delete mode 100644 t/t4018/golang-interface
 delete mode 100644 t/t4018/golang-long-func
 delete mode 100644 t/t4018/golang-struct
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc
 delete mode 100644 t/t4018/perl-skip-forward-decl
 delete mode 100644 t/t4018/perl-skip-sub-in-pod
 delete mode 100644 t/t4018/perl-sub-definition
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace
 create mode 100755 t/t4018/perl.sh
 create mode 100755 t/t4018/ruby.sh

Range-diff:
 1:  305fc646d0d =  1:  0be132b05e2 userdiff: refactor away the parse_bool() function
 2:  989438c46ae =  2:  d1e00a739ac userdiff style: re-order drivers in alphabetical order
 3:  4c48e5532ce =  3:  b99bd158d45 userdiff style: declare patterns with consistent style
 4:  f41fa5b316f =  4:  9ce6d47021c userdiff style: normalize pascal regex declaration
 5:  0875d5205c1 =  5:  369fbdcee83 userdiff: add and use for_each_userdiff_driver()
 6:  638247d04d5 =  6:  70d62a97211 userdiff tests: explicitly test "default" pattern
 7:  219043a4881 !  7:  792421a2f8b userdiff tests: list builtin drivers via test-tool
    @@ t/helper/test-userdiff.c (new)
     +static int driver_cb(struct userdiff_driver *driver,
     +		     enum userdiff_driver_type type, void *priv)
     +{
    -+	puts(driver->name);
    ++	if (driver->funcname.pattern)
    ++		puts(driver->name);
     +	return 0;
     +}
     +
    @@ t/t4018-diff-funcname.sh: test_description='Test custom diff function name patte
      . ./test-lib.sh
      
      test_expect_success 'setup' '
    -+	builtin_drivers=$(test-tool userdiff list-builtin-drivers) &&
    -+	test -n "$builtin_drivers" &&
    ++	# Make sure additions to builtin_drivers are sorted
    ++	test_when_finished "rm builtin-drivers.sorted" &&
    ++	test-tool userdiff list-builtin-drivers >builtin-drivers &&
    ++	test_file_not_empty builtin-drivers &&
    ++	sort <builtin-drivers >builtin-drivers.sorted &&
    ++	test_cmp builtin-drivers.sorted builtin-drivers &&
     +
      	# a non-trivial custom pattern
      	git config diff.custom1.funcname "!static
    @@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
     -	rust
     -	tex
     -	default
    -+	$builtin_drivers
    ++	$(cat builtin-drivers)
      	custom1
      	custom2
      	custom3
 8:  eb66160aac7 =  8:  9081e2a152e userdiff: remove support for "broken" tests
 9:  c6c54039e27 <  -:  ----------- userdiff tests: match full hunk headers
10:  1c6ddf96f61 !  9:  d3652f95d5e blame tests: don't rely on t/t4018/ directory
    @@ Commit message
         with userdiff driver, 2020-11-01) so that the blame tests don't rely
         on stealing the contents of "t/t4018/fortran-external-function".
     
    -    I'm about to refactor that directory to delete that file, just moving
    -    the relevant test file here inline is the easiest solution, and I
    -    think also the most readable.
    +    I'm about to change that file in a subsequent commit. Just moving the
    +    relevant test file here inline is the easiest solution, and I think
    +    also the most readable.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
11:  8a883d87990 ! 10:  35d12779ea1 blame tests: simplify userdiff driver test
    @@ Commit message
         added in 999cfc4f45 (test-lib functions: add --author support to
         test_commit, 2021-01-12).
     
    -    We also did not need the full fortran-external-function content, let's
    +    We also did not need the full fortran-external-function content. Let's
         cut it down to just the important parts, and further modify it to
         demonstrate that the fortran-specific userdiff function is in effect
    -    by adding "WRONG" lines surrounding the "RIGHT" one.
    +    by adding "DO NOT MATCH ..." and "AS THE ..." lines surrounding the
    +    "RIGHT" one. This is to check that we're using the userdiff "fortran"
    +    driver, as opposed to the default driver.
     
         The test also left behind a .gitattributes files, let's clean it up
         with "test_when_finished".
    @@ t/annotate-tests.sh: test_expect_success 'blame -L ^:RE (absolute: end-of-file)'
     -	cat >$fortran_file <<-\EOF &&
     +test_expect_success 'blame -L :funcname with userdiff driver' '
     +	cat >file.template <<-\EOF &&
    -+	def WRONG begin end
    ++	DO NOT MATCH THIS LINE
      	function RIGHT(a, b) result(c)
    -+	int WRONG(void) {}
    ++	AS THE DEFAULT DRIVER WOULD
      
      	integer, intent(in) :: ChangeMe
     -	integer, intent(in) :: b
    @@ t/annotate-tests.sh: test_expect_success 'blame -L ^:RE (absolute: end-of-file)'
     +	echo "$fortran_file diff=fortran" >.gitattributes &&
     +
     +	test_commit --author "A <A@test.git>" \
    -+		"add" $fortran_file \
    ++		"add" "$fortran_file" \
     +		"$(cat file.template)" &&
     +	test_commit --author "B <B@test.git>" \
    -+		"change" $fortran_file \
    ++		"change" "$fortran_file" \
     +		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
    -+	check_count -f $fortran_file -L:RIGHT A 3 B 1
    ++	check_count -f "$fortran_file" -L:RIGHT A 3 B 1
      '
      
      test_expect_success 'setup incremental' '
12:  e56a7a6b5f4 <  -:  ----------- userdiff tests: rewrite hunk header test infrastructure
 -:  ----------- > 11:  4bd8a0daa25 userdiff tests: match full hunk headers
 -:  ----------- > 12:  d2d74476f2a userdiff tests: change setup loop to individual test setup
21:  9a18506aff8 ! 13:  8db95a69924 userdiff tests: factor out test_diff_funcname() logic
    @@ Commit message
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## t/t4018-diff-funcname.sh ##
    -@@ t/t4018-diff-funcname.sh: test_expect_success 'last regexp must not be negated' '
    - 	test_i18ngrep ": Last expression must not be negated:" msg
    +@@ t/t4018-diff-funcname.sh: test_expect_success 'setup hunk header tests' '
    + 	git -C t4018 add .
      '
      
     +do_change_me () {
    @@ t/t4018-diff-funcname.sh: test_expect_success 'last regexp must not be negated'
     +	sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <$file
     +}
     +
    - test_diff_funcname () {
    - 	desc=$1
    - 	cat <&8 >arg.header &&
    -@@ t/t4018-diff-funcname.sh: test_diff_funcname () {
    - 		cp arg.test "$what" &&
    - 		cp arg.header expected &&
    - 		git add "$what" &&
    --		sed -e "s/ChangeMe/IWasChanged/" <"$what" >tmp &&
    --		mv tmp "$what"
    -+		do_change_me "$what"
    - 	' &&
    + # check each individual file
    + for i in $(git -C t4018 ls-files)
    + do
    +@@ t/t4018-diff-funcname.sh: do
      
    - 	test_expect_success "$desc" '
    - 		git diff -U1 "$what" >diff &&
    --		sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <diff >actual &&
    -+		last_diff_context_line diff >actual &&
    - 		test_cmp expected actual
    - 	' &&
    + 		# add test file to the index
    + 		git add \"$i\" &&
    +-		# place modified file in the worktree
    +-		sed -e 's/ChangeMe/IWasChanged/' <\"t4018/$i.content\" >\"$i\"
    ++		do_change_me \"$i\"
    + 	"
      
    + 	test_expect_success "hunk header: $i" "
    + 		git diff -U1 $i >diff &&
    +-		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
    ++		last_diff_context_line diff >ctx &&
    + 		test_cmp t4018/$i.header ctx
    + 	"
    + done
 -:  ----------- > 14:  e64a00d020e userdiff tests: add alternative hunk header test infrastructure
 -:  ----------- > 15:  3dab65bf394 userdiff tests: add a test with multiple tests in a LANG file
13:  84d20a7cd0c ! 16:  6eff13d01d3 userdiff tests: do config teardown in test_diff_funcname()
    @@ Commit message
      ## t/t4018-diff-funcname.sh ##
     @@ t/t4018-diff-funcname.sh: test_diff_funcname () {
      		git diff -U1 "$what" >diff &&
    - 		sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <diff >actual &&
    + 		last_diff_context_line diff >actual &&
      		test_cmp expected actual
     +	' &&
     +
14:  70fc9fa565b ! 17:  48f15aed4e4 userdiff tests: move custom patterns into one test file
    @@ Commit message
     
      ## t/t4018-diff-funcname.sh ##
     @@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
    - 	builtin_drivers=$(test-tool userdiff list-builtin-drivers) &&
    - 	test -n "$builtin_drivers" &&
    + 	sort <builtin-drivers >builtin-drivers.sorted &&
    + 	test_cmp builtin-drivers.sorted builtin-drivers &&
      
     -	# a non-trivial custom pattern
     -	git config diff.custom1.funcname "!static
    @@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
     @@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
      
      diffpatterns="
    - 	$builtin_drivers
    + 	$(cat builtin-drivers)
     -	custom1
     -	custom2
     -	custom3
15:  8539d6d464e = 18:  11556fe0967 userdiff tests: remove hack for "RIGHT" token
16:  121e5d6dfaf ! 19:  1b96e89843c userdiff tests: do not do compile tests on "custom" pattern
    @@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
      '
      
     -diffpatterns="
    --	$builtin_drivers
    +-	$(cat builtin-drivers)
     -	custom
     -"
     -
     -for p in $diffpatterns
    -+for p in $builtin_drivers
    ++for p in $(cat builtin-drivers)
      do
      	test_expect_success "builtin $p pattern compiles" '
      		echo "*.java diff=$p" >.gitattributes &&
    -@@ t/t4018-diff-funcname.sh: test_diff_funcname () {
    - 	'
    - }
    +@@ t/t4018-diff-funcname.sh: test_expect_success 'last regexp must not be negated' '
    + 	test_i18ngrep ": Last expression must not be negated:" msg
    + '
      
    --for what in $diffpatterns
    -+for what in $builtin_drivers custom
    - do
    - 	test="$TEST_DIRECTORY/t4018/$what.sh"
    - 	if ! test -e "$test"
    ++diffpatterns="
    ++	$(cat builtin-drivers)
    ++	custom
    ++"
    ++
    + test_expect_success 'setup hunk header tests' '
    + 	for i in $diffpatterns
    + 	do
 -:  ----------- > 20:  d3cbfc4354d userdiff tests: assert that new built-in drivers have tests
17:  451b7ae453d = 21:  fd6c51ac6db userdiff tests + docs: document & test "diff.<driver>.x?funcname"
18:  5a402bb9bf1 = 22:  43d818be785 gitattributes doc: reword discussion of built-in userdiff patterns
 -:  ----------- > 23:  37d54d77755 userdiff tests: move perl tests to perl.sh
 -:  ----------- > 24:  b6f4f613857 userdiff tests: move away from "RIGHT" in perl.sh
19:  a3badb1a3ee ! 25:  a39ba8e2545 gitattributes doc: document multi-line userdiff patterns
    @@ Documentation/gitattributes.txt: backslashes; the pattern above picks a line tha
     -There are built-in patterns shipped as part of git itself. A more
     -advanced version of the `tex` pattern discussed above is one of them.
     +Multiple patterns can be supplied by listing them one per line
    -+separated by `\n`. They will be matched one at a time from left to
    -+right. Do not supply a trailing "\n" for the last pattern. E.g.:
    ++separated by `\n`. They will be matched one line at a time, e.g.:
     +
     +------------------------
     +[diff "perl"]
     +	xfuncname = "!^=head\n^[^ ]+.*"
     +------------------------
     +
    -+Patterns in in a list of multiple that begin with "!" are negated. A
    -+matching negated pattern will cause the matched line to be
    ++Patterns in a list of multiple patterns that begin with "!" are
    ++negated. A matching negated pattern will cause the matched line to be
     +skipped. Use it to skip a later pattern that would otherwise match. It
     +is an error if one or more negated patterns aren't followed by a
     +non-negated pattern.
    @@ Documentation/gitattributes.txt: backslashes; the pattern above picks a line tha
     +construct that will match a literal "!" without "!" being the first
     +character on that line, such as "[!]".
     +
    ++If the last pattern in a list of multiple patterns ends with "\n" it
    ++will be interpreted as an empty pattern, and will match the first
    ++empty line. It's almost always a logic error to provide a list of
    ++multiple patterns ending with "\n", but it's permitted in case you
    ++genuinely want to match an empty line.
    ++
     +If the pattern contains a `$1` capture it will be used instead of the
     +entire matching line (`$0`) to display the hunk header. This can be
     +used e.g. to strip whitespace from the beginning of the line, or to
    @@ t/t4018/custom.sh: ChangeMe
     +foo
     +EOF_HUNK
     +sub foo;
    ++
     +=head1
    ++
    ++ChangeMe
    ++
    ++EOF_TEST
    ++
    ++test_expect_success 'custom: multiple patterns ending with \n' '
    ++	git config diff.custom.xfuncname "!^=head
    ++^sub ([^;]+)
    ++"
    ++'
    ++
    ++test_diff_funcname 'custom: multiple patterns ending with \n' \
    ++	8<<\EOF_HUNK 9<<\EOF_TEST
    ++
    ++EOF_HUNK
    ++sub foo;
    ++
    ++=head1
    ++
     +ChangeMe
     +
     +EOF_TEST
     
      ## t/t4018/perl.sh ##
    -@@ t/t4018/perl.sh: sub RIGHT
    +@@ t/t4018/perl.sh: sub asub
      	print "ChangeMe\n";
      }
      EOF_TEST
    @@ t/t4018/perl.sh: sub RIGHT
     +EOF_HUNK
     +sub foo;
     +=head1
    -+ChangeMe
     +
    ++ChangeMe
     +EOF_TEST
 -:  ----------- > 26:  27394c6c2a4 userdiff tests: switch to -U0 by default
20:  1b46726e85f = 27:  91ab863a298 userdiff tests: remove "funcname" from custom3 test
22:  24548fb680e <  -:  ----------- userdiff tests: test hunk headers on accumulated files
23:  48f00a59d5e <  -:  ----------- userdiff tests: test hunk header selection with -U0
24:  05a01990c9c ! 28:  b68133ce5f7 userdiff tests: assert empty hunk header context on -U<large>
    @@ Commit message
     
         Assert the existing behavior that under -U<large> we'll show no hunk
         header context, where <large> takes us past the potential hunk header
    -    we'd have extracted. I'm just picking a number over nine thousand as a
    -    really large number we're unlikely to exceed in these tests.
    +    we'd have extracted.
    +
    +    I'm just picking a number over nine thousand as a really large number
    +    we're unlikely to exceed in these tests.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    @@ t/t4018-diff-funcname.sh: test_diff_funcname () {
      		test_cmp expected actual
      	' &&
      
    -+	test_expect_success "$desc -U9001 (accumulated)" '
    -+		git diff -U9001 "$what".acc >diff &&
    -+		last_diff_context_line diff >actual.lines &&
    -+		tail -n 1 actual.lines >actual &&
    ++	test_expect_success "$desc -U9001" '
    ++		git diff -U9001 "$what" >diff &&
    ++		last_diff_context_line diff >actual &&
     +		echo >blank &&
     +		test_cmp blank actual
     +	' &&
25:  3d2f42d7041 <  -:  ----------- userdiff: match "package" in diff=golang
 -:  ----------- > 29:  9f3a7ca788b userdiff tests: test for a bug in 1dbf0c0ad6c
 -:  ----------- > 30:  43ee24e554b userdiff golang: simplify and correct matching regex
 -:  ----------- > 31:  70a2e7ca70b userdiff golang: don't over-match intented constructs
 -:  ----------- > 32:  6b942cd651b userdiff golang: add a rule to match "package"
 -:  ----------- > 33:  f45d35387d9 userdiff golang: match multi-line "const" and "import"
26:  b2e16ade06c ! 34:  c67c3e160f3 userdiff tests: add basic test for ada
    @@ Commit message
         1. https://rosettacode.org/wiki/99_bottles_of_beer#Ada
         2. https://en.wikibooks.org/wiki/Ada_Programming/Tasking
     
    + ## t/t4018-diff-funcname.sh ##
    +@@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
    + 	# Do not add anything to this list. New built-in drivers should have
    + 	# tests
    + 	cat >drivers-no-tests <<-\EOF &&
    +-	ada
    + 	bibtex
    + 	csharp
    + 	html
    +
      ## t/t4018/ada.sh (new) ##
     @@
     +#!/bin/sh
27:  826b6f4d6ae ! 35:  e2aedd738ef userdiff tests: add basic test for ruby
    @@ Commit message
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    + ## t/t4018-diff-funcname.sh ##
    +@@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
    + 	html
    + 	objc
    + 	pascal
    +-	ruby
    + 	tex
    + 	EOF
    + 
    +
      ## t/t4018/ruby.sh (new) ##
     @@
     +#!/bin/sh
    @@ t/t4018/ruby.sh (new)
     +EOF_TEST
     +
     +test_diff_funcname 'ruby: picks first "class/module/def" before changed context' \
    -+	"class Two" \
    ++	'-U1' \
     +	8<<\EOF_HUNK 9<<\EOF_TEST
     +class One
     +EOF_HUNK
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 00/35] 20210215154427.32693-1-avarab@gmail.com

[Johannes Sixt]

Am 24.02.21 um 20:50 schrieb Ævar Arnfjörð Bjarmason:
> Addresses feedback on v2. Since Junio & Johannes expressed a desire to
> keep the existing test scheme in t4018/* it's still there, but it's
> also possible to add *.sh tests in that directory to use the more
> familiar test framework used elsewhere in the test suite.
> 
> The tests added here make use of it to e.g. supply custom -U<n>
> arguments, set config before the tests etc.
> 
> I also improved that existing test support so you can have N tests in
> one file with (mostly) the existing test syntax. See the "userdiff
> tests: add a test with multiple tests in a LANG file" patch.

I've read through all patches and had a comment here and there. I like a
lot that we can now put more than one test into a single file.

However, I do not like the shell script version of tests, because the
syntax is so hard to read. Also, it looks to me that they are only
needed for a few tests that could just as well be coded as one-off tests
outside the framework.

I've now pulled avar/t4018-diff-hunk-header-regex-tests-3 from your
github repo and will check again if I missed some cruicial points.

-- Hannes

Re: [PATCH v3 00/35] 20210215154427.32693-1-avarab@gmail.com

[Johannes Sixt]

Am 27.02.21 um 08:47 schrieb Johannes Sixt:
> Am 24.02.21 um 20:50 schrieb Ævar Arnfjörð Bjarmason:
>> Addresses feedback on v2. Since Junio & Johannes expressed a desire to
>> keep the existing test scheme in t4018/* it's still there, but it's
>> also possible to add *.sh tests in that directory to use the more
>> familiar test framework used elsewhere in the test suite.
>>
>> The tests added here make use of it to e.g. supply custom -U<n>
>> arguments, set config before the tests etc.
>>
>> I also improved that existing test support so you can have N tests in
>> one file with (mostly) the existing test syntax. See the "userdiff
>> tests: add a test with multiple tests in a LANG file" patch.
> 
> I've read through all patches and had a comment here and there. I like a
> lot that we can now put more than one test into a single file.
> 
> However, I do not like the shell script version of tests, because the
> syntax is so hard to read. Also, it looks to me that they are only
> needed for a few tests that could just as well be coded as one-off tests
> outside the framework.
> 
> I've now pulled avar/t4018-diff-hunk-header-regex-tests-3 from your
> github repo and will check again if I missed some cruicial points.


I've now looked through the patch series again.

I appreciate that you dug through the history and discovered and fixed a
few deficiencies and loose ends. The way to throw all test cases for a
language driver into a single file I like a lot, too. The way to specify
expected results is manageable (modulo the dependency on the test
number, t4018, that Junio mentioned).

But I do not see the need for the framework provided by the new
test_diff_funcname. At the end of the series, it is only used by Perl
and custom driver tests. (I discount the new ADA and Ruby tests as they
can easily migrated to the simple test scheme.) But then the Perl tests
do not do anything special, either. The multi-line pattern test is just
a nice add-on but not strictly needed. In the end, the Perl test is just
as straight-forward as all others.

That leaves the custom driver tests. I don't see the need for a new
framework just for that. It would be sufficient to integrate them in
main test file t4018-diff-funcname.sh if needed, or leave them in the
existing framework if possible.

-- Hannes

Re: [PATCH v3 00/35] 20210215154427.32693-1-avarab@gmail.com

[Ævar Arnfjörð Bjarmason]

On Sun, Feb 28 2021, Johannes Sixt wrote:

> Am 27.02.21 um 08:47 schrieb Johannes Sixt:
>> Am 24.02.21 um 20:50 schrieb Ævar Arnfjörð Bjarmason:
>>> Addresses feedback on v2. Since Junio & Johannes expressed a desire to
>>> keep the existing test scheme in t4018/* it's still there, but it's
>>> also possible to add *.sh tests in that directory to use the more
>>> familiar test framework used elsewhere in the test suite.
>>>
>>> The tests added here make use of it to e.g. supply custom -U<n>
>>> arguments, set config before the tests etc.
>>>
>>> I also improved that existing test support so you can have N tests in
>>> one file with (mostly) the existing test syntax. See the "userdiff
>>> tests: add a test with multiple tests in a LANG file" patch.
>> 
>> I've read through all patches and had a comment here and there. I like a
>> lot that we can now put more than one test into a single file.
>> 
>> However, I do not like the shell script version of tests, because the
>> syntax is so hard to read. Also, it looks to me that they are only
>> needed for a few tests that could just as well be coded as one-off tests
>> outside the framework.
>> 
>> I've now pulled avar/t4018-diff-hunk-header-regex-tests-3 from your
>> github repo and will check again if I missed some cruicial points.
>
>
> I've now looked through the patch series again.
>
> I appreciate that you dug through the history and discovered and fixed a
> few deficiencies and loose ends. The way to throw all test cases for a
> language driver into a single file I like a lot, too. The way to specify
> expected results is manageable (modulo the dependency on the test
> number, t4018, that Junio mentioned).

Yes, maybe just "HEADER:" or something would be better. I was trying to
come up with something guaranteed not to conflict with the language in
question. Maybe:

    => HEADER      = 
    => description = 

> But I do not see the need for the framework provided by the new
> test_diff_funcname. At the end of the series, it is only used by Perl
> and custom driver tests. (I discount the new ADA and Ruby tests as they
> can easily migrated to the simple test scheme.) But then the Perl tests
> do not do anything special, either. The multi-line pattern test is just
> a nice add-on but not strictly needed. In the end, the Perl test is just
> as straight-forward as all others.

The benefit now is:

 1. Unlike the new plain-text "all test cases for a language driver" in
    the same file mechanism you can have test descriptions. The
    "description" in the golang one is just for show, you won't get
    anything informative from test-lib.sh when your test fails.

 2. I think having #1 beats not having test descriptions at all, or
     having to shove a description like the Ruby:

    "picks first "class/module/def" before changed context"

    into something that would make all the FS's we have to support
    happy.

 3. A test in the new perl.sh one sets config. I think in both that case
    and custom.sh it's more readable to carry such config setting with
    the test, rather than at a distance in the main setup of t4018-*.sh.

That being said I'd like to improve the syntax a bit, in particular
instead of having a wrapper for test_expect_success I think it makes
sense just to have the test call test_expect_success, and then provide a
couple of helper functions.

Re: [PATCH v3 00/35] 20210215154427.32693-1-avarab@gmail.com

[Johannes Sixt]

Am 28.02.21 um 17:07 schrieb Ævar Arnfjörð Bjarmason:
> On Sun, Feb 28 2021, Johannes Sixt wrote:
>> But I do not see the need for the framework provided by the new
>> test_diff_funcname. At the end of the series, it is only used by Perl
>> and custom driver tests. (I discount the new ADA and Ruby tests as they
>> can easily migrated to the simple test scheme.) But then the Perl tests
>> do not do anything special, either. The multi-line pattern test is just
>> a nice add-on but not strictly needed. In the end, the Perl test is just
>> as straight-forward as all others.
> 
> The benefit now is:
> 
>  1. Unlike the new plain-text "all test cases for a language driver" in
>     the same file mechanism you can have test descriptions. The
>     "description" in the golang one is just for show, you won't get
>     anything informative from test-lib.sh when your test fails.
> 
>  2. I think having #1 beats not having test descriptions at all, or
>      having to shove a description like the Ruby:
> 
>     "picks first "class/module/def" before changed context"
> 
>     into something that would make all the FS's we have to support
>     happy.

I missed that the descriptions are gone now that many test cases are
shoved into a single file when the simple framework is used. That is
indeed a disadvantage. But please keep in mind that code is more often
read than written. *If* we have to grow a new framework, then it must
not suffer from unreadability.

> 
>  3. A test in the new perl.sh one sets config. I think in both that case
>     and custom.sh it's more readable to carry such config setting with
>     the test, rather than at a distance in the main setup of t4018-*.sh.

As I said, the config in perl.sh is only used for a multi-line pattern
test. That is dispensable as there is already a similar test with custom
patterns. And the tests for custom patterns can be moved to the main
test file entirely.

> That being said I'd like to improve the syntax a bit, in particular
> instead of having a wrapper for test_expect_success I think it makes
> sense just to have the test call test_expect_success, and then provide a
> couple of helper functions.

That sounds like an improvement.

-- Hannes

[PATCH v4 00/10] userdiff: refactor + test improvements

[Ævar Arnfjörð Bjarmason]

This is a restart of the 35-patch v3 of this topic at
https://lore.kernel.org/git/20210224195129.4004-1-avarab@gmail.com/

I still plan on submitting the rest of it, but wanted to start with
the early parts of that series that hasn't been controversial or has
outstanding feedback I haven't addressed.

The range-diff to v3 is just for those patches I'm re-rolling here.

Ævar Arnfjörð Bjarmason (10):
  userdiff: refactor away the parse_bool() function
  userdiff style: re-order drivers in alphabetical order
  userdiff style: declare patterns with consistent style
  userdiff style: normalize pascal regex declaration
  userdiff: add and use for_each_userdiff_driver()
  userdiff tests: explicitly test "default" pattern
  userdiff tests: list builtin drivers via test-tool
  userdiff: remove support for "broken" tests
  blame tests: don't rely on t/t4018/ directory
  blame tests: simplify userdiff driver test

 Makefile                 |   1 +
 t/annotate-tests.sh      |  34 ++++----
 t/helper/test-tool.c     |   1 +
 t/helper/test-tool.h     |   1 +
 t/helper/test-userdiff.c |  31 +++++++
 t/t4018-diff-funcname.sh |  39 ++-------
 t/t4018/README           |   3 -
 userdiff.c               | 178 ++++++++++++++++++++++++---------------
 userdiff.h               |  15 ++++
 9 files changed, 186 insertions(+), 117 deletions(-)
 create mode 100644 t/helper/test-userdiff.c

Range-diff:
 1:  0be132b05e2 =  1:  fb7346cd296 userdiff: refactor away the parse_bool() function
 2:  d1e00a739ac =  2:  149387155bc userdiff style: re-order drivers in alphabetical order
 3:  b99bd158d45 =  3:  faf1a824f05 userdiff style: declare patterns with consistent style
 4:  9ce6d47021c =  4:  1e9ddcd1a9a userdiff style: normalize pascal regex declaration
 5:  369fbdcee83 =  5:  64ea5e8443f userdiff: add and use for_each_userdiff_driver()
 6:  70d62a97211 =  6:  862f6ab5d66 userdiff tests: explicitly test "default" pattern
 7:  792421a2f8b =  7:  22a07591b76 userdiff tests: list builtin drivers via test-tool
 8:  9081e2a152e !  8:  7755db95014 userdiff: remove support for "broken" tests
    @@ Commit message
     
         There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve
         Fortran xfuncname regex, 2020-08-12). Let's remove the test support
    -    for them, this is in preparation for a more general refactoring of the
    -    tests.
    +    for them.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
 9:  d3652f95d5e !  9:  4e0b4b42e16 blame tests: don't rely on t/t4018/ directory
    @@ Commit message
         with userdiff driver, 2020-11-01) so that the blame tests don't rely
         on stealing the contents of "t/t4018/fortran-external-function".
     
    -    I'm about to change that file in a subsequent commit. Just moving the
    -    relevant test file here inline is the easiest solution, and I think
    -    also the most readable.
    +    I have another patch series that'll possibly (or not) refactor that
    +    file, but having this test inter-dependency makes things simple in any
    +    case by making this test more readable.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
10:  35d12779ea1 ! 10:  ce98c61bf40 blame tests: simplify userdiff driver test
    @@ Commit message
         test_commit, 2021-01-12).
     
         We also did not need the full fortran-external-function content. Let's
    -    cut it down to just the important parts, and further modify it to
    -    demonstrate that the fortran-specific userdiff function is in effect
    -    by adding "DO NOT MATCH ..." and "AS THE ..." lines surrounding the
    -    "RIGHT" one. This is to check that we're using the userdiff "fortran"
    -    driver, as opposed to the default driver.
    +    cut it down to just the important parts.
     
    -    The test also left behind a .gitattributes files, let's clean it up
    -    with "test_when_finished".
    +    I'm modifying it to demonstrate that the fortran-specific userdiff
    +    function is in effect by adding "DO NOT MATCH ..." and "AS THE ..."
    +    lines surrounding the "RIGHT" one.
    +
    +    This is to check that we're using the userdiff "fortran" driver, as
    +    opposed to the default driver which would match on those lines as part
    +    of the general heuristic of matching a line that doesn't begin with
    +    whitespace.
    +
    +    The test had also been leaving behind a .gitattributes file for later
    +    tests to possibly trip over, let's clean it up with
    +    "test_when_finished".
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 01/10] userdiff: refactor away the parse_bool() function

[Ævar Arnfjörð Bjarmason]

Since 6680a0874f (drop odd return value semantics from
userdiff_config, 2012-02-07) we have not cared about the return values
of parse_tristate() or git_config_bool() v.s. falling through in
userdiff_config(), so let's do so in those cases to make the code
easier to read.

Having a wrapper function for git_config_bool() dates back to
d9bae1a178 (diff: cache textconv output, 2010-04-01) and
122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05), both of
which predated the change in 6680a0874f which made their return values
redundant.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 3f81a2261c5..c147bcbb173 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -275,19 +275,12 @@ static int parse_funcname(struct userdiff_funcname *f, const char *k,
 	return 0;
 }
 
-static int parse_tristate(int *b, const char *k, const char *v)
+static void parse_tristate(int *b, const char *k, const char *v)
 {
 	if (v && !strcasecmp(v, "auto"))
 		*b = -1;
 	else
 		*b = git_config_bool(k, v);
-	return 0;
-}
-
-static int parse_bool(int *b, const char *k, const char *v)
-{
-	*b = git_config_bool(k, v);
-	return 0;
 }
 
 int userdiff_config(const char *k, const char *v)
@@ -312,16 +305,17 @@ int userdiff_config(const char *k, const char *v)
 		return parse_funcname(&drv->funcname, k, v, 0);
 	if (!strcmp(type, "xfuncname"))
 		return parse_funcname(&drv->funcname, k, v, REG_EXTENDED);
-	if (!strcmp(type, "binary"))
-		return parse_tristate(&drv->binary, k, v);
 	if (!strcmp(type, "command"))
 		return git_config_string(&drv->external, k, v);
 	if (!strcmp(type, "textconv"))
 		return git_config_string(&drv->textconv, k, v);
-	if (!strcmp(type, "cachetextconv"))
-		return parse_bool(&drv->textconv_want_cache, k, v);
 	if (!strcmp(type, "wordregex"))
 		return git_config_string(&drv->word_regex, k, v);
+	/* Don't care about the parse errors for these, fallthrough */
+	if (!strcmp(type, "cachetextconv"))
+		drv->textconv_want_cache = git_config_bool(k, v);
+	if (!strcmp(type, "binary"))
+		parse_tristate(&drv->binary, k, v);
 
 	return 0;
 }
-- 
2.31.0.366.ga80606b22c1

Re: [PATCH v4 01/10] userdiff: refactor away the parse_bool() function

[Jeff King]

On Wed, Mar 24, 2021 at 02:48:43AM +0100, Ævar Arnfjörð Bjarmason wrote:

> Since 6680a0874f (drop odd return value semantics from
> userdiff_config, 2012-02-07) we have not cared about the return values
> of parse_tristate() or git_config_bool() v.s. falling through in
> userdiff_config(), so let's do so in those cases to make the code
> easier to read.
> 
> Having a wrapper function for git_config_bool() dates back to
> d9bae1a178 (diff: cache textconv output, 2010-04-01) and
> 122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05), both of
> which predated the change in 6680a0874f which made their return values
> redundant.

I think this cleanup makes sense.

>  int userdiff_config(const char *k, const char *v)
> @@ -312,16 +305,17 @@ int userdiff_config(const char *k, const char *v)
>  		return parse_funcname(&drv->funcname, k, v, 0);
>  	if (!strcmp(type, "xfuncname"))
>  		return parse_funcname(&drv->funcname, k, v, REG_EXTENDED);
> -	if (!strcmp(type, "binary"))
> -		return parse_tristate(&drv->binary, k, v);
>  	if (!strcmp(type, "command"))
>  		return git_config_string(&drv->external, k, v);
>  	if (!strcmp(type, "textconv"))
>  		return git_config_string(&drv->textconv, k, v);
> -	if (!strcmp(type, "cachetextconv"))
> -		return parse_bool(&drv->textconv_want_cache, k, v);
>  	if (!strcmp(type, "wordregex"))
>  		return git_config_string(&drv->word_regex, k, v);
> +	/* Don't care about the parse errors for these, fallthrough */
> +	if (!strcmp(type, "cachetextconv"))
> +		drv->textconv_want_cache = git_config_bool(k, v);
> +	if (!strcmp(type, "binary"))
> +		parse_tristate(&drv->binary, k, v);

The original returned early, which short-circuited the rest of the
strcmp(). that probably doesn't matter much in practice (after all, an
unrecognized value is inherently O(n)). But perhaps:

  if (...)
     assign...;
  else if (...)
     assign...;

would make the comment unnecessary. I don't feel strongly either way,
though.

-Peff

[PATCH v4 02/10] userdiff style: re-order drivers in alphabetical order

[Ævar Arnfjörð Bjarmason]

Address some old code smell and move around the built-in userdiff
drivers so they're both in alphabetical order, and now in the same
order they appear in the gitattributes(5) documentation.

The two started drifting in be58e70dba (diff: unify external diff and
funcname parsing code, 2008-10-05), and then even further in
80c49c3de2 (color-words: make regex configurable via attributes,
2009-01-17) when the "cpp" pattern was added.

There are no functional changes here, and as --color-moved will show
only moved existing lines.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 76 +++++++++++++++++++++++++++---------------------------
 1 file changed, 38 insertions(+), 38 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c147bcbb173..c92cbcc0540 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,6 +44,44 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
+PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 "[={}\"]|[^={}\" \t]+"),
+PATTERNS("cpp",
+	 /* Jump targets or access declarations */
+	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
+	 /* functions/methods, variables, and compounds at top level */
+	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
+PATTERNS("csharp",
+	 /* Keywords */
+	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
+	 /* Methods and constructors */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
+	 /* Properties */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
+	 /* Type definitions */
+	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
+	 /* Namespace */
+	 "^[ \t]*(namespace[ \t]+.*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
+IPATTERN("css",
+	 "![:;][[:space:]]*$\n"
+	 "^[:[@.#]?[_a-z0-9].*$",
+	 /* -- */
+	 /*
+	  * This regex comes from W3C CSS specs. Should theoretically also
+	  * allow ISO 10646 characters U+00A0 and higher,
+	  * but they are not handled in this regex.
+	  */
+	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
+	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
+),
 PATTERNS("dts",
 	 "!;\n"
 	 "!=\n"
@@ -191,46 +229,8 @@ PATTERNS("rust",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[0-9][0-9_a-fA-Fiosuxz]*(\\.([0-9]*[eE][+-]?)?[0-9_fF]*)?"
 	 "|[-+*\\/<>%&^|=!:]=|<<=?|>>=?|&&|\\|\\||->|=>|\\.{2}=|\\.{3}|::"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
-	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("tex", "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$",
 	 "\\\\[a-zA-Z@]+|\\\\.|[a-zA-Z0-9\x80-\xff]+"),
-PATTERNS("cpp",
-	 /* Jump targets or access declarations */
-	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
-	 /* functions/methods, variables, and compounds at top level */
-	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
-PATTERNS("csharp",
-	 /* Keywords */
-	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
-	 /* Methods and constructors */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
-	 /* Properties */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
-	 /* Type definitions */
-	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
-	 /* Namespace */
-	 "^[ \t]*(namespace[ \t]+.*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
-IPATTERN("css",
-	 "![:;][[:space:]]*$\n"
-	 "^[:[@.#]?[_a-z0-9].*$",
-	 /* -- */
-	 /*
-	  * This regex comes from W3C CSS specs. Should theoretically also
-	  * allow ISO 10646 characters U+00A0 and higher,
-	  * but they are not handled in this regex.
-	  */
-	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
-	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
-),
 { "default", NULL, -1, { NULL, 0 } },
 };
 #undef PATTERNS
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 03/10] userdiff style: declare patterns with consistent style

[Ævar Arnfjörð Bjarmason]

Change those patterns which were declared with a regex on the same
line as the "PATTERNS()" line to put that regex on the next line, and
add missing "/* -- */" separator comments between the pattern and
word_regex.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c92cbcc0540..c7aaf7094f8 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,7 +44,9 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+PATTERNS("bibtex",
+	 "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 /* -- */
 	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("cpp",
 	 /* Jump targets or access declarations */
@@ -121,7 +123,9 @@ IPATTERN("fortran",
 	  * they would have been matched above as a variable anyway. */
 	 "|[-+]?[0-9.]+([AaIiDdEeFfLlTtXx][Ss]?[-+]?[0-9.]*)?(_[a-zA-Z0-9][a-zA-Z0-9_]*)?"
 	 "|//|\\*\\*|::|[/<>=]="),
-IPATTERN("fountain", "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+IPATTERN("fountain",
+	 "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+	 /* -- */
 	 "[^ \t-]+"),
 PATTERNS("golang",
 	 /* Functions */
@@ -132,7 +136,9 @@ PATTERNS("golang",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.eE]+i?|0[xX]?[0-9a-fA-F]+i?"
 	 "|[-+*/<>%&^|=!:]=|--|\\+\\+|<<=?|>>=?|&\\^=?|&&|\\|\\||<-|\\.{3}"),
-PATTERNS("html", "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+PATTERNS("html",
+	 "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("java",
 	 "!^[ \t]*(catch|do|for|if|instanceof|new|return|switch|throw|while)\n"
@@ -144,6 +150,7 @@ PATTERNS("java",
 	 "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"),
 PATTERNS("markdown",
 	 "^ {0,3}#{1,6}[ \t].*",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("matlab",
 	 /*
@@ -152,6 +159,7 @@ PATTERNS("matlab",
 	  * that is understood by both.
 	  */
 	 "^[[:space:]]*((classdef|function)[[:space:]].*)$|^(%%%?|##)[[:space:]].*$",
+	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*|[-+0-9.e]+|[=~<>]=|\\.[*/\\^']|\\|\\||&&"),
 PATTERNS("objc",
 	 /* Negate C statements that can look like functions */
@@ -212,13 +220,15 @@ PATTERNS("php",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+"
 	 "|[-+*/<>%&^|=!.]=|--|\\+\\+|<<=?|>>=?|===|&&|\\|\\||::|->"),
-PATTERNS("python", "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
+PATTERNS("python",
+	 "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+[jJlL]?|0[xX]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|//=?|<<=?|>>=?|\\*\\*=?"),
 	 /* -- */
-PATTERNS("ruby", "^[ \t]*((class|module|def)[ \t].*)$",
+PATTERNS("ruby",
+	 "^[ \t]*((class|module|def)[ \t].*)$",
 	 /* -- */
 	 "(@|@@|\\$)?[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+|\\?(\\\\C-)?(\\\\M-)?."
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 04/10] userdiff style: normalize pascal regex declaration

[Ævar Arnfjörð Bjarmason]

Declare the pascal pattern consistently with how we declare the
others, not having "\n" on one line by itself, but as part of the
pattern, and when there are alterations have the "|" at the start, not
end of the line.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c7aaf7094f8..10a02d36209 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -175,9 +175,8 @@ PATTERNS("objc",
 	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
 PATTERNS("pascal",
-	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface|"
-		"implementation|initialization|finalization)[ \t]*.*)$"
-	 "\n"
+	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface"
+	 "|implementation|initialization|finalization)[ \t]*.*)$\n"
 	 "^(.*=[ \t]*(class|record).*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

Refactor the userdiff_find_by_namelen() function so that a new
for_each_userdiff_driver() API function does most of the work.

This will be useful for the same reason we've got other for_each_*()
API functions as part of various APIs, and will be used in a follow-up
commit.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 61 +++++++++++++++++++++++++++++++++++++++++++-----------
 userdiff.h | 15 ++++++++++++++
 2 files changed, 64 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 10a02d36209..55f4f769bd3 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -259,20 +259,32 @@ static struct userdiff_driver driver_false = {
 	{ NULL, 0 }
 };
 
-static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+struct for_each_userdiff_driver_cb {
+	const char *k;
+	size_t len;
+	struct userdiff_driver *driver;
+};
+
+static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
+				       enum userdiff_driver_type type, void *priv)
 {
-	int i;
-	for (i = 0; i < ndrivers; i++) {
-		struct userdiff_driver *drv = drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
-	}
-	for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
-		struct userdiff_driver *drv = builtin_drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
+	struct for_each_userdiff_driver_cb *cb_data = priv;
+
+	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
+	    !driver->name[cb_data->len]) {
+		cb_data->driver = driver;
+		return -1; /* found it! */
 	}
-	return NULL;
+	return 0;
+}
+
+static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+{
+	struct for_each_userdiff_driver_cb udcbdata = { .k = k, .len = len, .driver = NULL };
+
+	for_each_userdiff_driver(userdiff_find_by_namelen_cb,
+				 USERDIFF_DRIVER_TYPE_UNSPECIFIED, &udcbdata);
+	return udcbdata.driver;
 }
 
 static int parse_funcname(struct userdiff_funcname *f, const char *k,
@@ -373,3 +385,28 @@ struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 
 	return driver;
 }
+
+int for_each_userdiff_driver(each_userdiff_driver_fn fn,
+			     enum userdiff_driver_type type, void *cb_data)
+{
+	int i, ret;
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {
+
+		for (i = 0; i < ndrivers; i++) {
+			struct userdiff_driver *drv = drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_CUSTOM, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_BUILTIN)) {
+
+		for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
+			struct userdiff_driver *drv = builtin_drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_BUILTIN, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	return 0;
+}
diff --git a/userdiff.h b/userdiff.h
index 203057e13e5..fe14014a775 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -21,6 +21,13 @@ struct userdiff_driver {
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
 };
+enum userdiff_driver_type {
+	USERDIFF_DRIVER_TYPE_UNSPECIFIED = 1<<0,
+	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<1,
+	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<2,
+};
+typedef int (*each_userdiff_driver_fn)(struct userdiff_driver *,
+				       enum userdiff_driver_type, void *);
 
 int userdiff_config(const char *k, const char *v);
 struct userdiff_driver *userdiff_find_by_name(const char *name);
@@ -34,4 +41,12 @@ struct userdiff_driver *userdiff_find_by_path(struct index_state *istate,
 struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 					      struct userdiff_driver *driver);
 
+/*
+ * Iterate over each driver of type userdiff_driver_type, or
+ * USERDIFF_DRIVER_TYPE_UNSPECIFIED for all of them. Return non-zero
+ * to exit from the loop.
+ */
+int for_each_userdiff_driver(each_userdiff_driver_fn,
+			     enum userdiff_driver_type, void *);
+
 #endif /* USERDIFF */
-- 
2.31.0.366.ga80606b22c1

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> +struct for_each_userdiff_driver_cb {
> +	const char *k;
> +	size_t len;
> +	struct userdiff_driver *driver;
> +};

Makes me wonder if we want to rename s/k/name/;

> +static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
> +				       enum userdiff_driver_type type, void *priv)
>  {
> +	struct for_each_userdiff_driver_cb *cb_data = priv;
> +
> +	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
> +	    !driver->name[cb_data->len]) {
> +		cb_data->driver = driver;
> +		return -1; /* found it! */
>  	}
> -	return NULL;
> +	return 0;
> +}

Makes sense.

> +static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
> +{
> +	struct for_each_userdiff_driver_cb udcbdata = { .k = k, .len = len, .driver = NULL };

No need to explicitly spell the zero initialization.  Wrapping it
like this:

	struct for_each_userdiff_driver_cb udcbdata = {
		.k = k, .len = len
	};

would avoid the overlong line.

> +
> +	for_each_userdiff_driver(userdiff_find_by_namelen_cb,
> +				 USERDIFF_DRIVER_TYPE_UNSPECIFIED, &udcbdata);
> +	return udcbdata.driver;


> @@ -373,3 +385,28 @@ struct userdiff_driver *userdiff_get_textconv(struct repository *r,
>  
>  	return driver;
>  }
> +
> +int for_each_userdiff_driver(each_userdiff_driver_fn fn,
> +			     enum userdiff_driver_type type, void *cb_data)
> +{
> +	int i, ret;
> +	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {

I presume that the concrete ones are bitmask (i.e. BUILTIN occupies
bit #0 while CUSTOM occupies bit #1, or something like that).  Then

    #define USERDIFF_DRIVER_TYPE_UNSPECIFIED (-1)

would make this (and the other) condition far easier to read, i.e.

	if (type & USERDIFF_DRIVER_TYPE_CUSTOM) {
		... if the caller wants to iterate over "custom" drivers
		... do these things

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Jeff King]

On Wed, Mar 24, 2021 at 02:48:47AM +0100, Ævar Arnfjörð Bjarmason wrote:

> Refactor the userdiff_find_by_namelen() function so that a new
> for_each_userdiff_driver() API function does most of the work.
> 
> This will be useful for the same reason we've got other for_each_*()
> API functions as part of various APIs, and will be used in a follow-up
> commit.

The refactorings up to here all made sense, but TBH this one makes the
code more confusing to follow to me.

Perhaps part of it is just that the diff is messy, but I had to read it
several times to understand what's going on. Here's what I think were
the tricky parts:

> -static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
> +struct for_each_userdiff_driver_cb {
> +	const char *k;
> +	size_t len;
> +	struct userdiff_driver *driver;
> +};

Our callback function does _one_ type of selection (based on a "type"
parameter), but not another (based on the name). That feels
inconsistent, but is also the reason we have this awkward struct.  Part
of my confusion is the name: this is not something to be generically
used with for_each_userdiff_driver(), but rather a type unique to
find_by_namelen() to be passed through the opaque void pointer.

So "struct find_by_namelen_data" would have been a lot more
enlightening.

The fact that callbacks are awkward in general in C might not be
solvable, at least not without duplicating some iteration code.

> +static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
> +				       enum userdiff_driver_type type, void *priv)
>  {
> [...]
> +	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
> +	    !driver->name[cb_data->len]) {
> +		cb_data->driver = driver;
> +		return -1; /* found it! */
>  	}

This "return -1" took me a while to grok, and the comment didn't help
all that much. The point is to stop traversing the list, but "-1" to me
signals error. I think returning "1" might be a bit more idiomatic, but
also a comment that says "tell the caller to stop iterating" would have
been more clear.

> +int for_each_userdiff_driver(each_userdiff_driver_fn fn,
> +			     enum userdiff_driver_type type, void *cb_data)
> +{
> +	int i, ret;
> +	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {
> +
> +		for (i = 0; i < ndrivers; i++) {
> +			struct userdiff_driver *drv = drivers + i;
> +			ret = fn(drv, USERDIFF_DRIVER_TYPE_CUSTOM, cb_data);
> +			if (ret)
> +				return ret;
> +		}
> +	}
> +	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_BUILTIN)) {
> +
> +		for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
> +			struct userdiff_driver *drv = builtin_drivers + i;
> +			ret = fn(drv, USERDIFF_DRIVER_TYPE_BUILTIN, cb_data);
> +			if (ret)
> +				return ret;
> +		}
> +	}
> +	return 0;
> +}

I spent a while scratching my head at these types, and what they would
be used for, since this caller doesn't introduce any. Looking at patch 7
helped, though it's unclear to me why we need to distinguish between
custom and builtin drivers there. As you note there, nobody calls
list-custom-drivers nor list-drivers. And if we haven't configured
anything, then wouldn't list-drivers be the same as list-builtin-drivers?
Or for the purposes of that test, if we _did_ configure something,

  As an aside, it feels like this is something we ought to be able to
  ask git-config about, rather than having a test-helper. This is
  basically "baked-in" config, and if we represented it as such, and
  parsed it into a struct just like regular config, then probably "git
  config --list --source" could be used to find it (and differentiate it
  from user-provided config). Possible downsides:

    1. Would people find it confusing that "git config --list" suddenly
       gets way bigger? Maybe we'd want an "--include-baked-in" option
       or something.

    2. Is the cost of parsing the config measurably bad? Obviously a
       user could provide the same content and we'd have to parse it,
       but there's a lot more rules here than most users would probably
       provide.

> +enum userdiff_driver_type {
> +	USERDIFF_DRIVER_TYPE_UNSPECIFIED = 1<<0,
> +	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<1,
> +	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<2,
> +};

I was confused by these being bits, because some of them seem mutually
exclusive (e.g., UNSPECIFIED and anything else).

Perhaps it would make more sense as:

  USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
  USERDIFF_DRIVER_TYPE_CUSTOM = 1<<0,
  USERDIFF_DRIVER_TYPE_ALL = USERDIFF_DRIVER_TYPE_BUILTIN | USERDIFF_DRIVER_TYPE_CUSTOM

Or the one caller who wants "ALL" could even do the OR themselves.

I do kind of wonder if there's much value in having a single function
with a type field at all, though, given that there's no overlap in the
implementation. Would separate "for_each_custom" and "for_each_builtin"
functions make sense? And then the existing caller would just call them
sequentially.

I dunno. I know a lot of this is nit-picking, and I don't think there's
anything incorrect in this patch. I just found it surprisingly hard to
read for something that purports to be refactoring / cleaning the code.

-Peff

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Our callback function does _one_ type of selection (based on a "type"
> parameter), but not another (based on the name). That feels
> inconsistent, but is also the reason we have this awkward struct.

I wrote a very similar review but did not send it out, as I couldn't
pinpoint where the awkwardness come from exactly.

> Part
> of my confusion is the name: this is not something to be generically
> used with for_each_userdiff_driver(), but rather a type unique to
> find_by_namelen() to be passed through the opaque void pointer.
>
> So "struct find_by_namelen_data" would have been a lot more
> enlightening.

Yes.  The callback parameter being "void *" is the API, and it is
just this user that uses this particular structure.

And while "type" could also be made a part of this structure and
have the API always iterate over all entries regardless of "type",
i.e. the callback function could be the one responsible for finding
the entries in the table for one particular type, it is understandable
that potential callers can be helped by having the pre-filtering
based on the "type" thing on the API side.

>> +	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
>> +	    !driver->name[cb_data->len]) {
>> +		cb_data->driver = driver;
>> +		return -1; /* found it! */
>>  	}
>
> This "return -1" took me a while to grok, and the comment didn't help
> all that much. The point is to stop traversing the list, but "-1" to me
> signals error. I think returning "1" might be a bit more idiomatic, but
> also a comment that says "tell the caller to stop iterating" would have
> been more clear.

And the fact that it becomes the return value of "for_each_()" iterator
does not quite help to use a negative value, implying there was some
error condition X-<.

> Perhaps it would make more sense as:
>
>   USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
>   USERDIFF_DRIVER_TYPE_CUSTOM = 1<<0,
>   USERDIFF_DRIVER_TYPE_ALL = USERDIFF_DRIVER_TYPE_BUILTIN | USERDIFF_DRIVER_TYPE_CUSTOM
>
> Or the one caller who wants "ALL" could even do the OR themselves.

Great minds think alike ;-)

> I do kind of wonder if there's much value in having a single function
> with a type field at all, though, given that there's no overlap in the
> implementation. Would separate "for_each_custom" and "for_each_builtin"
> functions make sense? And then the existing caller would just call them
> sequentially.

Or a single for_each_driver() that gives <name, length, type> tuple
to the callback function, iterating over all drivers regardless of
the type.

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Jeff King]

On Wed, Mar 24, 2021 at 12:57:15PM -0700, Junio C Hamano wrote:

> And while "type" could also be made a part of this structure and
> have the API always iterate over all entries regardless of "type",
> i.e. the callback function could be the one responsible for finding
> the entries in the table for one particular type, it is understandable
> that potential callers can be helped by having the pre-filtering
> based on the "type" thing on the API side.

Yeah. We _could_ also have the for-each function filter by name (if
present). And then quit when it finds a matching name (because it knows
the names are supposed to be unique).

That is the opposite of:

> Or a single for_each_driver() that gives <name, length, type> tuple
> to the callback function, iterating over all drivers regardless of
> the type.

I'd be fine with that, too. I don't have a huge preference either way,
but it does feel like it would fix the weird asymmetry (though as I
said, I don't think the asymmetry is _wrong_, and it may not be worth
over-engineering this tiny corner of the interface).

Thanks for spelling out both of these directions clearly. My response to
Ævar was a little muddled as I was having trouble figuring out just what
it was that left me so puzzled by the patch. :)


By the way, one thing I wondered about all of this: could we have an
entry in both the custom and builtin lists? I think the answer is "no",
because any custom config for a builtin type would get placed inside the
existing struct in the builtin_drivers array (which is sadly a reason we
must leak any old string values when we parse diff.*.funcname, etc; we
don't know if they are string literals or heap values from previously
parsed config).

I also notice that the "builtin" for-each does not count the boolean
"diff" or "!diff" attribute structs. That is perhaps reasonable, as they
do not have interesting names nor values in the first place. I do still
wonder whether "builtin vs custom" is even a useful distinction (i.e.,
those builtins are less builtin_drivers[] than drivers[]).

-Peff

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

On Wed, Mar 24 2021, Jeff King wrote:

> On Wed, Mar 24, 2021 at 02:48:47AM +0100, Ævar Arnfjörð Bjarmason wrote:
>
>> Refactor the userdiff_find_by_namelen() function so that a new
>> for_each_userdiff_driver() API function does most of the work.
>> 
>> This will be useful for the same reason we've got other for_each_*()
>> API functions as part of various APIs, and will be used in a follow-up
>> commit.
>
> The refactorings up to here all made sense, but TBH this one makes the
> code more confusing to follow to me.
>
> Perhaps part of it is just that the diff is messy, but I had to read it
> several times to understand what's going on. Here's what I think were
> the tricky parts:
>
>> -static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
>> +struct for_each_userdiff_driver_cb {
>> +	const char *k;
>> +	size_t len;
>> +	struct userdiff_driver *driver;
>> +};
>
> Our callback function does _one_ type of selection (based on a "type"
> parameter), but not another (based on the name). That feels
> inconsistent, but is also the reason we have this awkward struct.  Part
> of my confusion is the name: this is not something to be generically
> used with for_each_userdiff_driver(), but rather a type unique to
> find_by_namelen() to be passed through the opaque void pointer.
>
> So "struct find_by_namelen_data" would have been a lot more
> enlightening.
>
> The fact that callbacks are awkward in general in C might not be
> solvable, at least not without duplicating some iteration code.
>
>> +static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
>> +				       enum userdiff_driver_type type, void *priv)
>>  {
>> [...]
>> +	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
>> +	    !driver->name[cb_data->len]) {
>> +		cb_data->driver = driver;
>> +		return -1; /* found it! */
>>  	}
>
> This "return -1" took me a while to grok, and the comment didn't help
> all that much. The point is to stop traversing the list, but "-1" to me
> signals error. I think returning "1" might be a bit more idiomatic, but
> also a comment that says "tell the caller to stop iterating" would have
> been more clear.

*nod*

Also thanks for all the reviewing so far both, I'm not replying to all
of it point-by-point here, will respond with a re-roll at some point.

>> +int for_each_userdiff_driver(each_userdiff_driver_fn fn,
>> +			     enum userdiff_driver_type type, void *cb_data)
>> +{
>> +	int i, ret;
>> +	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {
>> +
>> +		for (i = 0; i < ndrivers; i++) {
>> +			struct userdiff_driver *drv = drivers + i;
>> +			ret = fn(drv, USERDIFF_DRIVER_TYPE_CUSTOM, cb_data);
>> +			if (ret)
>> +				return ret;
>> +		}
>> +	}
>> +	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_BUILTIN)) {
>> +
>> +		for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
>> +			struct userdiff_driver *drv = builtin_drivers + i;
>> +			ret = fn(drv, USERDIFF_DRIVER_TYPE_BUILTIN, cb_data);
>> +			if (ret)
>> +				return ret;
>> +		}
>> +	}
>> +	return 0;
>> +}
>
> I spent a while scratching my head at these types, and what they would
> be used for, since this caller doesn't introduce any. Looking at patch 7
> helped, though it's unclear to me why we need to distinguish between
> custom and builtin drivers there. As you note there, nobody calls
> list-custom-drivers nor list-drivers. And if we haven't configured
> anything, then wouldn't list-drivers be the same as list-builtin-drivers?
> Or for the purposes of that test, if we _did_ configure something,
>
>   As an aside, it feels like this is something we ought to be able to
>   ask git-config about, rather than having a test-helper. This is
>   basically "baked-in" config, and if we represented it as such, and
>   parsed it into a struct just like regular config, then probably "git
>   config --list --source" could be used to find it (and differentiate it
>   from user-provided config). Possible downsides:
>
>     1. Would people find it confusing that "git config --list" suddenly
>        gets way bigger? Maybe we'd want an "--include-baked-in" option
>        or something.
>
>     2. Is the cost of parsing the config measurably bad? Obviously a
>        user could provide the same content and we'd have to parse it,
>        but there's a lot more rules here than most users would probably
>        provide.

Also:

 3. Only the PATTERNS() macro translates as-is to config syntax. We
    don't have a way to do what IPATTERN() does in the config syntax
    currently.

    We could add a ifuncname and xifuncname or whatever for it I guess,
    but currently the ICASE behavior in the C code is magic.

>> +enum userdiff_driver_type {
>> +	USERDIFF_DRIVER_TYPE_UNSPECIFIED = 1<<0,
>> +	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<1,
>> +	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<2,
>> +};
>
> I was confused by these being bits, because some of them seem mutually
> exclusive (e.g., UNSPECIFIED and anything else).
>
> Perhaps it would make more sense as:
>
>   USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
>   USERDIFF_DRIVER_TYPE_CUSTOM = 1<<0,
>   USERDIFF_DRIVER_TYPE_ALL = USERDIFF_DRIVER_TYPE_BUILTIN | USERDIFF_DRIVER_TYPE_CUSTOM
>
> Or the one caller who wants "ALL" could even do the OR themselves.
>
> I do kind of wonder if there's much value in having a single function
> with a type field at all, though, given that there's no overlap in the
> implementation. Would separate "for_each_custom" and "for_each_builtin"
> functions make sense? And then the existing caller would just call them
> sequentially.
>
> I dunno. I know a lot of this is nit-picking, and I don't think there's
> anything incorrect in this patch. I just found it surprisingly hard to
> read for something that purports to be refactoring / cleaning the code.
>
> -Peff

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Jeff King]

On Thu, Mar 25, 2021 at 12:05:09AM +0100, Ævar Arnfjörð Bjarmason wrote:

> >   As an aside, it feels like this is something we ought to be able to
> >   ask git-config about, rather than having a test-helper. This is
> >   basically "baked-in" config, and if we represented it as such, and
> >   parsed it into a struct just like regular config, then probably "git
> >   config --list --source" could be used to find it (and differentiate it
> >   from user-provided config). Possible downsides:
> >
> >     1. Would people find it confusing that "git config --list" suddenly
> >        gets way bigger? Maybe we'd want an "--include-baked-in" option
> >        or something.
> >
> >     2. Is the cost of parsing the config measurably bad? Obviously a
> >        user could provide the same content and we'd have to parse it,
> >        but there's a lot more rules here than most users would probably
> >        provide.
> 
> Also:
> 
>  3. Only the PATTERNS() macro translates as-is to config syntax. We
>     don't have a way to do what IPATTERN() does in the config syntax
>     currently.
> 
>     We could add a ifuncname and xifuncname or whatever for it I guess,
>     but currently the ICASE behavior in the C code is magic.

Good point. IMHO that is something we should consider fixing
independently. It was a mistake to add builtins that couldn't be
replicated via the config (though I notice it happened quite a while
ago, and nobody seems to have cared, so perhaps it isn't that
important).

I have also wondered if we should just ship a file which could be
installed as /etc/gitconfig.filetypes and include it the stock
/etc/gitconfig. That is effectively the same as "baked in", but
hopefully makes it more clear to users how they can modify things.  But
all of that is somewhat orthogonal to what you're doing here.

-Peff

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

On Thu, Mar 25 2021, Jeff King wrote:

> On Thu, Mar 25, 2021 at 12:05:09AM +0100, Ævar Arnfjörð Bjarmason wrote:
>
>> >   As an aside, it feels like this is something we ought to be able to
>> >   ask git-config about, rather than having a test-helper. This is
>> >   basically "baked-in" config, and if we represented it as such, and
>> >   parsed it into a struct just like regular config, then probably "git
>> >   config --list --source" could be used to find it (and differentiate it
>> >   from user-provided config). Possible downsides:
>> >
>> >     1. Would people find it confusing that "git config --list" suddenly
>> >        gets way bigger? Maybe we'd want an "--include-baked-in" option
>> >        or something.
>> >
>> >     2. Is the cost of parsing the config measurably bad? Obviously a
>> >        user could provide the same content and we'd have to parse it,
>> >        but there's a lot more rules here than most users would probably
>> >        provide.
>> 
>> Also:
>> 
>>  3. Only the PATTERNS() macro translates as-is to config syntax. We
>>     don't have a way to do what IPATTERN() does in the config syntax
>>     currently.
>> 
>>     We could add a ifuncname and xifuncname or whatever for it I guess,
>>     but currently the ICASE behavior in the C code is magic.
>
> Good point. IMHO that is something we should consider fixing
> independently. It was a mistake to add builtins that couldn't be
> replicated via the config (though I notice it happened quite a while
> ago, and nobody seems to have cared, so perhaps it isn't that
> important).

I'm conspiring to eventually optimistically replace these ERE patterns
with PCRE if we build with that at some point.

Then you could just prefix your pattern with (?i) here in this and other
things that want icase...

> I have also wondered if we should just ship a file which could be
> installed as /etc/gitconfig.filetypes and include it the stock
> /etc/gitconfig. That is effectively the same as "baked in", but
> hopefully makes it more clear to users how they can modify things.  But
> all of that is somewhat orthogonal to what you're doing here.

In theory I'm with you on that, in practice this is just the sort of
thing that requires opt-in effort from every person packaging git
(installing system-wide-config).

A good number of those people will decide "default config values? In
some ~200 line file included in /etc/gitconfig?? I don't need that! This
is a minimal install!".

And then their web UI that calls "git diff" under the hood won't do the
right thing with a .gitattributes config that expects these built-in
drivers anymore.

But yeah, there's no reason your earlier suggestion of injecting global
defaults into "git config -l" wouldn't work, and for our own C APIs such
as this one to rely on that happening.

It would even make git more discoverable, as all the "this is set to xyz
by default" docs in git-config(1) could become reflective, you could
just run "git config -l --show-origin | grep ^default:" or something to
see all default values.

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Jeff King]

On Thu, Mar 25, 2021 at 02:26:21AM +0100, Ævar Arnfjörð Bjarmason wrote:

> >>     We could add a ifuncname and xifuncname or whatever for it I guess,
> >>     but currently the ICASE behavior in the C code is magic.
> >
> > Good point. IMHO that is something we should consider fixing
> > independently. It was a mistake to add builtins that couldn't be
> > replicated via the config (though I notice it happened quite a while
> > ago, and nobody seems to have cared, so perhaps it isn't that
> > important).
> 
> I'm conspiring to eventually optimistically replace these ERE patterns
> with PCRE if we build with that at some point.
> 
> Then you could just prefix your pattern with (?i) here in this and other
> things that want icase...

I really like PCRE myself, but is it portable/common enough for us to
start using it for baked-in funcname patterns? I sort of assumed there
were exotic platforms where libpcre wouldn't build (or at least it would
be inconvenient or uncommon to have it). And it would be nice to degrade
those gracefully (or at least better than "I guess you don't get any
builtin funcnames. Tough luck").

> > I have also wondered if we should just ship a file which could be
> > installed as /etc/gitconfig.filetypes and include it the stock
> > /etc/gitconfig. That is effectively the same as "baked in", but
> > hopefully makes it more clear to users how they can modify things.  But
> > all of that is somewhat orthogonal to what you're doing here.
> 
> In theory I'm with you on that, in practice this is just the sort of
> thing that requires opt-in effort from every person packaging git
> (installing system-wide-config).

I assumed that we'd install it with "make install", and packaging would
pick it up from there. But you're right that it is likely to create extra
headaches.

> But yeah, there's no reason your earlier suggestion of injecting global
> defaults into "git config -l" wouldn't work, and for our own C APIs such
> as this one to rely on that happening.
> 
> It would even make git more discoverable, as all the "this is set to xyz
> by default" docs in git-config(1) could become reflective, you could
> just run "git config -l --show-origin | grep ^default:" or something to
> see all default values.

Yeah, exactly. The discoverability is the real value IMHO. I just worry
about overwhelming the user who runs "git config" with --show-origin. I
guess it could avoid showing baked-in config unless an extra option is
given, but then that makes discoverability slightly harder (though still
better than it is today).

I dunno. This is mostly orthogonal to your patch series, so I don't mind
at all punting on it for now. Though if we _did_ do the baked-in config
then, then a lot of this userdiff code would want to be converted to it.

-Peff

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

On Fri, Mar 26 2021, Jeff King wrote:

> On Thu, Mar 25, 2021 at 02:26:21AM +0100, Ævar Arnfjörð Bjarmason wrote:
>
>> >>     We could add a ifuncname and xifuncname or whatever for it I guess,
>> >>     but currently the ICASE behavior in the C code is magic.
>> >
>> > Good point. IMHO that is something we should consider fixing
>> > independently. It was a mistake to add builtins that couldn't be
>> > replicated via the config (though I notice it happened quite a while
>> > ago, and nobody seems to have cared, so perhaps it isn't that
>> > important).
>> 
>> I'm conspiring to eventually optimistically replace these ERE patterns
>> with PCRE if we build with that at some point.
>> 
>> Then you could just prefix your pattern with (?i) here in this and other
>> things that want icase...
>
> I really like PCRE myself, but is it portable/common enough for us to
> start using it for baked-in funcname patterns? I sort of assumed there
> were exotic platforms where libpcre wouldn't build (or at least it would
> be inconvenient or uncommon to have it). And it would be nice to degrade
> those gracefully (or at least better than "I guess you don't get any
> builtin funcnames. Tough luck").

I think it's as portable as git, the JIT backend isn't, but PCRE itself
is portably C code, and e.g. everything that python, PHP and any other
number of things that depend on PCRE has had PCRE ported to it.

That plan involves an "git rm -r compat/regex" and a compat/pcre
instead, I have some long-left-over patches for that.

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Jeff King]

On Fri, Apr 09, 2021 at 09:44:57PM +0200, Ævar Arnfjörð Bjarmason wrote:

> > I really like PCRE myself, but is it portable/common enough for us to
> > start using it for baked-in funcname patterns? I sort of assumed there
> > were exotic platforms where libpcre wouldn't build (or at least it would
> > be inconvenient or uncommon to have it). And it would be nice to degrade
> > those gracefully (or at least better than "I guess you don't get any
> > builtin funcnames. Tough luck").
> 
> I think it's as portable as git, the JIT backend isn't, but PCRE itself
> is portably C code, and e.g. everything that python, PHP and any other
> number of things that depend on PCRE has had PCRE ported to it.
> 
> That plan involves an "git rm -r compat/regex" and a compat/pcre
> instead, I have some long-left-over patches for that.

OK. I was more worried about platforms where it was cumbersome to
install pcre. If we are shipping it as a vendored library, then at least
the dependency management is on us, and not the user. I do worry a
little about running into complications with building or debugging it.
It would be importing a lot of new code (that in theory we never need to
look at, but that is not always how it works...). And while it may build
portably everywhere, that may involve extra work and not integrate with
our build knobs (e.g., it looks like it uses autoconf)

-Peff

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>> That plan involves an "git rm -r compat/regex" and a compat/pcre
>> instead, I have some long-left-over patches for that.
>
> OK. I was more worried about platforms where it was cumbersome to
> install pcre. If we are shipping it as a vendored library, then at least
> the dependency management is on us, and not the user. I do worry a
> little about running into complications with building or debugging it.

Please don't.  I prefer not shipping compat/pcre ourselves; having
to worry about version skew and picking up upstream security fixes
in time, etc. for an external library that is well maintained is not
what we need.

Re: [PATCH v4 05/10] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

On Sat, Apr 10 2021, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
>
>>> That plan involves an "git rm -r compat/regex" and a compat/pcre
>>> instead, I have some long-left-over patches for that.
>>
>> OK. I was more worried about platforms where it was cumbersome to
>> install pcre. If we are shipping it as a vendored library, then at least
>> the dependency management is on us, and not the user. I do worry a
>> little about running into complications with building or debugging it.
>
> Please don't.  I prefer not shipping compat/pcre ourselves; having
> to worry about version skew and picking up upstream security fixes
> in time, etc. for an external library that is well maintained is not
> what we need.

I'm not submitting patches for this now, I personally don't care much if
PCRE became a required dep (as in you need the library) and we didn't
ship it.

I just assumed if I'd propose that that we'd want "make" to work on more
bare-bones systems, since we e.g. carry a copy of
sha1collisiondetection/ partially to facilitate that.

But FWIW, and to the extent I've got time etc. my rough plan for this
larger PCRE conpsiary was:

 1. Get the current pickaxe-prep-for-PCRE patches to land.

 2. I've got patches on top of that which move over to PCRE, so as with
    the existing grep.c code we could drop kwset entirely (declare that
    if you need optimized fast search, you can install PCRE).

 3. git rm kwset.[ch]

 4. Migrate more things to some light wrapper "do a regex match" API
    (just small modifications to grep.c) that'll optimistically be able
    to do BRE/ERE/PCRE matches. If you've got PCRE then PCRE can handle
    them all (it's got a RX dialecttranslation interface).

 5. All our regcomp/regexec should be gone in favor if this simple
    replacement API.

Which is where this "PCRE everywhere" would come in, at this point we'll
be carrying a compat/regex/ which we could just as well replace with a
compat/pcre/.

The reason it would matter to have a hard prereq on PCRE is because it
would simplify and speed up a lot of code in various places to be able
to make an assumption that it's there. E.g. for things that do "match a
line" like pickaxe we could do clever anchoring and match the whole diff
at once, and also things like streaming the diff out as we consume it
into PCRE's stream matching API.

We also have a lot of things purely on the C level that would be
simpler, e.g. (just one thing I remember) grep.c has 40 lines of code to
emulate \b<word>\b, but we could ... just use \b.

To Jeff's comment upthread:

> And while it may build portably everywhere, that may involve extra
> work and not integrate with our build knobs (e.g., it looks like it
> uses autoconf)

I already did that work in an old WIP branch I have. I haven't run it
again so maybe this summary is inaccurate, but here's the commit:
https://github.com/avar/git/commit/79256ee4a1

So basically it won't build with autoconf or whatever, just with our own
build system after we'd wget the relevant PCREv2 source files into
compat/pcre/.

The only things PCRE really needed autoconf or its own build system for
was to define 20-ish macros which are either options we can switch, or
(looking at this now) 4x probes like "HAVE_STDINT_H" which we could add
to compat.mak.uname and friends.

If we wanted it to Just Work we could run that script and "git commit"
it, or just leave it at a make warning saying "you don't have PCRE,
either install it using your pkg system, or run this handy shellscript".

But yeah, if Junio's paranoid about the security aspect we could skip
that "./get-pcre.sh && git commit" step.

I'm getting some deja-vu writing this next part, I'm pretty sure I've
written some version of the same E-Mail before, in any case:

 A. This wouldn't be anything new. We have some old bitrotting copy of
    GNU awk's copy of glibc's regex engine in compat/regex. It's trivial
    to e.g. get it to exhaust all memory on your system.

    Sure, PCRE could have bugs, but at least we *could* update it. As
    you may remember I abandoned my last attempt to update compat/regex/

 B. Even if you assume some bug like that, I think realistically no user
    anywhere would be vulnerable to a copy we'd ship in compat/pcre/,
    any vendor would ignore it and build the prereq themselves. It's
    purely a developer aid.

    Of course it could have some CVE, but it wouldn't be mean a git
    point release.

 C. Even if the regex engine has an CVE/RCE we have no feature anywhere
    to execute arbitrary user regexes, anyone who could inject one can
    already run arbitrary git commands.

 D. The *one* exception to C) is gitweb's not-on-by-default "grep the
    source tree" feature.

    So first its users would be better off than they are now now. You
    can trivially DoS any box that has it exposed, secondly the union of
    gitweb users + those that would turn that on + build their own git
    for running a public-facing site is probably 0 people these days.

 E. Even if you assume some combination of B..D isn't true I genuinely
    don't know why this concern of PCRE being a special
    security/vulnerability concern comes from, it seems to be brought up
    any time the topic is discussed.

    The implicit context is that we already process regexes using a
    plethora of engines, and are using anything from AIXs, SunOS's,
    Window's, OSX's, GNU libc's etc. regex engines to do so.

    I haven't seen anything to suggest that PCRE's security record is
    notably worse than even one of those pieces of software, and right
    now we're exposing any bugs in the union of all those engines.

    It's quite the opposite, unlike those engines PCRE actually gives
    you really good features to limit resource consumption and regex
    features in the event that you don't trust your regex or your
    input.

    I'd actually trust for things like processing a "git grep on the
    web" with appropriate resource constraints.

    The two solutions in that space are basically to use DFA engine as
    e.g. Google's re2 does. That means living with a severely restricted
    set of regex features, or to have bells and whistles as PCRE's NFA
    does, but one that has tweakable resource limits.

[PATCH v4 06/10] userdiff tests: explicitly test "default" pattern

[Ævar Arnfjörð Bjarmason]

Since 122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05)
the internals of the userdiff.c code have understood a "default" name,
which is invoked as userdiff_find_by_name("default") and present in
the "builtin_drivers" struct. Let's test for this special case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 9675bc17db2..cefe329aea7 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -49,6 +49,7 @@ diffpatterns="
 	ruby
 	rust
 	tex
+	default
 	custom1
 	custom2
 	custom3
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 07/10] userdiff tests: list builtin drivers via test-tool

[Ævar Arnfjörð Bjarmason]

Change the userdiff test to list the builtin drivers via the
test-tool, using the new for_each_userdiff_driver() API function.

This gets rid of the need to modify this part of the test every time a
new pattern is added, see 2ff6c34612 (userdiff: support Bash,
2020-10-22) and 09dad9256a (userdiff: support Markdown, 2020-05-02)
for two recent examples.

I only need the "list-builtin-drivers "argument here, but let's add
"list-custom-drivers" and "list-drivers" too, just because it's easy.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile                 |  1 +
 t/helper/test-tool.c     |  1 +
 t/helper/test-tool.h     |  1 +
 t/helper/test-userdiff.c | 31 +++++++++++++++++++++++++++++++
 t/t4018-diff-funcname.sh | 32 ++++++++------------------------
 5 files changed, 42 insertions(+), 24 deletions(-)
 create mode 100644 t/helper/test-userdiff.c

diff --git a/Makefile b/Makefile
index f3dc2178324..3e2568d68da 100644
--- a/Makefile
+++ b/Makefile
@@ -744,6 +744,7 @@ TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o
 TEST_BUILTINS_OBJS += test-subprocess.o
 TEST_BUILTINS_OBJS += test-trace2.o
 TEST_BUILTINS_OBJS += test-urlmatch-normalization.o
+TEST_BUILTINS_OBJS += test-userdiff.o
 TEST_BUILTINS_OBJS += test-wildmatch.o
 TEST_BUILTINS_OBJS += test-windows-named-pipe.o
 TEST_BUILTINS_OBJS += test-write-cache.o
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index f97cd9f48a6..dcb05ca6e5e 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -71,6 +71,7 @@ static struct test_cmd cmds[] = {
 	{ "submodule-nested-repo-config", cmd__submodule_nested_repo_config },
 	{ "subprocess", cmd__subprocess },
 	{ "trace2", cmd__trace2 },
+	{ "userdiff", cmd__userdiff },
 	{ "urlmatch-normalization", cmd__urlmatch_normalization },
 	{ "xml-encode", cmd__xml_encode },
 	{ "wildmatch", cmd__wildmatch },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index 28072c0ad5a..589f2e8ac67 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -61,6 +61,7 @@ int cmd__submodule_config(int argc, const char **argv);
 int cmd__submodule_nested_repo_config(int argc, const char **argv);
 int cmd__subprocess(int argc, const char **argv);
 int cmd__trace2(int argc, const char **argv);
+int cmd__userdiff(int argc, const char **argv);
 int cmd__urlmatch_normalization(int argc, const char **argv);
 int cmd__xml_encode(int argc, const char **argv);
 int cmd__wildmatch(int argc, const char **argv);
diff --git a/t/helper/test-userdiff.c b/t/helper/test-userdiff.c
new file mode 100644
index 00000000000..f173a4f18af
--- /dev/null
+++ b/t/helper/test-userdiff.c
@@ -0,0 +1,31 @@
+#include "test-tool.h"
+#include "cache.h"
+#include "userdiff.h"
+
+static int driver_cb(struct userdiff_driver *driver,
+		     enum userdiff_driver_type type, void *priv)
+{
+	if (driver->funcname.pattern)
+		puts(driver->name);
+	return 0;
+}
+
+static int list_what(enum userdiff_driver_type type)
+{
+	return for_each_userdiff_driver(driver_cb, type, NULL);
+}
+
+int cmd__userdiff(int argc, const char **argv)
+{
+	if (argc != 2)
+		return 1;
+
+	if (!strcmp(argv[1], "list-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_UNSPECIFIED);
+	else if (!strcmp(argv[1], "list-builtin-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_BUILTIN);
+	else if (!strcmp(argv[1], "list-custom-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_CUSTOM);
+	else
+		return error("unknown argument %s", argv[1]);
+}
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index cefe329aea7..5bd82e09ab3 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -8,6 +8,13 @@ test_description='Test custom diff function name patterns'
 . ./test-lib.sh
 
 test_expect_success 'setup' '
+	# Make sure additions to builtin_drivers are sorted
+	test_when_finished "rm builtin-drivers.sorted" &&
+	test-tool userdiff list-builtin-drivers >builtin-drivers &&
+	test_file_not_empty builtin-drivers &&
+	sort <builtin-drivers >builtin-drivers.sorted &&
+	test_cmp builtin-drivers.sorted builtin-drivers &&
+
 	# a non-trivial custom pattern
 	git config diff.custom1.funcname "!static
 !String
@@ -26,30 +33,7 @@ test_expect_success 'setup' '
 '
 
 diffpatterns="
-	ada
-	bash
-	bibtex
-	cpp
-	csharp
-	css
-	dts
-	elixir
-	fortran
-	fountain
-	golang
-	html
-	java
-	markdown
-	matlab
-	objc
-	pascal
-	perl
-	php
-	python
-	ruby
-	rust
-	tex
-	default
+	$(cat builtin-drivers)
 	custom1
 	custom2
 	custom3
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 08/10] userdiff: remove support for "broken" tests

[Ævar Arnfjörð Bjarmason]

There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve
Fortran xfuncname regex, 2020-08-12). Let's remove the test support
for them.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 8 +-------
 t/t4018/README           | 3 ---
 2 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 5bd82e09ab3..9aec9f8e6de 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -86,13 +86,7 @@ test_expect_success 'setup hunk header tests' '
 # check each individual file
 for i in $(git ls-files)
 do
-	if grep broken "$i" >/dev/null 2>&1
-	then
-		result=failure
-	else
-		result=success
-	fi
-	test_expect_$result "hunk header: $i" "
+	test_expect_success "hunk header: $i" "
 		git diff -U1 $i >actual &&
 		grep '@@ .* @@.*RIGHT' actual
 	"
diff --git a/t/t4018/README b/t/t4018/README
index 283e01cca1a..2d25b2b4fc9 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -7,9 +7,6 @@ at least two lines from the line that must appear in the hunk header.
 The text that must appear in the hunk header must contain the word
 "right", but in all upper-case, like in the title above.
 
-To mark a test case that highlights a malfunction, insert the word
-BROKEN in all lower-case somewhere in the file.
-
 This text is a bit twisted and out of order, but it is itself a
 test case for the default hunk header pattern. Know what you are doing
 if you change it.
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 09/10] blame tests: don't rely on t/t4018/ directory

[Ævar Arnfjörð Bjarmason]

Refactor a test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) so that the blame tests don't rely
on stealing the contents of "t/t4018/fortran-external-function".

I have another patch series that'll possibly (or not) refactor that
file, but having this test inter-dependency makes things simple in any
case by making this test more readable.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 29ce89090d8..04a2c58594c 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -482,12 +482,22 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 test_expect_success 'setup -L :funcname with userdiff driver' '
 	echo "fortran-* diff=fortran" >.gitattributes &&
 	fortran_file=fortran-external-function &&
-	orig_file="$TEST_DIRECTORY/t4018/$fortran_file" &&
-	cp "$orig_file" . &&
+	cat >$fortran_file <<-\EOF &&
+	function RIGHT(a, b) result(c)
+
+	integer, intent(in) :: ChangeMe
+	integer, intent(in) :: b
+	integer, intent(out) :: c
+
+	c = a+b
+
+	end function RIGHT
+	EOF
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
 	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$orig_file" >"$fortran_file" &&
+	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
+	mv "$fortran_file".tmp "$fortran_file" &&
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
 	git commit -m "change fortran file"
-- 
2.31.0.366.ga80606b22c1

[PATCH v4 10/10] blame tests: simplify userdiff driver test

[Ævar Arnfjörð Bjarmason]

Simplify the test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) to use the --author support recently
added in 999cfc4f45 (test-lib functions: add --author support to
test_commit, 2021-01-12).

We also did not need the full fortran-external-function content. Let's
cut it down to just the important parts.

I'm modifying it to demonstrate that the fortran-specific userdiff
function is in effect by adding "DO NOT MATCH ..." and "AS THE ..."
lines surrounding the "RIGHT" one.

This is to check that we're using the userdiff "fortran" driver, as
opposed to the default driver which would match on those lines as part
of the general heuristic of matching a line that doesn't begin with
whitespace.

The test had also been leaving behind a .gitattributes file for later
tests to possibly trip over, let's clean it up with
"test_when_finished".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 36 +++++++++++++++---------------------
 1 file changed, 15 insertions(+), 21 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 04a2c58594c..d3b299e75cb 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -479,32 +479,26 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 	check_count -f hello.c -L$n -L^:ma.. F 4 G 1 H 1
 '
 
-test_expect_success 'setup -L :funcname with userdiff driver' '
-	echo "fortran-* diff=fortran" >.gitattributes &&
-	fortran_file=fortran-external-function &&
-	cat >$fortran_file <<-\EOF &&
+test_expect_success 'blame -L :funcname with userdiff driver' '
+	cat >file.template <<-\EOF &&
+	DO NOT MATCH THIS LINE
 	function RIGHT(a, b) result(c)
+	AS THE DEFAULT DRIVER WOULD
 
 	integer, intent(in) :: ChangeMe
-	integer, intent(in) :: b
-	integer, intent(out) :: c
-
-	c = a+b
-
-	end function RIGHT
 	EOF
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
-	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
-	mv "$fortran_file".tmp "$fortran_file" &&
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
-	git commit -m "change fortran file"
-'
 
-test_expect_success 'blame -L :funcname with userdiff driver' '
-	check_count -f fortran-external-function -L:RIGHT A 7 B 1
+	fortran_file=file.f03 &&
+	test_when_finished "rm .gitattributes" &&
+	echo "$fortran_file diff=fortran" >.gitattributes &&
+
+	test_commit --author "A <A@test.git>" \
+		"add" "$fortran_file" \
+		"$(cat file.template)" &&
+	test_commit --author "B <B@test.git>" \
+		"change" "$fortran_file" \
+		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
+	check_count -f "$fortran_file" -L:RIGHT A 3 B 1
 '
 
 test_expect_success 'setup incremental' '
-- 
2.31.0.366.ga80606b22c1

Re: [PATCH v4 00/10] userdiff: refactor + test improvements

[Johannes Sixt]

Am 24.03.21 um 02:48 schrieb Ævar Arnfjörð Bjarmason:
> This is a restart of the 35-patch v3 of this topic at
> https://lore.kernel.org/git/20210224195129.4004-1-avarab@gmail.com/
> 
> I still plan on submitting the rest of it, but wanted to start with
> the early parts of that series that hasn't been controversial or has
> outstanding feedback I haven't addressed.
> 
> The range-diff to v3 is just for those patches I'm re-rolling here.

I'm mostly relying on the interdiff below. I think I had no comments on
these patches in the earlier round, so:

Acked-by: Johannes Sixt <j6t@kdbg.org>

> 
> Ævar Arnfjörð Bjarmason (10):
>   userdiff: refactor away the parse_bool() function
>   userdiff style: re-order drivers in alphabetical order
>   userdiff style: declare patterns with consistent style
>   userdiff style: normalize pascal regex declaration
>   userdiff: add and use for_each_userdiff_driver()
>   userdiff tests: explicitly test "default" pattern
>   userdiff tests: list builtin drivers via test-tool
>   userdiff: remove support for "broken" tests
>   blame tests: don't rely on t/t4018/ directory
>   blame tests: simplify userdiff driver test
> 
>  Makefile                 |   1 +
>  t/annotate-tests.sh      |  34 ++++----
>  t/helper/test-tool.c     |   1 +
>  t/helper/test-tool.h     |   1 +
>  t/helper/test-userdiff.c |  31 +++++++
>  t/t4018-diff-funcname.sh |  39 ++-------
>  t/t4018/README           |   3 -
>  userdiff.c               | 178 ++++++++++++++++++++++++---------------
>  userdiff.h               |  15 ++++
>  9 files changed, 186 insertions(+), 117 deletions(-)
>  create mode 100644 t/helper/test-userdiff.c
> 
> Range-diff:
>  1:  0be132b05e2 =  1:  fb7346cd296 userdiff: refactor away the parse_bool() function
>  2:  d1e00a739ac =  2:  149387155bc userdiff style: re-order drivers in alphabetical order
>  3:  b99bd158d45 =  3:  faf1a824f05 userdiff style: declare patterns with consistent style
>  4:  9ce6d47021c =  4:  1e9ddcd1a9a userdiff style: normalize pascal regex declaration
>  5:  369fbdcee83 =  5:  64ea5e8443f userdiff: add and use for_each_userdiff_driver()
>  6:  70d62a97211 =  6:  862f6ab5d66 userdiff tests: explicitly test "default" pattern
>  7:  792421a2f8b =  7:  22a07591b76 userdiff tests: list builtin drivers via test-tool
>  8:  9081e2a152e !  8:  7755db95014 userdiff: remove support for "broken" tests
>     @@ Commit message
>      
>          There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve
>          Fortran xfuncname regex, 2020-08-12). Let's remove the test support
>     -    for them, this is in preparation for a more general refactoring of the
>     -    tests.
>     +    for them.
>      
>          Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>      
>  9:  d3652f95d5e !  9:  4e0b4b42e16 blame tests: don't rely on t/t4018/ directory
>     @@ Commit message
>          with userdiff driver, 2020-11-01) so that the blame tests don't rely
>          on stealing the contents of "t/t4018/fortran-external-function".
>      
>     -    I'm about to change that file in a subsequent commit. Just moving the
>     -    relevant test file here inline is the easiest solution, and I think
>     -    also the most readable.
>     +    I have another patch series that'll possibly (or not) refactor that
>     +    file, but having this test inter-dependency makes things simple in any
>     +    case by making this test more readable.
>      
>          Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>      
> 10:  35d12779ea1 ! 10:  ce98c61bf40 blame tests: simplify userdiff driver test
>     @@ Commit message
>          test_commit, 2021-01-12).
>      
>          We also did not need the full fortran-external-function content. Let's
>     -    cut it down to just the important parts, and further modify it to
>     -    demonstrate that the fortran-specific userdiff function is in effect
>     -    by adding "DO NOT MATCH ..." and "AS THE ..." lines surrounding the
>     -    "RIGHT" one. This is to check that we're using the userdiff "fortran"
>     -    driver, as opposed to the default driver.
>     +    cut it down to just the important parts.
>      
>     -    The test also left behind a .gitattributes files, let's clean it up
>     -    with "test_when_finished".
>     +    I'm modifying it to demonstrate that the fortran-specific userdiff
>     +    function is in effect by adding "DO NOT MATCH ..." and "AS THE ..."
>     +    lines surrounding the "RIGHT" one.
>     +
>     +    This is to check that we're using the userdiff "fortran" driver, as
>     +    opposed to the default driver which would match on those lines as part
>     +    of the general heuristic of matching a line that doesn't begin with
>     +    whitespace.
>     +
>     +    The test had also been leaving behind a .gitattributes file for later
>     +    tests to possibly trip over, let's clean it up with
>     +    "test_when_finished".
>      
>          Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>      
> 

Re: [PATCH v4 00/10] userdiff: refactor + test improvements

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> Am 24.03.21 um 02:48 schrieb Ævar Arnfjörð Bjarmason:
>> This is a restart of the 35-patch v3 of this topic at
>> https://lore.kernel.org/git/20210224195129.4004-1-avarab@gmail.com/
>> 
>> I still plan on submitting the rest of it, but wanted to start with
>> the early parts of that series that hasn't been controversial or has
>> outstanding feedback I haven't addressed.
>> 
>> The range-diff to v3 is just for those patches I'm re-rolling here.
>
> I'm mostly relying on the interdiff below. I think I had no comments on
> these patches in the earlier round, so:
>
> Acked-by: Johannes Sixt <j6t@kdbg.org>

Thanks.  I've read through everything in the patch this round,
ignoring anything that came before, and them looked mostly fine.
Peff's comment on 01/10 to make it if/else if cascade does make
sense to me, too, though.

Re: [PATCH v4 00/10] userdiff: refactor + test improvements

[Jeff King]

On Wed, Mar 24, 2021 at 12:02:54PM -0700, Junio C Hamano wrote:

> Johannes Sixt <j6t@kdbg.org> writes:
> 
> > Am 24.03.21 um 02:48 schrieb Ævar Arnfjörð Bjarmason:
> >> This is a restart of the 35-patch v3 of this topic at
> >> https://lore.kernel.org/git/20210224195129.4004-1-avarab@gmail.com/
> >> 
> >> I still plan on submitting the rest of it, but wanted to start with
> >> the early parts of that series that hasn't been controversial or has
> >> outstanding feedback I haven't addressed.
> >> 
> >> The range-diff to v3 is just for those patches I'm re-rolling here.
> >
> > I'm mostly relying on the interdiff below. I think I had no comments on
> > these patches in the earlier round, so:
> >
> > Acked-by: Johannes Sixt <j6t@kdbg.org>
> 
> Thanks.  I've read through everything in the patch this round,
> ignoring anything that came before, and them looked mostly fine.
> Peff's comment on 01/10 to make it if/else if cascade does make
> sense to me, too, though.

I left some other comments on patch 4, mostly about clarity. But just to
be clear, I don't think there's anything incorrect there, and I wouldn't
be offended if it gets picked up as-is.

-Peff

[PATCH v5 0/9] userdiff: refactor + test improvements

[Ævar Arnfjörð Bjarmason]

As noted in v3[1] this is the start of some wider userdiff test
improvements.

Since v3 I've ejected the parse_bool() change at the start of the
series. I split that into another series[2]:

This one applies on master, and doesn't conflict with the tiny
unrelated change in userdiff.c done in that series either textually or
semantically.

Changes since v3:

 * Jeff and Junio suggested refactoring the userdiff.c callback
   facility. I've done that using some union of their suggetions, I
   think it's easier to read this time around.

   Basically, the "for each userdiff" callback doesn't take a "I want
   this type", that's now handled in the user callback, which
   simplifies things a lot.

 * Other minor changes related to the above, e.g. renaming the CB
   struct per Jeff's suggestion.

 * I found that the "list-custom-drivers" I'd added for completeness
   to the helper didn't work, I needed to setup the .git dir and
   config in the helper. That's now done, and the tests check that the
   helper does the right thing.

1. https://lore.kernel.org/git/20210224195129.4004-1-avarab@gmail.com/
2. https://lore.kernel.org/git/cover-0.6-0000000000-20210408T133125Z-avarab@gmail.com/

Ævar Arnfjörð Bjarmason (9):
  userdiff style: re-order drivers in alphabetical order
  userdiff style: declare patterns with consistent style
  userdiff style: normalize pascal regex declaration
  userdiff: add and use for_each_userdiff_driver()
  userdiff tests: explicitly test "default" pattern
  userdiff tests: list builtin drivers via test-tool
  userdiff: remove support for "broken" tests
  blame tests: don't rely on t/t4018/ directory
  blame tests: simplify userdiff driver test

 Makefile                 |   1 +
 t/annotate-tests.sh      |  34 ++++----
 t/helper/test-tool.c     |   1 +
 t/helper/test-tool.h     |   1 +
 t/helper/test-userdiff.c |  46 +++++++++++
 t/t4018-diff-funcname.sh |  53 +++++-------
 t/t4018/README           |   3 -
 userdiff.c               | 169 ++++++++++++++++++++++++++-------------
 userdiff.h               |  13 +++
 9 files changed, 213 insertions(+), 108 deletions(-)
 create mode 100644 t/helper/test-userdiff.c

Range-diff:
 1:  fb7346cd29 <  -:  ---------- userdiff: refactor away the parse_bool() function
 2:  149387155b =  1:  2f7a7b2897 userdiff style: re-order drivers in alphabetical order
 3:  faf1a824f0 =  2:  22e48f33b6 userdiff style: declare patterns with consistent style
 4:  1e9ddcd1a9 =  3:  dff2aa0d38 userdiff style: normalize pascal regex declaration
 5:  64ea5e8443 !  4:  1a3a61e389 userdiff: add and use for_each_userdiff_driver()
    @@ userdiff.c: static struct userdiff_driver driver_false = {
      };
      
     -static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
    -+struct for_each_userdiff_driver_cb {
    -+	const char *k;
    ++struct find_by_namelen_data {
    ++	const char *name;
     +	size_t len;
     +	struct userdiff_driver *driver;
     +};
    @@ userdiff.c: static struct userdiff_driver driver_false = {
     -		struct userdiff_driver *drv = builtin_drivers + i;
     -		if (!strncmp(drv->name, k, len) && !drv->name[len])
     -			return drv;
    -+	struct for_each_userdiff_driver_cb *cb_data = priv;
    ++	struct find_by_namelen_data *cb_data = priv;
     +
    -+	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
    ++	if (!strncmp(driver->name, cb_data->name, cb_data->len) &&
     +	    !driver->name[cb_data->len]) {
     +		cb_data->driver = driver;
    -+		return -1; /* found it! */
    ++		return 1; /* tell the caller to stop iterating */
      	}
     -	return NULL;
     +	return 0;
     +}
     +
    -+static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
    ++static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t len)
     +{
    -+	struct for_each_userdiff_driver_cb udcbdata = { .k = k, .len = len, .driver = NULL };
    -+
    -+	for_each_userdiff_driver(userdiff_find_by_namelen_cb,
    -+				 USERDIFF_DRIVER_TYPE_UNSPECIFIED, &udcbdata);
    ++	struct find_by_namelen_data udcbdata = {
    ++		.name = name,
    ++		.len = len,
    ++	};
    ++	for_each_userdiff_driver(userdiff_find_by_namelen_cb, &udcbdata);
     +	return udcbdata.driver;
      }
      
    @@ userdiff.c: struct userdiff_driver *userdiff_get_textconv(struct repository *r,
      	return driver;
      }
     +
    -+int for_each_userdiff_driver(each_userdiff_driver_fn fn,
    -+			     enum userdiff_driver_type type, void *cb_data)
    ++static int for_each_userdiff_driver_list(each_userdiff_driver_fn fn,
    ++					 enum userdiff_driver_type type, void *cb_data,
    ++					 struct userdiff_driver *drv,
    ++					 int drv_size)
     +{
    -+	int i, ret;
    -+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {
    -+
    -+		for (i = 0; i < ndrivers; i++) {
    -+			struct userdiff_driver *drv = drivers + i;
    -+			ret = fn(drv, USERDIFF_DRIVER_TYPE_CUSTOM, cb_data);
    -+			if (ret)
    -+				return ret;
    -+		}
    ++	int i;
    ++	int ret;
    ++	for (i = 0; i < drv_size; i++) {
    ++		struct userdiff_driver *item = drv + i;
    ++		if ((ret = fn(item, type, cb_data)))
    ++			return ret;
     +	}
    -+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_BUILTIN)) {
    ++	return 0;
    ++}
    ++
    ++int for_each_userdiff_driver(each_userdiff_driver_fn fn, void *cb_data)
    ++{
    ++	int ret;
    ++
    ++	ret = for_each_userdiff_driver_list(fn, USERDIFF_DRIVER_TYPE_CUSTOM,
    ++					    cb_data, drivers, ndrivers);
    ++	if (ret)
    ++		return ret;
    ++
    ++	ret = for_each_userdiff_driver_list(fn, USERDIFF_DRIVER_TYPE_BUILTIN,
    ++					    cb_data, builtin_drivers,
    ++					    ARRAY_SIZE(builtin_drivers));
    ++	if (ret)
    ++		return ret;
     +
    -+		for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
    -+			struct userdiff_driver *drv = builtin_drivers + i;
    -+			ret = fn(drv, USERDIFF_DRIVER_TYPE_BUILTIN, cb_data);
    -+			if (ret)
    -+				return ret;
    -+		}
    -+	}
     +	return 0;
     +}
     
    @@ userdiff.h: struct userdiff_driver {
      	int textconv_want_cache;
      };
     +enum userdiff_driver_type {
    -+	USERDIFF_DRIVER_TYPE_UNSPECIFIED = 1<<0,
    -+	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<1,
    -+	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<2,
    ++	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
    ++	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<1,
     +};
     +typedef int (*each_userdiff_driver_fn)(struct userdiff_driver *,
     +				       enum userdiff_driver_type, void *);
    @@ userdiff.h: struct userdiff_driver *userdiff_find_by_path(struct index_state *is
      					      struct userdiff_driver *driver);
      
     +/*
    -+ * Iterate over each driver of type userdiff_driver_type, or
    -+ * USERDIFF_DRIVER_TYPE_UNSPECIFIED for all of them. Return non-zero
    -+ * to exit from the loop.
    ++ * Iterate over all userdiff drivers. The userdiff_driver_type
    ++ * argument to each_userdiff_driver_fn indicates their type. Return
    ++ * non-zero to exit early from the loop.
     + */
    -+int for_each_userdiff_driver(each_userdiff_driver_fn,
    -+			     enum userdiff_driver_type, void *);
    ++int for_each_userdiff_driver(each_userdiff_driver_fn, void *);
     +
      #endif /* USERDIFF */
 6:  862f6ab5d6 =  5:  3eb7abd121 userdiff tests: explicitly test "default" pattern
 7:  22a07591b7 !  6:  e90758a978 userdiff tests: list builtin drivers via test-tool
    @@ t/helper/test-userdiff.c (new)
     +#include "test-tool.h"
     +#include "cache.h"
     +#include "userdiff.h"
    ++#include "config.h"
     +
     +static int driver_cb(struct userdiff_driver *driver,
     +		     enum userdiff_driver_type type, void *priv)
     +{
    -+	if (driver->funcname.pattern)
    ++	enum userdiff_driver_type *want_type = priv;
    ++	if (type & *want_type && driver->funcname.pattern)
     +		puts(driver->name);
     +	return 0;
     +}
     +
    -+static int list_what(enum userdiff_driver_type type)
    ++static int cmd__userdiff_config(const char *var, const char *value, void *cb)
     +{
    -+	return for_each_userdiff_driver(driver_cb, type, NULL);
    ++	if (userdiff_config(var, value) < 0)
    ++		return -1;
    ++	return 0;
     +}
     +
     +int cmd__userdiff(int argc, const char **argv)
     +{
    ++	enum userdiff_driver_type want = 0;
     +	if (argc != 2)
     +		return 1;
     +
     +	if (!strcmp(argv[1], "list-drivers"))
    -+		return list_what(USERDIFF_DRIVER_TYPE_UNSPECIFIED);
    ++		want = (USERDIFF_DRIVER_TYPE_BUILTIN |
    ++			USERDIFF_DRIVER_TYPE_CUSTOM);
     +	else if (!strcmp(argv[1], "list-builtin-drivers"))
    -+		return list_what(USERDIFF_DRIVER_TYPE_BUILTIN);
    ++		want = USERDIFF_DRIVER_TYPE_BUILTIN;
     +	else if (!strcmp(argv[1], "list-custom-drivers"))
    -+		return list_what(USERDIFF_DRIVER_TYPE_CUSTOM);
    ++		want = USERDIFF_DRIVER_TYPE_CUSTOM;
     +	else
     +		return error("unknown argument %s", argv[1]);
    ++
    ++	if (want & USERDIFF_DRIVER_TYPE_CUSTOM) {
    ++		setup_git_directory();
    ++		git_config(cmd__userdiff_config, NULL);
    ++	}
    ++
    ++	for_each_userdiff_driver(driver_cb, &want);
    ++
    ++	return 0;
     +}
     
      ## t/t4018-diff-funcname.sh ##
    -@@ t/t4018-diff-funcname.sh: test_description='Test custom diff function name patterns'
    - . ./test-lib.sh
    +@@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
    + 	echo B >B.java
    + '
      
    - test_expect_success 'setup' '
    ++test_expect_success 'setup: test-tool userdiff' '
     +	# Make sure additions to builtin_drivers are sorted
     +	test_when_finished "rm builtin-drivers.sorted" &&
     +	test-tool userdiff list-builtin-drivers >builtin-drivers &&
    @@ t/t4018-diff-funcname.sh: test_description='Test custom diff function name patte
     +	sort <builtin-drivers >builtin-drivers.sorted &&
     +	test_cmp builtin-drivers.sorted builtin-drivers &&
     +
    - 	# a non-trivial custom pattern
    - 	git config diff.custom1.funcname "!static
    - !String
    -@@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
    - '
    - 
    ++	# Ditto, but "custom" requires the .git directory and config
    ++	# to be setup and read.
    ++	test_when_finished "rm custom-drivers.sorted" &&
    ++	test-tool userdiff list-custom-drivers >custom-drivers &&
    ++	test_file_not_empty custom-drivers &&
    ++	sort <custom-drivers >custom-drivers.sorted &&
    ++	test_cmp custom-drivers.sorted custom-drivers
    ++'
    ++
      diffpatterns="
     -	ada
     -	bash
    @@ t/t4018-diff-funcname.sh: test_expect_success 'setup' '
     -	rust
     -	tex
     -	default
    +-	custom1
    +-	custom2
    +-	custom3
     +	$(cat builtin-drivers)
    - 	custom1
    - 	custom2
    - 	custom3
    ++	$(cat custom-drivers)
    + "
    + 
    + for p in $diffpatterns
 8:  7755db9501 =  7:  04bce275ab userdiff: remove support for "broken" tests
 9:  4e0b4b42e1 =  8:  3583078715 blame tests: don't rely on t/t4018/ directory
10:  ce98c61bf4 =  9:  548673260b blame tests: simplify userdiff driver test
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 1/9] userdiff style: re-order drivers in alphabetical order

[Ævar Arnfjörð Bjarmason]

Address some old code smell and move around the built-in userdiff
drivers so they're both in alphabetical order, and now in the same
order they appear in the gitattributes(5) documentation.

The two started drifting in be58e70dba (diff: unify external diff and
funcname parsing code, 2008-10-05), and then even further in
80c49c3de2 (color-words: make regex configurable via attributes,
2009-01-17) when the "cpp" pattern was added.

There are no functional changes here, and as --color-moved will show
only moved existing lines.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 76 +++++++++++++++++++++++++++---------------------------
 1 file changed, 38 insertions(+), 38 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 3f81a2261c..650f421d63 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,6 +44,44 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
+PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 "[={}\"]|[^={}\" \t]+"),
+PATTERNS("cpp",
+	 /* Jump targets or access declarations */
+	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
+	 /* functions/methods, variables, and compounds at top level */
+	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
+PATTERNS("csharp",
+	 /* Keywords */
+	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
+	 /* Methods and constructors */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
+	 /* Properties */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
+	 /* Type definitions */
+	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
+	 /* Namespace */
+	 "^[ \t]*(namespace[ \t]+.*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
+IPATTERN("css",
+	 "![:;][[:space:]]*$\n"
+	 "^[:[@.#]?[_a-z0-9].*$",
+	 /* -- */
+	 /*
+	  * This regex comes from W3C CSS specs. Should theoretically also
+	  * allow ISO 10646 characters U+00A0 and higher,
+	  * but they are not handled in this regex.
+	  */
+	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
+	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
+),
 PATTERNS("dts",
 	 "!;\n"
 	 "!=\n"
@@ -191,46 +229,8 @@ PATTERNS("rust",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[0-9][0-9_a-fA-Fiosuxz]*(\\.([0-9]*[eE][+-]?)?[0-9_fF]*)?"
 	 "|[-+*\\/<>%&^|=!:]=|<<=?|>>=?|&&|\\|\\||->|=>|\\.{2}=|\\.{3}|::"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
-	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("tex", "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$",
 	 "\\\\[a-zA-Z@]+|\\\\.|[a-zA-Z0-9\x80-\xff]+"),
-PATTERNS("cpp",
-	 /* Jump targets or access declarations */
-	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
-	 /* functions/methods, variables, and compounds at top level */
-	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
-PATTERNS("csharp",
-	 /* Keywords */
-	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
-	 /* Methods and constructors */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
-	 /* Properties */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
-	 /* Type definitions */
-	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
-	 /* Namespace */
-	 "^[ \t]*(namespace[ \t]+.*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
-IPATTERN("css",
-	 "![:;][[:space:]]*$\n"
-	 "^[:[@.#]?[_a-z0-9].*$",
-	 /* -- */
-	 /*
-	  * This regex comes from W3C CSS specs. Should theoretically also
-	  * allow ISO 10646 characters U+00A0 and higher,
-	  * but they are not handled in this regex.
-	  */
-	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
-	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
-),
 { "default", NULL, -1, { NULL, 0 } },
 };
 #undef PATTERNS
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 2/9] userdiff style: declare patterns with consistent style

[Ævar Arnfjörð Bjarmason]

Change those patterns which were declared with a regex on the same
line as the "PATTERNS()" line to put that regex on the next line, and
add missing "/* -- */" separator comments between the pattern and
word_regex.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 650f421d63..33b0ce4020 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,7 +44,9 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+PATTERNS("bibtex",
+	 "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 /* -- */
 	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("cpp",
 	 /* Jump targets or access declarations */
@@ -121,7 +123,9 @@ IPATTERN("fortran",
 	  * they would have been matched above as a variable anyway. */
 	 "|[-+]?[0-9.]+([AaIiDdEeFfLlTtXx][Ss]?[-+]?[0-9.]*)?(_[a-zA-Z0-9][a-zA-Z0-9_]*)?"
 	 "|//|\\*\\*|::|[/<>=]="),
-IPATTERN("fountain", "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+IPATTERN("fountain",
+	 "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+	 /* -- */
 	 "[^ \t-]+"),
 PATTERNS("golang",
 	 /* Functions */
@@ -132,7 +136,9 @@ PATTERNS("golang",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.eE]+i?|0[xX]?[0-9a-fA-F]+i?"
 	 "|[-+*/<>%&^|=!:]=|--|\\+\\+|<<=?|>>=?|&\\^=?|&&|\\|\\||<-|\\.{3}"),
-PATTERNS("html", "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+PATTERNS("html",
+	 "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("java",
 	 "!^[ \t]*(catch|do|for|if|instanceof|new|return|switch|throw|while)\n"
@@ -144,6 +150,7 @@ PATTERNS("java",
 	 "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"),
 PATTERNS("markdown",
 	 "^ {0,3}#{1,6}[ \t].*",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("matlab",
 	 /*
@@ -152,6 +159,7 @@ PATTERNS("matlab",
 	  * that is understood by both.
 	  */
 	 "^[[:space:]]*((classdef|function)[[:space:]].*)$|^(%%%?|##)[[:space:]].*$",
+	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*|[-+0-9.e]+|[=~<>]=|\\.[*/\\^']|\\|\\||&&"),
 PATTERNS("objc",
 	 /* Negate C statements that can look like functions */
@@ -212,13 +220,15 @@ PATTERNS("php",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+"
 	 "|[-+*/<>%&^|=!.]=|--|\\+\\+|<<=?|>>=?|===|&&|\\|\\||::|->"),
-PATTERNS("python", "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
+PATTERNS("python",
+	 "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+[jJlL]?|0[xX]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|//=?|<<=?|>>=?|\\*\\*=?"),
 	 /* -- */
-PATTERNS("ruby", "^[ \t]*((class|module|def)[ \t].*)$",
+PATTERNS("ruby",
+	 "^[ \t]*((class|module|def)[ \t].*)$",
 	 /* -- */
 	 "(@|@@|\\$)?[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+|\\?(\\\\C-)?(\\\\M-)?."
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 3/9] userdiff style: normalize pascal regex declaration

[Ævar Arnfjörð Bjarmason]

Declare the pascal pattern consistently with how we declare the
others, not having "\n" on one line by itself, but as part of the
pattern, and when there are alterations have the "|" at the start, not
end of the line.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 33b0ce4020..978ae64155 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -175,9 +175,8 @@ PATTERNS("objc",
 	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
 PATTERNS("pascal",
-	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface|"
-		"implementation|initialization|finalization)[ \t]*.*)$"
-	 "\n"
+	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface"
+	 "|implementation|initialization|finalization)[ \t]*.*)$\n"
 	 "^(.*=[ \t]*(class|record).*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 4/9] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

Refactor the userdiff_find_by_namelen() function so that a new
for_each_userdiff_driver() API function does most of the work.

This will be useful for the same reason we've got other for_each_*()
API functions as part of various APIs, and will be used in a follow-up
commit.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 70 ++++++++++++++++++++++++++++++++++++++++++++----------
 userdiff.h | 13 ++++++++++
 2 files changed, 71 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 978ae64155..a667ccaa8c 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -259,20 +259,33 @@ static struct userdiff_driver driver_false = {
 	{ NULL, 0 }
 };
 
-static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+struct find_by_namelen_data {
+	const char *name;
+	size_t len;
+	struct userdiff_driver *driver;
+};
+
+static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
+				       enum userdiff_driver_type type, void *priv)
 {
-	int i;
-	for (i = 0; i < ndrivers; i++) {
-		struct userdiff_driver *drv = drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
-	}
-	for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
-		struct userdiff_driver *drv = builtin_drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
+	struct find_by_namelen_data *cb_data = priv;
+
+	if (!strncmp(driver->name, cb_data->name, cb_data->len) &&
+	    !driver->name[cb_data->len]) {
+		cb_data->driver = driver;
+		return 1; /* tell the caller to stop iterating */
 	}
-	return NULL;
+	return 0;
+}
+
+static struct userdiff_driver *userdiff_find_by_namelen(const char *name, size_t len)
+{
+	struct find_by_namelen_data udcbdata = {
+		.name = name,
+		.len = len,
+	};
+	for_each_userdiff_driver(userdiff_find_by_namelen_cb, &udcbdata);
+	return udcbdata.driver;
 }
 
 static int parse_funcname(struct userdiff_funcname *f, const char *k,
@@ -379,3 +392,36 @@ struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 
 	return driver;
 }
+
+static int for_each_userdiff_driver_list(each_userdiff_driver_fn fn,
+					 enum userdiff_driver_type type, void *cb_data,
+					 struct userdiff_driver *drv,
+					 int drv_size)
+{
+	int i;
+	int ret;
+	for (i = 0; i < drv_size; i++) {
+		struct userdiff_driver *item = drv + i;
+		if ((ret = fn(item, type, cb_data)))
+			return ret;
+	}
+	return 0;
+}
+
+int for_each_userdiff_driver(each_userdiff_driver_fn fn, void *cb_data)
+{
+	int ret;
+
+	ret = for_each_userdiff_driver_list(fn, USERDIFF_DRIVER_TYPE_CUSTOM,
+					    cb_data, drivers, ndrivers);
+	if (ret)
+		return ret;
+
+	ret = for_each_userdiff_driver_list(fn, USERDIFF_DRIVER_TYPE_BUILTIN,
+					    cb_data, builtin_drivers,
+					    ARRAY_SIZE(builtin_drivers));
+	if (ret)
+		return ret;
+
+	return 0;
+}
diff --git a/userdiff.h b/userdiff.h
index 203057e13e..aee91bc77e 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -21,6 +21,12 @@ struct userdiff_driver {
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
 };
+enum userdiff_driver_type {
+	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
+	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<1,
+};
+typedef int (*each_userdiff_driver_fn)(struct userdiff_driver *,
+				       enum userdiff_driver_type, void *);
 
 int userdiff_config(const char *k, const char *v);
 struct userdiff_driver *userdiff_find_by_name(const char *name);
@@ -34,4 +40,11 @@ struct userdiff_driver *userdiff_find_by_path(struct index_state *istate,
 struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 					      struct userdiff_driver *driver);
 
+/*
+ * Iterate over all userdiff drivers. The userdiff_driver_type
+ * argument to each_userdiff_driver_fn indicates their type. Return
+ * non-zero to exit early from the loop.
+ */
+int for_each_userdiff_driver(each_userdiff_driver_fn, void *);
+
 #endif /* USERDIFF */
-- 
2.31.1.527.g9b8f7de2547

Re: [PATCH v5 4/9] userdiff: add and use for_each_userdiff_driver()

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> +static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
> +				       enum userdiff_driver_type type, void *priv)
>  {
> -	int i;
> -	for (i = 0; i < ndrivers; i++) {
> -		struct userdiff_driver *drv = drivers + i;
> -		if (!strncmp(drv->name, k, len) && !drv->name[len])
> -			return drv;
> -	}
> -	for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
> -		struct userdiff_driver *drv = builtin_drivers + i;
> -		if (!strncmp(drv->name, k, len) && !drv->name[len])
> -			return drv;
> +	struct find_by_namelen_data *cb_data = priv;
> +
> +	if (!strncmp(driver->name, cb_data->name, cb_data->len) &&
> +	    !driver->name[cb_data->len]) {
> +		cb_data->driver = driver;
> +		return 1; /* tell the caller to stop iterating */
>  	}
> -	return NULL;
> +	return 0;
> +}
> ...
> +enum userdiff_driver_type {
> +	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
> +	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<1,
> +};
> +typedef int (*each_userdiff_driver_fn)(struct userdiff_driver *,
> +				       enum userdiff_driver_type, void *);

Makes sense.

[PATCH v5 5/9] userdiff tests: explicitly test "default" pattern

[Ævar Arnfjörð Bjarmason]

Since 122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05)
the internals of the userdiff.c code have understood a "default" name,
which is invoked as userdiff_find_by_name("default") and present in
the "builtin_drivers" struct. Let's test for this special case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 9675bc17db..cefe329aea 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -49,6 +49,7 @@ diffpatterns="
 	ruby
 	rust
 	tex
+	default
 	custom1
 	custom2
 	custom3
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 6/9] userdiff tests: list builtin drivers via test-tool

[Ævar Arnfjörð Bjarmason]

Change the userdiff test to list the builtin drivers via the
test-tool, using the new for_each_userdiff_driver() API function.

This gets rid of the need to modify this part of the test every time a
new pattern is added, see 2ff6c34612 (userdiff: support Bash,
2020-10-22) and 09dad9256a (userdiff: support Markdown, 2020-05-02)
for two recent examples.

I only need the "list-builtin-drivers "argument here, but let's add
"list-custom-drivers" and "list-drivers" too, just because it's easy.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile                 |  1 +
 t/helper/test-tool.c     |  1 +
 t/helper/test-tool.h     |  1 +
 t/helper/test-userdiff.c | 46 ++++++++++++++++++++++++++++++++++++++++
 t/t4018-diff-funcname.sh | 46 +++++++++++++++++-----------------------
 5 files changed, 68 insertions(+), 27 deletions(-)
 create mode 100644 t/helper/test-userdiff.c

diff --git a/Makefile b/Makefile
index a6a73c5741..d2ba9bb402 100644
--- a/Makefile
+++ b/Makefile
@@ -752,6 +752,7 @@ TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o
 TEST_BUILTINS_OBJS += test-subprocess.o
 TEST_BUILTINS_OBJS += test-trace2.o
 TEST_BUILTINS_OBJS += test-urlmatch-normalization.o
+TEST_BUILTINS_OBJS += test-userdiff.o
 TEST_BUILTINS_OBJS += test-wildmatch.o
 TEST_BUILTINS_OBJS += test-windows-named-pipe.o
 TEST_BUILTINS_OBJS += test-write-cache.o
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index 287aa60023..bf13b1f8c1 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -72,6 +72,7 @@ static struct test_cmd cmds[] = {
 	{ "submodule-nested-repo-config", cmd__submodule_nested_repo_config },
 	{ "subprocess", cmd__subprocess },
 	{ "trace2", cmd__trace2 },
+	{ "userdiff", cmd__userdiff },
 	{ "urlmatch-normalization", cmd__urlmatch_normalization },
 	{ "xml-encode", cmd__xml_encode },
 	{ "wildmatch", cmd__wildmatch },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index 9ea4b31011..698f95bf46 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -62,6 +62,7 @@ int cmd__submodule_config(int argc, const char **argv);
 int cmd__submodule_nested_repo_config(int argc, const char **argv);
 int cmd__subprocess(int argc, const char **argv);
 int cmd__trace2(int argc, const char **argv);
+int cmd__userdiff(int argc, const char **argv);
 int cmd__urlmatch_normalization(int argc, const char **argv);
 int cmd__xml_encode(int argc, const char **argv);
 int cmd__wildmatch(int argc, const char **argv);
diff --git a/t/helper/test-userdiff.c b/t/helper/test-userdiff.c
new file mode 100644
index 0000000000..f013f8a31e
--- /dev/null
+++ b/t/helper/test-userdiff.c
@@ -0,0 +1,46 @@
+#include "test-tool.h"
+#include "cache.h"
+#include "userdiff.h"
+#include "config.h"
+
+static int driver_cb(struct userdiff_driver *driver,
+		     enum userdiff_driver_type type, void *priv)
+{
+	enum userdiff_driver_type *want_type = priv;
+	if (type & *want_type && driver->funcname.pattern)
+		puts(driver->name);
+	return 0;
+}
+
+static int cmd__userdiff_config(const char *var, const char *value, void *cb)
+{
+	if (userdiff_config(var, value) < 0)
+		return -1;
+	return 0;
+}
+
+int cmd__userdiff(int argc, const char **argv)
+{
+	enum userdiff_driver_type want = 0;
+	if (argc != 2)
+		return 1;
+
+	if (!strcmp(argv[1], "list-drivers"))
+		want = (USERDIFF_DRIVER_TYPE_BUILTIN |
+			USERDIFF_DRIVER_TYPE_CUSTOM);
+	else if (!strcmp(argv[1], "list-builtin-drivers"))
+		want = USERDIFF_DRIVER_TYPE_BUILTIN;
+	else if (!strcmp(argv[1], "list-custom-drivers"))
+		want = USERDIFF_DRIVER_TYPE_CUSTOM;
+	else
+		return error("unknown argument %s", argv[1]);
+
+	if (want & USERDIFF_DRIVER_TYPE_CUSTOM) {
+		setup_git_directory();
+		git_config(cmd__userdiff_config, NULL);
+	}
+
+	for_each_userdiff_driver(driver_cb, &want);
+
+	return 0;
+}
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index cefe329aea..409372f3a4 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -25,34 +25,26 @@ test_expect_success 'setup' '
 	echo B >B.java
 '
 
+test_expect_success 'setup: test-tool userdiff' '
+	# Make sure additions to builtin_drivers are sorted
+	test_when_finished "rm builtin-drivers.sorted" &&
+	test-tool userdiff list-builtin-drivers >builtin-drivers &&
+	test_file_not_empty builtin-drivers &&
+	sort <builtin-drivers >builtin-drivers.sorted &&
+	test_cmp builtin-drivers.sorted builtin-drivers &&
+
+	# Ditto, but "custom" requires the .git directory and config
+	# to be setup and read.
+	test_when_finished "rm custom-drivers.sorted" &&
+	test-tool userdiff list-custom-drivers >custom-drivers &&
+	test_file_not_empty custom-drivers &&
+	sort <custom-drivers >custom-drivers.sorted &&
+	test_cmp custom-drivers.sorted custom-drivers
+'
+
 diffpatterns="
-	ada
-	bash
-	bibtex
-	cpp
-	csharp
-	css
-	dts
-	elixir
-	fortran
-	fountain
-	golang
-	html
-	java
-	markdown
-	matlab
-	objc
-	pascal
-	perl
-	php
-	python
-	ruby
-	rust
-	tex
-	default
-	custom1
-	custom2
-	custom3
+	$(cat builtin-drivers)
+	$(cat custom-drivers)
 "
 
 for p in $diffpatterns
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 7/9] userdiff: remove support for "broken" tests

[Ævar Arnfjörð Bjarmason]

There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve
Fortran xfuncname regex, 2020-08-12). Let's remove the test support
for them.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 8 +-------
 t/t4018/README           | 3 ---
 2 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 409372f3a4..740696c8f7 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -94,13 +94,7 @@ test_expect_success 'setup hunk header tests' '
 # check each individual file
 for i in $(git ls-files)
 do
-	if grep broken "$i" >/dev/null 2>&1
-	then
-		result=failure
-	else
-		result=success
-	fi
-	test_expect_$result "hunk header: $i" "
+	test_expect_success "hunk header: $i" "
 		git diff -U1 $i >actual &&
 		grep '@@ .* @@.*RIGHT' actual
 	"
diff --git a/t/t4018/README b/t/t4018/README
index 283e01cca1..2d25b2b4fc 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -7,9 +7,6 @@ at least two lines from the line that must appear in the hunk header.
 The text that must appear in the hunk header must contain the word
 "right", but in all upper-case, like in the title above.
 
-To mark a test case that highlights a malfunction, insert the word
-BROKEN in all lower-case somewhere in the file.
-
 This text is a bit twisted and out of order, but it is itself a
 test case for the default hunk header pattern. Know what you are doing
 if you change it.
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 8/9] blame tests: don't rely on t/t4018/ directory

[Ævar Arnfjörð Bjarmason]

Refactor a test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) so that the blame tests don't rely
on stealing the contents of "t/t4018/fortran-external-function".

I have another patch series that'll possibly (or not) refactor that
file, but having this test inter-dependency makes things simple in any
case by making this test more readable.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 29ce89090d..04a2c58594 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -482,12 +482,22 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 test_expect_success 'setup -L :funcname with userdiff driver' '
 	echo "fortran-* diff=fortran" >.gitattributes &&
 	fortran_file=fortran-external-function &&
-	orig_file="$TEST_DIRECTORY/t4018/$fortran_file" &&
-	cp "$orig_file" . &&
+	cat >$fortran_file <<-\EOF &&
+	function RIGHT(a, b) result(c)
+
+	integer, intent(in) :: ChangeMe
+	integer, intent(in) :: b
+	integer, intent(out) :: c
+
+	c = a+b
+
+	end function RIGHT
+	EOF
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
 	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$orig_file" >"$fortran_file" &&
+	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
+	mv "$fortran_file".tmp "$fortran_file" &&
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
 	git commit -m "change fortran file"
-- 
2.31.1.527.g9b8f7de2547

[PATCH v5 9/9] blame tests: simplify userdiff driver test

[Ævar Arnfjörð Bjarmason]

Simplify the test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) to use the --author support recently
added in 999cfc4f45 (test-lib functions: add --author support to
test_commit, 2021-01-12).

We also did not need the full fortran-external-function content. Let's
cut it down to just the important parts.

I'm modifying it to demonstrate that the fortran-specific userdiff
function is in effect by adding "DO NOT MATCH ..." and "AS THE ..."
lines surrounding the "RIGHT" one.

This is to check that we're using the userdiff "fortran" driver, as
opposed to the default driver which would match on those lines as part
of the general heuristic of matching a line that doesn't begin with
whitespace.

The test had also been leaving behind a .gitattributes file for later
tests to possibly trip over, let's clean it up with
"test_when_finished".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 36 +++++++++++++++---------------------
 1 file changed, 15 insertions(+), 21 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 04a2c58594..d3b299e75c 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -479,32 +479,26 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 	check_count -f hello.c -L$n -L^:ma.. F 4 G 1 H 1
 '
 
-test_expect_success 'setup -L :funcname with userdiff driver' '
-	echo "fortran-* diff=fortran" >.gitattributes &&
-	fortran_file=fortran-external-function &&
-	cat >$fortran_file <<-\EOF &&
+test_expect_success 'blame -L :funcname with userdiff driver' '
+	cat >file.template <<-\EOF &&
+	DO NOT MATCH THIS LINE
 	function RIGHT(a, b) result(c)
+	AS THE DEFAULT DRIVER WOULD
 
 	integer, intent(in) :: ChangeMe
-	integer, intent(in) :: b
-	integer, intent(out) :: c
-
-	c = a+b
-
-	end function RIGHT
 	EOF
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
-	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
-	mv "$fortran_file".tmp "$fortran_file" &&
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
-	git commit -m "change fortran file"
-'
 
-test_expect_success 'blame -L :funcname with userdiff driver' '
-	check_count -f fortran-external-function -L:RIGHT A 7 B 1
+	fortran_file=file.f03 &&
+	test_when_finished "rm .gitattributes" &&
+	echo "$fortran_file diff=fortran" >.gitattributes &&
+
+	test_commit --author "A <A@test.git>" \
+		"add" "$fortran_file" \
+		"$(cat file.template)" &&
+	test_commit --author "B <B@test.git>" \
+		"change" "$fortran_file" \
+		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
+	check_count -f "$fortran_file" -L:RIGHT A 3 B 1
 '
 
 test_expect_success 'setup incremental' '
-- 
2.31.1.527.g9b8f7de2547

[PATCH v3 01/35] userdiff: refactor away the parse_bool() function

[Ævar Arnfjörð Bjarmason]

Since 6680a0874f (drop odd return value semantics from
userdiff_config, 2012-02-07) we have not cared about the return values
of parse_tristate() or git_config_bool() v.s. falling through in
userdiff_config(), so let's do so in those cases to make the code
easier to read.

Having a wrapper function for git_config_bool() dates back to
d9bae1a178 (diff: cache textconv output, 2010-04-01) and
122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05), both of
which predated the change in 6680a0874f which made their return values
redundant.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 3f81a2261c5..c147bcbb173 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -275,19 +275,12 @@ static int parse_funcname(struct userdiff_funcname *f, const char *k,
 	return 0;
 }
 
-static int parse_tristate(int *b, const char *k, const char *v)
+static void parse_tristate(int *b, const char *k, const char *v)
 {
 	if (v && !strcasecmp(v, "auto"))
 		*b = -1;
 	else
 		*b = git_config_bool(k, v);
-	return 0;
-}
-
-static int parse_bool(int *b, const char *k, const char *v)
-{
-	*b = git_config_bool(k, v);
-	return 0;
 }
 
 int userdiff_config(const char *k, const char *v)
@@ -312,16 +305,17 @@ int userdiff_config(const char *k, const char *v)
 		return parse_funcname(&drv->funcname, k, v, 0);
 	if (!strcmp(type, "xfuncname"))
 		return parse_funcname(&drv->funcname, k, v, REG_EXTENDED);
-	if (!strcmp(type, "binary"))
-		return parse_tristate(&drv->binary, k, v);
 	if (!strcmp(type, "command"))
 		return git_config_string(&drv->external, k, v);
 	if (!strcmp(type, "textconv"))
 		return git_config_string(&drv->textconv, k, v);
-	if (!strcmp(type, "cachetextconv"))
-		return parse_bool(&drv->textconv_want_cache, k, v);
 	if (!strcmp(type, "wordregex"))
 		return git_config_string(&drv->word_regex, k, v);
+	/* Don't care about the parse errors for these, fallthrough */
+	if (!strcmp(type, "cachetextconv"))
+		drv->textconv_want_cache = git_config_bool(k, v);
+	if (!strcmp(type, "binary"))
+		parse_tristate(&drv->binary, k, v);
 
 	return 0;
 }
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 02/35] userdiff style: re-order drivers in alphabetical order

[Ævar Arnfjörð Bjarmason]

Address some old code smell and move around the built-in userdiff
drivers so they're both in alphabetical order, and now in the same
order they appear in the gitattributes(5) documentation.

The two started drifting in be58e70dba (diff: unify external diff and
funcname parsing code, 2008-10-05), and then even further in
80c49c3de2 (color-words: make regex configurable via attributes,
2009-01-17) when the "cpp" pattern was added.

There are no functional changes here, and as --color-moved will show
only moved existing lines.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 76 +++++++++++++++++++++++++++---------------------------
 1 file changed, 38 insertions(+), 38 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c147bcbb173..c92cbcc0540 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,6 +44,44 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
+PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 "[={}\"]|[^={}\" \t]+"),
+PATTERNS("cpp",
+	 /* Jump targets or access declarations */
+	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
+	 /* functions/methods, variables, and compounds at top level */
+	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
+PATTERNS("csharp",
+	 /* Keywords */
+	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
+	 /* Methods and constructors */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
+	 /* Properties */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
+	 /* Type definitions */
+	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
+	 /* Namespace */
+	 "^[ \t]*(namespace[ \t]+.*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
+IPATTERN("css",
+	 "![:;][[:space:]]*$\n"
+	 "^[:[@.#]?[_a-z0-9].*$",
+	 /* -- */
+	 /*
+	  * This regex comes from W3C CSS specs. Should theoretically also
+	  * allow ISO 10646 characters U+00A0 and higher,
+	  * but they are not handled in this regex.
+	  */
+	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
+	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
+),
 PATTERNS("dts",
 	 "!;\n"
 	 "!=\n"
@@ -191,46 +229,8 @@ PATTERNS("rust",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[0-9][0-9_a-fA-Fiosuxz]*(\\.([0-9]*[eE][+-]?)?[0-9_fF]*)?"
 	 "|[-+*\\/<>%&^|=!:]=|<<=?|>>=?|&&|\\|\\||->|=>|\\.{2}=|\\.{3}|::"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
-	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("tex", "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$",
 	 "\\\\[a-zA-Z@]+|\\\\.|[a-zA-Z0-9\x80-\xff]+"),
-PATTERNS("cpp",
-	 /* Jump targets or access declarations */
-	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
-	 /* functions/methods, variables, and compounds at top level */
-	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
-PATTERNS("csharp",
-	 /* Keywords */
-	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
-	 /* Methods and constructors */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
-	 /* Properties */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
-	 /* Type definitions */
-	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
-	 /* Namespace */
-	 "^[ \t]*(namespace[ \t]+.*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
-IPATTERN("css",
-	 "![:;][[:space:]]*$\n"
-	 "^[:[@.#]?[_a-z0-9].*$",
-	 /* -- */
-	 /*
-	  * This regex comes from W3C CSS specs. Should theoretically also
-	  * allow ISO 10646 characters U+00A0 and higher,
-	  * but they are not handled in this regex.
-	  */
-	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
-	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
-),
 { "default", NULL, -1, { NULL, 0 } },
 };
 #undef PATTERNS
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 03/35] userdiff style: declare patterns with consistent style

[Ævar Arnfjörð Bjarmason]

Change those patterns which were declared with a regex on the same
line as the "PATTERNS()" line to put that regex on the next line, and
add missing "/* -- */" separator comments between the pattern and
word_regex.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c92cbcc0540..c7aaf7094f8 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,7 +44,9 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+PATTERNS("bibtex",
+	 "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 /* -- */
 	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("cpp",
 	 /* Jump targets or access declarations */
@@ -121,7 +123,9 @@ IPATTERN("fortran",
 	  * they would have been matched above as a variable anyway. */
 	 "|[-+]?[0-9.]+([AaIiDdEeFfLlTtXx][Ss]?[-+]?[0-9.]*)?(_[a-zA-Z0-9][a-zA-Z0-9_]*)?"
 	 "|//|\\*\\*|::|[/<>=]="),
-IPATTERN("fountain", "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+IPATTERN("fountain",
+	 "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+	 /* -- */
 	 "[^ \t-]+"),
 PATTERNS("golang",
 	 /* Functions */
@@ -132,7 +136,9 @@ PATTERNS("golang",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.eE]+i?|0[xX]?[0-9a-fA-F]+i?"
 	 "|[-+*/<>%&^|=!:]=|--|\\+\\+|<<=?|>>=?|&\\^=?|&&|\\|\\||<-|\\.{3}"),
-PATTERNS("html", "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+PATTERNS("html",
+	 "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("java",
 	 "!^[ \t]*(catch|do|for|if|instanceof|new|return|switch|throw|while)\n"
@@ -144,6 +150,7 @@ PATTERNS("java",
 	 "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"),
 PATTERNS("markdown",
 	 "^ {0,3}#{1,6}[ \t].*",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("matlab",
 	 /*
@@ -152,6 +159,7 @@ PATTERNS("matlab",
 	  * that is understood by both.
 	  */
 	 "^[[:space:]]*((classdef|function)[[:space:]].*)$|^(%%%?|##)[[:space:]].*$",
+	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*|[-+0-9.e]+|[=~<>]=|\\.[*/\\^']|\\|\\||&&"),
 PATTERNS("objc",
 	 /* Negate C statements that can look like functions */
@@ -212,13 +220,15 @@ PATTERNS("php",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+"
 	 "|[-+*/<>%&^|=!.]=|--|\\+\\+|<<=?|>>=?|===|&&|\\|\\||::|->"),
-PATTERNS("python", "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
+PATTERNS("python",
+	 "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+[jJlL]?|0[xX]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|//=?|<<=?|>>=?|\\*\\*=?"),
 	 /* -- */
-PATTERNS("ruby", "^[ \t]*((class|module|def)[ \t].*)$",
+PATTERNS("ruby",
+	 "^[ \t]*((class|module|def)[ \t].*)$",
 	 /* -- */
 	 "(@|@@|\\$)?[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+|\\?(\\\\C-)?(\\\\M-)?."
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 04/35] userdiff style: normalize pascal regex declaration

[Ævar Arnfjörð Bjarmason]

Declare the pascal pattern consistently with how we declare the
others, not having "\n" on one line by itself, but as part of the
pattern, and when there are alterations have the "|" at the start, not
end of the line.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c7aaf7094f8..10a02d36209 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -175,9 +175,8 @@ PATTERNS("objc",
 	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
 PATTERNS("pascal",
-	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface|"
-		"implementation|initialization|finalization)[ \t]*.*)$"
-	 "\n"
+	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface"
+	 "|implementation|initialization|finalization)[ \t]*.*)$\n"
 	 "^(.*=[ \t]*(class|record).*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 05/35] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

Refactor the userdiff_find_by_namelen() function so that a new
for_each_userdiff_driver() API function does most of the work.

This will be useful for the same reason we've got other for_each_*()
API functions as part of various APIs, and will be used in a follow-up
commit.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 61 +++++++++++++++++++++++++++++++++++++++++++-----------
 userdiff.h | 15 ++++++++++++++
 2 files changed, 64 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 10a02d36209..55f4f769bd3 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -259,20 +259,32 @@ static struct userdiff_driver driver_false = {
 	{ NULL, 0 }
 };
 
-static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+struct for_each_userdiff_driver_cb {
+	const char *k;
+	size_t len;
+	struct userdiff_driver *driver;
+};
+
+static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
+				       enum userdiff_driver_type type, void *priv)
 {
-	int i;
-	for (i = 0; i < ndrivers; i++) {
-		struct userdiff_driver *drv = drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
-	}
-	for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
-		struct userdiff_driver *drv = builtin_drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
+	struct for_each_userdiff_driver_cb *cb_data = priv;
+
+	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
+	    !driver->name[cb_data->len]) {
+		cb_data->driver = driver;
+		return -1; /* found it! */
 	}
-	return NULL;
+	return 0;
+}
+
+static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+{
+	struct for_each_userdiff_driver_cb udcbdata = { .k = k, .len = len, .driver = NULL };
+
+	for_each_userdiff_driver(userdiff_find_by_namelen_cb,
+				 USERDIFF_DRIVER_TYPE_UNSPECIFIED, &udcbdata);
+	return udcbdata.driver;
 }
 
 static int parse_funcname(struct userdiff_funcname *f, const char *k,
@@ -373,3 +385,28 @@ struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 
 	return driver;
 }
+
+int for_each_userdiff_driver(each_userdiff_driver_fn fn,
+			     enum userdiff_driver_type type, void *cb_data)
+{
+	int i, ret;
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {
+
+		for (i = 0; i < ndrivers; i++) {
+			struct userdiff_driver *drv = drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_CUSTOM, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_BUILTIN)) {
+
+		for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
+			struct userdiff_driver *drv = builtin_drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_BUILTIN, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	return 0;
+}
diff --git a/userdiff.h b/userdiff.h
index 203057e13e5..fe14014a775 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -21,6 +21,13 @@ struct userdiff_driver {
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
 };
+enum userdiff_driver_type {
+	USERDIFF_DRIVER_TYPE_UNSPECIFIED = 1<<0,
+	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<1,
+	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<2,
+};
+typedef int (*each_userdiff_driver_fn)(struct userdiff_driver *,
+				       enum userdiff_driver_type, void *);
 
 int userdiff_config(const char *k, const char *v);
 struct userdiff_driver *userdiff_find_by_name(const char *name);
@@ -34,4 +41,12 @@ struct userdiff_driver *userdiff_find_by_path(struct index_state *istate,
 struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 					      struct userdiff_driver *driver);
 
+/*
+ * Iterate over each driver of type userdiff_driver_type, or
+ * USERDIFF_DRIVER_TYPE_UNSPECIFIED for all of them. Return non-zero
+ * to exit from the loop.
+ */
+int for_each_userdiff_driver(each_userdiff_driver_fn,
+			     enum userdiff_driver_type, void *);
+
 #endif /* USERDIFF */
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 06/35] userdiff tests: explicitly test "default" pattern

[Ævar Arnfjörð Bjarmason]

Since 122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05)
the internals of the userdiff.c code have understood a "default" name,
which is invoked as userdiff_find_by_name("default") and present in
the "builtin_drivers" struct. Let's test for this special case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 9675bc17db2..cefe329aea7 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -49,6 +49,7 @@ diffpatterns="
 	ruby
 	rust
 	tex
+	default
 	custom1
 	custom2
 	custom3
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 07/35] userdiff tests: list builtin drivers via test-tool

[Ævar Arnfjörð Bjarmason]

Change the userdiff test to list the builtin drivers via the
test-tool, using the new for_each_userdiff_driver() API function.

This gets rid of the need to modify this part of the test every time a
new pattern is added, see 2ff6c34612 (userdiff: support Bash,
2020-10-22) and 09dad9256a (userdiff: support Markdown, 2020-05-02)
for two recent examples.

I only need the "list-builtin-drivers "argument here, but let's add
"list-custom-drivers" and "list-drivers" too, just because it's easy.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile                 |  1 +
 t/helper/test-tool.c     |  1 +
 t/helper/test-tool.h     |  1 +
 t/helper/test-userdiff.c | 31 +++++++++++++++++++++++++++++++
 t/t4018-diff-funcname.sh | 32 ++++++++------------------------
 5 files changed, 42 insertions(+), 24 deletions(-)
 create mode 100644 t/helper/test-userdiff.c

diff --git a/Makefile b/Makefile
index 5a239cac20e..710a0deaed0 100644
--- a/Makefile
+++ b/Makefile
@@ -741,6 +741,7 @@ TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o
 TEST_BUILTINS_OBJS += test-subprocess.o
 TEST_BUILTINS_OBJS += test-trace2.o
 TEST_BUILTINS_OBJS += test-urlmatch-normalization.o
+TEST_BUILTINS_OBJS += test-userdiff.o
 TEST_BUILTINS_OBJS += test-wildmatch.o
 TEST_BUILTINS_OBJS += test-windows-named-pipe.o
 TEST_BUILTINS_OBJS += test-write-cache.o
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index f97cd9f48a6..dcb05ca6e5e 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -71,6 +71,7 @@ static struct test_cmd cmds[] = {
 	{ "submodule-nested-repo-config", cmd__submodule_nested_repo_config },
 	{ "subprocess", cmd__subprocess },
 	{ "trace2", cmd__trace2 },
+	{ "userdiff", cmd__userdiff },
 	{ "urlmatch-normalization", cmd__urlmatch_normalization },
 	{ "xml-encode", cmd__xml_encode },
 	{ "wildmatch", cmd__wildmatch },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index 28072c0ad5a..589f2e8ac67 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -61,6 +61,7 @@ int cmd__submodule_config(int argc, const char **argv);
 int cmd__submodule_nested_repo_config(int argc, const char **argv);
 int cmd__subprocess(int argc, const char **argv);
 int cmd__trace2(int argc, const char **argv);
+int cmd__userdiff(int argc, const char **argv);
 int cmd__urlmatch_normalization(int argc, const char **argv);
 int cmd__xml_encode(int argc, const char **argv);
 int cmd__wildmatch(int argc, const char **argv);
diff --git a/t/helper/test-userdiff.c b/t/helper/test-userdiff.c
new file mode 100644
index 00000000000..f173a4f18af
--- /dev/null
+++ b/t/helper/test-userdiff.c
@@ -0,0 +1,31 @@
+#include "test-tool.h"
+#include "cache.h"
+#include "userdiff.h"
+
+static int driver_cb(struct userdiff_driver *driver,
+		     enum userdiff_driver_type type, void *priv)
+{
+	if (driver->funcname.pattern)
+		puts(driver->name);
+	return 0;
+}
+
+static int list_what(enum userdiff_driver_type type)
+{
+	return for_each_userdiff_driver(driver_cb, type, NULL);
+}
+
+int cmd__userdiff(int argc, const char **argv)
+{
+	if (argc != 2)
+		return 1;
+
+	if (!strcmp(argv[1], "list-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_UNSPECIFIED);
+	else if (!strcmp(argv[1], "list-builtin-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_BUILTIN);
+	else if (!strcmp(argv[1], "list-custom-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_CUSTOM);
+	else
+		return error("unknown argument %s", argv[1]);
+}
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index cefe329aea7..5bd82e09ab3 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -8,6 +8,13 @@ test_description='Test custom diff function name patterns'
 . ./test-lib.sh
 
 test_expect_success 'setup' '
+	# Make sure additions to builtin_drivers are sorted
+	test_when_finished "rm builtin-drivers.sorted" &&
+	test-tool userdiff list-builtin-drivers >builtin-drivers &&
+	test_file_not_empty builtin-drivers &&
+	sort <builtin-drivers >builtin-drivers.sorted &&
+	test_cmp builtin-drivers.sorted builtin-drivers &&
+
 	# a non-trivial custom pattern
 	git config diff.custom1.funcname "!static
 !String
@@ -26,30 +33,7 @@ test_expect_success 'setup' '
 '
 
 diffpatterns="
-	ada
-	bash
-	bibtex
-	cpp
-	csharp
-	css
-	dts
-	elixir
-	fortran
-	fountain
-	golang
-	html
-	java
-	markdown
-	matlab
-	objc
-	pascal
-	perl
-	php
-	python
-	ruby
-	rust
-	tex
-	default
+	$(cat builtin-drivers)
 	custom1
 	custom2
 	custom3
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 08/35] userdiff: remove support for "broken" tests

[Ævar Arnfjörð Bjarmason]

There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve
Fortran xfuncname regex, 2020-08-12). Let's remove the test support
for them, this is in preparation for a more general refactoring of the
tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 8 +-------
 t/t4018/README           | 3 ---
 2 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 5bd82e09ab3..9aec9f8e6de 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -86,13 +86,7 @@ test_expect_success 'setup hunk header tests' '
 # check each individual file
 for i in $(git ls-files)
 do
-	if grep broken "$i" >/dev/null 2>&1
-	then
-		result=failure
-	else
-		result=success
-	fi
-	test_expect_$result "hunk header: $i" "
+	test_expect_success "hunk header: $i" "
 		git diff -U1 $i >actual &&
 		grep '@@ .* @@.*RIGHT' actual
 	"
diff --git a/t/t4018/README b/t/t4018/README
index 283e01cca1a..2d25b2b4fc9 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -7,9 +7,6 @@ at least two lines from the line that must appear in the hunk header.
 The text that must appear in the hunk header must contain the word
 "right", but in all upper-case, like in the title above.
 
-To mark a test case that highlights a malfunction, insert the word
-BROKEN in all lower-case somewhere in the file.
-
 This text is a bit twisted and out of order, but it is itself a
 test case for the default hunk header pattern. Know what you are doing
 if you change it.
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 09/35] blame tests: don't rely on t/t4018/ directory

[Ævar Arnfjörð Bjarmason]

Refactor a test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) so that the blame tests don't rely
on stealing the contents of "t/t4018/fortran-external-function".

I'm about to change that file in a subsequent commit. Just moving the
relevant test file here inline is the easiest solution, and I think
also the most readable.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 29ce89090d8..04a2c58594c 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -482,12 +482,22 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 test_expect_success 'setup -L :funcname with userdiff driver' '
 	echo "fortran-* diff=fortran" >.gitattributes &&
 	fortran_file=fortran-external-function &&
-	orig_file="$TEST_DIRECTORY/t4018/$fortran_file" &&
-	cp "$orig_file" . &&
+	cat >$fortran_file <<-\EOF &&
+	function RIGHT(a, b) result(c)
+
+	integer, intent(in) :: ChangeMe
+	integer, intent(in) :: b
+	integer, intent(out) :: c
+
+	c = a+b
+
+	end function RIGHT
+	EOF
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
 	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$orig_file" >"$fortran_file" &&
+	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
+	mv "$fortran_file".tmp "$fortran_file" &&
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
 	git commit -m "change fortran file"
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 10/35] blame tests: simplify userdiff driver test

[Ævar Arnfjörð Bjarmason]

Simplify the test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) to use the --author support recently
added in 999cfc4f45 (test-lib functions: add --author support to
test_commit, 2021-01-12).

We also did not need the full fortran-external-function content. Let's
cut it down to just the important parts, and further modify it to
demonstrate that the fortran-specific userdiff function is in effect
by adding "DO NOT MATCH ..." and "AS THE ..." lines surrounding the
"RIGHT" one. This is to check that we're using the userdiff "fortran"
driver, as opposed to the default driver.

The test also left behind a .gitattributes files, let's clean it up
with "test_when_finished".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 36 +++++++++++++++---------------------
 1 file changed, 15 insertions(+), 21 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 04a2c58594c..d3b299e75cb 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -479,32 +479,26 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 	check_count -f hello.c -L$n -L^:ma.. F 4 G 1 H 1
 '
 
-test_expect_success 'setup -L :funcname with userdiff driver' '
-	echo "fortran-* diff=fortran" >.gitattributes &&
-	fortran_file=fortran-external-function &&
-	cat >$fortran_file <<-\EOF &&
+test_expect_success 'blame -L :funcname with userdiff driver' '
+	cat >file.template <<-\EOF &&
+	DO NOT MATCH THIS LINE
 	function RIGHT(a, b) result(c)
+	AS THE DEFAULT DRIVER WOULD
 
 	integer, intent(in) :: ChangeMe
-	integer, intent(in) :: b
-	integer, intent(out) :: c
-
-	c = a+b
-
-	end function RIGHT
 	EOF
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
-	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
-	mv "$fortran_file".tmp "$fortran_file" &&
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
-	git commit -m "change fortran file"
-'
 
-test_expect_success 'blame -L :funcname with userdiff driver' '
-	check_count -f fortran-external-function -L:RIGHT A 7 B 1
+	fortran_file=file.f03 &&
+	test_when_finished "rm .gitattributes" &&
+	echo "$fortran_file diff=fortran" >.gitattributes &&
+
+	test_commit --author "A <A@test.git>" \
+		"add" "$fortran_file" \
+		"$(cat file.template)" &&
+	test_commit --author "B <B@test.git>" \
+		"change" "$fortran_file" \
+		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
+	check_count -f "$fortran_file" -L:RIGHT A 3 B 1
 '
 
 test_expect_success 'setup incremental' '
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 11/35] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

Fix a regression in the test framework for userdiff added in
bfa7d01413 (t4018: an infrastructure to test hunk headers,
2014-03-21).

The testing infrastructure added in that change went overboard with
simplifying the tests, to the point where we lost test coverage.

Before that we'd been able to test the full context line, or ever
since the feature was originally added in f258475a6e (Per-path
attribute based hunk header selection., 2007-07-06).

After bfa7d01413 all we cared about was whether "RIGHT" appeared on
the line. We thus lost the information about whether or not "RIGHT"
was extracted from the line for the hunk header, or the line appeared
in full (or other subset of the line).

Let's bring back coverage for that by extending the custom test syntax
here to understand "t4018" control lines, we grep those out before
doing the add/diff, and understand "t4018 header: " lines to declare
the full context line we're expecting.

Now when we have failures we'll benefit from the full test_cmp output,
instead of just getting a non-zero exit code from "grep".

The "sed -n -e" here was originally a single 's/^.*@@\( \|$\)//p'
pattern, but the '\( \|$\)' part had portability issues on OSX and
AIX.

The one-off addition of headers here to the test files was done by
instrumenting the test itself:

    grep '@@ .* @@.*RIGHT' actual >header.right &&
    sed 's/^@@.*@@ //' header.right >$i.header.small &&
    mv \"$TEST_DIRECTORY\"/t4018/$i tmp &&
    printf 't4018 header: ' >\"$TEST_DIRECTORY\"/t4018/$i &&
    cat $i.header.small >> \"$TEST_DIRECTORY\"/t4018/$i &&
    cat tmp >>\"$TEST_DIRECTORY\"/t4018/$i

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh                   | 27 +++++++++++++---------
 t/t4018/README                             | 25 ++++++++++++--------
 t/t4018/bash-arithmetic-function           |  1 +
 t/t4018/bash-bashism-style-compact         |  1 +
 t/t4018/bash-bashism-style-function        |  1 +
 t/t4018/bash-bashism-style-whitespace      |  1 +
 t/t4018/bash-conditional-function          |  1 +
 t/t4018/bash-missing-parentheses           |  1 +
 t/t4018/bash-mixed-style-compact           |  1 +
 t/t4018/bash-mixed-style-function          |  1 +
 t/t4018/bash-nested-functions              |  1 +
 t/t4018/bash-other-characters              |  1 +
 t/t4018/bash-posix-style-compact           |  1 +
 t/t4018/bash-posix-style-function          |  1 +
 t/t4018/bash-posix-style-whitespace        |  1 +
 t/t4018/bash-subshell-function             |  1 +
 t/t4018/bash-trailing-comment              |  1 +
 t/t4018/cpp-c++-function                   |  1 +
 t/t4018/cpp-class-constructor              |  1 +
 t/t4018/cpp-class-constructor-mem-init     |  1 +
 t/t4018/cpp-class-definition               |  1 +
 t/t4018/cpp-class-definition-derived       |  1 +
 t/t4018/cpp-class-destructor               |  1 +
 t/t4018/cpp-function-returning-global-type |  1 +
 t/t4018/cpp-function-returning-nested      |  1 +
 t/t4018/cpp-function-returning-pointer     |  1 +
 t/t4018/cpp-function-returning-reference   |  1 +
 t/t4018/cpp-gnu-style-function             |  1 +
 t/t4018/cpp-namespace-definition           |  1 +
 t/t4018/cpp-operator-definition            |  1 +
 t/t4018/cpp-skip-access-specifiers         |  1 +
 t/t4018/cpp-skip-comment-block             |  1 +
 t/t4018/cpp-skip-labels                    |  1 +
 t/t4018/cpp-struct-definition              |  1 +
 t/t4018/cpp-struct-single-line             |  1 +
 t/t4018/cpp-template-function-definition   |  1 +
 t/t4018/cpp-union-definition               |  1 +
 t/t4018/cpp-void-c-function                |  1 +
 t/t4018/css-attribute-value-selector       |  1 +
 t/t4018/css-block-level-@-statements       |  1 +
 t/t4018/css-brace-in-col-1                 |  1 +
 t/t4018/css-class-selector                 |  1 +
 t/t4018/css-colon-eol                      |  1 +
 t/t4018/css-colon-selector                 |  1 +
 t/t4018/css-common                         |  1 +
 t/t4018/css-id-selector                    |  1 +
 t/t4018/css-long-selector-list             |  1 +
 t/t4018/css-prop-sans-indent               |  1 +
 t/t4018/css-root-selector                  |  1 +
 t/t4018/css-short-selector-list            |  1 +
 t/t4018/css-trailing-space                 |  1 +
 t/t4018/custom1-pattern                    |  1 +
 t/t4018/custom2-match-to-end-of-line       |  1 +
 t/t4018/custom3-alternation-in-pattern     |  1 +
 t/t4018/dts-labels                         |  1 +
 t/t4018/dts-node-unitless                  |  1 +
 t/t4018/dts-nodes                          |  1 +
 t/t4018/dts-nodes-boolean-prop             |  1 +
 t/t4018/dts-nodes-comment1                 |  1 +
 t/t4018/dts-nodes-comment2                 |  1 +
 t/t4018/dts-nodes-multiline-prop           |  1 +
 t/t4018/dts-reference                      |  1 +
 t/t4018/dts-root                           |  1 +
 t/t4018/dts-root-comment                   |  1 +
 t/t4018/elixir-do-not-pick-end             |  1 +
 t/t4018/elixir-ex-unit-test                |  1 +
 t/t4018/elixir-function                    |  1 +
 t/t4018/elixir-macro                       |  1 +
 t/t4018/elixir-module                      |  1 +
 t/t4018/elixir-module-func                 |  1 +
 t/t4018/elixir-nested-module               |  1 +
 t/t4018/elixir-private-function            |  1 +
 t/t4018/elixir-protocol                    |  1 +
 t/t4018/elixir-protocol-implementation     |  1 +
 t/t4018/fortran-block-data                 |  1 +
 t/t4018/fortran-comment                    |  1 +
 t/t4018/fortran-comment-keyword            |  1 +
 t/t4018/fortran-comment-legacy             |  1 +
 t/t4018/fortran-comment-legacy-star        |  1 +
 t/t4018/fortran-external-function          |  1 +
 t/t4018/fortran-external-subroutine        |  1 +
 t/t4018/fortran-module                     |  1 +
 t/t4018/fortran-module-procedure           |  1 +
 t/t4018/fortran-program                    |  1 +
 t/t4018/fountain-scene                     |  1 +
 t/t4018/golang-complex-function            |  1 +
 t/t4018/golang-func                        |  1 +
 t/t4018/golang-interface                   |  1 +
 t/t4018/golang-long-func                   |  1 +
 t/t4018/golang-struct                      |  1 +
 t/t4018/java-class-member-function         |  1 +
 t/t4018/markdown-heading-indented          |  1 +
 t/t4018/markdown-heading-non-headings      |  1 +
 t/t4018/matlab-class-definition            |  1 +
 t/t4018/matlab-function                    |  1 +
 t/t4018/matlab-octave-section-1            |  1 +
 t/t4018/matlab-octave-section-2            |  1 +
 t/t4018/matlab-section                     |  1 +
 t/t4018/perl-skip-end-of-heredoc           |  1 +
 t/t4018/perl-skip-forward-decl             |  1 +
 t/t4018/perl-skip-sub-in-pod               |  1 +
 t/t4018/perl-sub-definition                |  1 +
 t/t4018/perl-sub-definition-kr-brace       |  1 +
 t/t4018/php-abstract-class                 |  1 +
 t/t4018/php-abstract-method                |  1 +
 t/t4018/php-class                          |  1 +
 t/t4018/php-final-class                    |  1 +
 t/t4018/php-final-method                   |  1 +
 t/t4018/php-function                       |  1 +
 t/t4018/php-interface                      |  1 +
 t/t4018/php-method                         |  1 +
 t/t4018/php-trait                          |  1 +
 t/t4018/python-async-def                   |  1 +
 t/t4018/python-class                       |  1 +
 t/t4018/python-def                         |  1 +
 t/t4018/python-indented-async-def          |  1 +
 t/t4018/python-indented-class              |  1 +
 t/t4018/python-indented-def                |  1 +
 t/t4018/rust-fn                            |  1 +
 t/t4018/rust-impl                          |  1 +
 t/t4018/rust-macro-rules                   |  1 +
 t/t4018/rust-struct                        |  1 +
 t/t4018/rust-trait                         |  1 +
 123 files changed, 153 insertions(+), 20 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 9aec9f8e6de..15dcbe735ca 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -70,16 +70,20 @@ test_expect_success 'setup hunk header tests' '
 		echo "$i-* diff=$i"
 	done > .gitattributes &&
 
-	# add all test files to the index
-	(
-		cd "$TEST_DIRECTORY"/t4018 &&
-		git --git-dir="$TRASH_DIRECTORY/.git" add .
-	) &&
-
-	# place modified files in the worktree
-	for i in $(git ls-files)
+	cp -R "$TEST_DIRECTORY"/t4018 . &&
+	git init t4018 &&
+	git -C t4018 add . &&
+
+	for i in $(git -C t4018 ls-files)
 	do
-		sed -e "s/ChangeMe/IWasChanged/" <"$TEST_DIRECTORY/t4018/$i" >"$i" || return 1
+		grep -v "^t4018" "t4018/$i" >"t4018/$i.content" &&
+		sed -n -e "s/^t4018 header: //p" <"t4018/$i" >"t4018/$i.header" &&
+		cp "t4018/$i.content" "$i" &&
+
+		# add test file to the index
+		git add "$i" &&
+		# place modified file in the worktree
+		sed -e "s/ChangeMe/IWasChanged/" <"t4018/$i.content" >"$i" || return 1
 	done
 '
 
@@ -87,8 +91,9 @@ test_expect_success 'setup hunk header tests' '
 for i in $(git ls-files)
 do
 	test_expect_success "hunk header: $i" "
-		git diff -U1 $i >actual &&
-		grep '@@ .* @@.*RIGHT' actual
+		git diff -U1 $i >diff &&
+		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
+		test_cmp t4018/$i.header ctx
 	"
 done
 
diff --git a/t/t4018/README b/t/t4018/README
index 2d25b2b4fc9..0a246bbc10e 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -1,15 +1,22 @@
-How to write RIGHT test cases
-=============================
+t4018 header: description of the test.
+How to write test cases
+=======================
+
+Create test cases called "LANG-whatever" in this directory, where
+"LANG" is e.g. the userdiff driver name, where "whatever" is a brief
+description of the test.
 
 Insert the word "ChangeMe" (exactly this form) at a distance of
 at least two lines from the line that must appear in the hunk header.
 
-The text that must appear in the hunk header must contain the word
-"right", but in all upper-case, like in the title above.
+Any line starting with "t4018" is a control line for the test:
 
-This text is a bit twisted and out of order, but it is itself a
-test case for the default hunk header pattern. Know what you are doing
-if you change it.
+ - The "t4018 header:" line above specifies what text must appear in
+   the hunk header. We strip away the starting "@@ [...] @@" part of
+   the line for ease of not having to hardcode the line numbers and
+   offsets.
 
-BTW, this tests that the head line goes to the hunk header, not the line
-of equal signs.
+In many of the test cases the header line includes the token "RIGHT",
+this used to be part of the test syntax, but isn't anymore. Now we
+care about the "t4018 header:" line, not whatever line contains a
+"RIGHT" token.
diff --git a/t/t4018/bash-arithmetic-function b/t/t4018/bash-arithmetic-function
index c0b276cb50f..f5609bf49eb 100644
--- a/t/t4018/bash-arithmetic-function
+++ b/t/t4018/bash-arithmetic-function
@@ -1,3 +1,4 @@
+t4018 header: RIGHT()
 RIGHT() ((
 
     ChangeMe = "$x" + "$y"
diff --git a/t/t4018/bash-bashism-style-compact b/t/t4018/bash-bashism-style-compact
index 1ca3126f611..42611aad166 100644
--- a/t/t4018/bash-bashism-style-compact
+++ b/t/t4018/bash-bashism-style-compact
@@ -1,3 +1,4 @@
+t4018 header: function RIGHT {
 function RIGHT {
     function InvalidSyntax{
         :
diff --git a/t/t4018/bash-bashism-style-function b/t/t4018/bash-bashism-style-function
index f1de4fa831c..823b13a463b 100644
--- a/t/t4018/bash-bashism-style-function
+++ b/t/t4018/bash-bashism-style-function
@@ -1,3 +1,4 @@
+t4018 header: function RIGHT {
 function RIGHT {
     :
     echo 'ChangeMe'
diff --git a/t/t4018/bash-bashism-style-whitespace b/t/t4018/bash-bashism-style-whitespace
index ade85dd3a50..4f1add4a4e5 100644
--- a/t/t4018/bash-bashism-style-whitespace
+++ b/t/t4018/bash-bashism-style-whitespace
@@ -1,3 +1,4 @@
+t4018 header: function 	RIGHT 	( 	) 	{
 	 function 	RIGHT 	( 	) 	{
 
 	    ChangeMe
diff --git a/t/t4018/bash-conditional-function b/t/t4018/bash-conditional-function
index c5949e829ba..1e71a71a320 100644
--- a/t/t4018/bash-conditional-function
+++ b/t/t4018/bash-conditional-function
@@ -1,3 +1,4 @@
+t4018 header: RIGHT()
 RIGHT() [[ \
 
     "$a" > "$ChangeMe"
diff --git a/t/t4018/bash-missing-parentheses b/t/t4018/bash-missing-parentheses
index 8c8a05dd7ab..9233042d91f 100644
--- a/t/t4018/bash-missing-parentheses
+++ b/t/t4018/bash-missing-parentheses
@@ -1,3 +1,4 @@
+t4018 header: function RIGHT {
 function RIGHT {
     functionInvalidSyntax {
         :
diff --git a/t/t4018/bash-mixed-style-compact b/t/t4018/bash-mixed-style-compact
index d9364cba671..0f93c2be55a 100644
--- a/t/t4018/bash-mixed-style-compact
+++ b/t/t4018/bash-mixed-style-compact
@@ -1,3 +1,4 @@
+t4018 header: function RIGHT(){
 function RIGHT(){
     :
     echo 'ChangeMe'
diff --git a/t/t4018/bash-mixed-style-function b/t/t4018/bash-mixed-style-function
index 555f9b24667..b3024cdc6dc 100644
--- a/t/t4018/bash-mixed-style-function
+++ b/t/t4018/bash-mixed-style-function
@@ -1,3 +1,4 @@
+t4018 header: function RIGHT() {
 function RIGHT() {
 
     ChangeMe
diff --git a/t/t4018/bash-nested-functions b/t/t4018/bash-nested-functions
index 2c9237ead42..1eee6b9a830 100644
--- a/t/t4018/bash-nested-functions
+++ b/t/t4018/bash-nested-functions
@@ -1,3 +1,4 @@
+t4018 header: RIGHT()
 outer() {
     RIGHT() {
         :
diff --git a/t/t4018/bash-other-characters b/t/t4018/bash-other-characters
index a3f390d525d..02fa1ec822d 100644
--- a/t/t4018/bash-other-characters
+++ b/t/t4018/bash-other-characters
@@ -1,3 +1,4 @@
+t4018 header: _RIGHT_0n()
 _RIGHT_0n() {
 
     ChangeMe
diff --git a/t/t4018/bash-posix-style-compact b/t/t4018/bash-posix-style-compact
index 045bd2029b7..7ba61dea663 100644
--- a/t/t4018/bash-posix-style-compact
+++ b/t/t4018/bash-posix-style-compact
@@ -1,3 +1,4 @@
+t4018 header: RIGHT()
 RIGHT(){
 
     ChangeMe
diff --git a/t/t4018/bash-posix-style-function b/t/t4018/bash-posix-style-function
index a4d144856e9..8566c3f3838 100644
--- a/t/t4018/bash-posix-style-function
+++ b/t/t4018/bash-posix-style-function
@@ -1,3 +1,4 @@
+t4018 header: RIGHT()
 RIGHT() {
 
     ChangeMe
diff --git a/t/t4018/bash-posix-style-whitespace b/t/t4018/bash-posix-style-whitespace
index 4d984f0aa4d..dcc06da3f67 100644
--- a/t/t4018/bash-posix-style-whitespace
+++ b/t/t4018/bash-posix-style-whitespace
@@ -1,3 +1,4 @@
+t4018 header: RIGHT 	( 	)
 	 RIGHT 	( 	) 	{
 
 	    ChangeMe
diff --git a/t/t4018/bash-subshell-function b/t/t4018/bash-subshell-function
index 80baa09484e..f6b188679a2 100644
--- a/t/t4018/bash-subshell-function
+++ b/t/t4018/bash-subshell-function
@@ -1,3 +1,4 @@
+t4018 header: RIGHT()
 RIGHT() (
 
     ChangeMe=2
diff --git a/t/t4018/bash-trailing-comment b/t/t4018/bash-trailing-comment
index f1edbeda319..16ba9701f3f 100644
--- a/t/t4018/bash-trailing-comment
+++ b/t/t4018/bash-trailing-comment
@@ -1,3 +1,4 @@
+t4018 header: RIGHT()
 RIGHT() { # Comment
 
     ChangeMe
diff --git a/t/t4018/cpp-c++-function b/t/t4018/cpp-c++-function
index 9ee6bbef557..316ad0eb34f 100644
--- a/t/t4018/cpp-c++-function
+++ b/t/t4018/cpp-c++-function
@@ -1,3 +1,4 @@
+t4018 header: Item RIGHT::DoSomething( Args with_spaces )
 Item RIGHT::DoSomething( Args with_spaces )
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-class-constructor b/t/t4018/cpp-class-constructor
index ec4f115c250..38552023f00 100644
--- a/t/t4018/cpp-class-constructor
+++ b/t/t4018/cpp-class-constructor
@@ -1,3 +1,4 @@
+t4018 header: Item::Item(int RIGHT)
 Item::Item(int RIGHT)
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-class-constructor-mem-init b/t/t4018/cpp-class-constructor-mem-init
index 49a69f37e16..dc48c12ac25 100644
--- a/t/t4018/cpp-class-constructor-mem-init
+++ b/t/t4018/cpp-class-constructor-mem-init
@@ -1,3 +1,4 @@
+t4018 header: Item::Item(int RIGHT) :
 Item::Item(int RIGHT) :
 	member(0)
 {
diff --git a/t/t4018/cpp-class-definition b/t/t4018/cpp-class-definition
index 11b61da3b75..1f258c8f980 100644
--- a/t/t4018/cpp-class-definition
+++ b/t/t4018/cpp-class-definition
@@ -1,3 +1,4 @@
+t4018 header: class RIGHT
 class RIGHT
 {
 	int ChangeMe;
diff --git a/t/t4018/cpp-class-definition-derived b/t/t4018/cpp-class-definition-derived
index 3b98cd09ab5..1e5ec3b4837 100644
--- a/t/t4018/cpp-class-definition-derived
+++ b/t/t4018/cpp-class-definition-derived
@@ -1,3 +1,4 @@
+t4018 header: class RIGHT :
 class RIGHT :
 	public Baseclass
 {
diff --git a/t/t4018/cpp-class-destructor b/t/t4018/cpp-class-destructor
index 54876650965..a06a37169ad 100644
--- a/t/t4018/cpp-class-destructor
+++ b/t/t4018/cpp-class-destructor
@@ -1,3 +1,4 @@
+t4018 header: RIGHT::~RIGHT()
 RIGHT::~RIGHT()
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-function-returning-global-type b/t/t4018/cpp-function-returning-global-type
index 1084d5990ef..9bb8f57474f 100644
--- a/t/t4018/cpp-function-returning-global-type
+++ b/t/t4018/cpp-function-returning-global-type
@@ -1,3 +1,4 @@
+t4018 header: ::Item get::it::RIGHT()
 ::Item get::it::RIGHT()
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-function-returning-nested b/t/t4018/cpp-function-returning-nested
index d9750aa61a5..3a35a39816a 100644
--- a/t/t4018/cpp-function-returning-nested
+++ b/t/t4018/cpp-function-returning-nested
@@ -1,3 +1,4 @@
+t4018 header: get::Item get::it::RIGHT()
 get::Item get::it::RIGHT()
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-function-returning-pointer b/t/t4018/cpp-function-returning-pointer
index ef15657ea8f..9890c5488db 100644
--- a/t/t4018/cpp-function-returning-pointer
+++ b/t/t4018/cpp-function-returning-pointer
@@ -1,3 +1,4 @@
+t4018 header: const char *get_it_RIGHT(char *ptr)
 const char *get_it_RIGHT(char *ptr)
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-function-returning-reference b/t/t4018/cpp-function-returning-reference
index 01b051df701..7147ab9a74d 100644
--- a/t/t4018/cpp-function-returning-reference
+++ b/t/t4018/cpp-function-returning-reference
@@ -1,3 +1,4 @@
+t4018 header: string& get::it::RIGHT(char *ptr)
 string& get::it::RIGHT(char *ptr)
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-gnu-style-function b/t/t4018/cpp-gnu-style-function
index 08c7c7565ae..91e243f3869 100644
--- a/t/t4018/cpp-gnu-style-function
+++ b/t/t4018/cpp-gnu-style-function
@@ -1,3 +1,4 @@
+t4018 header: RIGHT(int arg)
 const char *
 RIGHT(int arg)
 {
diff --git a/t/t4018/cpp-namespace-definition b/t/t4018/cpp-namespace-definition
index 6749980241c..bd32988b3f9 100644
--- a/t/t4018/cpp-namespace-definition
+++ b/t/t4018/cpp-namespace-definition
@@ -1,3 +1,4 @@
+t4018 header: namespace RIGHT
 namespace RIGHT
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-operator-definition b/t/t4018/cpp-operator-definition
index 1acd8271592..7e8ba945281 100644
--- a/t/t4018/cpp-operator-definition
+++ b/t/t4018/cpp-operator-definition
@@ -1,3 +1,4 @@
+t4018 header: Value operator+(Value LEFT, Value RIGHT)
 Value operator+(Value LEFT, Value RIGHT)
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-skip-access-specifiers b/t/t4018/cpp-skip-access-specifiers
index 4d4a9dbb9db..d61c86d3703 100644
--- a/t/t4018/cpp-skip-access-specifiers
+++ b/t/t4018/cpp-skip-access-specifiers
@@ -1,3 +1,4 @@
+t4018 header: class RIGHT : public Baseclass
 class RIGHT : public Baseclass
 {
 public:
diff --git a/t/t4018/cpp-skip-comment-block b/t/t4018/cpp-skip-comment-block
index 3800b9967a5..ea4d9c25911 100644
--- a/t/t4018/cpp-skip-comment-block
+++ b/t/t4018/cpp-skip-comment-block
@@ -1,3 +1,4 @@
+t4018 header: struct item RIGHT(int i)
 struct item RIGHT(int i)
 // Do not
 // pick up
diff --git a/t/t4018/cpp-skip-labels b/t/t4018/cpp-skip-labels
index b9c10aba225..cf508647281 100644
--- a/t/t4018/cpp-skip-labels
+++ b/t/t4018/cpp-skip-labels
@@ -1,3 +1,4 @@
+t4018 header: void RIGHT (void)
 void RIGHT (void)
 {
 repeat:		// C++ comment
diff --git a/t/t4018/cpp-struct-definition b/t/t4018/cpp-struct-definition
index 521c59fd151..a0f9a16204d 100644
--- a/t/t4018/cpp-struct-definition
+++ b/t/t4018/cpp-struct-definition
@@ -1,3 +1,4 @@
+t4018 header: struct RIGHT {
 struct RIGHT {
 	unsigned
 	/* this bit field looks like a label and should not be picked up */
diff --git a/t/t4018/cpp-struct-single-line b/t/t4018/cpp-struct-single-line
index a0de5fb800f..0d2378a320a 100644
--- a/t/t4018/cpp-struct-single-line
+++ b/t/t4018/cpp-struct-single-line
@@ -1,3 +1,4 @@
+t4018 header: struct RIGHT_iterator_tag {};
 void wrong()
 {
 }
diff --git a/t/t4018/cpp-template-function-definition b/t/t4018/cpp-template-function-definition
index 0cdf5ba5bd4..f1ea1e54f5a 100644
--- a/t/t4018/cpp-template-function-definition
+++ b/t/t4018/cpp-template-function-definition
@@ -1,3 +1,4 @@
+t4018 header: template<class T> int RIGHT(T arg)
 template<class T> int RIGHT(T arg)
 {
 	ChangeMe;
diff --git a/t/t4018/cpp-union-definition b/t/t4018/cpp-union-definition
index 7ec94df6973..6c00ab3b430 100644
--- a/t/t4018/cpp-union-definition
+++ b/t/t4018/cpp-union-definition
@@ -1,3 +1,4 @@
+t4018 header: union RIGHT {
 union RIGHT {
 	double v;
 	int ChangeMe;
diff --git a/t/t4018/cpp-void-c-function b/t/t4018/cpp-void-c-function
index 153081e872c..08765096c96 100644
--- a/t/t4018/cpp-void-c-function
+++ b/t/t4018/cpp-void-c-function
@@ -1,3 +1,4 @@
+t4018 header: void RIGHT (void)
 void RIGHT (void)
 {
 	ChangeMe;
diff --git a/t/t4018/css-attribute-value-selector b/t/t4018/css-attribute-value-selector
index 918256b20c5..8fe8429eb02 100644
--- a/t/t4018/css-attribute-value-selector
+++ b/t/t4018/css-attribute-value-selector
@@ -1,3 +1,4 @@
+t4018 header: [class*="RIGHT"] {
 [class*="RIGHT"] {
     background : #000;
     border : 10px ChangeMe #C6C6C6;
diff --git a/t/t4018/css-block-level-@-statements b/t/t4018/css-block-level-@-statements
index d6755f2f3db..0ea1b9eb37c 100644
--- a/t/t4018/css-block-level-@-statements
+++ b/t/t4018/css-block-level-@-statements
@@ -1,3 +1,4 @@
+t4018 header: @keyframes RIGHT {
 @keyframes RIGHT {
     from {
         background : #000;
diff --git a/t/t4018/css-brace-in-col-1 b/t/t4018/css-brace-in-col-1
index 7831577506a..8bc58ce7098 100644
--- a/t/t4018/css-brace-in-col-1
+++ b/t/t4018/css-brace-in-col-1
@@ -1,3 +1,4 @@
+t4018 header: RIGHT label.control-label
 RIGHT label.control-label
 {
     margin-top: 10px!important;
diff --git a/t/t4018/css-class-selector b/t/t4018/css-class-selector
index f790a0062f4..9b7a2b80e72 100644
--- a/t/t4018/css-class-selector
+++ b/t/t4018/css-class-selector
@@ -1,3 +1,4 @@
+t4018 header: .RIGHT {
 .RIGHT {
     background : #000;
     border : 10px ChangeMe #C6C6C6;
diff --git a/t/t4018/css-colon-eol b/t/t4018/css-colon-eol
index 5a30553d291..665c9f6af5b 100644
--- a/t/t4018/css-colon-eol
+++ b/t/t4018/css-colon-eol
@@ -1,3 +1,4 @@
+t4018 header: RIGHT h1 {
 RIGHT h1 {
 color:
 ChangeMe;
diff --git a/t/t4018/css-colon-selector b/t/t4018/css-colon-selector
index c6d71fb42de..dd0e897155f 100644
--- a/t/t4018/css-colon-selector
+++ b/t/t4018/css-colon-selector
@@ -1,3 +1,4 @@
+t4018 header: RIGHT a:hover {
 RIGHT a:hover {
     margin-top:
     10px!important;
diff --git a/t/t4018/css-common b/t/t4018/css-common
index 84ed754b33b..8351eac792b 100644
--- a/t/t4018/css-common
+++ b/t/t4018/css-common
@@ -1,3 +1,4 @@
+t4018 header: RIGHT label.control-label {
 RIGHT label.control-label {
     margin-top: 10px!important;
     border : 10px ChangeMe #C6C6C6;
diff --git a/t/t4018/css-id-selector b/t/t4018/css-id-selector
index 17c5111052d..2a4351a4f0b 100644
--- a/t/t4018/css-id-selector
+++ b/t/t4018/css-id-selector
@@ -1,3 +1,4 @@
+t4018 header: #RIGHT {
 #RIGHT {
     background : #000;
     border : 10px ChangeMe #C6C6C6;
diff --git a/t/t4018/css-long-selector-list b/t/t4018/css-long-selector-list
index 7ccd25d9ed6..e5fba459290 100644
--- a/t/t4018/css-long-selector-list
+++ b/t/t4018/css-long-selector-list
@@ -1,3 +1,4 @@
+t4018 header: div ul#RIGHT {
 p.header,
 label.control-label,
 div ul#RIGHT {
diff --git a/t/t4018/css-prop-sans-indent b/t/t4018/css-prop-sans-indent
index a9e3c86b3c9..8ca742c91f8 100644
--- a/t/t4018/css-prop-sans-indent
+++ b/t/t4018/css-prop-sans-indent
@@ -1,3 +1,4 @@
+t4018 header: RIGHT, label.control-label {
 RIGHT, label.control-label {
 margin-top: 10px!important;
 padding: 0;
diff --git a/t/t4018/css-root-selector b/t/t4018/css-root-selector
index 22b958e3694..327d7630649 100644
--- a/t/t4018/css-root-selector
+++ b/t/t4018/css-root-selector
@@ -1,3 +1,4 @@
+t4018 header: :RIGHT {
 :RIGHT {
     background : #000;
     border : 10px ChangeMe #C6C6C6;
diff --git a/t/t4018/css-short-selector-list b/t/t4018/css-short-selector-list
index 6a0bdee336b..8a8937a5bd8 100644
--- a/t/t4018/css-short-selector-list
+++ b/t/t4018/css-short-selector-list
@@ -1,3 +1,4 @@
+t4018 header: label.control, div ul#RIGHT {
 label.control, div ul#RIGHT {
     margin-top: 10px!important;
     border : 10px ChangeMe #C6C6C6;
diff --git a/t/t4018/css-trailing-space b/t/t4018/css-trailing-space
index 32b5606c70f..68956baab70 100644
--- a/t/t4018/css-trailing-space
+++ b/t/t4018/css-trailing-space
@@ -1,3 +1,4 @@
+t4018 header: RIGHT label.control-label {
 RIGHT label.control-label {
     margin:10px;   
     padding:10px;
diff --git a/t/t4018/custom1-pattern b/t/t4018/custom1-pattern
index e8fd59f884d..37a3422384b 100644
--- a/t/t4018/custom1-pattern
+++ b/t/t4018/custom1-pattern
@@ -1,3 +1,4 @@
+t4018 header: int special, RIGHT;
 public class Beer
 {
 	int special, RIGHT;
diff --git a/t/t4018/custom2-match-to-end-of-line b/t/t4018/custom2-match-to-end-of-line
index f88ac318b79..4800bb1c568 100644
--- a/t/t4018/custom2-match-to-end-of-line
+++ b/t/t4018/custom2-match-to-end-of-line
@@ -1,3 +1,4 @@
+t4018 header: RIGHT_Beer
 public class RIGHT_Beer
 {
 	int special;
diff --git a/t/t4018/custom3-alternation-in-pattern b/t/t4018/custom3-alternation-in-pattern
index 5f3769c64fc..bf7df3d9a73 100644
--- a/t/t4018/custom3-alternation-in-pattern
+++ b/t/t4018/custom3-alternation-in-pattern
@@ -1,3 +1,4 @@
+t4018 header: public static void main(String RIGHT[])
 public class Beer
 {
 	int special;
diff --git a/t/t4018/dts-labels b/t/t4018/dts-labels
index b21ef8737bb..6b907dcdedf 100644
--- a/t/t4018/dts-labels
+++ b/t/t4018/dts-labels
@@ -1,3 +1,4 @@
+t4018 header: label2: RIGHT {
 / {
 	label_1: node1@ff00 {
 		label2: RIGHT {
diff --git a/t/t4018/dts-node-unitless b/t/t4018/dts-node-unitless
index c5287d91416..990730ef6f4 100644
--- a/t/t4018/dts-node-unitless
+++ b/t/t4018/dts-node-unitless
@@ -1,3 +1,4 @@
+t4018 header: RIGHT {
 / {
 	label_1: node1 {
 		RIGHT {
diff --git a/t/t4018/dts-nodes b/t/t4018/dts-nodes
index 5a4334bb164..4bcebc85e60 100644
--- a/t/t4018/dts-nodes
+++ b/t/t4018/dts-nodes
@@ -1,3 +1,4 @@
+t4018 header: RIGHT@deadf00,4000 {
 / {
 	label_1: node1@ff00 {
 		RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes-boolean-prop b/t/t4018/dts-nodes-boolean-prop
index afc6b5b404e..404cab6f0e1 100644
--- a/t/t4018/dts-nodes-boolean-prop
+++ b/t/t4018/dts-nodes-boolean-prop
@@ -1,3 +1,4 @@
+t4018 header: RIGHT@deadf00,4000 {
 / {
 	label_1: node1@ff00 {
 		RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes-comment1 b/t/t4018/dts-nodes-comment1
index 559dfce9b30..2e5cea26818 100644
--- a/t/t4018/dts-nodes-comment1
+++ b/t/t4018/dts-nodes-comment1
@@ -1,3 +1,4 @@
+t4018 header: RIGHT@deadf00,4000 /* &a comment */ {
 / {
 	label_1: node1@ff00 {
 		RIGHT@deadf00,4000 /* &a comment */ {
diff --git a/t/t4018/dts-nodes-comment2 b/t/t4018/dts-nodes-comment2
index 27e9718b31c..f6cc3177958 100644
--- a/t/t4018/dts-nodes-comment2
+++ b/t/t4018/dts-nodes-comment2
@@ -1,3 +1,4 @@
+t4018 header: RIGHT@deadf00,4000 { /* a trailing comment */
 / {
 	label_1: node1@ff00 {
 		RIGHT@deadf00,4000 { /* a trailing comment */ 
diff --git a/t/t4018/dts-nodes-multiline-prop b/t/t4018/dts-nodes-multiline-prop
index 072d58b69dc..ac07b404975 100644
--- a/t/t4018/dts-nodes-multiline-prop
+++ b/t/t4018/dts-nodes-multiline-prop
@@ -1,3 +1,4 @@
+t4018 header: RIGHT@deadf00,4000 {
 / {
 	label_1: node1@ff00 {
 		RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-reference b/t/t4018/dts-reference
index 8f0c87d8637..bf141229e93 100644
--- a/t/t4018/dts-reference
+++ b/t/t4018/dts-reference
@@ -1,3 +1,4 @@
+t4018 header: &RIGHT {
 &label_1 {
 	TEST = <455>;
 };
diff --git a/t/t4018/dts-root b/t/t4018/dts-root
index 4353b8220c9..4105b0bb48a 100644
--- a/t/t4018/dts-root
+++ b/t/t4018/dts-root
@@ -1,3 +1,4 @@
+t4018 header: / { RIGHT /* Technically just supposed to be a slash and brace */
 / { RIGHT /* Technically just supposed to be a slash and brace */
 	#size-cells = <1>;
 
diff --git a/t/t4018/dts-root-comment b/t/t4018/dts-root-comment
index 333a625c700..cdf2cee98db 100644
--- a/t/t4018/dts-root-comment
+++ b/t/t4018/dts-root-comment
@@ -1,3 +1,4 @@
+t4018 header: / { RIGHT /* Technically just supposed to be a slash and brace */
 / { RIGHT /* Technically just supposed to be a slash and brace */
 	#size-cells = <1>;
 
diff --git a/t/t4018/elixir-do-not-pick-end b/t/t4018/elixir-do-not-pick-end
index fae08ba7e8c..cf861f81e9a 100644
--- a/t/t4018/elixir-do-not-pick-end
+++ b/t/t4018/elixir-do-not-pick-end
@@ -1,3 +1,4 @@
+t4018 header: defmodule RIGHT do
 defmodule RIGHT do
 end
 #
diff --git a/t/t4018/elixir-ex-unit-test b/t/t4018/elixir-ex-unit-test
index 0560a2b6971..469f1a03b71 100644
--- a/t/t4018/elixir-ex-unit-test
+++ b/t/t4018/elixir-ex-unit-test
@@ -1,3 +1,4 @@
+t4018 header: test "RIGHT" do
 defmodule Test do
   test "RIGHT" do
     assert true == true
diff --git a/t/t4018/elixir-function b/t/t4018/elixir-function
index d452f495a7e..139a4e1da8e 100644
--- a/t/t4018/elixir-function
+++ b/t/t4018/elixir-function
@@ -1,3 +1,4 @@
+t4018 header: def function(RIGHT, arg) do
 def function(RIGHT, arg) do
   # comment
   # comment
diff --git a/t/t4018/elixir-macro b/t/t4018/elixir-macro
index 4f925e9ad46..0b0f7d721bb 100644
--- a/t/t4018/elixir-macro
+++ b/t/t4018/elixir-macro
@@ -1,3 +1,4 @@
+t4018 header: defmacro foo(RIGHT) do
 defmacro foo(RIGHT) do
   # Code
   # Code
diff --git a/t/t4018/elixir-module b/t/t4018/elixir-module
index 91a4e7aa200..a4773f4abae 100644
--- a/t/t4018/elixir-module
+++ b/t/t4018/elixir-module
@@ -1,3 +1,4 @@
+t4018 header: defmodule RIGHT do
 defmodule RIGHT do
   @moduledoc """
   Foo bar
diff --git a/t/t4018/elixir-module-func b/t/t4018/elixir-module-func
index c9910d06751..138493829b6 100644
--- a/t/t4018/elixir-module-func
+++ b/t/t4018/elixir-module-func
@@ -1,3 +1,4 @@
+t4018 header: def fun(RIGHT) do
 defmodule Foo do
   def fun(RIGHT) do
      # Code
diff --git a/t/t4018/elixir-nested-module b/t/t4018/elixir-nested-module
index 771ebc5c42a..296990dd2db 100644
--- a/t/t4018/elixir-nested-module
+++ b/t/t4018/elixir-nested-module
@@ -1,3 +1,4 @@
+t4018 header: defmodule MyApp.RIGHT do
 defmodule MyApp.RIGHT do
   @moduledoc """
   Foo bar
diff --git a/t/t4018/elixir-private-function b/t/t4018/elixir-private-function
index 1aabe33b7a9..94428e775b5 100644
--- a/t/t4018/elixir-private-function
+++ b/t/t4018/elixir-private-function
@@ -1,3 +1,4 @@
+t4018 header: defp function(RIGHT, arg) do
 defp function(RIGHT, arg) do
   # comment
   # comment
diff --git a/t/t4018/elixir-protocol b/t/t4018/elixir-protocol
index 7d9173691e3..1f1dc8cb199 100644
--- a/t/t4018/elixir-protocol
+++ b/t/t4018/elixir-protocol
@@ -1,3 +1,4 @@
+t4018 header: defprotocol RIGHT do
 defprotocol RIGHT do
   @doc """
   Calculates the size (and not the length!) of a data structure
diff --git a/t/t4018/elixir-protocol-implementation b/t/t4018/elixir-protocol-implementation
index f9234bbfc48..973226206d7 100644
--- a/t/t4018/elixir-protocol-implementation
+++ b/t/t4018/elixir-protocol-implementation
@@ -1,3 +1,4 @@
+t4018 header: defimpl RIGHT do
 defimpl RIGHT do
   # Docs
   # Docs
diff --git a/t/t4018/fortran-block-data b/t/t4018/fortran-block-data
index 63d4e21d0ad..eb0a77afade 100644
--- a/t/t4018/fortran-block-data
+++ b/t/t4018/fortran-block-data
@@ -1,3 +1,4 @@
+t4018 header: BLOCK DATA RIGHT
        BLOCK DATA RIGHT
        
        COMMON /B/ C, ChangeMe
diff --git a/t/t4018/fortran-comment b/t/t4018/fortran-comment
index 7b10d176588..e5e82331ebc 100644
--- a/t/t4018/fortran-comment
+++ b/t/t4018/fortran-comment
@@ -1,3 +1,4 @@
+t4018 header: subroutine RIGHT
       module a
 
       contains
diff --git a/t/t4018/fortran-comment-keyword b/t/t4018/fortran-comment-keyword
index e9206a53799..143aff89d73 100644
--- a/t/t4018/fortran-comment-keyword
+++ b/t/t4018/fortran-comment-keyword
@@ -1,3 +1,4 @@
+t4018 header: subroutine RIGHT (funcA, funcB)
       module a
 
       contains
diff --git a/t/t4018/fortran-comment-legacy b/t/t4018/fortran-comment-legacy
index 53cd062c1e5..edbac0b4148 100644
--- a/t/t4018/fortran-comment-legacy
+++ b/t/t4018/fortran-comment-legacy
@@ -1,3 +1,4 @@
+t4018 header: subroutine RIGHT
       module a
 
       contains
diff --git a/t/t4018/fortran-comment-legacy-star b/t/t4018/fortran-comment-legacy-star
index 2cbcdc3d8ab..be918f7b030 100644
--- a/t/t4018/fortran-comment-legacy-star
+++ b/t/t4018/fortran-comment-legacy-star
@@ -1,3 +1,4 @@
+t4018 header: subroutine RIGHT
       module a
 
       contains
diff --git a/t/t4018/fortran-external-function b/t/t4018/fortran-external-function
index 5a2d85d3aa4..ea4d8bcedc4 100644
--- a/t/t4018/fortran-external-function
+++ b/t/t4018/fortran-external-function
@@ -1,3 +1,4 @@
+t4018 header: function RIGHT(a, b) result(c)
 function RIGHT(a, b) result(c)
 
 integer, intent(in) :: ChangeMe
diff --git a/t/t4018/fortran-external-subroutine b/t/t4018/fortran-external-subroutine
index 4ce85fea132..9f662dd26b4 100644
--- a/t/t4018/fortran-external-subroutine
+++ b/t/t4018/fortran-external-subroutine
@@ -1,3 +1,4 @@
+t4018 header: subroutine RIGHT
 subroutine RIGHT
 
 real ChangeMe
diff --git a/t/t4018/fortran-module b/t/t4018/fortran-module
index c4b737dac3f..1882b3cd338 100644
--- a/t/t4018/fortran-module
+++ b/t/t4018/fortran-module
@@ -1,3 +1,4 @@
+t4018 header: module RIGHT
 module RIGHT
 
 use ChangeMe
diff --git a/t/t4018/fortran-module-procedure b/t/t4018/fortran-module-procedure
index 1ce6d854c22..51d368ba9be 100644
--- a/t/t4018/fortran-module-procedure
+++ b/t/t4018/fortran-module-procedure
@@ -1,3 +1,4 @@
+t4018 header: module RIGHT
  module RIGHT
 
    implicit none
diff --git a/t/t4018/fortran-program b/t/t4018/fortran-program
index 4616895e4b5..1c84e1ff0f8 100644
--- a/t/t4018/fortran-program
+++ b/t/t4018/fortran-program
@@ -1,3 +1,4 @@
+t4018 header: program RIGHT
 program RIGHT
 
 call ChangeMe
diff --git a/t/t4018/fountain-scene b/t/t4018/fountain-scene
index 6b3257d6803..2ffcf799087 100644
--- a/t/t4018/fountain-scene
+++ b/t/t4018/fountain-scene
@@ -1,3 +1,4 @@
+t4018 header: EXT. STREET RIGHT OUTSIDE - DAY
 EXT. STREET RIGHT OUTSIDE - DAY
 
 CHARACTER
diff --git a/t/t4018/golang-complex-function b/t/t4018/golang-complex-function
index e057dcefed6..0574ba912e6 100644
--- a/t/t4018/golang-complex-function
+++ b/t/t4018/golang-complex-function
@@ -1,3 +1,4 @@
+t4018 header: func (t *Test) RIGHT(a Type) (Type, error) {
 type Test struct {
 	a Type
 }
diff --git a/t/t4018/golang-func b/t/t4018/golang-func
index 8e9c9ac7c3f..0472cfd9798 100644
--- a/t/t4018/golang-func
+++ b/t/t4018/golang-func
@@ -1,3 +1,4 @@
+t4018 header: func RIGHT() {
 func RIGHT() {
 	a := 5
 	b := ChangeMe
diff --git a/t/t4018/golang-interface b/t/t4018/golang-interface
index 553bedec962..3160a1d4524 100644
--- a/t/t4018/golang-interface
+++ b/t/t4018/golang-interface
@@ -1,3 +1,4 @@
+t4018 header: type RIGHT interface {
 type RIGHT interface {
 	a() Type
 	b() ChangeMe
diff --git a/t/t4018/golang-long-func b/t/t4018/golang-long-func
index ac3a77b5c41..de83aaafca5 100644
--- a/t/t4018/golang-long-func
+++ b/t/t4018/golang-long-func
@@ -1,3 +1,4 @@
+t4018 header: func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
 func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
 	anotherLongVariableName AnotherLongType) {
 	a := 5
diff --git a/t/t4018/golang-struct b/t/t4018/golang-struct
index 5deda77feec..fc8022537b2 100644
--- a/t/t4018/golang-struct
+++ b/t/t4018/golang-struct
@@ -1,3 +1,4 @@
+t4018 header: type RIGHT struct {
 type RIGHT struct {
 	a Type
 	b ChangeMe
diff --git a/t/t4018/java-class-member-function b/t/t4018/java-class-member-function
index 298bc7a71b2..df77bd71f79 100644
--- a/t/t4018/java-class-member-function
+++ b/t/t4018/java-class-member-function
@@ -1,3 +1,4 @@
+t4018 header: public static void main(String RIGHT[])
 public class Beer
 {
 	int special;
diff --git a/t/t4018/markdown-heading-indented b/t/t4018/markdown-heading-indented
index 1991c2bd456..e68801c7c06 100644
--- a/t/t4018/markdown-heading-indented
+++ b/t/t4018/markdown-heading-indented
@@ -1,3 +1,4 @@
+t4018 header:    ### RIGHT
 Indented headings are allowed, as long as the indent is no more than 3 spaces.
 
    ### RIGHT
diff --git a/t/t4018/markdown-heading-non-headings b/t/t4018/markdown-heading-non-headings
index c479c1a3f1e..00d677ebe95 100644
--- a/t/t4018/markdown-heading-non-headings
+++ b/t/t4018/markdown-heading-non-headings
@@ -1,3 +1,4 @@
+t4018 header: # RIGHT
 Headings can be right next to other lines of the file:
 # RIGHT
 Indents of four or more spaces make a code block:
diff --git a/t/t4018/matlab-class-definition b/t/t4018/matlab-class-definition
index 84daedfb4e5..86b48eebe0a 100644
--- a/t/t4018/matlab-class-definition
+++ b/t/t4018/matlab-class-definition
@@ -1,3 +1,4 @@
+t4018 header: classdef RIGHT
 classdef RIGHT
     properties
         ChangeMe
diff --git a/t/t4018/matlab-function b/t/t4018/matlab-function
index 897a9b13ff4..9a93efffb99 100644
--- a/t/t4018/matlab-function
+++ b/t/t4018/matlab-function
@@ -1,3 +1,4 @@
+t4018 header: function y = RIGHT()
 function y = RIGHT()
 x = 5;
 y = ChangeMe + x;
diff --git a/t/t4018/matlab-octave-section-1 b/t/t4018/matlab-octave-section-1
index 3bb6c4670e2..b896cb07ff1 100644
--- a/t/t4018/matlab-octave-section-1
+++ b/t/t4018/matlab-octave-section-1
@@ -1,3 +1,4 @@
+t4018 header: %%% RIGHT section
 %%% RIGHT section
 # this is octave script
 ChangeMe = 1;
diff --git a/t/t4018/matlab-octave-section-2 b/t/t4018/matlab-octave-section-2
index ab2980f7f29..4e75942678f 100644
--- a/t/t4018/matlab-octave-section-2
+++ b/t/t4018/matlab-octave-section-2
@@ -1,3 +1,4 @@
+t4018 header: ## RIGHT section
 ## RIGHT section
 # this is octave script
 ChangeMe = 1;
diff --git a/t/t4018/matlab-section b/t/t4018/matlab-section
index 5ea59a5de00..6b45ae20009 100644
--- a/t/t4018/matlab-section
+++ b/t/t4018/matlab-section
@@ -1,3 +1,4 @@
+t4018 header: %% RIGHT section
 %% RIGHT section
 % this is understood by both matlab and octave
 ChangeMe = 1;
diff --git a/t/t4018/perl-skip-end-of-heredoc b/t/t4018/perl-skip-end-of-heredoc
index c22d39b2567..8f90cca7314 100644
--- a/t/t4018/perl-skip-end-of-heredoc
+++ b/t/t4018/perl-skip-end-of-heredoc
@@ -1,3 +1,4 @@
+t4018 header: sub RIGHTwithheredocument {
 sub RIGHTwithheredocument {
 	print <<"EOF"
 decoy here-doc
diff --git a/t/t4018/perl-skip-forward-decl b/t/t4018/perl-skip-forward-decl
index a98cb8bdad0..ff1f6d14735 100644
--- a/t/t4018/perl-skip-forward-decl
+++ b/t/t4018/perl-skip-forward-decl
@@ -1,3 +1,4 @@
+t4018 header: package RIGHT;
 package RIGHT;
 
 use strict;
diff --git a/t/t4018/perl-skip-sub-in-pod b/t/t4018/perl-skip-sub-in-pod
index e39f02462e2..ff1c65b28fc 100644
--- a/t/t4018/perl-skip-sub-in-pod
+++ b/t/t4018/perl-skip-sub-in-pod
@@ -1,3 +1,4 @@
+t4018 header: =head1 SYNOPSIS_RIGHT
 =head1 NAME
 
 Beer - subroutine to output fragment of a drinking song
diff --git a/t/t4018/perl-sub-definition b/t/t4018/perl-sub-definition
index a507d1f6452..22e16ad5363 100644
--- a/t/t4018/perl-sub-definition
+++ b/t/t4018/perl-sub-definition
@@ -1,3 +1,4 @@
+t4018 header: sub RIGHT {
 sub RIGHT {
 	my ($n) = @_;
 	print "ChangeMe";
diff --git a/t/t4018/perl-sub-definition-kr-brace b/t/t4018/perl-sub-definition-kr-brace
index 330b3df1142..6c94e6a62dd 100644
--- a/t/t4018/perl-sub-definition-kr-brace
+++ b/t/t4018/perl-sub-definition-kr-brace
@@ -1,3 +1,4 @@
+t4018 header: sub RIGHT
 sub RIGHT
 {
 	print "ChangeMe\n";
diff --git a/t/t4018/php-abstract-class b/t/t4018/php-abstract-class
index 5213e124946..fbc97843c6b 100644
--- a/t/t4018/php-abstract-class
+++ b/t/t4018/php-abstract-class
@@ -1,3 +1,4 @@
+t4018 header: abstract class RIGHT
 abstract class RIGHT
 {
     const FOO = 'ChangeMe';
diff --git a/t/t4018/php-abstract-method b/t/t4018/php-abstract-method
index ce215df75a4..22f5120d3b7 100644
--- a/t/t4018/php-abstract-method
+++ b/t/t4018/php-abstract-method
@@ -1,3 +1,4 @@
+t4018 header: abstract public function RIGHT(): ?string
 abstract class Klass
 {
     abstract public function RIGHT(): ?string
diff --git a/t/t4018/php-class b/t/t4018/php-class
index 7785b6303c7..42456cda768 100644
--- a/t/t4018/php-class
+++ b/t/t4018/php-class
@@ -1,3 +1,4 @@
+t4018 header: class RIGHT
 class RIGHT
 {
     const FOO = 'ChangeMe';
diff --git a/t/t4018/php-final-class b/t/t4018/php-final-class
index 69f57105529..ccf2d28a2d8 100644
--- a/t/t4018/php-final-class
+++ b/t/t4018/php-final-class
@@ -1,3 +1,4 @@
+t4018 header: final class RIGHT
 final class RIGHT
 {
     const FOO = 'ChangeMe';
diff --git a/t/t4018/php-final-method b/t/t4018/php-final-method
index 537fb8ad9ae..55107faef98 100644
--- a/t/t4018/php-final-method
+++ b/t/t4018/php-final-method
@@ -1,3 +1,4 @@
+t4018 header: final public function RIGHT(): string
 class Klass
 {
     final public function RIGHT(): string
diff --git a/t/t4018/php-function b/t/t4018/php-function
index 35717c51c3b..f021285e385 100644
--- a/t/t4018/php-function
+++ b/t/t4018/php-function
@@ -1,3 +1,4 @@
+t4018 header: function RIGHT()
 function RIGHT()
 {
     return 'ChangeMe';
diff --git a/t/t4018/php-interface b/t/t4018/php-interface
index 86b49ad5d9e..ef48244eaa8 100644
--- a/t/t4018/php-interface
+++ b/t/t4018/php-interface
@@ -1,3 +1,4 @@
+t4018 header: interface RIGHT
 interface RIGHT
 {
     public function foo($ChangeMe);
diff --git a/t/t4018/php-method b/t/t4018/php-method
index 03af1a6d9d7..cbc171a99b9 100644
--- a/t/t4018/php-method
+++ b/t/t4018/php-method
@@ -1,3 +1,4 @@
+t4018 header: public static function RIGHT()
 class Klass
 {
     public static function RIGHT()
diff --git a/t/t4018/php-trait b/t/t4018/php-trait
index 65b8c82a616..cba65191d39 100644
--- a/t/t4018/php-trait
+++ b/t/t4018/php-trait
@@ -1,3 +1,4 @@
+t4018 header: trait RIGHT
 trait RIGHT
 {
     public function foo($ChangeMe)
diff --git a/t/t4018/python-async-def b/t/t4018/python-async-def
index 87640e03d21..facfaeef373 100644
--- a/t/t4018/python-async-def
+++ b/t/t4018/python-async-def
@@ -1,3 +1,4 @@
+t4018 header: async def RIGHT(pi: int = 3.14):
 async def RIGHT(pi: int = 3.14):
     while True:
         break
diff --git a/t/t4018/python-class b/t/t4018/python-class
index ba9e741430f..87153873b4f 100644
--- a/t/t4018/python-class
+++ b/t/t4018/python-class
@@ -1,3 +1,4 @@
+t4018 header: class RIGHT(int, str):
 class RIGHT(int, str):
     # comment
     # another comment
diff --git a/t/t4018/python-def b/t/t4018/python-def
index e50b31b0ad5..08fb9a6b4ec 100644
--- a/t/t4018/python-def
+++ b/t/t4018/python-def
@@ -1,3 +1,4 @@
+t4018 header: def RIGHT(pi: int = 3.14):
 def RIGHT(pi: int = 3.14):
     while True:
         break
diff --git a/t/t4018/python-indented-async-def b/t/t4018/python-indented-async-def
index f5d03258af4..f604d08028f 100644
--- a/t/t4018/python-indented-async-def
+++ b/t/t4018/python-indented-async-def
@@ -1,3 +1,4 @@
+t4018 header: async def RIGHT(self, x: int):
 class Foo:
     async def RIGHT(self, x: int):
         return [
diff --git a/t/t4018/python-indented-class b/t/t4018/python-indented-class
index 19b4f35c4ca..65c07f74f6f 100644
--- a/t/t4018/python-indented-class
+++ b/t/t4018/python-indented-class
@@ -1,3 +1,4 @@
+t4018 header: class RIGHT:
 if TYPE_CHECKING:
     class RIGHT:
         # comment
diff --git a/t/t4018/python-indented-def b/t/t4018/python-indented-def
index 208fbadd2be..be87764e31b 100644
--- a/t/t4018/python-indented-def
+++ b/t/t4018/python-indented-def
@@ -1,3 +1,4 @@
+t4018 header: def RIGHT(self, x: int):
 class Foo:
     def RIGHT(self, x: int):
         return [
diff --git a/t/t4018/rust-fn b/t/t4018/rust-fn
index cbe02155f11..939b131ea80 100644
--- a/t/t4018/rust-fn
+++ b/t/t4018/rust-fn
@@ -1,3 +1,4 @@
+t4018 header: pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
 pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
     let _ = x;
     // a comment
diff --git a/t/t4018/rust-impl b/t/t4018/rust-impl
index 09df3cd93b2..1f798a43d38 100644
--- a/t/t4018/rust-impl
+++ b/t/t4018/rust-impl
@@ -1,3 +1,4 @@
+t4018 header: impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
 impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
 
     pub fn ChangeMe(&self) -> () {
diff --git a/t/t4018/rust-macro-rules b/t/t4018/rust-macro-rules
index ec610c5b62b..3990daf0e90 100644
--- a/t/t4018/rust-macro-rules
+++ b/t/t4018/rust-macro-rules
@@ -1,3 +1,4 @@
+t4018 header: macro_rules! RIGHT {
 macro_rules! RIGHT {
     () => {
         // a comment
diff --git a/t/t4018/rust-struct b/t/t4018/rust-struct
index 76aff1c0d8e..8c901a437cb 100644
--- a/t/t4018/rust-struct
+++ b/t/t4018/rust-struct
@@ -1,3 +1,4 @@
+t4018 header: pub(super) struct RIGHT<'a> {
 #[derive(Debug)]
 pub(super) struct RIGHT<'a> {
     name: &'a str,
diff --git a/t/t4018/rust-trait b/t/t4018/rust-trait
index ea397f09ed1..4cc9714e8f0 100644
--- a/t/t4018/rust-trait
+++ b/t/t4018/rust-trait
@@ -1,3 +1,4 @@
+t4018 header: unsafe trait RIGHT<T> {
 unsafe trait RIGHT<T> {
     fn len(&self) -> u32;
     fn ChangeMe(&self, n: u32) -> T;
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 11/35] userdiff tests: match full hunk headers

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index 9aec9f8e6de..15dcbe735ca 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -70,16 +70,20 @@ test_expect_success 'setup hunk header tests' '
>  		echo "$i-* diff=$i"
>  	done > .gitattributes &&
>  
> -	# add all test files to the index
> -	(
> -		cd "$TEST_DIRECTORY"/t4018 &&
> -		git --git-dir="$TRASH_DIRECTORY/.git" add .
> -	) &&
> -
> -	# place modified files in the worktree
> -	for i in $(git ls-files)
> +	cp -R "$TEST_DIRECTORY"/t4018 . &&

Is this because otherwise we'll mix leftover cruft in the source
directory?  I guess this is OK as it is just once per t4018 test
run.

> +	git init t4018 &&
> +	git -C t4018 add . &&
> +
> +	for i in $(git -C t4018 ls-files)
>  	do
> -		sed -e "s/ChangeMe/IWasChanged/" <"$TEST_DIRECTORY/t4018/$i" >"$i" || return 1
> +		grep -v "^t4018" "t4018/$i" >"t4018/$i.content" &&
> +		sed -n -e "s/^t4018 header: //p" <"t4018/$i" >"t4018/$i.header" &&

For now this would do, but I am assuming that the "t4018" prefix
(which by the way is a way to make it harder to renumber these
tests) was invented so that we can add more magic than just
"header:" in the futhre (either in this series or later).  I am not
sure if we want to run one invocation of "sed" for each new magic we
invent---perhaps the plan is that we'd keep the framework simple for
now (while we have only one "magic") and then revamp it when we gain
the second one, in which case I am perfectly fine with it.

> +		cp "t4018/$i.content" "$i" &&
> +
> +		# add test file to the index
> +		git add "$i" &&
> +		# place modified file in the worktree
> +		sed -e "s/ChangeMe/IWasChanged/" <"t4018/$i.content" >"$i" || return 1

OK.

>  	done
>  '
>  
> @@ -87,8 +91,9 @@ test_expect_success 'setup hunk header tests' '
>  for i in $(git ls-files)
>  do
>  	test_expect_success "hunk header: $i" "
> -		git diff -U1 $i >actual &&
> -		grep '@@ .* @@.*RIGHT' actual
> +		git diff -U1 $i >diff &&
> +		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&

The original was already loose but this makes it even looser.  Can
we tighten the pattern to strip the line number information from the
hunk header to something more like like this, please?

	/^@@[-+0-9, ]*@@/

Thanks.

[PATCH v3 12/35] userdiff tests: change setup loop to individual test setup

[Ævar Arnfjörð Bjarmason]

Change the recently amended setup loop in "setup hunk header tests" to
instead set up the test data as we test each individual hunk header
test.

This means we can get rid of the "|| return 1" case and the previous
for-loop, and won't spend time on setting up all the data only to
e.g. fail on the 1st test when running under "-i".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 25 ++++++++++++-------------
 1 file changed, 12 insertions(+), 13 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 15dcbe735ca..2365f0e361e 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -72,24 +72,23 @@ test_expect_success 'setup hunk header tests' '
 
 	cp -R "$TEST_DIRECTORY"/t4018 . &&
 	git init t4018 &&
-	git -C t4018 add . &&
+	git -C t4018 add .
+'
 
-	for i in $(git -C t4018 ls-files)
-	do
-		grep -v "^t4018" "t4018/$i" >"t4018/$i.content" &&
-		sed -n -e "s/^t4018 header: //p" <"t4018/$i" >"t4018/$i.header" &&
-		cp "t4018/$i.content" "$i" &&
+# check each individual file
+for i in $(git -C t4018 ls-files)
+do
+	test_expect_success "setup hunk header: $i" "
+		grep -v '^t4018' \"t4018/$i\" >\"t4018/$i.content\" &&
+		sed -n -e 's/^t4018 header: //p' <\"t4018/$i\" >\"t4018/$i.header\" &&
+		cp \"t4018/$i.content\" \"$i\" &&
 
 		# add test file to the index
-		git add "$i" &&
+		git add \"$i\" &&
 		# place modified file in the worktree
-		sed -e "s/ChangeMe/IWasChanged/" <"t4018/$i.content" >"$i" || return 1
-	done
-'
+		sed -e 's/ChangeMe/IWasChanged/' <\"t4018/$i.content\" >\"$i\"
+	"
 
-# check each individual file
-for i in $(git ls-files)
-do
 	test_expect_success "hunk header: $i" "
 		git diff -U1 $i >diff &&
 		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 12/35] userdiff tests: change setup loop to individual test setup

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> +# check each individual file
> +for i in $(git -C t4018 ls-files)
> +do
> +	test_expect_success "setup hunk header: $i" "
> +		grep -v '^t4018' \"t4018/$i\" >\"t4018/$i.content\" &&
> +		sed -n -e 's/^t4018 header: //p' <\"t4018/$i\" >\"t4018/$i.header\" &&
> +		cp \"t4018/$i.content\" \"$i\" &&
>  
>  		# add test file to the index
> -		git add "$i" &&
> +		git add \"$i\" &&
>  		# place modified file in the worktree
> -		sed -e "s/ChangeMe/IWasChanged/" <"t4018/$i.content" >"$i" || return 1
> -	done
> -'
> +		sed -e 's/ChangeMe/IWasChanged/' <\"t4018/$i.content\" >\"$i\"
> +	"

Please use '' around the second argument (i.e. test body) of the
test_expect_success, and use "" inside it.  "$i" that is used in the
loop is visible perfectly fine inside the test body when it is
eval'ed, and we won't have to count ugly backslashes that way.

>  
> -# check each individual file
> -for i in $(git ls-files)
> -do
>  	test_expect_success "hunk header: $i" "
>  		git diff -U1 $i >diff &&
>  		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&

Re: [PATCH v3 12/35] userdiff tests: change setup loop to individual test setup

[Johannes Sixt]

Am 25.02.21 um 03:52 schrieb Junio C Hamano:
> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
> 
>> +# check each individual file
>> +for i in $(git -C t4018 ls-files)
>> +do
>> +	test_expect_success "setup hunk header: $i" "
>> +		grep -v '^t4018' \"t4018/$i\" >\"t4018/$i.content\" &&
>> +		sed -n -e 's/^t4018 header: //p' <\"t4018/$i\" >\"t4018/$i.header\" &&
>> +		cp \"t4018/$i.content\" \"$i\" &&
>>  
>>  		# add test file to the index
>> -		git add "$i" &&
>> +		git add \"$i\" &&
>>  		# place modified file in the worktree
>> -		sed -e "s/ChangeMe/IWasChanged/" <"t4018/$i.content" >"$i" || return 1
>> -	done
>> -'
>> +		sed -e 's/ChangeMe/IWasChanged/' <\"t4018/$i.content\" >\"$i\"
>> +	"
> 
> Please use '' around the second argument (i.e. test body) of the
> test_expect_success, and use "" inside it.  "$i" that is used in the
> loop is visible perfectly fine inside the test body when it is
> eval'ed, and we won't have to count ugly backslashes that way.

If we do that, then we better be sure that the implementation of
test_expect_success does not clobber $i. Looks like we are OK at this time.

-- Hannes

Re: [PATCH v3 12/35] userdiff tests: change setup loop to individual test setup

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> Am 25.02.21 um 03:52 schrieb Junio C Hamano:
>> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>> 
>>> +# check each individual file
>>> +for i in $(git -C t4018 ls-files)
>>> +do
>>> +	test_expect_success "setup hunk header: $i" "
>>> +		grep -v '^t4018' \"t4018/$i\" >\"t4018/$i.content\" &&
>>> +		sed -n -e 's/^t4018 header: //p' <\"t4018/$i\" >\"t4018/$i.header\" &&
>>> +		cp \"t4018/$i.content\" \"$i\" &&
>>>  
>>>  		# add test file to the index
>>> -		git add "$i" &&
>>> +		git add \"$i\" &&
>>>  		# place modified file in the worktree
>>> -		sed -e "s/ChangeMe/IWasChanged/" <"t4018/$i.content" >"$i" || return 1
>>> -	done
>>> -'
>>> +		sed -e 's/ChangeMe/IWasChanged/' <\"t4018/$i.content\" >\"$i\"
>>> +	"
>> 
>> Please use '' around the second argument (i.e. test body) of the
>> test_expect_success, and use "" inside it.  "$i" that is used in the
>> loop is visible perfectly fine inside the test body when it is
>> eval'ed, and we won't have to count ugly backslashes that way.
>
> If we do that, then we better be sure that the implementation of
> test_expect_success does not clobber $i. Looks like we are OK at this time.

That's a good point.  Often we use more specific variable name than
$i to take advantage of the 'eval'-ed nature of the test body, and
it may probably be wise to do so here, too.

[PATCH v3 13/35] userdiff tests: factor out test_diff_funcname() logic

[Ævar Arnfjörð Bjarmason]

Factor out logic in test_diff_funcname() into two helper functions,
these will be useful in a follow-up commit where we'll do this munging
in more than one place.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 2365f0e361e..8a8a7a99c88 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -75,6 +75,17 @@ test_expect_success 'setup hunk header tests' '
 	git -C t4018 add .
 '
 
+do_change_me () {
+	file=$1
+	sed -e "s/ChangeMe/IWasChanged/" <"$file" >tmp &&
+	mv tmp "$file"
+}
+
+last_diff_context_line () {
+	file=$1
+	sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <$file
+}
+
 # check each individual file
 for i in $(git -C t4018 ls-files)
 do
@@ -85,13 +96,12 @@ do
 
 		# add test file to the index
 		git add \"$i\" &&
-		# place modified file in the worktree
-		sed -e 's/ChangeMe/IWasChanged/' <\"t4018/$i.content\" >\"$i\"
+		do_change_me \"$i\"
 	"
 
 	test_expect_success "hunk header: $i" "
 		git diff -U1 $i >diff &&
-		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
+		last_diff_context_line diff >ctx &&
 		test_cmp t4018/$i.header ctx
 	"
 done
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 13/35] userdiff tests: factor out test_diff_funcname() logic

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> Factor out logic in test_diff_funcname() into two helper functions,
> these will be useful in a follow-up commit where we'll do this munging
> in more than one place.
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  t/t4018-diff-funcname.sh | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index 2365f0e361e..8a8a7a99c88 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -75,6 +75,17 @@ test_expect_success 'setup hunk header tests' '
>  	git -C t4018 add .
>  '
>  
> +do_change_me () {
> +	file=$1
> +	sed -e "s/ChangeMe/IWasChanged/" <"$file" >tmp &&
> +	mv tmp "$file"
> +}
> +
> +last_diff_context_line () {

What this helper computes looks more like header line for each and
every hunk, not just the last one, to me.  Misnamed?

> +	file=$1
> +	sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <$file

Style.

 - Redirection operators should be written with space before, but no
   space after them.  In other words, write 'echo test >"$file"'
   instead of 'echo test> $file' or 'echo test > $file'.  Note that
   even though it is not required by POSIX to double-quote the
   redirection target in a variable (as shown above), our code does so
   because some versions of bash issue a warning without the quotes.

In any case, I wonder if this kind of clean-up should have been done
immediately before step 11/35, not after 11/35 started rewriting the
framework.  Any touch-up done after 11/35 risks smelling like "oops,
we found a better way to write it after we did the big rewrite".

> +}
> +
>  # check each individual file
>  for i in $(git -C t4018 ls-files)
>  do
> @@ -85,13 +96,12 @@ do
>  
>  		# add test file to the index
>  		git add \"$i\" &&
> -		# place modified file in the worktree
> -		sed -e 's/ChangeMe/IWasChanged/' <\"t4018/$i.content\" >\"$i\"
> +		do_change_me \"$i\"
>  	"
>  
>  	test_expect_success "hunk header: $i" "
>  		git diff -U1 $i >diff &&
> -		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
> +		last_diff_context_line diff >ctx &&
>  		test_cmp t4018/$i.header ctx
>  	"
>  done

[PATCH v3 14/35] userdiff tests: add alternative hunk header test infrastructure

[Ævar Arnfjörð Bjarmason]

Add an alternative to the hunk header test infrastructure introduced
in bfa7d01413 (t4018: an infrastructure to test hunk headers,
2014-03-21). See c228a5c077 (Merge branch 'js/userdiff-cc',
2014-03-31) for the whole series that commit was part of.

As noted in an earlier commit that change introduced the regression of
not testing for the full hunk line, but just whether "RIGHT" appeared
on it.

A preceding commit fixed that specific issue, but we are still left
with the inflexibility of not being able to do any testing except that
which fits inside the narrow confines of this custom test syntax,
e.g. being unable to setup config variables before specific tests.

So introduce a new "test_diff_funcname()" function which is expected
to be used by tests in t4018/*.sh, and move the "custom" tests over to
that. As will be seen in follow-up changes we'll make use of this new
function infrastructure in the "custom" tests.

There's no reason for not moving all of the tests over to this new
infrastructure as far as test coverage goes, but Junio and Johannes
expressed a desire to keep the existing test mode [1], so I'm only
moving a narrow set of tests over to the new mode.

1. https://lore.kernel.org/git/xmqqsg5vrhha.fsf@gitster.c.googlers.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh                      | 38 +++++++++++++++-
 t/t4018/README                                | 44 ++++++++++++++++---
 t/t4018/{custom1-pattern => custom1.sh}       | 11 ++++-
 t/t4018/custom2-match-to-end-of-line          |  9 ----
 t/t4018/custom2.sh                            | 18 ++++++++
 ...tom3-alternation-in-pattern => custom3.sh} | 11 ++++-
 6 files changed, 113 insertions(+), 18 deletions(-)
 rename t/t4018/{custom1-pattern => custom1.sh} (71%)
 mode change 100644 => 100755
 delete mode 100644 t/t4018/custom2-match-to-end-of-line
 create mode 100755 t/t4018/custom2.sh
 rename t/t4018/{custom3-alternation-in-pattern => custom3.sh} (66%)
 mode change 100644 => 100755

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 8a8a7a99c88..6fd3dce1364 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -87,7 +87,7 @@ last_diff_context_line () {
 }
 
 # check each individual file
-for i in $(git -C t4018 ls-files)
+for i in $(git -C t4018 ls-files -- ':!*.sh')
 do
 	test_expect_success "setup hunk header: $i" "
 		grep -v '^t4018' \"t4018/$i\" >\"t4018/$i.content\" &&
@@ -106,4 +106,40 @@ do
 	"
 done
 
+test_diff_funcname () {
+	desc=$1
+	cat <&8 >arg.header &&
+	cat <&9 >arg.test &&
+	what=$(cat arg.what) &&
+
+	test_expect_success "setup: $desc" '
+		cp arg.test "$what" &&
+		cp arg.header expected &&
+		git add "$what" &&
+		do_change_me "$what"
+	' &&
+
+	test_expect_success "$desc" '
+		git diff -U1 "$what" >diff &&
+		last_diff_context_line diff >actual &&
+		test_cmp expected actual
+	'
+}
+
+for what in $diffpatterns
+do
+	test="$TEST_DIRECTORY/t4018/$what.sh"
+	if ! test -e "$test"
+	then
+		continue
+	fi &&
+
+	test_expect_success "setup: hunk header for $what" '
+		echo "$what diff=$what" >.gitattributes &&
+		echo "$what" >arg.what
+	' &&
+
+	. "$test"
+done
+
 test_done
diff --git a/t/t4018/README b/t/t4018/README
index 0a246bbc10e..54ae735d5f8 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -1,13 +1,45 @@
-t4018 header: description of the test.
+t4018 header: There are two ways of writing tests in this directory. In both cases
 How to write test cases
 =======================
 
-Create test cases called "LANG-whatever" in this directory, where
-"LANG" is e.g. the userdiff driver name, where "whatever" is a brief
-description of the test.
+There are two ways of writing tests in this directory. In both cases
+"LANG" is the userdiff driver name, e.g. "perl" or "cpp".
+
+The word "ChangeMe" (exactly this form) should appear at a distance of
+at least two lines from the line that must appear in the hunk
+header. See below sections.
+
+t4018 header: t/README.
+"LANG.sh" test cases
+====================
+
+These tests use the normal test-lib.sh syntax and environment, and are
+sourced by t4018-diff-funcname.sh. The "test_diff_funcname()" function
+is a thin wrapper around the "test_expect_success()" function. See
+t/README.
+
+The content of the "EOF_TEST" argument is used as-is, with the
+exception of the "ChangeMe" token discussed above.
+
+The advantage of using this over the "LANG-whatever" test cases (see
+below) are:
 
-Insert the word "ChangeMe" (exactly this form) at a distance of
-at least two lines from the line that must appear in the hunk header.
+ - The ability to do custom test setup/teardown, e.g. using
+   "test_config" before the test is run.
+
+ - The description of the test is a string, and doesn't need to be a
+   valid filename.
+
+ - All the tests for a given driver are present in one file. As
+   demonstrated with the two "t4018 header" lines in this file this is
+   also possible with the "LANG-whatever" tests, but those N tests in
+   one file won't benefit from different test descriptions.
+
+"LANG-whatever" test cases
+==========================
+
+Create test cases called "LANG-whatever" in this directory, where
+"whatever" is a brief description of the test.
 
 Any line starting with "t4018" is a control line for the test:
 
diff --git a/t/t4018/custom1-pattern b/t/t4018/custom1.sh
old mode 100644
new mode 100755
similarity index 71%
rename from t/t4018/custom1-pattern
rename to t/t4018/custom1.sh
index 37a3422384b..f8bbccadb47
--- a/t/t4018/custom1-pattern
+++ b/t/t4018/custom1.sh
@@ -1,4 +1,12 @@
-t4018 header: int special, RIGHT;
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom1: pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+int special, RIGHT;
+EOF_HUNK
 public class Beer
 {
 	int special, RIGHT;
@@ -16,3 +24,4 @@ public class Beer
 			+ "99 bottles of beer on the wall.\n");
 	}
 }
+EOF_TEST
diff --git a/t/t4018/custom2-match-to-end-of-line b/t/t4018/custom2-match-to-end-of-line
deleted file mode 100644
index 4800bb1c568..00000000000
--- a/t/t4018/custom2-match-to-end-of-line
+++ /dev/null
@@ -1,9 +0,0 @@
-t4018 header: RIGHT_Beer
-public class RIGHT_Beer
-{
-	int special;
-	public static void main(String args[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
diff --git a/t/t4018/custom2.sh b/t/t4018/custom2.sh
new file mode 100755
index 00000000000..c68421f788e
--- /dev/null
+++ b/t/t4018/custom2.sh
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom2: match to end of line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT_Beer
+EOF_HUNK
+public class RIGHT_Beer
+{
+	int special;
+	public static void main(String args[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
diff --git a/t/t4018/custom3-alternation-in-pattern b/t/t4018/custom3.sh
old mode 100644
new mode 100755
similarity index 66%
rename from t/t4018/custom3-alternation-in-pattern
rename to t/t4018/custom3.sh
index bf7df3d9a73..07c5c134ffe
--- a/t/t4018/custom3-alternation-in-pattern
+++ b/t/t4018/custom3.sh
@@ -1,4 +1,12 @@
-t4018 header: public static void main(String RIGHT[])
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom3: alternation in pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
 public class Beer
 {
 	int special;
@@ -16,3 +24,4 @@ public class Beer
 			+ "99 bottles of beer on the wall.\n");
 	}
 }
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 14/35] userdiff tests: add alternative hunk header test infrastructure

[Johannes Sixt]

Am 24.02.21 um 20:51 schrieb Ævar Arnfjörð Bjarmason:
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index 8a8a7a99c88..6fd3dce1364 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -87,7 +87,7 @@ last_diff_context_line () {
>  }
>  
>  # check each individual file
> -for i in $(git -C t4018 ls-files)
> +for i in $(git -C t4018 ls-files -- ':!*.sh')
>  do
>  	test_expect_success "setup hunk header: $i" "
>  		grep -v '^t4018' \"t4018/$i\" >\"t4018/$i.content\" &&
> @@ -106,4 +106,40 @@ do
>  	"
>  done
>  
> +test_diff_funcname () {
> +	desc=$1
> +	cat <&8 >arg.header &&
> +	cat <&9 >arg.test &&
> +	what=$(cat arg.what) &&
> +
> +	test_expect_success "setup: $desc" '
> +		cp arg.test "$what" &&
> +		cp arg.header expected &&
> +		git add "$what" &&
> +		do_change_me "$what"
> +	' &&

I think we should not chain test_expect_success with && because when one
of the tests early in the chain fails, test case numbers are shifted
compared to when all tests succeed. I don't know, though, whether that
is an important thing (or a thing at all).

> +
> +	test_expect_success "$desc" '
> +		git diff -U1 "$what" >diff &&
> +		last_diff_context_line diff >actual &&
> +		test_cmp expected actual
> +	'
> +}
> +
> +for what in $diffpatterns
> +do
> +	test="$TEST_DIRECTORY/t4018/$what.sh"
> +	if ! test -e "$test"
> +	then
> +		continue
> +	fi &&
> +
> +	test_expect_success "setup: hunk header for $what" '
> +		echo "$what diff=$what" >.gitattributes &&
> +		echo "$what" >arg.what
> +	' &&

Same here. And when this is removed, the && just above is not needed
anymore, either.

> +
> +	. "$test"
> +done
> +
>  test_done

-- Hannes

[PATCH v3 15/35] userdiff tests: add a test with multiple tests in a LANG file

[Ævar Arnfjörð Bjarmason]

Demonstrate that we can now have tests with multiple tests in a given
LANG file. This is useful to show rules that don't match, follow-up
commits will add some tests like that.

Let's move the "golang" test, which I'm going to be modifying soon
over to to this new convention.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh        |  2 +-
 t/t4018/README                  |  8 +++++++
 t/t4018/golang                  | 39 +++++++++++++++++++++++++++++++++
 t/t4018/golang-complex-function |  9 --------
 t/t4018/golang-func             |  5 -----
 t/t4018/golang-interface        |  5 -----
 t/t4018/golang-long-func        |  6 -----
 t/t4018/golang-struct           |  5 -----
 8 files changed, 48 insertions(+), 31 deletions(-)
 create mode 100644 t/t4018/golang
 delete mode 100644 t/t4018/golang-complex-function
 delete mode 100644 t/t4018/golang-func
 delete mode 100644 t/t4018/golang-interface
 delete mode 100644 t/t4018/golang-long-func
 delete mode 100644 t/t4018/golang-struct

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 6fd3dce1364..7fc4291f4be 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -67,7 +67,7 @@ test_expect_success 'last regexp must not be negated' '
 test_expect_success 'setup hunk header tests' '
 	for i in $diffpatterns
 	do
-		echo "$i-* diff=$i"
+		echo "$i* diff=$i"
 	done > .gitattributes &&
 
 	cp -R "$TEST_DIRECTORY"/t4018 . &&
diff --git a/t/t4018/README b/t/t4018/README
index 54ae735d5f8..a3220dd6374 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -41,6 +41,9 @@ below) are:
 Create test cases called "LANG-whatever" in this directory, where
 "whatever" is a brief description of the test.
 
+You can also stick all the tests into one "LANG" file. See "t4018
+description" below.
+
 Any line starting with "t4018" is a control line for the test:
 
  - The "t4018 header:" line above specifies what text must appear in
@@ -48,6 +51,11 @@ Any line starting with "t4018" is a control line for the test:
    the line for ease of not having to hardcode the line numbers and
    offsets.
 
+ - The "t4018 description:" line above the test is a convention to add
+   a human-readable description for the test. Unlike in the case of
+   the LANG.sh test cases these descriptions don't make it to
+   "test_expect_success", and won't be seen in the test output.
+
 In many of the test cases the header line includes the token "RIGHT",
 this used to be part of the test syntax, but isn't anymore. Now we
 care about the "t4018 header:" line, not whatever line contains a
diff --git a/t/t4018/golang b/t/t4018/golang
new file mode 100644
index 00000000000..000e66b1c7b
--- /dev/null
+++ b/t/t4018/golang
@@ -0,0 +1,39 @@
+t4018 description: complex function
+t4018 header: func (t *Test) RIGHT(a Type) (Type, error) {
+type Test struct {
+	a Type
+}
+
+func (t *Test) RIGHT(a Type) (Type, error) {
+	t.a = a
+	return ChangeMe, nil
+}
+
+t4018 description: func
+t4018 header: func RIGHT() {
+func RIGHT() {
+	a := 5
+	b := ChangeMe
+}
+
+t4018 description: interface
+t4018 header: type RIGHT interface {
+type RIGHT interface {
+	a() Type
+	b() ChangeMe
+}
+
+t4018 description: long func
+t4018 header: func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
+func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
+	anotherLongVariableName AnotherLongType) {
+	a := 5
+	b := ChangeMe
+}
+
+t4018 description: struct
+t4018 header: type RIGHT struct {
+type RIGHT struct {
+	a Type
+	b ChangeMe
+}
diff --git a/t/t4018/golang-complex-function b/t/t4018/golang-complex-function
deleted file mode 100644
index 0574ba912e6..00000000000
--- a/t/t4018/golang-complex-function
+++ /dev/null
@@ -1,9 +0,0 @@
-t4018 header: func (t *Test) RIGHT(a Type) (Type, error) {
-type Test struct {
-	a Type
-}
-
-func (t *Test) RIGHT(a Type) (Type, error) {
-	t.a = a
-	return ChangeMe, nil
-}
diff --git a/t/t4018/golang-func b/t/t4018/golang-func
deleted file mode 100644
index 0472cfd9798..00000000000
--- a/t/t4018/golang-func
+++ /dev/null
@@ -1,5 +0,0 @@
-t4018 header: func RIGHT() {
-func RIGHT() {
-	a := 5
-	b := ChangeMe
-}
diff --git a/t/t4018/golang-interface b/t/t4018/golang-interface
deleted file mode 100644
index 3160a1d4524..00000000000
--- a/t/t4018/golang-interface
+++ /dev/null
@@ -1,5 +0,0 @@
-t4018 header: type RIGHT interface {
-type RIGHT interface {
-	a() Type
-	b() ChangeMe
-}
diff --git a/t/t4018/golang-long-func b/t/t4018/golang-long-func
deleted file mode 100644
index de83aaafca5..00000000000
--- a/t/t4018/golang-long-func
+++ /dev/null
@@ -1,6 +0,0 @@
-t4018 header: func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
-func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
-	anotherLongVariableName AnotherLongType) {
-	a := 5
-	b := ChangeMe
-}
diff --git a/t/t4018/golang-struct b/t/t4018/golang-struct
deleted file mode 100644
index fc8022537b2..00000000000
--- a/t/t4018/golang-struct
+++ /dev/null
@@ -1,5 +0,0 @@
-t4018 header: type RIGHT struct {
-type RIGHT struct {
-	a Type
-	b ChangeMe
-}
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 16/35] userdiff tests: do config teardown in test_diff_funcname()

[Ævar Arnfjörð Bjarmason]

Do a teardown of any custom "diff.<what>.x?funcname" config after a
test_diff_funcname() test runs. Nothing currently uses this, but a
follow-up commit will start setting custom config before certain
tests. Centralizing this teardown makes the tests simpler.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 7fc4291f4be..496313fc900 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -123,6 +123,13 @@ test_diff_funcname () {
 		git diff -U1 "$what" >diff &&
 		last_diff_context_line diff >actual &&
 		test_cmp expected actual
+	' &&
+
+	test_expect_success "teardown: $desc" '
+		# In case any custom config was set immediately before
+		# the test itself in the test file
+		test_unconfig "diff.$what.funcname" &&
+		test_unconfig "diff.$what.xfuncname"
 	'
 }
 
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 17/35] userdiff tests: move custom patterns into one test file

[Ævar Arnfjörð Bjarmason]

In a preceding commit the test infrastructure got rewritten so
"t/t4018/" are now normal test files which can do things like set
config, so let's make it responsible for setting up and tearing down
the config for its tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 16 +-------
 t/t4018/custom.sh        | 79 ++++++++++++++++++++++++++++++++++++++++
 t/t4018/custom1.sh       | 27 --------------
 t/t4018/custom2.sh       | 18 ---------
 t/t4018/custom3.sh       | 27 --------------
 5 files changed, 80 insertions(+), 87 deletions(-)
 create mode 100755 t/t4018/custom.sh
 delete mode 100755 t/t4018/custom1.sh
 delete mode 100755 t/t4018/custom2.sh
 delete mode 100755 t/t4018/custom3.sh

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 496313fc900..71a7b474cd4 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -15,18 +15,6 @@ test_expect_success 'setup' '
 	sort <builtin-drivers >builtin-drivers.sorted &&
 	test_cmp builtin-drivers.sorted builtin-drivers &&
 
-	# a non-trivial custom pattern
-	git config diff.custom1.funcname "!static
-!String
-[^ 	].*s.*" &&
-
-	# a custom pattern which matches to end of line
-	git config diff.custom2.funcname "......Beer\$" &&
-
-	# alternation in pattern
-	git config diff.custom3.funcname "Beer$" &&
-	git config diff.custom3.xfuncname "^[ 	]*((public|static).*)$" &&
-
 	# for regexp compilation tests
 	echo A >A.java &&
 	echo B >B.java
@@ -34,9 +22,7 @@ test_expect_success 'setup' '
 
 diffpatterns="
 	$(cat builtin-drivers)
-	custom1
-	custom2
-	custom3
+	custom
 "
 
 for p in $diffpatterns
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
new file mode 100755
index 00000000000..59d855c01c5
--- /dev/null
+++ b/t/t4018/custom.sh
@@ -0,0 +1,79 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_expect_success 'custom: setup non-trivial custom' '
+	git config diff.custom.funcname "!static
+!String
+[^ 	].*s.*"
+'
+
+test_diff_funcname 'custom: non-trivial custom pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+int special, RIGHT;
+EOF_HUNK
+public class Beer
+{
+	int special, RIGHT;
+	public static void main(String args[])
+	{
+		String s=" ";
+		for(int x = 99; x > 0; x--)
+		{
+			System.out.print(x + " bottles of beer on the wall "
+				+ x + " bottles of beer\n" // ChangeMe
+				+ "Take one down, pass it around, " + (x - 1)
+				+ " bottles of beer on the wall.\n");
+		}
+		System.out.print("Go to the store, buy some more,\n"
+			+ "99 bottles of beer on the wall.\n");
+	}
+}
+EOF_TEST
+
+test_expect_success 'custom: setup match to end of line' '
+	git config diff.custom.funcname "......Beer\$"
+'
+
+test_diff_funcname 'custom: match to end of line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT_Beer
+EOF_HUNK
+public class RIGHT_Beer
+{
+	int special;
+	public static void main(String args[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
+
+test_expect_success 'custom: setup alternation in pattern' '
+	git config diff.custom.funcname "Beer$" &&
+	git config diff.custom.xfuncname "^[ 	]*((public|static).*)$"
+'
+
+test_diff_funcname 'custom: alternation in pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
+public class Beer
+{
+	int special;
+	public static void main(String RIGHT[])
+	{
+		String s=" ";
+		for(int x = 99; x > 0; x--)
+		{
+			System.out.print(x + " bottles of beer on the wall "
+				+ x + " bottles of beer\n" // ChangeMe
+				+ "Take one down, pass it around, " + (x - 1)
+				+ " bottles of beer on the wall.\n");
+		}
+		System.out.print("Go to the store, buy some more,\n"
+			+ "99 bottles of beer on the wall.\n");
+	}
+}
+EOF_TEST
diff --git a/t/t4018/custom1.sh b/t/t4018/custom1.sh
deleted file mode 100755
index f8bbccadb47..00000000000
--- a/t/t4018/custom1.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom1: pattern' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-int special, RIGHT;
-EOF_HUNK
-public class Beer
-{
-	int special, RIGHT;
-	public static void main(String args[])
-	{
-		String s=" ";
-		for(int x = 99; x > 0; x--)
-		{
-			System.out.print(x + " bottles of beer on the wall "
-				+ x + " bottles of beer\n" // ChangeMe
-				+ "Take one down, pass it around, " + (x - 1)
-				+ " bottles of beer on the wall.\n");
-		}
-		System.out.print("Go to the store, buy some more,\n"
-			+ "99 bottles of beer on the wall.\n");
-	}
-}
-EOF_TEST
diff --git a/t/t4018/custom2.sh b/t/t4018/custom2.sh
deleted file mode 100755
index c68421f788e..00000000000
--- a/t/t4018/custom2.sh
+++ /dev/null
@@ -1,18 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom2: match to end of line' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-RIGHT_Beer
-EOF_HUNK
-public class RIGHT_Beer
-{
-	int special;
-	public static void main(String args[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
-EOF_TEST
diff --git a/t/t4018/custom3.sh b/t/t4018/custom3.sh
deleted file mode 100755
index 07c5c134ffe..00000000000
--- a/t/t4018/custom3.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom3: alternation in pattern' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-public static void main(String RIGHT[])
-EOF_HUNK
-public class Beer
-{
-	int special;
-	public static void main(String RIGHT[])
-	{
-		String s=" ";
-		for(int x = 99; x > 0; x--)
-		{
-			System.out.print(x + " bottles of beer on the wall "
-				+ x + " bottles of beer\n" // ChangeMe
-				+ "Take one down, pass it around, " + (x - 1)
-				+ " bottles of beer on the wall.\n");
-		}
-		System.out.print("Go to the store, buy some more,\n"
-			+ "99 bottles of beer on the wall.\n");
-	}
-}
-EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 18/35] userdiff tests: remove hack for "RIGHT" token

[Ævar Arnfjörð Bjarmason]

Now that the "RIGHT" token isn't how we select the desired hunk header
line in the test anymore we can revert a hack added in
f1b75fbaf1 (t4018: convert custom pattern test to the new
infrastructure, 2014-03-21) and go back to the regular expression we
were testing before that change.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/custom.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 59d855c01c5..b208a771d28 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -33,14 +33,14 @@ public class Beer
 EOF_TEST
 
 test_expect_success 'custom: setup match to end of line' '
-	git config diff.custom.funcname "......Beer\$"
+	git config diff.custom.funcname "Beer\$"
 '
 
 test_diff_funcname 'custom: match to end of line' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-RIGHT_Beer
+Beer
 EOF_HUNK
-public class RIGHT_Beer
+public class Beer
 {
 	int special;
 	public static void main(String args[])
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 19/35] userdiff tests: do not do compile tests on "custom" pattern

[Ævar Arnfjörð Bjarmason]

Since f1b75fbaf1 (t4018: convert custom pattern test to the new
infrastructure, 2014-03-21) we have been doing the basic sanity check
of whether patterns in userdiff.c compile on the "custom" patterns.

That we were doing this was an emergent effect of that change and an
earlier refactoring in bfa7d01413 (t4018: an infrastructure to test
hunk headers, 2014-03-21).

This was never intended by the test added in
e3bf5e43fd (t4018-diff-funcname: test syntax of builtin xfuncname
patterns, 2008-09-22), nor is there any point in doing this. We'll
error out in the custom.sh test itself if those patterns don't
compile.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 71a7b474cd4..b80546b4d7f 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -20,12 +20,7 @@ test_expect_success 'setup' '
 	echo B >B.java
 '
 
-diffpatterns="
-	$(cat builtin-drivers)
-	custom
-"
-
-for p in $diffpatterns
+for p in $(cat builtin-drivers)
 do
 	test_expect_success "builtin $p pattern compiles" '
 		echo "*.java diff=$p" >.gitattributes &&
@@ -50,6 +45,11 @@ test_expect_success 'last regexp must not be negated' '
 	test_i18ngrep ": Last expression must not be negated:" msg
 '
 
+diffpatterns="
+	$(cat builtin-drivers)
+	custom
+"
+
 test_expect_success 'setup hunk header tests' '
 	for i in $diffpatterns
 	do
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 20/35] userdiff tests: assert that new built-in drivers have tests

[Ævar Arnfjörð Bjarmason]

Add an assertion to the userdiff test framework to check that
everything except a narrow whitelist of existing built-in patterns has
tests.

Since this test framework was added we've added new patterns without
any tests. Let's make it obvious in the future in the diff for such
patches that they should have those tests.

For anything with tests we can skip the "does the pattern compile?"
test, as the actual tests will check that for us.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index b80546b4d7f..a3058fda130 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -15,6 +15,19 @@ test_expect_success 'setup' '
 	sort <builtin-drivers >builtin-drivers.sorted &&
 	test_cmp builtin-drivers.sorted builtin-drivers &&
 
+	# Do not add anything to this list. New built-in drivers should have
+	# tests
+	cat >drivers-no-tests <<-\EOF &&
+	ada
+	bibtex
+	csharp
+	html
+	objc
+	pascal
+	ruby
+	tex
+	EOF
+
 	# for regexp compilation tests
 	echo A >A.java &&
 	echo B >B.java
@@ -22,7 +35,12 @@ test_expect_success 'setup' '
 
 for p in $(cat builtin-drivers)
 do
-	test_expect_success "builtin $p pattern compiles" '
+	P=$(echo $p | tr 'a-z' 'A-Z')
+	if grep -q $p drivers-no-tests
+	then
+		test_set_prereq NO_TEST_FOR_DRIVER_$P
+	fi
+	test_expect_success NO_TEST_FOR_DRIVER_$P "builtin $p pattern compiles" '
 		echo "*.java diff=$p" >.gitattributes &&
 		test_expect_code 1 git diff --no-index \
 			A.java B.java 2>msg &&
@@ -119,11 +137,17 @@ test_diff_funcname () {
 	'
 }
 
+>drivers-had-no-tests
 for what in $diffpatterns
 do
 	test="$TEST_DIRECTORY/t4018/$what.sh"
 	if ! test -e "$test"
 	then
+		git -C t4018 ls-files ':!*.sh' "$what*" >other-tests &&
+		if ! test -s other-tests
+		then
+			echo $what >>drivers-had-no-tests
+		fi
 		continue
 	fi &&
 
@@ -135,4 +159,8 @@ do
 	. "$test"
 done
 
+test_expect_success 'we should not have new built-in drivers without tests' '
+	test_cmp drivers-no-tests drivers-had-no-tests
+'
+
 test_done
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 20/35] userdiff tests: assert that new built-in drivers have tests

[Johannes Sixt]

Am 24.02.21 um 20:51 schrieb Ævar Arnfjörð Bjarmason:
> Add an assertion to the userdiff test framework to check that
> everything except a narrow whitelist of existing built-in patterns has
> tests.
> 
> Since this test framework was added we've added new patterns without
> any tests. Let's make it obvious in the future in the diff for such
> patches that they should have those tests.
> 
> For anything with tests we can skip the "does the pattern compile?"
> test, as the actual tests will check that for us.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  t/t4018-diff-funcname.sh | 30 +++++++++++++++++++++++++++++-
>  1 file changed, 29 insertions(+), 1 deletion(-)
> 
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index b80546b4d7f..a3058fda130 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -15,6 +15,19 @@ test_expect_success 'setup' '
>  	sort <builtin-drivers >builtin-drivers.sorted &&
>  	test_cmp builtin-drivers.sorted builtin-drivers &&
>  
> +	# Do not add anything to this list. New built-in drivers should have
> +	# tests
> +	cat >drivers-no-tests <<-\EOF &&
> +	ada
> +	bibtex
> +	csharp
> +	html
> +	objc
> +	pascal
> +	ruby
> +	tex
> +	EOF
> +
>  	# for regexp compilation tests
>  	echo A >A.java &&
>  	echo B >B.java
> @@ -22,7 +35,12 @@ test_expect_success 'setup' '
>  
>  for p in $(cat builtin-drivers)
>  do
> -	test_expect_success "builtin $p pattern compiles" '
> +	P=$(echo $p | tr 'a-z' 'A-Z')
> +	if grep -q $p drivers-no-tests
> +	then
> +		test_set_prereq NO_TEST_FOR_DRIVER_$P
> +	fi
> +	test_expect_success NO_TEST_FOR_DRIVER_$P "builtin $p pattern compiles" '
>  		echo "*.java diff=$p" >.gitattributes &&
>  		test_expect_code 1 git diff --no-index \
>  			A.java B.java 2>msg &&

Please drop this hunk. It is extremly distracting to see, e.g.,

ok 8 # skip builtin cpp pattern compiles (missing NO_TEST_FOR_DRIVER_CPP)
ok 9 - builtin cpp wordRegex pattern compiles

It says "NO_TEST_FOR_DRIVER_CPP", but I know we have tests. I have to
waste time to check that something not related to "we have tests for the
driver" is meant here. It may be just a matter of naming the
prerequisite, but I think we do not need this optimization.

> @@ -119,11 +137,17 @@ test_diff_funcname () {
>  	'
>  }
>  
> +>drivers-had-no-tests
>  for what in $diffpatterns
>  do
>  	test="$TEST_DIRECTORY/t4018/$what.sh"
>  	if ! test -e "$test"
>  	then
> +		git -C t4018 ls-files ':!*.sh' "$what*" >other-tests &&
> +		if ! test -s other-tests
> +		then
> +			echo $what >>drivers-had-no-tests
> +		fi
>  		continue
>  	fi &&
>  
> @@ -135,4 +159,8 @@ do
>  	. "$test"
>  done
>  
> +test_expect_success 'we should not have new built-in drivers without tests' '
> +	test_cmp drivers-no-tests drivers-had-no-tests
> +'
> +
>  test_done
> 

Re: [PATCH v3 20/35] userdiff tests: assert that new built-in drivers have tests

[Ævar Arnfjörð Bjarmason]

On Sun, Feb 28 2021, Johannes Sixt wrote:

> Am 24.02.21 um 20:51 schrieb Ævar Arnfjörð Bjarmason:
>> Add an assertion to the userdiff test framework to check that
>> everything except a narrow whitelist of existing built-in patterns has
>> tests.
>> 
>> Since this test framework was added we've added new patterns without
>> any tests. Let's make it obvious in the future in the diff for such
>> patches that they should have those tests.
>> 
>> For anything with tests we can skip the "does the pattern compile?"
>> test, as the actual tests will check that for us.
>> 
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>>  t/t4018-diff-funcname.sh | 30 +++++++++++++++++++++++++++++-
>>  1 file changed, 29 insertions(+), 1 deletion(-)
>> 
>> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
>> index b80546b4d7f..a3058fda130 100755
>> --- a/t/t4018-diff-funcname.sh
>> +++ b/t/t4018-diff-funcname.sh
>> @@ -15,6 +15,19 @@ test_expect_success 'setup' '
>>  	sort <builtin-drivers >builtin-drivers.sorted &&
>>  	test_cmp builtin-drivers.sorted builtin-drivers &&
>>  
>> +	# Do not add anything to this list. New built-in drivers should have
>> +	# tests
>> +	cat >drivers-no-tests <<-\EOF &&
>> +	ada
>> +	bibtex
>> +	csharp
>> +	html
>> +	objc
>> +	pascal
>> +	ruby
>> +	tex
>> +	EOF
>> +
>>  	# for regexp compilation tests
>>  	echo A >A.java &&
>>  	echo B >B.java
>> @@ -22,7 +35,12 @@ test_expect_success 'setup' '
>>  
>>  for p in $(cat builtin-drivers)
>>  do
>> -	test_expect_success "builtin $p pattern compiles" '
>> +	P=$(echo $p | tr 'a-z' 'A-Z')
>> +	if grep -q $p drivers-no-tests
>> +	then
>> +		test_set_prereq NO_TEST_FOR_DRIVER_$P
>> +	fi
>> +	test_expect_success NO_TEST_FOR_DRIVER_$P "builtin $p pattern compiles" '
>>  		echo "*.java diff=$p" >.gitattributes &&
>>  		test_expect_code 1 git diff --no-index \
>>  			A.java B.java 2>msg &&
>
> Please drop this hunk. It is extremly distracting to see, e.g.,
>
> ok 8 # skip builtin cpp pattern compiles (missing NO_TEST_FOR_DRIVER_CPP)
> ok 9 - builtin cpp wordRegex pattern compiles
>
> It says "NO_TEST_FOR_DRIVER_CPP", but I know we have tests. I have to
> waste time to check that something not related to "we have tests for the
> driver" is meant here. It may be just a matter of naming the
> prerequisite, but I think we do not need this optimization.

Perhaps some other way to do this is better. I did the prerequisite just
to avoid indenting that whole part and I figured it was more readable,
but apparently not on the "more readable".

I do think it makes sense to keep this in some form, i.e. not test the
compilation if we know we have later tests to compile the regex. It's
providing self-documenting code to show that once we add tests for the
few missing drivers we can delete that test entirely.

And if a later change does the same check on the t4034/* files the
--word-diff check can also go.

>> @@ -119,11 +137,17 @@ test_diff_funcname () {
>>  	'
>>  }
>>  
>> +>drivers-had-no-tests
>>  for what in $diffpatterns
>>  do
>>  	test="$TEST_DIRECTORY/t4018/$what.sh"
>>  	if ! test -e "$test"
>>  	then
>> +		git -C t4018 ls-files ':!*.sh' "$what*" >other-tests &&
>> +		if ! test -s other-tests
>> +		then
>> +			echo $what >>drivers-had-no-tests
>> +		fi
>>  		continue
>>  	fi &&
>>  
>> @@ -135,4 +159,8 @@ do
>>  	. "$test"
>>  done
>>  
>> +test_expect_success 'we should not have new built-in drivers without tests' '
>> +	test_cmp drivers-no-tests drivers-had-no-tests
>> +'
>> +
>>  test_done
>> 

[PATCH v3 21/35] userdiff tests + docs: document & test "diff.<driver>.x?funcname"

[Ævar Arnfjörð Bjarmason]

Add the missing documentation for "diff.<driver>.funcname" and test
for how it and "diff.<driver>.xfuncname" interact.

Between the introduction of the "diff.<driver>.xfuncname" form in
45d9414fa5 (diff.*.xfuncname which uses "extended" regex's for hunk
header selection, 2008-09-18) and when this documentation was written
in 90b94c26f7 (Documentation: Add diff.<driver>.* to config,
2011-04-07) we forgot to document the existence of
"diff.<driver>.funcname".

Let's make a mention of it here, we could also partially revert the
former commit and discuss the more verbose form in gitattributes(5),
but let's stop short of that. It makes sense to guide users towards
ERE over BRE whenever possible.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/config/diff.txt | 11 +++++++++++
 t/t4018/custom.sh             | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt
index 2d3331f55c2..6f39ef1da93 100644
--- a/Documentation/config/diff.txt
+++ b/Documentation/config/diff.txt
@@ -153,10 +153,21 @@ diff.<driver>.command::
 	The custom diff driver command.  See linkgit:gitattributes[5]
 	for details.
 
+diff.<driver>.funcname::
 diff.<driver>.xfuncname::
 	The regular expression that the diff driver should use to
 	recognize the hunk header.  A built-in pattern may also be used.
 	See linkgit:gitattributes[5] for details.
++
+When provided as `diff.<driver>.funcname` the regular expression is
+interpreted as a basic regular expression. With
+`diff.<driver>.xfuncname` it's interpreted as an extended regular
+expression.
++
+The `*.funcname` and `*.xfuncname` variables behave as if though they
+were one configuration variable for the purposes of what value
+eventually gets used. Setting `*.funcname` will override an earlier
+`*.xfuncname` and vice-versa.
 
 diff.<driver>.binary::
 	Set this option to true to make the diff driver treat files as
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index b208a771d28..72d38dad686 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -77,3 +77,37 @@ public class Beer
 	}
 }
 EOF_TEST
+
+test_expect_success 'custom: setup config precedence' '
+	git config diff.custom.funcname "foo" &&
+	git config diff.custom.xfuncname "bar"
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+bar
+EOF_HUNK
+foo
+bar
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: setup config precedence' '
+	git config diff.custom.xfuncname "bar" &&
+	git config diff.custom.funcname "foo"
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+foo
+EOF_HUNK
+foo
+bar
+
+ChangeMe
+
+baz
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 22/35] gitattributes doc: reword discussion of built-in userdiff patterns

[Ævar Arnfjörð Bjarmason]

Reword the discussion of the built-in userdiff patterns to make it
more natural to precede it with a discussion about the semantics of
pattern matching, instead of assuming that it follows right after the
"diff.tex.xfuncname" example which now immediately precedes it. This
will make a follow-up commit smaller.

Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/gitattributes.txt | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index e84e104f932..62c1147ba97 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -794,11 +794,17 @@ backslashes; the pattern above picks a line that begins with a
 backslash, and zero or more occurrences of `sub` followed by
 `section` followed by open brace, to the end of line.
 
-There are a few built-in patterns to make this easier, and `tex`
-is one of them, so you do not have to write the above in your
-configuration file (you still need to enable this with the
-attribute mechanism, via `.gitattributes`).  The following built in
-patterns are available:
+There are built-in patterns shipped as part of git itself. A more
+advanced version of the `tex` pattern discussed above is one of them.
+
+For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
+configuration file as discussed above, but if present, it will
+override a built-in pattern.
+
+Nevertheless, you need to enable built-in patterns via .gitattributes`
+for the pattern to take effect.
+
+The following built-in patterns are available:
 
 - `ada` suitable for source code in the Ada language.
 
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 23/35] userdiff tests: move perl tests to perl.sh

[Ævar Arnfjörð Bjarmason]

Move the perl tests to perl.sh, a follow-up change will piggy-back on
these tests for updating the userdiff documentation. This will require
the new test *.sh test framework.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/perl-skip-end-of-heredoc     |  9 ----
 t/t4018/perl-skip-forward-decl       | 11 ----
 t/t4018/perl-skip-sub-in-pod         | 19 -------
 t/t4018/perl-sub-definition          |  5 --
 t/t4018/perl-sub-definition-kr-brace |  5 --
 t/t4018/perl.sh                      | 78 ++++++++++++++++++++++++++++
 6 files changed, 78 insertions(+), 49 deletions(-)
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc
 delete mode 100644 t/t4018/perl-skip-forward-decl
 delete mode 100644 t/t4018/perl-skip-sub-in-pod
 delete mode 100644 t/t4018/perl-sub-definition
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace
 create mode 100755 t/t4018/perl.sh

diff --git a/t/t4018/perl-skip-end-of-heredoc b/t/t4018/perl-skip-end-of-heredoc
deleted file mode 100644
index 8f90cca7314..00000000000
--- a/t/t4018/perl-skip-end-of-heredoc
+++ /dev/null
@@ -1,9 +0,0 @@
-t4018 header: sub RIGHTwithheredocument {
-sub RIGHTwithheredocument {
-	print <<"EOF"
-decoy here-doc
-EOF
-	# some lines of context
-	# to pad it out
-	print "ChangeMe\n";
-}
diff --git a/t/t4018/perl-skip-forward-decl b/t/t4018/perl-skip-forward-decl
deleted file mode 100644
index ff1f6d14735..00000000000
--- a/t/t4018/perl-skip-forward-decl
+++ /dev/null
@@ -1,11 +0,0 @@
-t4018 header: package RIGHT;
-package RIGHT;
-
-use strict;
-use warnings;
-use parent qw(Exporter);
-our @EXPORT_OK = qw(round finalround);
-
-sub other; # forward declaration
-
-# ChangeMe
diff --git a/t/t4018/perl-skip-sub-in-pod b/t/t4018/perl-skip-sub-in-pod
deleted file mode 100644
index ff1c65b28fc..00000000000
--- a/t/t4018/perl-skip-sub-in-pod
+++ /dev/null
@@ -1,19 +0,0 @@
-t4018 header: =head1 SYNOPSIS_RIGHT
-=head1 NAME
-
-Beer - subroutine to output fragment of a drinking song
-
-=head1 SYNOPSIS_RIGHT
-
-	use Beer qw(round finalround);
-
-	sub song {
-		for (my $i = 99; $i > 0; $i--) {
-			round $i;
-		}
-		finalround;
-	}
-
-	ChangeMe;
-
-=cut
diff --git a/t/t4018/perl-sub-definition b/t/t4018/perl-sub-definition
deleted file mode 100644
index 22e16ad5363..00000000000
--- a/t/t4018/perl-sub-definition
+++ /dev/null
@@ -1,5 +0,0 @@
-t4018 header: sub RIGHT {
-sub RIGHT {
-	my ($n) = @_;
-	print "ChangeMe";
-}
diff --git a/t/t4018/perl-sub-definition-kr-brace b/t/t4018/perl-sub-definition-kr-brace
deleted file mode 100644
index 6c94e6a62dd..00000000000
--- a/t/t4018/perl-sub-definition-kr-brace
+++ /dev/null
@@ -1,5 +0,0 @@
-t4018 header: sub RIGHT
-sub RIGHT
-{
-	print "ChangeMe\n";
-}
diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
new file mode 100755
index 00000000000..ac8fff7417a
--- /dev/null
+++ b/t/t4018/perl.sh
@@ -0,0 +1,78 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'perl: skip end of heredoc' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHTwithheredocument {
+EOF_HUNK
+sub RIGHTwithheredocument {
+	print <<"EOF"
+decoy here-doc
+EOF
+	# some lines of context
+	# to pad it out
+	print "ChangeMe\n";
+}
+EOF_TEST
+
+test_diff_funcname 'perl: skip forward decl' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+package RIGHT;
+EOF_HUNK
+package RIGHT;
+
+use strict;
+use warnings;
+use parent qw(Exporter);
+our @EXPORT_OK = qw(round finalround);
+
+sub other; # forward declaration
+
+# ChangeMe
+EOF_TEST
+
+test_diff_funcname 'perl: skip sub in pod' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+=head1 SYNOPSIS_RIGHT
+EOF_HUNK
+=head1 NAME
+
+Beer - subroutine to output fragment of a drinking song
+
+=head1 SYNOPSIS_RIGHT
+
+	use Beer qw(round finalround);
+
+	sub song {
+		for (my $i = 99; $i > 0; $i--) {
+			round $i;
+		}
+		finalround;
+	}
+
+	ChangeMe;
+
+=cut
+EOF_TEST
+
+test_diff_funcname 'perl: sub definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHT {
+EOF_HUNK
+sub RIGHT {
+	my ($n) = @_;
+	print "ChangeMe";
+}
+EOF_TEST
+
+test_diff_funcname 'perl: sub definition kr brace' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHT
+EOF_HUNK
+sub RIGHT
+{
+	print "ChangeMe\n";
+}
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 24/35] userdiff tests: move away from "RIGHT" in perl.sh

[Ævar Arnfjörð Bjarmason]

The "RIGHT" token is no longer magical, see recent changes to
t/t4018/README. Let's change it in the recently moved perl.sh
tests. This change was done separately so the earlier commit could
benefit from the "diff --color-moved" detection.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/perl.sh | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
index ac8fff7417a..b53b759353b 100755
--- a/t/t4018/perl.sh
+++ b/t/t4018/perl.sh
@@ -5,9 +5,9 @@
 
 test_diff_funcname 'perl: skip end of heredoc' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-sub RIGHTwithheredocument {
+sub withheredocument {
 EOF_HUNK
-sub RIGHTwithheredocument {
+sub withheredocument {
 	print <<"EOF"
 decoy here-doc
 EOF
@@ -19,9 +19,9 @@ EOF_TEST
 
 test_diff_funcname 'perl: skip forward decl' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-package RIGHT;
+package Some::Package;
 EOF_HUNK
-package RIGHT;
+package Some::Package;
 
 use strict;
 use warnings;
@@ -35,13 +35,13 @@ EOF_TEST
 
 test_diff_funcname 'perl: skip sub in pod' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-=head1 SYNOPSIS_RIGHT
+=head1 SYNOPSIS
 EOF_HUNK
 =head1 NAME
 
 Beer - subroutine to output fragment of a drinking song
 
-=head1 SYNOPSIS_RIGHT
+=head1 SYNOPSIS
 
 	use Beer qw(round finalround);
 
@@ -59,9 +59,9 @@ EOF_TEST
 
 test_diff_funcname 'perl: sub definition' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-sub RIGHT {
+sub asub {
 EOF_HUNK
-sub RIGHT {
+sub asub {
 	my ($n) = @_;
 	print "ChangeMe";
 }
@@ -69,9 +69,9 @@ EOF_TEST
 
 test_diff_funcname 'perl: sub definition kr brace' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-sub RIGHT
+sub asub
 EOF_HUNK
-sub RIGHT
+sub asub
 {
 	print "ChangeMe\n";
 }
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 25/35] gitattributes doc: document multi-line userdiff patterns

[Ævar Arnfjörð Bjarmason]

Document the multi-line userdiff patterns and how their matching and
the negation syntax works.

These patterns have been supported since f258475a6e (Per-path
attribute based hunk header selection., 2007-07-06), and have had
their current semantics ever since 3d8dccd74a (diff: fix "multiple
regexp" semantics to find hunk header comment, 2008-09-20).

But we had no documentation for them, let's fix that, and also add
tests showing how some of the things being discussed here work.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/gitattributes.txt | 38 ++++++++++++++++--
 t/t4018/custom.sh               | 70 +++++++++++++++++++++++++++++++++
 t/t4018/perl.sh                 | 16 ++++++++
 3 files changed, 120 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index 62c1147ba97..8082ff1ee73 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -794,12 +794,42 @@ backslashes; the pattern above picks a line that begins with a
 backslash, and zero or more occurrences of `sub` followed by
 `section` followed by open brace, to the end of line.
 
-There are built-in patterns shipped as part of git itself. A more
-advanced version of the `tex` pattern discussed above is one of them.
+Multiple patterns can be supplied by listing them one per line
+separated by `\n`. They will be matched one line at a time, e.g.:
+
+------------------------
+[diff "perl"]
+	xfuncname = "!^=head\n^[^ ]+.*"
+------------------------
+
+Patterns in a list of multiple patterns that begin with "!" are
+negated. A matching negated pattern will cause the matched line to be
+skipped. Use it to skip a later pattern that would otherwise match. It
+is an error if one or more negated patterns aren't followed by a
+non-negated pattern.
+
+To match a literal "!" at the start of a line, use some other regex
+construct that will match a literal "!" without "!" being the first
+character on that line, such as "[!]".
+
+If the last pattern in a list of multiple patterns ends with "\n" it
+will be interpreted as an empty pattern, and will match the first
+empty line. It's almost always a logic error to provide a list of
+multiple patterns ending with "\n", but it's permitted in case you
+genuinely want to match an empty line.
+
+If the pattern contains a `$1` capture it will be used instead of the
+entire matching line (`$0`) to display the hunk header. This can be
+used e.g. to strip whitespace from the beginning of the line, or to
+only display the function name as part of a longer function
+definition.
+
+There are built-in patterns shipped as part of git itself, see the
+full listing below.
 
 For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
-configuration file as discussed above, but if present, it will
-override a built-in pattern.
+configuration file. If present, it will override a built-in pattern,
+as shown in the `diff.perl.xfuncname` example above.
 
 Nevertheless, you need to enable built-in patterns via .gitattributes`
 for the pattern to take effect.
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 72d38dad686..127524afda3 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -111,3 +111,73 @@ ChangeMe
 
 baz
 EOF_TEST
+
+test_expect_success 'custom: setup negation syntax, ! is magic' '
+	git config diff.custom.xfuncname "!negation
+line"
+'
+
+test_diff_funcname 'custom: negation syntax, ! is magic' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+line
+EOF_HUNK
+line
+!negation
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: setup negation syntax, use [!] to override ! magic' '
+	git config diff.custom.xfuncname "[!]negation
+line"
+'
+
+test_diff_funcname 'custom: negation syntax, use [!] to override ! magic' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+!negation
+EOF_HUNK
+line
+!negation
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: setup captures in multiple patterns' '
+	git config diff.custom.xfuncname "!^=head
+^format ([^ ]+)
+^sub ([^;]+)"
+'
+
+test_diff_funcname 'custom: captures in multiple patterns' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+foo
+EOF_HUNK
+sub foo;
+
+=head1
+
+ChangeMe
+
+EOF_TEST
+
+test_expect_success 'custom: multiple patterns ending with \n' '
+	git config diff.custom.xfuncname "!^=head
+^sub ([^;]+)
+"
+'
+
+test_diff_funcname 'custom: multiple patterns ending with \n' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+
+EOF_HUNK
+sub foo;
+
+=head1
+
+ChangeMe
+
+EOF_TEST
diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
index b53b759353b..ba11241750b 100755
--- a/t/t4018/perl.sh
+++ b/t/t4018/perl.sh
@@ -76,3 +76,19 @@ sub asub
 	print "ChangeMe\n";
 }
 EOF_TEST
+
+
+test_expect_success 'custom: setup config overrides built-in patterns' '
+	git config diff.perl.xfuncname "!^=head
+^[^ ]+.*"
+'
+
+test_diff_funcname 'custom: config overrides built-in patterns' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub foo;
+EOF_HUNK
+sub foo;
+=head1
+
+ChangeMe
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 26/35] userdiff tests: switch to -U0 by default

[Ævar Arnfjörð Bjarmason]

Change the -U1 default in the userdiff tests to -U0. There's no reason
to use -U1, it just causes confusion and the need to work around the
test semantics themselves by not having the matching line immediately
precede the changed line.

The -U1 was initially added in f12c66b9bb8 (userdiff/perl: anchor
"sub" and "package" patterns on the left, 2011-05-21), seemingly to
appease a specific test I'd reported as failing with the "perl"
pattern. This was then documented in bfa7d01413b (t4018: an
infrastructure to test hunk headers, 2014-03-21).

There's no special logic in xdiff/xemit.c that would depend on -U1
v.s. -U0 here. So let's use -U0 for simplicity, we're interested in
what lines match most of the time, not at which line we start the
search.

A couple of test cases depended on the -U1, most confusingly one of
the custom.sh test cases.

Let's extend "test_diff_funcname()" to take an argument to "git diff"
and test "-U1" there, both because rewriting the testcase would be
painful to understand in the context of reviewing this change, and so
that we explicitly test that we're using -U0.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 5 +++--
 t/t4018/README           | 7 +++----
 t/t4018/custom.sh        | 1 +
 t/t4018/perl.sh          | 1 -
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index a3058fda130..ca23d156666 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -104,7 +104,7 @@ do
 	"
 
 	test_expect_success "hunk header: $i" "
-		git diff -U1 $i >diff &&
+		git diff -U0 $i >diff &&
 		last_diff_context_line diff >ctx &&
 		test_cmp t4018/$i.header ctx
 	"
@@ -112,6 +112,7 @@ done
 
 test_diff_funcname () {
 	desc=$1
+	diff_opts=${2:--U0} &&
 	cat <&8 >arg.header &&
 	cat <&9 >arg.test &&
 	what=$(cat arg.what) &&
@@ -124,7 +125,7 @@ test_diff_funcname () {
 	' &&
 
 	test_expect_success "$desc" '
-		git diff -U1 "$what" >diff &&
+		git diff $diff_opts "$what" >diff &&
 		last_diff_context_line diff >actual &&
 		test_cmp expected actual
 	' &&
diff --git a/t/t4018/README b/t/t4018/README
index a3220dd6374..cef7d3c0e17 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -5,11 +5,10 @@ How to write test cases
 There are two ways of writing tests in this directory. In both cases
 "LANG" is the userdiff driver name, e.g. "perl" or "cpp".
 
-The word "ChangeMe" (exactly this form) should appear at a distance of
-at least two lines from the line that must appear in the hunk
-header. See below sections.
+The word "ChangeMe" below the line that must appear in the hunk header
+(we run the diff with -U0). See below sections.
 
-t4018 header: t/README.
+t4018 header: The content of the "EOF_TEST" argument is used as-is, with the
 "LANG.sh" test cases
 ====================
 
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 127524afda3..97f310c02fb 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -10,6 +10,7 @@ test_expect_success 'custom: setup non-trivial custom' '
 '
 
 test_diff_funcname 'custom: non-trivial custom pattern' \
+	'-U1' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
 int special, RIGHT;
 EOF_HUNK
diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
index ba11241750b..63f373d1a43 100755
--- a/t/t4018/perl.sh
+++ b/t/t4018/perl.sh
@@ -89,6 +89,5 @@ sub foo;
 EOF_HUNK
 sub foo;
 =head1
-
 ChangeMe
 EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 27/35] userdiff tests: remove "funcname" from custom3 test

[Ævar Arnfjörð Bjarmason]

We can only have one "funcname" or "xfuncname", any later definition
overrides the earlier one, so this configuration wasn't doing
anything.

When this test was originally added in 3632cfc248 (Use compatibility
regex library for OSX/Darwin, 2008-09-07) we had no such definition of
two patters for this test. Back then this was setting the
"diff.java.funcname" configuration variable.

The stage for that second pattern being set got set later. In
45d9414fa5 (diff.*.xfuncname which uses "extended" regex's for hunk
header selection, 2008-09-18) the pattern got converted from
"funcname" to "xfuncname".

Soon after in b19d288b4d (t4018-diff-funcname: demonstrate end of line
funcname matching flaw, 2008-10-15) another test immediately preceding
this one got added, using "diff.java.funcname" for its configuration.

Then f792a0b88e (t4018 (funcname patterns): make configuration easier
to track, 2011-05-21) came along and codified this whole thing when
converting the two tests from "git config" to "test_config".

Since this was never the intent of the test let's just remove this,
the rationale in f792a0b88e for having some test for the clobbering
behavior makes sense, but I'll do that in another follow-up test, not
as a hard to read side-effect of this one.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/custom.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 97f310c02fb..58187c2293b 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -52,7 +52,6 @@ public class Beer
 EOF_TEST
 
 test_expect_success 'custom: setup alternation in pattern' '
-	git config diff.custom.funcname "Beer$" &&
 	git config diff.custom.xfuncname "^[ 	]*((public|static).*)$"
 '
 
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 28/35] userdiff tests: assert empty hunk header context on -U<large>

[Ævar Arnfjörð Bjarmason]

Assert the existing behavior that under -U<large> we'll show no hunk
header context, where <large> takes us past the potential hunk header
we'd have extracted.

I'm just picking a number over nine thousand as a really large number
we're unlikely to exceed in these tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index ca23d156666..ba10d1f5313 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -130,6 +130,13 @@ test_diff_funcname () {
 		test_cmp expected actual
 	' &&
 
+	test_expect_success "$desc -U9001" '
+		git diff -U9001 "$what" >diff &&
+		last_diff_context_line diff >actual &&
+		echo >blank &&
+		test_cmp blank actual
+	' &&
+
 	test_expect_success "teardown: $desc" '
 		# In case any custom config was set immediately before
 		# the test itself in the test file
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 29/35] userdiff tests: test for a bug in 1dbf0c0ad6c

[Ævar Arnfjörð Bjarmason]

Add a test for a bug in 1dbf0c0ad6c (userdiff: add built-in pattern
for golang, 2018-03-01), we'd ignore everything after the "{" only for
"type" lines, but not "func".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/golang | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/t/t4018/golang b/t/t4018/golang
index 000e66b1c7b..70bf0d936bb 100644
--- a/t/t4018/golang
+++ b/t/t4018/golang
@@ -16,6 +16,13 @@ func RIGHT() {
 	b := ChangeMe
 }
 
+t4018 description: func with comment after {
+t4018 header: func name() { // comment
+func name() { // comment
+	a := 5
+	b := ChangeMe
+}
+
 t4018 description: interface
 t4018 header: type RIGHT interface {
 type RIGHT interface {
@@ -37,3 +44,10 @@ type RIGHT struct {
 	a Type
 	b ChangeMe
 }
+
+t4018 description: struct with comment after {
+t4018 header: type some struct {
+type some struct { // comment
+	a Type
+	b ChangeMe
+}
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 30/35] userdiff golang: simplify and correct matching regex

[Ævar Arnfjörð Bjarmason]

Simplify the regex for the golang driver added in
1dbf0c0ad6c (userdiff: add built-in pattern for golang, 2018-03-01) to
remove redundant constructs.

There's no point in having a regex like this:

    .*(foo)?

When we can just write:

    .*

In the "func" case, since the "(foo?)" match isn't mandatory it won't
matter for the end result, and in this case we're not using the
capture pattern. Not that it would matter since it's followed by a
greedy .*, so we'd only get the empty string.

In the "type" case we would stop at the "{", since it was not preceded
by a ".*". This was a bug, if we have a comment or something else on
that line we should include it.

I'm also changing the "func[ \t]*.*" rule to "func[ \t].*" while I'm
at it. We should always get whitespace after "func", so this narrows
down our match. Let's do the same in the new "type" rule.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/golang | 2 +-
 userdiff.c     | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/t/t4018/golang b/t/t4018/golang
index 70bf0d936bb..72a35d66008 100644
--- a/t/t4018/golang
+++ b/t/t4018/golang
@@ -46,7 +46,7 @@ type RIGHT struct {
 }
 
 t4018 description: struct with comment after {
-t4018 header: type some struct {
+t4018 header: type some struct { // comment
 type some struct { // comment
 	a Type
 	b ChangeMe
diff --git a/userdiff.c b/userdiff.c
index 55f4f769bd3..698eca5ad35 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -129,9 +129,9 @@ IPATTERN("fountain",
 	 "[^ \t-]+"),
 PATTERNS("golang",
 	 /* Functions */
-	 "^[ \t]*(func[ \t]*.*(\\{[ \t]*)?)\n"
+	 "^[ \t]*(func[ \t].*)\n"
 	 /* Structs and interfaces */
-	 "^[ \t]*(type[ \t].*(struct|interface)[ \t]*(\\{[ \t]*)?)",
+	 "^[ \t]*(type[ \t].*(struct|interface)[ \t].*)",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.eE]+i?|0[xX]?[0-9a-fA-F]+i?"
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 31/35] userdiff golang: don't over-match intented constructs

[Ævar Arnfjörð Bjarmason]

Fix a bug introduced when the "golang" driver was added in
1dbf0c0ad6c (userdiff: add built-in pattern for golang, 2018-03-01).

Unlike the default def_ff() driver in xemit.c it would match "type"
declarations inside functions. Let's make it mandatory that a "func"
or "type" must be at the beginning of the line with no whitespace to
get around this.

Go is such a regularly formatted language that I think this can be
counted on. I think the whitespace matching was probably copy/pasted
from an earlier userdiff.c pattern.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/golang | 20 ++++++++++++++++++++
 userdiff.c     |  4 ++--
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/t/t4018/golang b/t/t4018/golang
index 72a35d66008..252b6049da4 100644
--- a/t/t4018/golang
+++ b/t/t4018/golang
@@ -51,3 +51,23 @@ type some struct { // comment
 	a Type
 	b ChangeMe
 }
+
+t4018 description: func combined with type
+t4018 header: func myfunc() {
+func myfunc() {
+	type mystruct struct {
+		a Foo
+		b Bar
+	}
+	ChangeMe
+
+t4018 description: anonymous indented func()
+t4018 header: func SomeThing() bool {
+func SomeThing() bool {
+	func() {
+		defer func() {
+			fmt.Println("hello")
+		}()
+	}()
+
+	ChangeMe
diff --git a/userdiff.c b/userdiff.c
index 698eca5ad35..704af241e44 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -129,9 +129,9 @@ IPATTERN("fountain",
 	 "[^ \t-]+"),
 PATTERNS("golang",
 	 /* Functions */
-	 "^[ \t]*(func[ \t].*)\n"
+	 "^(func[ \t].*)\n"
 	 /* Structs and interfaces */
-	 "^[ \t]*(type[ \t].*(struct|interface)[ \t].*)",
+	 "^(type[ \t].*[ \t](struct|interface)[ \t].*)",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.eE]+i?|0[xX]?[0-9a-fA-F]+i?"
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 31/35] userdiff golang: don't over-match intented constructs

[Johannes Sixt]

Am 24.02.21 um 20:51 schrieb Ævar Arnfjörð Bjarmason:
> Fix a bug introduced when the "golang" driver was added in
> 1dbf0c0ad6c (userdiff: add built-in pattern for golang, 2018-03-01).
> 
> Unlike the default def_ff() driver in xemit.c it would match "type"
> declarations inside functions. Let's make it mandatory that a "func"
> or "type" must be at the beginning of the line with no whitespace to
> get around this.
> 
> Go is such a regularly formatted language that I think this can be
> counted on. I think the whitespace matching was probably copy/pasted
> from an earlier userdiff.c pattern.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  t/t4018/golang | 20 ++++++++++++++++++++
>  userdiff.c     |  4 ++--
>  2 files changed, 22 insertions(+), 2 deletions(-)
> 
> diff --git a/t/t4018/golang b/t/t4018/golang
> index 72a35d66008..252b6049da4 100644
> --- a/t/t4018/golang
> +++ b/t/t4018/golang
> @@ -51,3 +51,23 @@ type some struct { // comment
>  	a Type
>  	b ChangeMe
>  }
> +
> +t4018 description: func combined with type
> +t4018 header: func myfunc() {
> +func myfunc() {
> +	type mystruct struct {
> +		a Foo
> +		b Bar
> +	}
> +	ChangeMe
> +
> +t4018 description: anonymous indented func()
> +t4018 header: func SomeThing() bool {
> +func SomeThing() bool {
> +	func() {

Wait! With the unmodified pattern, this indented "func()" cannot match
because it is not followed by a space. So, this test case does not
demonstrate that the modified pattern makes a difference. In the commit
message of 30/35 you say that "we should always get whitespace after
'func'", but here you show a case where that is not true. Did you forget
to write a name after "func"?

> +		defer func() {
> +			fmt.Println("hello")
> +		}()
> +	}()
> +
> +	ChangeMe
> diff --git a/userdiff.c b/userdiff.c
> index 698eca5ad35..704af241e44 100644
> --- a/userdiff.c
> +++ b/userdiff.c
> @@ -129,9 +129,9 @@ IPATTERN("fountain",
>  	 "[^ \t-]+"),
>  PATTERNS("golang",
>  	 /* Functions */
> -	 "^[ \t]*(func[ \t].*)\n"
> +	 "^(func[ \t].*)\n"
>  	 /* Structs and interfaces */
> -	 "^[ \t]*(type[ \t].*(struct|interface)[ \t].*)",
> +	 "^(type[ \t].*[ \t](struct|interface)[ \t].*)",
>  	 /* -- */
>  	 "[a-zA-Z_][a-zA-Z0-9_]*"
>  	 "|[-+0-9.eE]+i?|0[xX]?[0-9a-fA-F]+i?"
> 

[PATCH v3 32/35] userdiff golang: add a rule to match "package"

[Ævar Arnfjörð Bjarmason]

Improve the "golang" built-in pattern to match "package" lines, as
they weren't matched before changing e.g. the imports would commonly
result in an empty hunk header, now we'll instead show the package
name.

I used https://blog.golang.org/package-names as a guide here, but
e.g. "foo_bar" is still valid syntax, so let's let it pass but veer on
the side of not having false positives.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/golang | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 userdiff.c     |  2 ++
 2 files changed, 51 insertions(+)

diff --git a/t/t4018/golang b/t/t4018/golang
index 252b6049da4..38f254cd269 100644
--- a/t/t4018/golang
+++ b/t/t4018/golang
@@ -1,3 +1,52 @@
+t4018 description: package
+t4018 header: package main
+package main
+
+import "fmt"
+// ChangeMe
+
+t4018 description: package regex is selective -- ALLCAPS
+t4018 header: package main
+package ALLCAPS
+
+import "fmt"
+// ChangeMe
+
+t4018 description: package regex is selective -- CamelCase
+t4018 header: package main
+package CamelCase
+
+import "fmt"
+// ChangeMe
+
+t4018 description: package regex is selective -- 123
+t4018 header: package main
+package 123
+
+import "fmt"
+// ChangeMe
+
+t4018 description: package regex is not overly selective -- x509
+t4018 header: package x509
+package x509
+
+import "fmt"
+// ChangeMe
+
+t4018 description: package regex is not overly selective -- underbars
+t4018 header: package not_conventional
+package not_conventional
+
+import "fmt"
+// ChangeMe
+
+t4018 description: package regex is not overly selective -- camelCase
+t4018 header: package camelCase
+package camelCase
+
+import "fmt"
+// ChangeMe
+
 t4018 description: complex function
 t4018 header: func (t *Test) RIGHT(a Type) (Type, error) {
 type Test struct {
diff --git a/userdiff.c b/userdiff.c
index 704af241e44..bbbbfa33e0a 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -128,6 +128,8 @@ IPATTERN("fountain",
 	 /* -- */
 	 "[^ \t-]+"),
 PATTERNS("golang",
+	 /* Packages */
+	 "^(package[ \t][a-z][A-Za-z0-9_]+)[ \t]*\n"
 	 /* Functions */
 	 "^(func[ \t].*)\n"
 	 /* Structs and interfaces */
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 32/35] userdiff golang: add a rule to match "package"

[Johannes Sixt]

Am 24.02.21 um 20:51 schrieb Ævar Arnfjörð Bjarmason:
> Improve the "golang" built-in pattern to match "package" lines, as
> they weren't matched before changing e.g. the imports would commonly
> result in an empty hunk header, now we'll instead show the package
> name.
> 
> I used https://blog.golang.org/package-names as a guide here, but
> e.g. "foo_bar" is still valid syntax, so let's let it pass but veer on
> the side of not having false positives.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  t/t4018/golang | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
>  userdiff.c     |  2 ++
>  2 files changed, 51 insertions(+)
> 
> diff --git a/t/t4018/golang b/t/t4018/golang
> index 252b6049da4..38f254cd269 100644
> --- a/t/t4018/golang
> +++ b/t/t4018/golang
> @@ -1,3 +1,52 @@
> +t4018 description: package
> +t4018 header: package main
> +package main
> +
> +import "fmt"
> +// ChangeMe
> +
> +t4018 description: package regex is selective -- ALLCAPS
> +t4018 header: package main
> +package ALLCAPS
> +
> +import "fmt"
> +// ChangeMe
> +
> +t4018 description: package regex is selective -- CamelCase
> +t4018 header: package main
> +package CamelCase
> +
> +import "fmt"
> +// ChangeMe
> +
> +t4018 description: package regex is selective -- 123
> +t4018 header: package main
> +package 123
> +
> +import "fmt"
> +// ChangeMe
> +
> +t4018 description: package regex is not overly selective -- x509
> +t4018 header: package x509
> +package x509
> +
> +import "fmt"
> +// ChangeMe
> +
> +t4018 description: package regex is not overly selective -- underbars
> +t4018 header: package not_conventional
> +package not_conventional
> +
> +import "fmt"
> +// ChangeMe
> +
> +t4018 description: package regex is not overly selective -- camelCase
> +t4018 header: package camelCase
> +package camelCase
> +
> +import "fmt"
> +// ChangeMe
> +
>  t4018 description: complex function
>  t4018 header: func (t *Test) RIGHT(a Type) (Type, error) {
>  type Test struct {
> diff --git a/userdiff.c b/userdiff.c
> index 704af241e44..bbbbfa33e0a 100644
> --- a/userdiff.c
> +++ b/userdiff.c
> @@ -128,6 +128,8 @@ IPATTERN("fountain",
>  	 /* -- */
>  	 "[^ \t-]+"),
>  PATTERNS("golang",
> +	 /* Packages */
> +	 "^(package[ \t][a-z][A-Za-z0-9_]+)[ \t]*\n"

Just a single whitespace character is permitted between the keyword and
the name?

>  	 /* Functions */
>  	 "^(func[ \t].*)\n"
>  	 /* Structs and interfaces */
> 

[PATCH v3 33/35] userdiff golang: match multi-line "const" and "import"

[Ævar Arnfjörð Bjarmason]

Add matching rules for the very common pattern of having a multi-line
"import" or "const" declaration near the start of the "package"
line. Before this change we'd skip this and match whatever came before
it, e.g. the "package" line.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/golang | 26 ++++++++++++++++++++++++++
 userdiff.c     |  2 ++
 2 files changed, 28 insertions(+)

diff --git a/t/t4018/golang b/t/t4018/golang
index 38f254cd269..7f51fa02203 100644
--- a/t/t4018/golang
+++ b/t/t4018/golang
@@ -47,6 +47,32 @@ package camelCase
 import "fmt"
 // ChangeMe
 
+t4018 description: import (
+t4018 header: import (
+package somePackage
+
+import (
+	"os"
+	ChangeMe
+)
+
+t4018 description: const (
+t4018 header: const (
+package somePackage
+
+const (
+	Foo = 1
+	Bar = ChangeMe
+)
+
+t4018 description: const rule is selective
+t4018 header: package main
+package main
+
+const Foo = "Bar"
+
+// ChangeMe
+
 t4018 description: complex function
 t4018 header: func (t *Test) RIGHT(a Type) (Type, error) {
 type Test struct {
diff --git a/userdiff.c b/userdiff.c
index bbbbfa33e0a..c4a2bfaed70 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -132,6 +132,8 @@ PATTERNS("golang",
 	 "^(package[ \t][a-z][A-Za-z0-9_]+)[ \t]*\n"
 	 /* Functions */
 	 "^(func[ \t].*)\n"
+	 /* const & import */
+	 "^((import|const)[ \t]*\\([ \t]*)\n"
 	 /* Structs and interfaces */
 	 "^(type[ \t].*[ \t](struct|interface)[ \t].*)",
 	 /* -- */
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v3 34/35] userdiff tests: add basic test for ada

[Ævar Arnfjörð Bjarmason]

Add test for the ada userdiff pattern added in e90d065e64 (Add
userdiff patterns for Ada, 2012-09-16).

I don't know the ada language itself, I just stole a couple of
examples of code that used tokens we're matching[1][2]. Both test
examples stress our negative and positive matching rules.

1. https://rosettacode.org/wiki/99_bottles_of_beer#Ada
2. https://en.wikibooks.org/wiki/Ada_Programming/Tasking
---
 t/t4018-diff-funcname.sh |  1 -
 t/t4018/ada.sh           | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+), 1 deletion(-)
 create mode 100755 t/t4018/ada.sh

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index ba10d1f5313..b0c2782d067 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -18,7 +18,6 @@ test_expect_success 'setup' '
 	# Do not add anything to this list. New built-in drivers should have
 	# tests
 	cat >drivers-no-tests <<-\EOF &&
-	ada
 	bibtex
 	csharp
 	html
diff --git a/t/t4018/ada.sh b/t/t4018/ada.sh
new file mode 100755
index 00000000000..45fc2c7a3b2
--- /dev/null
+++ b/t/t4018/ada.sh
@@ -0,0 +1,37 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'ada: "procedure" over "with"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+procedure Bottles is
+EOF_HUNK
+with Ada.Text_Io; use Ada.Text_Io;
+ procedure Bottles is
+ begin
+    for X in reverse 1..99 loop
+       Put_Line(Integer'Image(X) & " bottles of beer on the wall");
+       Put_Line(Integer'Image(X) & " bottles of beer"); -- ChangeMe
+       Put_Line("Take one down, pass it around");
+       Put_Line(Integer'Image(X - 1) & " bottles of beer on the wall");
+       New_Line;
+    end loop;
+ end Bottles;
+EOF_TEST
+
+test_diff_funcname 'ada: "task" over "procedure"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+task body Check_CPU is
+EOF_HUNK
+procedure Housekeeping is
+  task Check_CPU;
+  task Backup_Disk;
+
+  task body Check_CPU is
+    -- Comment for spacing with
+    -- the above "task" for -U1
+    ChangeMe
+  end Check_CPU;
+end Housekeeping;
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 34/35] userdiff tests: add basic test for ada

[Johannes Sixt]

Am 24.02.21 um 20:51 schrieb Ævar Arnfjörð Bjarmason:
> Add test for the ada userdiff pattern added in e90d065e64 (Add
> userdiff patterns for Ada, 2012-09-16).
> 
> I don't know the ada language itself, I just stole a couple of
> examples of code that used tokens we're matching[1][2]. Both test
> examples stress our negative and positive matching rules.
> 
> 1. https://rosettacode.org/wiki/99_bottles_of_beer#Ada
> 2. https://en.wikibooks.org/wiki/Ada_Programming/Tasking

Missing Signed-off-by.

> ---
>  t/t4018-diff-funcname.sh |  1 -
>  t/t4018/ada.sh           | 37 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 37 insertions(+), 1 deletion(-)
>  create mode 100755 t/t4018/ada.sh
> 
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index ba10d1f5313..b0c2782d067 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -18,7 +18,6 @@ test_expect_success 'setup' '
>  	# Do not add anything to this list. New built-in drivers should have
>  	# tests
>  	cat >drivers-no-tests <<-\EOF &&
> -	ada
>  	bibtex
>  	csharp
>  	html
> diff --git a/t/t4018/ada.sh b/t/t4018/ada.sh
> new file mode 100755
> index 00000000000..45fc2c7a3b2
> --- /dev/null
> +++ b/t/t4018/ada.sh
> @@ -0,0 +1,37 @@
> +#!/bin/sh
> +#
> +# See ../t4018-diff-funcname.sh's test_diff_funcname()
> +#
> +
> +test_diff_funcname 'ada: "procedure" over "with"' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +procedure Bottles is
> +EOF_HUNK
> +with Ada.Text_Io; use Ada.Text_Io;
> + procedure Bottles is
> + begin
> +    for X in reverse 1..99 loop
> +       Put_Line(Integer'Image(X) & " bottles of beer on the wall");
> +       Put_Line(Integer'Image(X) & " bottles of beer"); -- ChangeMe
> +       Put_Line("Take one down, pass it around");
> +       Put_Line(Integer'Image(X - 1) & " bottles of beer on the wall");
> +       New_Line;
> +    end loop;
> + end Bottles;
> +EOF_TEST
> +
> +test_diff_funcname 'ada: "task" over "procedure"' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +task body Check_CPU is
> +EOF_HUNK
> +procedure Housekeeping is
> +  task Check_CPU;
> +  task Backup_Disk;
> +
> +  task body Check_CPU is
> +    -- Comment for spacing with
> +    -- the above "task" for -U1

Aren't we using -U0 by now?

> +    ChangeMe
> +  end Check_CPU;
> +end Housekeeping;
> +EOF_TEST
> 

I don't see anything special in these tests. Couldn't you just use the
simplified version of tests, which are much easier to read than this
shell script with EOF markers?

-- Hannes

[PATCH v3 35/35] userdiff tests: add basic test for ruby

[Ævar Arnfjörð Bjarmason]

Add a test for the Ruby pattern added way back in ad8c1d9260 (diff:
add ruby funcname pattern, 2008-07-31).

The "One/Two" picking demonstrates existing behavior, and a general
case where we may not do what the user expects since we're not aware
of the indentation level.

The code is modified from the Ruby code we have in-tree at
Documentation/asciidoctor-extensions.rb.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh |  1 -
 t/t4018/ruby.sh          | 58 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+), 1 deletion(-)
 create mode 100755 t/t4018/ruby.sh

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index b0c2782d067..7793d7652d5 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -23,7 +23,6 @@ test_expect_success 'setup' '
 	html
 	objc
 	pascal
-	ruby
 	tex
 	EOF
 
diff --git a/t/t4018/ruby.sh b/t/t4018/ruby.sh
new file mode 100755
index 00000000000..ef8a154421a
--- /dev/null
+++ b/t/t4018/ruby.sh
@@ -0,0 +1,58 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'ruby: "def" over "class/module"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def process(parent)
+EOF_HUNK
+require 'asciidoctor'
+
+module Git
+  module Documentation
+    class SomeClass
+      use_some
+
+      def process(parent)
+        puts("hello")
+	puts(ChangeMe)
+      end
+    end
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'ruby: "class" over "class/module"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class Two
+EOF_HUNK
+module Git
+  module Documentation
+    class One
+    end
+
+    class Two
+      # Spacing for -U1
+      ChangeMe
+    end
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'ruby: picks first "class/module/def" before changed context' \
+	'-U1' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class One
+EOF_HUNK
+module Git
+  module Documentation
+    class One
+    end
+
+    class Two
+      ChangeMe
+    end
+  end
+end
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v3 35/35] userdiff tests: add basic test for ruby

[Johannes Sixt]

Am 24.02.21 um 20:51 schrieb Ævar Arnfjörð Bjarmason:
> Add a test for the Ruby pattern added way back in ad8c1d9260 (diff:
> add ruby funcname pattern, 2008-07-31).
> 
> The "One/Two" picking demonstrates existing behavior, and a general
> case where we may not do what the user expects since we're not aware
> of the indentation level.
> 
> The code is modified from the Ruby code we have in-tree at
> Documentation/asciidoctor-extensions.rb.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  t/t4018-diff-funcname.sh |  1 -
>  t/t4018/ruby.sh          | 58 ++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 58 insertions(+), 1 deletion(-)
>  create mode 100755 t/t4018/ruby.sh
> 
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index b0c2782d067..7793d7652d5 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -23,7 +23,6 @@ test_expect_success 'setup' '
>  	html
>  	objc
>  	pascal
> -	ruby
>  	tex
>  	EOF
>  
> diff --git a/t/t4018/ruby.sh b/t/t4018/ruby.sh
> new file mode 100755
> index 00000000000..ef8a154421a
> --- /dev/null
> +++ b/t/t4018/ruby.sh
> @@ -0,0 +1,58 @@
> +#!/bin/sh
> +#
> +# See ../t4018-diff-funcname.sh's test_diff_funcname()
> +#
> +
> +test_diff_funcname 'ruby: "def" over "class/module"' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +def process(parent)
> +EOF_HUNK
> +require 'asciidoctor'
> +
> +module Git
> +  module Documentation
> +    class SomeClass
> +      use_some
> +
> +      def process(parent)
> +        puts("hello")
> +	puts(ChangeMe)
> +      end
> +    end
> +  end
> +end
> +EOF_TEST
> +
> +test_diff_funcname 'ruby: "class" over "class/module"' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +class Two
> +EOF_HUNK
> +module Git
> +  module Documentation
> +    class One
> +    end
> +
> +    class Two
> +      # Spacing for -U1

Again: -U0 is the default by now.

> +      ChangeMe
> +    end
> +  end
> +end
> +EOF_TEST
> +
> +test_diff_funcname 'ruby: picks first "class/module/def" before changed context' \
> +	'-U1' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +class One
> +EOF_HUNK
> +module Git
> +  module Documentation
> +    class One
> +    end
> +
> +    class Two
> +      ChangeMe
> +    end
> +  end
> +end
> +EOF_TEST

Why does this test need -U1? Wouldn't it be sufficient to change the
expected text to "class Two"? And if the answer is "yes", I again do not
see the need for a shell script test here; the simplified version would do.

-- Hannes

[PATCH v2 01/27] userdiff: refactor away the parse_bool() function

[Ævar Arnfjörð Bjarmason]

Since 6680a0874f (drop odd return value semantics from
userdiff_config, 2012-02-07) we have not cared about the return values
of parse_tristate() or git_config_bool() v.s. falling through in
userdiff_config(), so let's do so in those cases to make the code
easier to read.

Having a wrapper function for git_config_bool() dates back to
d9bae1a178 (diff: cache textconv output, 2010-04-01) and
122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05), both of
which predated the change in 6680a0874f which made their return values
redundant.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 3f81a2261c..c147bcbb17 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -275,19 +275,12 @@ static int parse_funcname(struct userdiff_funcname *f, const char *k,
 	return 0;
 }
 
-static int parse_tristate(int *b, const char *k, const char *v)
+static void parse_tristate(int *b, const char *k, const char *v)
 {
 	if (v && !strcasecmp(v, "auto"))
 		*b = -1;
 	else
 		*b = git_config_bool(k, v);
-	return 0;
-}
-
-static int parse_bool(int *b, const char *k, const char *v)
-{
-	*b = git_config_bool(k, v);
-	return 0;
 }
 
 int userdiff_config(const char *k, const char *v)
@@ -312,16 +305,17 @@ int userdiff_config(const char *k, const char *v)
 		return parse_funcname(&drv->funcname, k, v, 0);
 	if (!strcmp(type, "xfuncname"))
 		return parse_funcname(&drv->funcname, k, v, REG_EXTENDED);
-	if (!strcmp(type, "binary"))
-		return parse_tristate(&drv->binary, k, v);
 	if (!strcmp(type, "command"))
 		return git_config_string(&drv->external, k, v);
 	if (!strcmp(type, "textconv"))
 		return git_config_string(&drv->textconv, k, v);
-	if (!strcmp(type, "cachetextconv"))
-		return parse_bool(&drv->textconv_want_cache, k, v);
 	if (!strcmp(type, "wordregex"))
 		return git_config_string(&drv->word_regex, k, v);
+	/* Don't care about the parse errors for these, fallthrough */
+	if (!strcmp(type, "cachetextconv"))
+		drv->textconv_want_cache = git_config_bool(k, v);
+	if (!strcmp(type, "binary"))
+		parse_tristate(&drv->binary, k, v);
 
 	return 0;
 }
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 02/27] userdiff style: re-order drivers in alphabetical order

[Ævar Arnfjörð Bjarmason]

Address some old code smell and move around the built-in userdiff
drivers so they're both in alphabetical order, and now in the same
order they appear in the gitattributes(5) documentation.

The two started drifting in be58e70dba (diff: unify external diff and
funcname parsing code, 2008-10-05), and then even further in
80c49c3de2 (color-words: make regex configurable via attributes,
2009-01-17) when the "cpp" pattern was added.

There are no functional changes here, and as --color-moved will show
only moved existing lines.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 76 +++++++++++++++++++++++++++---------------------------
 1 file changed, 38 insertions(+), 38 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c147bcbb17..c92cbcc054 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,6 +44,44 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
+PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 "[={}\"]|[^={}\" \t]+"),
+PATTERNS("cpp",
+	 /* Jump targets or access declarations */
+	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
+	 /* functions/methods, variables, and compounds at top level */
+	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
+PATTERNS("csharp",
+	 /* Keywords */
+	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
+	 /* Methods and constructors */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
+	 /* Properties */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
+	 /* Type definitions */
+	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
+	 /* Namespace */
+	 "^[ \t]*(namespace[ \t]+.*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
+IPATTERN("css",
+	 "![:;][[:space:]]*$\n"
+	 "^[:[@.#]?[_a-z0-9].*$",
+	 /* -- */
+	 /*
+	  * This regex comes from W3C CSS specs. Should theoretically also
+	  * allow ISO 10646 characters U+00A0 and higher,
+	  * but they are not handled in this regex.
+	  */
+	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
+	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
+),
 PATTERNS("dts",
 	 "!;\n"
 	 "!=\n"
@@ -191,46 +229,8 @@ PATTERNS("rust",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[0-9][0-9_a-fA-Fiosuxz]*(\\.([0-9]*[eE][+-]?)?[0-9_fF]*)?"
 	 "|[-+*\\/<>%&^|=!:]=|<<=?|>>=?|&&|\\|\\||->|=>|\\.{2}=|\\.{3}|::"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
-	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("tex", "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$",
 	 "\\\\[a-zA-Z@]+|\\\\.|[a-zA-Z0-9\x80-\xff]+"),
-PATTERNS("cpp",
-	 /* Jump targets or access declarations */
-	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
-	 /* functions/methods, variables, and compounds at top level */
-	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
-PATTERNS("csharp",
-	 /* Keywords */
-	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
-	 /* Methods and constructors */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
-	 /* Properties */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
-	 /* Type definitions */
-	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
-	 /* Namespace */
-	 "^[ \t]*(namespace[ \t]+.*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
-IPATTERN("css",
-	 "![:;][[:space:]]*$\n"
-	 "^[:[@.#]?[_a-z0-9].*$",
-	 /* -- */
-	 /*
-	  * This regex comes from W3C CSS specs. Should theoretically also
-	  * allow ISO 10646 characters U+00A0 and higher,
-	  * but they are not handled in this regex.
-	  */
-	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
-	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
-),
 { "default", NULL, -1, { NULL, 0 } },
 };
 #undef PATTERNS
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 03/27] userdiff style: declare patterns with consistent style

[Ævar Arnfjörð Bjarmason]

Change those patterns which were declared with a regex on the same
line as the "PATTERNS()" line to put that regex on the next line, and
add missing "/* -- */" separator comments between the pattern and
word_regex.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c92cbcc054..c7aaf7094f 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,7 +44,9 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+PATTERNS("bibtex",
+	 "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 /* -- */
 	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("cpp",
 	 /* Jump targets or access declarations */
@@ -121,7 +123,9 @@ IPATTERN("fortran",
 	  * they would have been matched above as a variable anyway. */
 	 "|[-+]?[0-9.]+([AaIiDdEeFfLlTtXx][Ss]?[-+]?[0-9.]*)?(_[a-zA-Z0-9][a-zA-Z0-9_]*)?"
 	 "|//|\\*\\*|::|[/<>=]="),
-IPATTERN("fountain", "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+IPATTERN("fountain",
+	 "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
+	 /* -- */
 	 "[^ \t-]+"),
 PATTERNS("golang",
 	 /* Functions */
@@ -132,7 +136,9 @@ PATTERNS("golang",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.eE]+i?|0[xX]?[0-9a-fA-F]+i?"
 	 "|[-+*/<>%&^|=!:]=|--|\\+\\+|<<=?|>>=?|&\\^=?|&&|\\|\\||<-|\\.{3}"),
-PATTERNS("html", "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+PATTERNS("html",
+	 "^[ \t]*(<[Hh][1-6]([ \t].*)?>.*)$",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("java",
 	 "!^[ \t]*(catch|do|for|if|instanceof|new|return|switch|throw|while)\n"
@@ -144,6 +150,7 @@ PATTERNS("java",
 	 "|--|\\+\\+|<<=?|>>>?=?|&&|\\|\\|"),
 PATTERNS("markdown",
 	 "^ {0,3}#{1,6}[ \t].*",
+	 /* -- */
 	 "[^<>= \t]+"),
 PATTERNS("matlab",
 	 /*
@@ -152,6 +159,7 @@ PATTERNS("matlab",
 	  * that is understood by both.
 	  */
 	 "^[[:space:]]*((classdef|function)[[:space:]].*)$|^(%%%?|##)[[:space:]].*$",
+	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*|[-+0-9.e]+|[=~<>]=|\\.[*/\\^']|\\|\\||&&"),
 PATTERNS("objc",
 	 /* Negate C statements that can look like functions */
@@ -212,13 +220,15 @@ PATTERNS("php",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+"
 	 "|[-+*/<>%&^|=!.]=|--|\\+\\+|<<=?|>>=?|===|&&|\\|\\||::|->"),
-PATTERNS("python", "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
+PATTERNS("python",
+	 "^[ \t]*((class|(async[ \t]+)?def)[ \t].*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+[jJlL]?|0[xX]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|//=?|<<=?|>>=?|\\*\\*=?"),
 	 /* -- */
-PATTERNS("ruby", "^[ \t]*((class|module|def)[ \t].*)$",
+PATTERNS("ruby",
+	 "^[ \t]*((class|module|def)[ \t].*)$",
 	 /* -- */
 	 "(@|@@|\\$)?[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[-+0-9.e]+|0[xXbB]?[0-9a-fA-F]+|\\?(\\\\C-)?(\\\\M-)?."
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 04/27] userdiff style: normalize pascal regex declaration

[Ævar Arnfjörð Bjarmason]

Declare the pascal pattern consistently with how we declare the
others, not having "\n" on one line by itself, but as part of the
pattern, and when there are alterations have the "|" at the start, not
end of the line.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c7aaf7094f..10a02d3620 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -175,9 +175,8 @@ PATTERNS("objc",
 	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
 	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
 PATTERNS("pascal",
-	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface|"
-		"implementation|initialization|finalization)[ \t]*.*)$"
-	 "\n"
+	 "^(((class[ \t]+)?(procedure|function)|constructor|destructor|interface"
+	 "|implementation|initialization|finalization)[ \t]*.*)$\n"
 	 "^(.*=[ \t]*(class|record).*)$",
 	 /* -- */
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 05/27] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

Refactor the userdiff_find_by_namelen() function so that a new
for_each_userdiff_driver() API function does most of the work.

This will be useful for the same reason we've got other for_each_*()
API functions as part of various APIs, and will be used in a follow-up
commit.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 61 +++++++++++++++++++++++++++++++++++++++++++-----------
 userdiff.h | 15 ++++++++++++++
 2 files changed, 64 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 10a02d3620..55f4f769bd 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -259,20 +259,32 @@ static struct userdiff_driver driver_false = {
 	{ NULL, 0 }
 };
 
-static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+struct for_each_userdiff_driver_cb {
+	const char *k;
+	size_t len;
+	struct userdiff_driver *driver;
+};
+
+static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
+				       enum userdiff_driver_type type, void *priv)
 {
-	int i;
-	for (i = 0; i < ndrivers; i++) {
-		struct userdiff_driver *drv = drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
-	}
-	for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
-		struct userdiff_driver *drv = builtin_drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
+	struct for_each_userdiff_driver_cb *cb_data = priv;
+
+	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
+	    !driver->name[cb_data->len]) {
+		cb_data->driver = driver;
+		return -1; /* found it! */
 	}
-	return NULL;
+	return 0;
+}
+
+static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+{
+	struct for_each_userdiff_driver_cb udcbdata = { .k = k, .len = len, .driver = NULL };
+
+	for_each_userdiff_driver(userdiff_find_by_namelen_cb,
+				 USERDIFF_DRIVER_TYPE_UNSPECIFIED, &udcbdata);
+	return udcbdata.driver;
 }
 
 static int parse_funcname(struct userdiff_funcname *f, const char *k,
@@ -373,3 +385,28 @@ struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 
 	return driver;
 }
+
+int for_each_userdiff_driver(each_userdiff_driver_fn fn,
+			     enum userdiff_driver_type type, void *cb_data)
+{
+	int i, ret;
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {
+
+		for (i = 0; i < ndrivers; i++) {
+			struct userdiff_driver *drv = drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_CUSTOM, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_BUILTIN)) {
+
+		for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
+			struct userdiff_driver *drv = builtin_drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_BUILTIN, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	return 0;
+}
diff --git a/userdiff.h b/userdiff.h
index 203057e13e..fe14014a77 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -21,6 +21,13 @@ struct userdiff_driver {
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
 };
+enum userdiff_driver_type {
+	USERDIFF_DRIVER_TYPE_UNSPECIFIED = 1<<0,
+	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<1,
+	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<2,
+};
+typedef int (*each_userdiff_driver_fn)(struct userdiff_driver *,
+				       enum userdiff_driver_type, void *);
 
 int userdiff_config(const char *k, const char *v);
 struct userdiff_driver *userdiff_find_by_name(const char *name);
@@ -34,4 +41,12 @@ struct userdiff_driver *userdiff_find_by_path(struct index_state *istate,
 struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 					      struct userdiff_driver *driver);
 
+/*
+ * Iterate over each driver of type userdiff_driver_type, or
+ * USERDIFF_DRIVER_TYPE_UNSPECIFIED for all of them. Return non-zero
+ * to exit from the loop.
+ */
+int for_each_userdiff_driver(each_userdiff_driver_fn,
+			     enum userdiff_driver_type, void *);
+
 #endif /* USERDIFF */
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 06/27] userdiff tests: explicitly test "default" pattern

[Ævar Arnfjörð Bjarmason]

Since 122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05)
the internals of the userdiff.c code have understood a "default" name,
which is invoked as userdiff_find_by_name("default") and present in
the "builtin_drivers" struct. Let's test for this special case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 9675bc17db..cefe329aea 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -49,6 +49,7 @@ diffpatterns="
 	ruby
 	rust
 	tex
+	default
 	custom1
 	custom2
 	custom3
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 07/27] userdiff tests: list builtin drivers via test-tool

[Ævar Arnfjörð Bjarmason]

Change the userdiff test to list the builtin drivers via the
test-tool, using the new for_each_userdiff_driver() API function.

This gets rid of the need to modify this part of the test every time a
new pattern is added, see 2ff6c34612 (userdiff: support Bash,
2020-10-22) and 09dad9256a (userdiff: support Markdown, 2020-05-02)
for two recent examples.

I only need the "list-builtin-drivers "argument here, but let's add
"list-custom-drivers" and "list-drivers" too, just because it's easy.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile                 |  1 +
 t/helper/test-tool.c     |  1 +
 t/helper/test-tool.h     |  1 +
 t/helper/test-userdiff.c | 30 ++++++++++++++++++++++++++++++
 t/t4018-diff-funcname.sh | 28 ++++------------------------
 5 files changed, 37 insertions(+), 24 deletions(-)
 create mode 100644 t/helper/test-userdiff.c

diff --git a/Makefile b/Makefile
index 5a239cac20..710a0deaed 100644
--- a/Makefile
+++ b/Makefile
@@ -741,6 +741,7 @@ TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o
 TEST_BUILTINS_OBJS += test-subprocess.o
 TEST_BUILTINS_OBJS += test-trace2.o
 TEST_BUILTINS_OBJS += test-urlmatch-normalization.o
+TEST_BUILTINS_OBJS += test-userdiff.o
 TEST_BUILTINS_OBJS += test-wildmatch.o
 TEST_BUILTINS_OBJS += test-windows-named-pipe.o
 TEST_BUILTINS_OBJS += test-write-cache.o
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index f97cd9f48a..dcb05ca6e5 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -71,6 +71,7 @@ static struct test_cmd cmds[] = {
 	{ "submodule-nested-repo-config", cmd__submodule_nested_repo_config },
 	{ "subprocess", cmd__subprocess },
 	{ "trace2", cmd__trace2 },
+	{ "userdiff", cmd__userdiff },
 	{ "urlmatch-normalization", cmd__urlmatch_normalization },
 	{ "xml-encode", cmd__xml_encode },
 	{ "wildmatch", cmd__wildmatch },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index 28072c0ad5..589f2e8ac6 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -61,6 +61,7 @@ int cmd__submodule_config(int argc, const char **argv);
 int cmd__submodule_nested_repo_config(int argc, const char **argv);
 int cmd__subprocess(int argc, const char **argv);
 int cmd__trace2(int argc, const char **argv);
+int cmd__userdiff(int argc, const char **argv);
 int cmd__urlmatch_normalization(int argc, const char **argv);
 int cmd__xml_encode(int argc, const char **argv);
 int cmd__wildmatch(int argc, const char **argv);
diff --git a/t/helper/test-userdiff.c b/t/helper/test-userdiff.c
new file mode 100644
index 0000000000..1e17aeb265
--- /dev/null
+++ b/t/helper/test-userdiff.c
@@ -0,0 +1,30 @@
+#include "test-tool.h"
+#include "cache.h"
+#include "userdiff.h"
+
+static int driver_cb(struct userdiff_driver *driver,
+		     enum userdiff_driver_type type, void *priv)
+{
+	puts(driver->name);
+	return 0;
+}
+
+static int list_what(enum userdiff_driver_type type)
+{
+	return for_each_userdiff_driver(driver_cb, type, NULL);
+}
+
+int cmd__userdiff(int argc, const char **argv)
+{
+	if (argc != 2)
+		return 1;
+
+	if (!strcmp(argv[1], "list-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_UNSPECIFIED);
+	else if (!strcmp(argv[1], "list-builtin-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_BUILTIN);
+	else if (!strcmp(argv[1], "list-custom-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_CUSTOM);
+	else
+		return error("unknown argument %s", argv[1]);
+}
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index cefe329aea..6faa719536 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -8,6 +8,9 @@ test_description='Test custom diff function name patterns'
 . ./test-lib.sh
 
 test_expect_success 'setup' '
+	builtin_drivers=$(test-tool userdiff list-builtin-drivers) &&
+	test -n "$builtin_drivers" &&
+
 	# a non-trivial custom pattern
 	git config diff.custom1.funcname "!static
 !String
@@ -26,30 +29,7 @@ test_expect_success 'setup' '
 '
 
 diffpatterns="
-	ada
-	bash
-	bibtex
-	cpp
-	csharp
-	css
-	dts
-	elixir
-	fortran
-	fountain
-	golang
-	html
-	java
-	markdown
-	matlab
-	objc
-	pascal
-	perl
-	php
-	python
-	ruby
-	rust
-	tex
-	default
+	$builtin_drivers
 	custom1
 	custom2
 	custom3
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 08/27] userdiff: remove support for "broken" tests

[Ævar Arnfjörð Bjarmason]

There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve
Fortran xfuncname regex, 2020-08-12). Let's remove the test support
for them, this is in preparation for a more general refactoring of the
tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 8 +-------
 t/t4018/README           | 3 ---
 2 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 6faa719536..5994c5b47a 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -82,13 +82,7 @@ test_expect_success 'setup hunk header tests' '
 # check each individual file
 for i in $(git ls-files)
 do
-	if grep broken "$i" >/dev/null 2>&1
-	then
-		result=failure
-	else
-		result=success
-	fi
-	test_expect_$result "hunk header: $i" "
+	test_expect_success "hunk header: $i" "
 		git diff -U1 $i >actual &&
 		grep '@@ .* @@.*RIGHT' actual
 	"
diff --git a/t/t4018/README b/t/t4018/README
index 283e01cca1..2d25b2b4fc 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -7,9 +7,6 @@ at least two lines from the line that must appear in the hunk header.
 The text that must appear in the hunk header must contain the word
 "right", but in all upper-case, like in the title above.
 
-To mark a test case that highlights a malfunction, insert the word
-BROKEN in all lower-case somewhere in the file.
-
 This text is a bit twisted and out of order, but it is itself a
 test case for the default hunk header pattern. Know what you are doing
 if you change it.
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 09/27] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

Fix a regression in the test framework for userdiff added in
bfa7d01413 (t4018: an infrastructure to test hunk headers,
2014-03-21).

The testing infrastructure added in that change went overboard with
simplifying the tests, to the point where we lost test coverage.

Before that we'd been able to test the full context line, or ever
since the feature was originally added in f258475a6e (Per-path
attribute based hunk header selection., 2007-07-06).

After bfa7d01413 all we cared about was whether "RIGHT" appeared on
the line. We thus lost the information about whether or not "RIGHT"
was extracted from the line for the hunk header, or the line appeared
in full (or other subset of the line).

Let's bring back coverage for that by adding corresponding *.ctx
files, this has the added advantage that we're doing a "test_cmp", so
when we have failures it's just a non-zero exit code from "grep",
we'll actually have something meaningful in the "-v" output.

As we'll see in a follow-up commit this is an intermediate step
towards even better test coverage. I'm structuring these tests in such
a way as to benefit from the diff.colorMove detection.

The "sed -n -e" here was originally a single 's/^.*@@\( \|$\)//p'
pattern, but the '\( \|$\)' part had portability issues on OSX and
AIX.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh                      |  7 +++---
 t/t4018/README                                | 22 +++++++++----------
 t/t4018/README.ctx                            |  1 +
 t/t4018/bash-arithmetic-function.ctx          |  1 +
 t/t4018/bash-bashism-style-compact.ctx        |  1 +
 t/t4018/bash-bashism-style-function.ctx       |  1 +
 t/t4018/bash-bashism-style-whitespace.ctx     |  1 +
 t/t4018/bash-conditional-function.ctx         |  1 +
 t/t4018/bash-missing-parentheses.ctx          |  1 +
 t/t4018/bash-mixed-style-compact.ctx          |  1 +
 t/t4018/bash-mixed-style-function.ctx         |  1 +
 t/t4018/bash-nested-functions.ctx             |  1 +
 t/t4018/bash-other-characters.ctx             |  1 +
 t/t4018/bash-posix-style-compact.ctx          |  1 +
 t/t4018/bash-posix-style-function.ctx         |  1 +
 t/t4018/bash-posix-style-whitespace.ctx       |  1 +
 t/t4018/bash-subshell-function.ctx            |  1 +
 t/t4018/bash-trailing-comment.ctx             |  1 +
 t/t4018/cpp-c++-function.ctx                  |  1 +
 t/t4018/cpp-class-constructor-mem-init.ctx    |  1 +
 t/t4018/cpp-class-constructor.ctx             |  1 +
 t/t4018/cpp-class-definition-derived.ctx      |  1 +
 t/t4018/cpp-class-definition.ctx              |  1 +
 t/t4018/cpp-class-destructor.ctx              |  1 +
 .../cpp-function-returning-global-type.ctx    |  1 +
 t/t4018/cpp-function-returning-nested.ctx     |  1 +
 t/t4018/cpp-function-returning-pointer.ctx    |  1 +
 t/t4018/cpp-function-returning-reference.ctx  |  1 +
 t/t4018/cpp-gnu-style-function.ctx            |  1 +
 t/t4018/cpp-namespace-definition.ctx          |  1 +
 t/t4018/cpp-operator-definition.ctx           |  1 +
 t/t4018/cpp-skip-access-specifiers.ctx        |  1 +
 t/t4018/cpp-skip-comment-block.ctx            |  1 +
 t/t4018/cpp-skip-labels.ctx                   |  1 +
 t/t4018/cpp-struct-definition.ctx             |  1 +
 t/t4018/cpp-struct-single-line.ctx            |  1 +
 t/t4018/cpp-template-function-definition.ctx  |  1 +
 t/t4018/cpp-union-definition.ctx              |  1 +
 t/t4018/cpp-void-c-function.ctx               |  1 +
 t/t4018/css-attribute-value-selector.ctx      |  1 +
 t/t4018/css-block-level-@-statements.ctx      |  1 +
 t/t4018/css-brace-in-col-1.ctx                |  1 +
 t/t4018/css-class-selector.ctx                |  1 +
 t/t4018/css-colon-eol.ctx                     |  1 +
 t/t4018/css-colon-selector.ctx                |  1 +
 t/t4018/css-common.ctx                        |  1 +
 t/t4018/css-id-selector.ctx                   |  1 +
 t/t4018/css-long-selector-list.ctx            |  1 +
 t/t4018/css-prop-sans-indent.ctx              |  1 +
 t/t4018/css-root-selector.ctx                 |  1 +
 t/t4018/css-short-selector-list.ctx           |  1 +
 t/t4018/css-trailing-space.ctx                |  1 +
 t/t4018/custom1-pattern.ctx                   |  1 +
 t/t4018/custom2-match-to-end-of-line.ctx      |  1 +
 t/t4018/custom3-alternation-in-pattern.ctx    |  1 +
 t/t4018/dts-labels.ctx                        |  1 +
 t/t4018/dts-node-unitless.ctx                 |  1 +
 t/t4018/dts-nodes-boolean-prop.ctx            |  1 +
 t/t4018/dts-nodes-comment1.ctx                |  1 +
 t/t4018/dts-nodes-comment2.ctx                |  1 +
 t/t4018/dts-nodes-multiline-prop.ctx          |  1 +
 t/t4018/dts-nodes.ctx                         |  1 +
 t/t4018/dts-reference.ctx                     |  1 +
 t/t4018/dts-root-comment.ctx                  |  1 +
 t/t4018/dts-root.ctx                          |  1 +
 t/t4018/elixir-do-not-pick-end.ctx            |  1 +
 t/t4018/elixir-ex-unit-test.ctx               |  1 +
 t/t4018/elixir-function.ctx                   |  1 +
 t/t4018/elixir-macro.ctx                      |  1 +
 t/t4018/elixir-module-func.ctx                |  1 +
 t/t4018/elixir-module.ctx                     |  1 +
 t/t4018/elixir-nested-module.ctx              |  1 +
 t/t4018/elixir-private-function.ctx           |  1 +
 t/t4018/elixir-protocol-implementation.ctx    |  1 +
 t/t4018/elixir-protocol.ctx                   |  1 +
 t/t4018/fortran-block-data.ctx                |  1 +
 t/t4018/fortran-comment-keyword.ctx           |  1 +
 t/t4018/fortran-comment-legacy-star.ctx       |  1 +
 t/t4018/fortran-comment-legacy.ctx            |  1 +
 t/t4018/fortran-comment.ctx                   |  1 +
 t/t4018/fortran-external-function.ctx         |  1 +
 t/t4018/fortran-external-subroutine.ctx       |  1 +
 t/t4018/fortran-module-procedure.ctx          |  1 +
 t/t4018/fortran-module.ctx                    |  1 +
 t/t4018/fortran-program.ctx                   |  1 +
 t/t4018/fountain-scene.ctx                    |  1 +
 t/t4018/golang-complex-function.ctx           |  1 +
 t/t4018/golang-func.ctx                       |  1 +
 t/t4018/golang-interface.ctx                  |  1 +
 t/t4018/golang-long-func.ctx                  |  1 +
 t/t4018/golang-struct.ctx                     |  1 +
 t/t4018/java-class-member-function.ctx        |  1 +
 t/t4018/markdown-heading-indented.ctx         |  1 +
 t/t4018/markdown-heading-non-headings.ctx     |  1 +
 t/t4018/matlab-class-definition.ctx           |  1 +
 t/t4018/matlab-function.ctx                   |  1 +
 t/t4018/matlab-octave-section-1.ctx           |  1 +
 t/t4018/matlab-octave-section-2.ctx           |  1 +
 t/t4018/matlab-section.ctx                    |  1 +
 t/t4018/perl-skip-end-of-heredoc.ctx          |  1 +
 t/t4018/perl-skip-forward-decl.ctx            |  1 +
 t/t4018/perl-skip-sub-in-pod.ctx              |  1 +
 t/t4018/perl-sub-definition-kr-brace.ctx      |  1 +
 t/t4018/perl-sub-definition.ctx               |  1 +
 t/t4018/php-abstract-class.ctx                |  1 +
 t/t4018/php-abstract-method.ctx               |  1 +
 t/t4018/php-class.ctx                         |  1 +
 t/t4018/php-final-class.ctx                   |  1 +
 t/t4018/php-final-method.ctx                  |  1 +
 t/t4018/php-function.ctx                      |  1 +
 t/t4018/php-interface.ctx                     |  1 +
 t/t4018/php-method.ctx                        |  1 +
 t/t4018/php-trait.ctx                         |  1 +
 t/t4018/python-async-def.ctx                  |  1 +
 t/t4018/python-class.ctx                      |  1 +
 t/t4018/python-def.ctx                        |  1 +
 t/t4018/python-indented-async-def.ctx         |  1 +
 t/t4018/python-indented-class.ctx             |  1 +
 t/t4018/python-indented-def.ctx               |  1 +
 t/t4018/rust-fn.ctx                           |  1 +
 t/t4018/rust-impl.ctx                         |  1 +
 t/t4018/rust-macro-rules.ctx                  |  1 +
 t/t4018/rust-struct.ctx                       |  1 +
 t/t4018/rust-trait.ctx                        |  1 +
 124 files changed, 137 insertions(+), 14 deletions(-)
 create mode 100644 t/t4018/README.ctx
 create mode 100644 t/t4018/bash-arithmetic-function.ctx
 create mode 100644 t/t4018/bash-bashism-style-compact.ctx
 create mode 100644 t/t4018/bash-bashism-style-function.ctx
 create mode 100644 t/t4018/bash-bashism-style-whitespace.ctx
 create mode 100644 t/t4018/bash-conditional-function.ctx
 create mode 100644 t/t4018/bash-missing-parentheses.ctx
 create mode 100644 t/t4018/bash-mixed-style-compact.ctx
 create mode 100644 t/t4018/bash-mixed-style-function.ctx
 create mode 100644 t/t4018/bash-nested-functions.ctx
 create mode 100644 t/t4018/bash-other-characters.ctx
 create mode 100644 t/t4018/bash-posix-style-compact.ctx
 create mode 100644 t/t4018/bash-posix-style-function.ctx
 create mode 100644 t/t4018/bash-posix-style-whitespace.ctx
 create mode 100644 t/t4018/bash-subshell-function.ctx
 create mode 100644 t/t4018/bash-trailing-comment.ctx
 create mode 100644 t/t4018/cpp-c++-function.ctx
 create mode 100644 t/t4018/cpp-class-constructor-mem-init.ctx
 create mode 100644 t/t4018/cpp-class-constructor.ctx
 create mode 100644 t/t4018/cpp-class-definition-derived.ctx
 create mode 100644 t/t4018/cpp-class-definition.ctx
 create mode 100644 t/t4018/cpp-class-destructor.ctx
 create mode 100644 t/t4018/cpp-function-returning-global-type.ctx
 create mode 100644 t/t4018/cpp-function-returning-nested.ctx
 create mode 100644 t/t4018/cpp-function-returning-pointer.ctx
 create mode 100644 t/t4018/cpp-function-returning-reference.ctx
 create mode 100644 t/t4018/cpp-gnu-style-function.ctx
 create mode 100644 t/t4018/cpp-namespace-definition.ctx
 create mode 100644 t/t4018/cpp-operator-definition.ctx
 create mode 100644 t/t4018/cpp-skip-access-specifiers.ctx
 create mode 100644 t/t4018/cpp-skip-comment-block.ctx
 create mode 100644 t/t4018/cpp-skip-labels.ctx
 create mode 100644 t/t4018/cpp-struct-definition.ctx
 create mode 100644 t/t4018/cpp-struct-single-line.ctx
 create mode 100644 t/t4018/cpp-template-function-definition.ctx
 create mode 100644 t/t4018/cpp-union-definition.ctx
 create mode 100644 t/t4018/cpp-void-c-function.ctx
 create mode 100644 t/t4018/css-attribute-value-selector.ctx
 create mode 100644 t/t4018/css-block-level-@-statements.ctx
 create mode 100644 t/t4018/css-brace-in-col-1.ctx
 create mode 100644 t/t4018/css-class-selector.ctx
 create mode 100644 t/t4018/css-colon-eol.ctx
 create mode 100644 t/t4018/css-colon-selector.ctx
 create mode 100644 t/t4018/css-common.ctx
 create mode 100644 t/t4018/css-id-selector.ctx
 create mode 100644 t/t4018/css-long-selector-list.ctx
 create mode 100644 t/t4018/css-prop-sans-indent.ctx
 create mode 100644 t/t4018/css-root-selector.ctx
 create mode 100644 t/t4018/css-short-selector-list.ctx
 create mode 100644 t/t4018/css-trailing-space.ctx
 create mode 100644 t/t4018/custom1-pattern.ctx
 create mode 100644 t/t4018/custom2-match-to-end-of-line.ctx
 create mode 100644 t/t4018/custom3-alternation-in-pattern.ctx
 create mode 100644 t/t4018/dts-labels.ctx
 create mode 100644 t/t4018/dts-node-unitless.ctx
 create mode 100644 t/t4018/dts-nodes-boolean-prop.ctx
 create mode 100644 t/t4018/dts-nodes-comment1.ctx
 create mode 100644 t/t4018/dts-nodes-comment2.ctx
 create mode 100644 t/t4018/dts-nodes-multiline-prop.ctx
 create mode 100644 t/t4018/dts-nodes.ctx
 create mode 100644 t/t4018/dts-reference.ctx
 create mode 100644 t/t4018/dts-root-comment.ctx
 create mode 100644 t/t4018/dts-root.ctx
 create mode 100644 t/t4018/elixir-do-not-pick-end.ctx
 create mode 100644 t/t4018/elixir-ex-unit-test.ctx
 create mode 100644 t/t4018/elixir-function.ctx
 create mode 100644 t/t4018/elixir-macro.ctx
 create mode 100644 t/t4018/elixir-module-func.ctx
 create mode 100644 t/t4018/elixir-module.ctx
 create mode 100644 t/t4018/elixir-nested-module.ctx
 create mode 100644 t/t4018/elixir-private-function.ctx
 create mode 100644 t/t4018/elixir-protocol-implementation.ctx
 create mode 100644 t/t4018/elixir-protocol.ctx
 create mode 100644 t/t4018/fortran-block-data.ctx
 create mode 100644 t/t4018/fortran-comment-keyword.ctx
 create mode 100644 t/t4018/fortran-comment-legacy-star.ctx
 create mode 100644 t/t4018/fortran-comment-legacy.ctx
 create mode 100644 t/t4018/fortran-comment.ctx
 create mode 100644 t/t4018/fortran-external-function.ctx
 create mode 100644 t/t4018/fortran-external-subroutine.ctx
 create mode 100644 t/t4018/fortran-module-procedure.ctx
 create mode 100644 t/t4018/fortran-module.ctx
 create mode 100644 t/t4018/fortran-program.ctx
 create mode 100644 t/t4018/fountain-scene.ctx
 create mode 100644 t/t4018/golang-complex-function.ctx
 create mode 100644 t/t4018/golang-func.ctx
 create mode 100644 t/t4018/golang-interface.ctx
 create mode 100644 t/t4018/golang-long-func.ctx
 create mode 100644 t/t4018/golang-struct.ctx
 create mode 100644 t/t4018/java-class-member-function.ctx
 create mode 100644 t/t4018/markdown-heading-indented.ctx
 create mode 100644 t/t4018/markdown-heading-non-headings.ctx
 create mode 100644 t/t4018/matlab-class-definition.ctx
 create mode 100644 t/t4018/matlab-function.ctx
 create mode 100644 t/t4018/matlab-octave-section-1.ctx
 create mode 100644 t/t4018/matlab-octave-section-2.ctx
 create mode 100644 t/t4018/matlab-section.ctx
 create mode 100644 t/t4018/perl-skip-end-of-heredoc.ctx
 create mode 100644 t/t4018/perl-skip-forward-decl.ctx
 create mode 100644 t/t4018/perl-skip-sub-in-pod.ctx
 create mode 100644 t/t4018/perl-sub-definition-kr-brace.ctx
 create mode 100644 t/t4018/perl-sub-definition.ctx
 create mode 100644 t/t4018/php-abstract-class.ctx
 create mode 100644 t/t4018/php-abstract-method.ctx
 create mode 100644 t/t4018/php-class.ctx
 create mode 100644 t/t4018/php-final-class.ctx
 create mode 100644 t/t4018/php-final-method.ctx
 create mode 100644 t/t4018/php-function.ctx
 create mode 100644 t/t4018/php-interface.ctx
 create mode 100644 t/t4018/php-method.ctx
 create mode 100644 t/t4018/php-trait.ctx
 create mode 100644 t/t4018/python-async-def.ctx
 create mode 100644 t/t4018/python-class.ctx
 create mode 100644 t/t4018/python-def.ctx
 create mode 100644 t/t4018/python-indented-async-def.ctx
 create mode 100644 t/t4018/python-indented-class.ctx
 create mode 100644 t/t4018/python-indented-def.ctx
 create mode 100644 t/t4018/rust-fn.ctx
 create mode 100644 t/t4018/rust-impl.ctx
 create mode 100644 t/t4018/rust-macro-rules.ctx
 create mode 100644 t/t4018/rust-struct.ctx
 create mode 100644 t/t4018/rust-trait.ctx

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 5994c5b47a..3941316682 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -80,11 +80,12 @@ test_expect_success 'setup hunk header tests' '
 '
 
 # check each individual file
-for i in $(git ls-files)
+for i in $(git ls-files -- ':!*.ctx')
 do
 	test_expect_success "hunk header: $i" "
-		git diff -U1 $i >actual &&
-		grep '@@ .* @@.*RIGHT' actual
+		git diff -U1 $i >diff &&
+		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
+		test_cmp $i.ctx ctx
 	"
 done
 
diff --git a/t/t4018/README b/t/t4018/README
index 2d25b2b4fc..d0619f76d4 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -1,15 +1,15 @@
-How to write RIGHT test cases
-=============================
+How to write test cases
+=======================
+
+Create test cases called "LANG-whatever" in this directory, where
+"LANG" is e.g. the userdiff driver name, where "whatever" is a brief
+description of the test.
 
 Insert the word "ChangeMe" (exactly this form) at a distance of
 at least two lines from the line that must appear in the hunk header.
 
-The text that must appear in the hunk header must contain the word
-"right", but in all upper-case, like in the title above.
-
-This text is a bit twisted and out of order, but it is itself a
-test case for the default hunk header pattern. Know what you are doing
-if you change it.
-
-BTW, this tests that the head line goes to the hunk header, not the line
-of equal signs.
+The text that must appear in the hunk header must contains the word
+"RIGHT" by convention. The "LANG-whatever.ctx" file contains what we
+expect to appear in the hunk header. We munged away the starting "@@
+[...] @@" part of the line for ease of not having to hardcode the line
+numbers and offsets.
diff --git a/t/t4018/README.ctx b/t/t4018/README.ctx
new file mode 100644
index 0000000000..cd79384b04
--- /dev/null
+++ b/t/t4018/README.ctx
@@ -0,0 +1 @@
+description of the test.
diff --git a/t/t4018/bash-arithmetic-function.ctx b/t/t4018/bash-arithmetic-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-arithmetic-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-bashism-style-compact.ctx b/t/t4018/bash-bashism-style-compact.ctx
new file mode 100644
index 0000000000..4f8eac48c6
--- /dev/null
+++ b/t/t4018/bash-bashism-style-compact.ctx
@@ -0,0 +1 @@
+function RIGHT {
diff --git a/t/t4018/bash-bashism-style-function.ctx b/t/t4018/bash-bashism-style-function.ctx
new file mode 100644
index 0000000000..4f8eac48c6
--- /dev/null
+++ b/t/t4018/bash-bashism-style-function.ctx
@@ -0,0 +1 @@
+function RIGHT {
diff --git a/t/t4018/bash-bashism-style-whitespace.ctx b/t/t4018/bash-bashism-style-whitespace.ctx
new file mode 100644
index 0000000000..35dbd0220e
--- /dev/null
+++ b/t/t4018/bash-bashism-style-whitespace.ctx
@@ -0,0 +1 @@
+function 	RIGHT 	( 	) 	{
diff --git a/t/t4018/bash-conditional-function.ctx b/t/t4018/bash-conditional-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-conditional-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-missing-parentheses.ctx b/t/t4018/bash-missing-parentheses.ctx
new file mode 100644
index 0000000000..4f8eac48c6
--- /dev/null
+++ b/t/t4018/bash-missing-parentheses.ctx
@@ -0,0 +1 @@
+function RIGHT {
diff --git a/t/t4018/bash-mixed-style-compact.ctx b/t/t4018/bash-mixed-style-compact.ctx
new file mode 100644
index 0000000000..bba11074eb
--- /dev/null
+++ b/t/t4018/bash-mixed-style-compact.ctx
@@ -0,0 +1 @@
+function RIGHT(){
diff --git a/t/t4018/bash-mixed-style-function.ctx b/t/t4018/bash-mixed-style-function.ctx
new file mode 100644
index 0000000000..922b87a4aa
--- /dev/null
+++ b/t/t4018/bash-mixed-style-function.ctx
@@ -0,0 +1 @@
+function RIGHT() {
diff --git a/t/t4018/bash-nested-functions.ctx b/t/t4018/bash-nested-functions.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-nested-functions.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-other-characters.ctx b/t/t4018/bash-other-characters.ctx
new file mode 100644
index 0000000000..6a55317fdf
--- /dev/null
+++ b/t/t4018/bash-other-characters.ctx
@@ -0,0 +1 @@
+_RIGHT_0n()
diff --git a/t/t4018/bash-posix-style-compact.ctx b/t/t4018/bash-posix-style-compact.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-posix-style-compact.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-posix-style-function.ctx b/t/t4018/bash-posix-style-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-posix-style-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-posix-style-whitespace.ctx b/t/t4018/bash-posix-style-whitespace.ctx
new file mode 100644
index 0000000000..28f8698e14
--- /dev/null
+++ b/t/t4018/bash-posix-style-whitespace.ctx
@@ -0,0 +1 @@
+RIGHT 	( 	)
diff --git a/t/t4018/bash-subshell-function.ctx b/t/t4018/bash-subshell-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-subshell-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-trailing-comment.ctx b/t/t4018/bash-trailing-comment.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-trailing-comment.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/cpp-c++-function.ctx b/t/t4018/cpp-c++-function.ctx
new file mode 100644
index 0000000000..337b49dbb3
--- /dev/null
+++ b/t/t4018/cpp-c++-function.ctx
@@ -0,0 +1 @@
+Item RIGHT::DoSomething( Args with_spaces )
diff --git a/t/t4018/cpp-class-constructor-mem-init.ctx b/t/t4018/cpp-class-constructor-mem-init.ctx
new file mode 100644
index 0000000000..6664b8b3d8
--- /dev/null
+++ b/t/t4018/cpp-class-constructor-mem-init.ctx
@@ -0,0 +1 @@
+Item::Item(int RIGHT) :
diff --git a/t/t4018/cpp-class-constructor.ctx b/t/t4018/cpp-class-constructor.ctx
new file mode 100644
index 0000000000..2dcadfc0ba
--- /dev/null
+++ b/t/t4018/cpp-class-constructor.ctx
@@ -0,0 +1 @@
+Item::Item(int RIGHT)
diff --git a/t/t4018/cpp-class-definition-derived.ctx b/t/t4018/cpp-class-definition-derived.ctx
new file mode 100644
index 0000000000..146f0a7b7c
--- /dev/null
+++ b/t/t4018/cpp-class-definition-derived.ctx
@@ -0,0 +1 @@
+class RIGHT :
diff --git a/t/t4018/cpp-class-definition.ctx b/t/t4018/cpp-class-definition.ctx
new file mode 100644
index 0000000000..54bff816d6
--- /dev/null
+++ b/t/t4018/cpp-class-definition.ctx
@@ -0,0 +1 @@
+class RIGHT
diff --git a/t/t4018/cpp-class-destructor.ctx b/t/t4018/cpp-class-destructor.ctx
new file mode 100644
index 0000000000..5390c17cf6
--- /dev/null
+++ b/t/t4018/cpp-class-destructor.ctx
@@ -0,0 +1 @@
+RIGHT::~RIGHT()
diff --git a/t/t4018/cpp-function-returning-global-type.ctx b/t/t4018/cpp-function-returning-global-type.ctx
new file mode 100644
index 0000000000..4dcdde25f4
--- /dev/null
+++ b/t/t4018/cpp-function-returning-global-type.ctx
@@ -0,0 +1 @@
+::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-nested.ctx b/t/t4018/cpp-function-returning-nested.ctx
new file mode 100644
index 0000000000..6ef73c8368
--- /dev/null
+++ b/t/t4018/cpp-function-returning-nested.ctx
@@ -0,0 +1 @@
+get::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-pointer.ctx b/t/t4018/cpp-function-returning-pointer.ctx
new file mode 100644
index 0000000000..bb0acce5c7
--- /dev/null
+++ b/t/t4018/cpp-function-returning-pointer.ctx
@@ -0,0 +1 @@
+const char *get_it_RIGHT(char *ptr)
diff --git a/t/t4018/cpp-function-returning-reference.ctx b/t/t4018/cpp-function-returning-reference.ctx
new file mode 100644
index 0000000000..76afe381fd
--- /dev/null
+++ b/t/t4018/cpp-function-returning-reference.ctx
@@ -0,0 +1 @@
+string& get::it::RIGHT(char *ptr)
diff --git a/t/t4018/cpp-gnu-style-function.ctx b/t/t4018/cpp-gnu-style-function.ctx
new file mode 100644
index 0000000000..1858287812
--- /dev/null
+++ b/t/t4018/cpp-gnu-style-function.ctx
@@ -0,0 +1 @@
+RIGHT(int arg)
diff --git a/t/t4018/cpp-namespace-definition.ctx b/t/t4018/cpp-namespace-definition.ctx
new file mode 100644
index 0000000000..14c29c4638
--- /dev/null
+++ b/t/t4018/cpp-namespace-definition.ctx
@@ -0,0 +1 @@
+namespace RIGHT
diff --git a/t/t4018/cpp-operator-definition.ctx b/t/t4018/cpp-operator-definition.ctx
new file mode 100644
index 0000000000..5b56778961
--- /dev/null
+++ b/t/t4018/cpp-operator-definition.ctx
@@ -0,0 +1 @@
+Value operator+(Value LEFT, Value RIGHT)
diff --git a/t/t4018/cpp-skip-access-specifiers.ctx b/t/t4018/cpp-skip-access-specifiers.ctx
new file mode 100644
index 0000000000..075bcd883b
--- /dev/null
+++ b/t/t4018/cpp-skip-access-specifiers.ctx
@@ -0,0 +1 @@
+class RIGHT : public Baseclass
diff --git a/t/t4018/cpp-skip-comment-block.ctx b/t/t4018/cpp-skip-comment-block.ctx
new file mode 100644
index 0000000000..656c59893d
--- /dev/null
+++ b/t/t4018/cpp-skip-comment-block.ctx
@@ -0,0 +1 @@
+struct item RIGHT(int i)
diff --git a/t/t4018/cpp-skip-labels.ctx b/t/t4018/cpp-skip-labels.ctx
new file mode 100644
index 0000000000..6b0635f7f7
--- /dev/null
+++ b/t/t4018/cpp-skip-labels.ctx
@@ -0,0 +1 @@
+void RIGHT (void)
diff --git a/t/t4018/cpp-struct-definition.ctx b/t/t4018/cpp-struct-definition.ctx
new file mode 100644
index 0000000000..48ed893279
--- /dev/null
+++ b/t/t4018/cpp-struct-definition.ctx
@@ -0,0 +1 @@
+struct RIGHT {
diff --git a/t/t4018/cpp-struct-single-line.ctx b/t/t4018/cpp-struct-single-line.ctx
new file mode 100644
index 0000000000..e3bc9d5017
--- /dev/null
+++ b/t/t4018/cpp-struct-single-line.ctx
@@ -0,0 +1 @@
+struct RIGHT_iterator_tag {};
diff --git a/t/t4018/cpp-template-function-definition.ctx b/t/t4018/cpp-template-function-definition.ctx
new file mode 100644
index 0000000000..c9da39cf65
--- /dev/null
+++ b/t/t4018/cpp-template-function-definition.ctx
@@ -0,0 +1 @@
+template<class T> int RIGHT(T arg)
diff --git a/t/t4018/cpp-union-definition.ctx b/t/t4018/cpp-union-definition.ctx
new file mode 100644
index 0000000000..2fc7b54fb8
--- /dev/null
+++ b/t/t4018/cpp-union-definition.ctx
@@ -0,0 +1 @@
+union RIGHT {
diff --git a/t/t4018/cpp-void-c-function.ctx b/t/t4018/cpp-void-c-function.ctx
new file mode 100644
index 0000000000..6b0635f7f7
--- /dev/null
+++ b/t/t4018/cpp-void-c-function.ctx
@@ -0,0 +1 @@
+void RIGHT (void)
diff --git a/t/t4018/css-attribute-value-selector.ctx b/t/t4018/css-attribute-value-selector.ctx
new file mode 100644
index 0000000000..7f8956251c
--- /dev/null
+++ b/t/t4018/css-attribute-value-selector.ctx
@@ -0,0 +1 @@
+[class*="RIGHT"] {
diff --git a/t/t4018/css-block-level-@-statements.ctx b/t/t4018/css-block-level-@-statements.ctx
new file mode 100644
index 0000000000..7f5e90468c
--- /dev/null
+++ b/t/t4018/css-block-level-@-statements.ctx
@@ -0,0 +1 @@
+@keyframes RIGHT {
diff --git a/t/t4018/css-brace-in-col-1.ctx b/t/t4018/css-brace-in-col-1.ctx
new file mode 100644
index 0000000000..91a9105c6a
--- /dev/null
+++ b/t/t4018/css-brace-in-col-1.ctx
@@ -0,0 +1 @@
+RIGHT label.control-label
diff --git a/t/t4018/css-class-selector.ctx b/t/t4018/css-class-selector.ctx
new file mode 100644
index 0000000000..ac7367d7f4
--- /dev/null
+++ b/t/t4018/css-class-selector.ctx
@@ -0,0 +1 @@
+.RIGHT {
diff --git a/t/t4018/css-colon-eol.ctx b/t/t4018/css-colon-eol.ctx
new file mode 100644
index 0000000000..b68493b9b0
--- /dev/null
+++ b/t/t4018/css-colon-eol.ctx
@@ -0,0 +1 @@
+RIGHT h1 {
diff --git a/t/t4018/css-colon-selector.ctx b/t/t4018/css-colon-selector.ctx
new file mode 100644
index 0000000000..00b1a5aefe
--- /dev/null
+++ b/t/t4018/css-colon-selector.ctx
@@ -0,0 +1 @@
+RIGHT a:hover {
diff --git a/t/t4018/css-common.ctx b/t/t4018/css-common.ctx
new file mode 100644
index 0000000000..43686b4081
--- /dev/null
+++ b/t/t4018/css-common.ctx
@@ -0,0 +1 @@
+RIGHT label.control-label {
diff --git a/t/t4018/css-id-selector.ctx b/t/t4018/css-id-selector.ctx
new file mode 100644
index 0000000000..ce19f6d8dc
--- /dev/null
+++ b/t/t4018/css-id-selector.ctx
@@ -0,0 +1 @@
+#RIGHT {
diff --git a/t/t4018/css-long-selector-list.ctx b/t/t4018/css-long-selector-list.ctx
new file mode 100644
index 0000000000..bc8d0fb62c
--- /dev/null
+++ b/t/t4018/css-long-selector-list.ctx
@@ -0,0 +1 @@
+div ul#RIGHT {
diff --git a/t/t4018/css-prop-sans-indent.ctx b/t/t4018/css-prop-sans-indent.ctx
new file mode 100644
index 0000000000..cc880b2f44
--- /dev/null
+++ b/t/t4018/css-prop-sans-indent.ctx
@@ -0,0 +1 @@
+RIGHT, label.control-label {
diff --git a/t/t4018/css-root-selector.ctx b/t/t4018/css-root-selector.ctx
new file mode 100644
index 0000000000..3010cded2a
--- /dev/null
+++ b/t/t4018/css-root-selector.ctx
@@ -0,0 +1 @@
+:RIGHT {
diff --git a/t/t4018/css-short-selector-list.ctx b/t/t4018/css-short-selector-list.ctx
new file mode 100644
index 0000000000..9e5d87d126
--- /dev/null
+++ b/t/t4018/css-short-selector-list.ctx
@@ -0,0 +1 @@
+label.control, div ul#RIGHT {
diff --git a/t/t4018/css-trailing-space.ctx b/t/t4018/css-trailing-space.ctx
new file mode 100644
index 0000000000..43686b4081
--- /dev/null
+++ b/t/t4018/css-trailing-space.ctx
@@ -0,0 +1 @@
+RIGHT label.control-label {
diff --git a/t/t4018/custom1-pattern.ctx b/t/t4018/custom1-pattern.ctx
new file mode 100644
index 0000000000..d1609cc9a6
--- /dev/null
+++ b/t/t4018/custom1-pattern.ctx
@@ -0,0 +1 @@
+int special, RIGHT;
diff --git a/t/t4018/custom2-match-to-end-of-line.ctx b/t/t4018/custom2-match-to-end-of-line.ctx
new file mode 100644
index 0000000000..8294c6e49b
--- /dev/null
+++ b/t/t4018/custom2-match-to-end-of-line.ctx
@@ -0,0 +1 @@
+RIGHT_Beer
diff --git a/t/t4018/custom3-alternation-in-pattern.ctx b/t/t4018/custom3-alternation-in-pattern.ctx
new file mode 100644
index 0000000000..2125474b68
--- /dev/null
+++ b/t/t4018/custom3-alternation-in-pattern.ctx
@@ -0,0 +1 @@
+public static void main(String RIGHT[])
diff --git a/t/t4018/dts-labels.ctx b/t/t4018/dts-labels.ctx
new file mode 100644
index 0000000000..48d9373cab
--- /dev/null
+++ b/t/t4018/dts-labels.ctx
@@ -0,0 +1 @@
+label2: RIGHT {
diff --git a/t/t4018/dts-node-unitless.ctx b/t/t4018/dts-node-unitless.ctx
new file mode 100644
index 0000000000..82c8683fa1
--- /dev/null
+++ b/t/t4018/dts-node-unitless.ctx
@@ -0,0 +1 @@
+RIGHT {
diff --git a/t/t4018/dts-nodes-boolean-prop.ctx b/t/t4018/dts-nodes-boolean-prop.ctx
new file mode 100644
index 0000000000..3a0232d55d
--- /dev/null
+++ b/t/t4018/dts-nodes-boolean-prop.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes-comment1.ctx b/t/t4018/dts-nodes-comment1.ctx
new file mode 100644
index 0000000000..ec364600b1
--- /dev/null
+++ b/t/t4018/dts-nodes-comment1.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 /* &a comment */ {
diff --git a/t/t4018/dts-nodes-comment2.ctx b/t/t4018/dts-nodes-comment2.ctx
new file mode 100644
index 0000000000..75f0d75258
--- /dev/null
+++ b/t/t4018/dts-nodes-comment2.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 { /* a trailing comment */
diff --git a/t/t4018/dts-nodes-multiline-prop.ctx b/t/t4018/dts-nodes-multiline-prop.ctx
new file mode 100644
index 0000000000..3a0232d55d
--- /dev/null
+++ b/t/t4018/dts-nodes-multiline-prop.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes.ctx b/t/t4018/dts-nodes.ctx
new file mode 100644
index 0000000000..3a0232d55d
--- /dev/null
+++ b/t/t4018/dts-nodes.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-reference.ctx b/t/t4018/dts-reference.ctx
new file mode 100644
index 0000000000..c1e13409ee
--- /dev/null
+++ b/t/t4018/dts-reference.ctx
@@ -0,0 +1 @@
+&RIGHT {
diff --git a/t/t4018/dts-root-comment.ctx b/t/t4018/dts-root-comment.ctx
new file mode 100644
index 0000000000..656053dd42
--- /dev/null
+++ b/t/t4018/dts-root-comment.ctx
@@ -0,0 +1 @@
+/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/dts-root.ctx b/t/t4018/dts-root.ctx
new file mode 100644
index 0000000000..656053dd42
--- /dev/null
+++ b/t/t4018/dts-root.ctx
@@ -0,0 +1 @@
+/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/elixir-do-not-pick-end.ctx b/t/t4018/elixir-do-not-pick-end.ctx
new file mode 100644
index 0000000000..8f28a7a689
--- /dev/null
+++ b/t/t4018/elixir-do-not-pick-end.ctx
@@ -0,0 +1 @@
+defmodule RIGHT do
diff --git a/t/t4018/elixir-ex-unit-test.ctx b/t/t4018/elixir-ex-unit-test.ctx
new file mode 100644
index 0000000000..a55e3de2cc
--- /dev/null
+++ b/t/t4018/elixir-ex-unit-test.ctx
@@ -0,0 +1 @@
+test "RIGHT" do
diff --git a/t/t4018/elixir-function.ctx b/t/t4018/elixir-function.ctx
new file mode 100644
index 0000000000..62aee9c8b1
--- /dev/null
+++ b/t/t4018/elixir-function.ctx
@@ -0,0 +1 @@
+def function(RIGHT, arg) do
diff --git a/t/t4018/elixir-macro.ctx b/t/t4018/elixir-macro.ctx
new file mode 100644
index 0000000000..fc1d3b85e8
--- /dev/null
+++ b/t/t4018/elixir-macro.ctx
@@ -0,0 +1 @@
+defmacro foo(RIGHT) do
diff --git a/t/t4018/elixir-module-func.ctx b/t/t4018/elixir-module-func.ctx
new file mode 100644
index 0000000000..8239214386
--- /dev/null
+++ b/t/t4018/elixir-module-func.ctx
@@ -0,0 +1 @@
+def fun(RIGHT) do
diff --git a/t/t4018/elixir-module.ctx b/t/t4018/elixir-module.ctx
new file mode 100644
index 0000000000..8f28a7a689
--- /dev/null
+++ b/t/t4018/elixir-module.ctx
@@ -0,0 +1 @@
+defmodule RIGHT do
diff --git a/t/t4018/elixir-nested-module.ctx b/t/t4018/elixir-nested-module.ctx
new file mode 100644
index 0000000000..3ffbdd18b1
--- /dev/null
+++ b/t/t4018/elixir-nested-module.ctx
@@ -0,0 +1 @@
+defmodule MyApp.RIGHT do
diff --git a/t/t4018/elixir-private-function.ctx b/t/t4018/elixir-private-function.ctx
new file mode 100644
index 0000000000..1c4eba44f7
--- /dev/null
+++ b/t/t4018/elixir-private-function.ctx
@@ -0,0 +1 @@
+defp function(RIGHT, arg) do
diff --git a/t/t4018/elixir-protocol-implementation.ctx b/t/t4018/elixir-protocol-implementation.ctx
new file mode 100644
index 0000000000..efb758aea6
--- /dev/null
+++ b/t/t4018/elixir-protocol-implementation.ctx
@@ -0,0 +1 @@
+defimpl RIGHT do
diff --git a/t/t4018/elixir-protocol.ctx b/t/t4018/elixir-protocol.ctx
new file mode 100644
index 0000000000..d0204e9f14
--- /dev/null
+++ b/t/t4018/elixir-protocol.ctx
@@ -0,0 +1 @@
+defprotocol RIGHT do
diff --git a/t/t4018/fortran-block-data.ctx b/t/t4018/fortran-block-data.ctx
new file mode 100644
index 0000000000..c3db084ccc
--- /dev/null
+++ b/t/t4018/fortran-block-data.ctx
@@ -0,0 +1 @@
+BLOCK DATA RIGHT
diff --git a/t/t4018/fortran-comment-keyword.ctx b/t/t4018/fortran-comment-keyword.ctx
new file mode 100644
index 0000000000..0b9220b355
--- /dev/null
+++ b/t/t4018/fortran-comment-keyword.ctx
@@ -0,0 +1 @@
+subroutine RIGHT (funcA, funcB)
diff --git a/t/t4018/fortran-comment-legacy-star.ctx b/t/t4018/fortran-comment-legacy-star.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-comment-legacy-star.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-comment-legacy.ctx b/t/t4018/fortran-comment-legacy.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-comment-legacy.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-comment.ctx b/t/t4018/fortran-comment.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-comment.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-external-function.ctx b/t/t4018/fortran-external-function.ctx
new file mode 100644
index 0000000000..56ec4d8eca
--- /dev/null
+++ b/t/t4018/fortran-external-function.ctx
@@ -0,0 +1 @@
+function RIGHT(a, b) result(c)
diff --git a/t/t4018/fortran-external-subroutine.ctx b/t/t4018/fortran-external-subroutine.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-external-subroutine.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-module-procedure.ctx b/t/t4018/fortran-module-procedure.ctx
new file mode 100644
index 0000000000..4f5ff2e4b8
--- /dev/null
+++ b/t/t4018/fortran-module-procedure.ctx
@@ -0,0 +1 @@
+module RIGHT
diff --git a/t/t4018/fortran-module.ctx b/t/t4018/fortran-module.ctx
new file mode 100644
index 0000000000..4f5ff2e4b8
--- /dev/null
+++ b/t/t4018/fortran-module.ctx
@@ -0,0 +1 @@
+module RIGHT
diff --git a/t/t4018/fortran-program.ctx b/t/t4018/fortran-program.ctx
new file mode 100644
index 0000000000..c4e844df30
--- /dev/null
+++ b/t/t4018/fortran-program.ctx
@@ -0,0 +1 @@
+program RIGHT
diff --git a/t/t4018/fountain-scene.ctx b/t/t4018/fountain-scene.ctx
new file mode 100644
index 0000000000..bf10171418
--- /dev/null
+++ b/t/t4018/fountain-scene.ctx
@@ -0,0 +1 @@
+EXT. STREET RIGHT OUTSIDE - DAY
diff --git a/t/t4018/golang-complex-function.ctx b/t/t4018/golang-complex-function.ctx
new file mode 100644
index 0000000000..8e8d5582ff
--- /dev/null
+++ b/t/t4018/golang-complex-function.ctx
@@ -0,0 +1 @@
+func (t *Test) RIGHT(a Type) (Type, error) {
diff --git a/t/t4018/golang-func.ctx b/t/t4018/golang-func.ctx
new file mode 100644
index 0000000000..88bc823813
--- /dev/null
+++ b/t/t4018/golang-func.ctx
@@ -0,0 +1 @@
+func RIGHT() {
diff --git a/t/t4018/golang-interface.ctx b/t/t4018/golang-interface.ctx
new file mode 100644
index 0000000000..2d07f5a383
--- /dev/null
+++ b/t/t4018/golang-interface.ctx
@@ -0,0 +1 @@
+type RIGHT interface {
diff --git a/t/t4018/golang-long-func.ctx b/t/t4018/golang-long-func.ctx
new file mode 100644
index 0000000000..25635e712e
--- /dev/null
+++ b/t/t4018/golang-long-func.ctx
@@ -0,0 +1 @@
+func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
diff --git a/t/t4018/golang-struct.ctx b/t/t4018/golang-struct.ctx
new file mode 100644
index 0000000000..8a1240699d
--- /dev/null
+++ b/t/t4018/golang-struct.ctx
@@ -0,0 +1 @@
+type RIGHT struct {
diff --git a/t/t4018/java-class-member-function.ctx b/t/t4018/java-class-member-function.ctx
new file mode 100644
index 0000000000..2125474b68
--- /dev/null
+++ b/t/t4018/java-class-member-function.ctx
@@ -0,0 +1 @@
+public static void main(String RIGHT[])
diff --git a/t/t4018/markdown-heading-indented.ctx b/t/t4018/markdown-heading-indented.ctx
new file mode 100644
index 0000000000..5938336743
--- /dev/null
+++ b/t/t4018/markdown-heading-indented.ctx
@@ -0,0 +1 @@
+   ### RIGHT
diff --git a/t/t4018/markdown-heading-non-headings.ctx b/t/t4018/markdown-heading-non-headings.ctx
new file mode 100644
index 0000000000..7e2165be6e
--- /dev/null
+++ b/t/t4018/markdown-heading-non-headings.ctx
@@ -0,0 +1 @@
+# RIGHT
diff --git a/t/t4018/matlab-class-definition.ctx b/t/t4018/matlab-class-definition.ctx
new file mode 100644
index 0000000000..5dd5b45628
--- /dev/null
+++ b/t/t4018/matlab-class-definition.ctx
@@ -0,0 +1 @@
+classdef RIGHT
diff --git a/t/t4018/matlab-function.ctx b/t/t4018/matlab-function.ctx
new file mode 100644
index 0000000000..72d2350b13
--- /dev/null
+++ b/t/t4018/matlab-function.ctx
@@ -0,0 +1 @@
+function y = RIGHT()
diff --git a/t/t4018/matlab-octave-section-1.ctx b/t/t4018/matlab-octave-section-1.ctx
new file mode 100644
index 0000000000..ca9b349f94
--- /dev/null
+++ b/t/t4018/matlab-octave-section-1.ctx
@@ -0,0 +1 @@
+%%% RIGHT section
diff --git a/t/t4018/matlab-octave-section-2.ctx b/t/t4018/matlab-octave-section-2.ctx
new file mode 100644
index 0000000000..5cbb77faf5
--- /dev/null
+++ b/t/t4018/matlab-octave-section-2.ctx
@@ -0,0 +1 @@
+## RIGHT section
diff --git a/t/t4018/matlab-section.ctx b/t/t4018/matlab-section.ctx
new file mode 100644
index 0000000000..e83fee6f4d
--- /dev/null
+++ b/t/t4018/matlab-section.ctx
@@ -0,0 +1 @@
+%% RIGHT section
diff --git a/t/t4018/perl-skip-end-of-heredoc.ctx b/t/t4018/perl-skip-end-of-heredoc.ctx
new file mode 100644
index 0000000000..c15f4b78bd
--- /dev/null
+++ b/t/t4018/perl-skip-end-of-heredoc.ctx
@@ -0,0 +1 @@
+sub RIGHTwithheredocument {
diff --git a/t/t4018/perl-skip-forward-decl.ctx b/t/t4018/perl-skip-forward-decl.ctx
new file mode 100644
index 0000000000..e0c51599ad
--- /dev/null
+++ b/t/t4018/perl-skip-forward-decl.ctx
@@ -0,0 +1 @@
+package RIGHT;
diff --git a/t/t4018/perl-skip-sub-in-pod.ctx b/t/t4018/perl-skip-sub-in-pod.ctx
new file mode 100644
index 0000000000..abddd76655
--- /dev/null
+++ b/t/t4018/perl-skip-sub-in-pod.ctx
@@ -0,0 +1 @@
+=head1 SYNOPSIS_RIGHT
diff --git a/t/t4018/perl-sub-definition-kr-brace.ctx b/t/t4018/perl-sub-definition-kr-brace.ctx
new file mode 100644
index 0000000000..7e5aee5cde
--- /dev/null
+++ b/t/t4018/perl-sub-definition-kr-brace.ctx
@@ -0,0 +1 @@
+sub RIGHT
diff --git a/t/t4018/perl-sub-definition.ctx b/t/t4018/perl-sub-definition.ctx
new file mode 100644
index 0000000000..d49a63598e
--- /dev/null
+++ b/t/t4018/perl-sub-definition.ctx
@@ -0,0 +1 @@
+sub RIGHT {
diff --git a/t/t4018/php-abstract-class.ctx b/t/t4018/php-abstract-class.ctx
new file mode 100644
index 0000000000..f572d2129b
--- /dev/null
+++ b/t/t4018/php-abstract-class.ctx
@@ -0,0 +1 @@
+abstract class RIGHT
diff --git a/t/t4018/php-abstract-method.ctx b/t/t4018/php-abstract-method.ctx
new file mode 100644
index 0000000000..14cb6df42e
--- /dev/null
+++ b/t/t4018/php-abstract-method.ctx
@@ -0,0 +1 @@
+abstract public function RIGHT(): ?string
diff --git a/t/t4018/php-class.ctx b/t/t4018/php-class.ctx
new file mode 100644
index 0000000000..54bff816d6
--- /dev/null
+++ b/t/t4018/php-class.ctx
@@ -0,0 +1 @@
+class RIGHT
diff --git a/t/t4018/php-final-class.ctx b/t/t4018/php-final-class.ctx
new file mode 100644
index 0000000000..4d59fb749b
--- /dev/null
+++ b/t/t4018/php-final-class.ctx
@@ -0,0 +1 @@
+final class RIGHT
diff --git a/t/t4018/php-final-method.ctx b/t/t4018/php-final-method.ctx
new file mode 100644
index 0000000000..b7da8f8082
--- /dev/null
+++ b/t/t4018/php-final-method.ctx
@@ -0,0 +1 @@
+final public function RIGHT(): string
diff --git a/t/t4018/php-function.ctx b/t/t4018/php-function.ctx
new file mode 100644
index 0000000000..c5f3e55302
--- /dev/null
+++ b/t/t4018/php-function.ctx
@@ -0,0 +1 @@
+function RIGHT()
diff --git a/t/t4018/php-interface.ctx b/t/t4018/php-interface.ctx
new file mode 100644
index 0000000000..a45fa0532a
--- /dev/null
+++ b/t/t4018/php-interface.ctx
@@ -0,0 +1 @@
+interface RIGHT
diff --git a/t/t4018/php-method.ctx b/t/t4018/php-method.ctx
new file mode 100644
index 0000000000..eb1659ff9f
--- /dev/null
+++ b/t/t4018/php-method.ctx
@@ -0,0 +1 @@
+public static function RIGHT()
diff --git a/t/t4018/php-trait.ctx b/t/t4018/php-trait.ctx
new file mode 100644
index 0000000000..57aa4c6267
--- /dev/null
+++ b/t/t4018/php-trait.ctx
@@ -0,0 +1 @@
+trait RIGHT
diff --git a/t/t4018/python-async-def.ctx b/t/t4018/python-async-def.ctx
new file mode 100644
index 0000000000..468c548bbe
--- /dev/null
+++ b/t/t4018/python-async-def.ctx
@@ -0,0 +1 @@
+async def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-class.ctx b/t/t4018/python-class.ctx
new file mode 100644
index 0000000000..a40b755e29
--- /dev/null
+++ b/t/t4018/python-class.ctx
@@ -0,0 +1 @@
+class RIGHT(int, str):
diff --git a/t/t4018/python-def.ctx b/t/t4018/python-def.ctx
new file mode 100644
index 0000000000..a1a9cbad63
--- /dev/null
+++ b/t/t4018/python-def.ctx
@@ -0,0 +1 @@
+def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-indented-async-def.ctx b/t/t4018/python-indented-async-def.ctx
new file mode 100644
index 0000000000..d393620a1e
--- /dev/null
+++ b/t/t4018/python-indented-async-def.ctx
@@ -0,0 +1 @@
+async def RIGHT(self, x: int):
diff --git a/t/t4018/python-indented-class.ctx b/t/t4018/python-indented-class.ctx
new file mode 100644
index 0000000000..0881c84dba
--- /dev/null
+++ b/t/t4018/python-indented-class.ctx
@@ -0,0 +1 @@
+class RIGHT:
diff --git a/t/t4018/python-indented-def.ctx b/t/t4018/python-indented-def.ctx
new file mode 100644
index 0000000000..6e5a44b391
--- /dev/null
+++ b/t/t4018/python-indented-def.ctx
@@ -0,0 +1 @@
+def RIGHT(self, x: int):
diff --git a/t/t4018/rust-fn.ctx b/t/t4018/rust-fn.ctx
new file mode 100644
index 0000000000..baa37cf253
--- /dev/null
+++ b/t/t4018/rust-fn.ctx
@@ -0,0 +1 @@
+pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
diff --git a/t/t4018/rust-impl.ctx b/t/t4018/rust-impl.ctx
new file mode 100644
index 0000000000..5344c35f3f
--- /dev/null
+++ b/t/t4018/rust-impl.ctx
@@ -0,0 +1 @@
+impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
diff --git a/t/t4018/rust-macro-rules.ctx b/t/t4018/rust-macro-rules.ctx
new file mode 100644
index 0000000000..7520463aa0
--- /dev/null
+++ b/t/t4018/rust-macro-rules.ctx
@@ -0,0 +1 @@
+macro_rules! RIGHT {
diff --git a/t/t4018/rust-struct.ctx b/t/t4018/rust-struct.ctx
new file mode 100644
index 0000000000..c1e09dc808
--- /dev/null
+++ b/t/t4018/rust-struct.ctx
@@ -0,0 +1 @@
+pub(super) struct RIGHT<'a> {
diff --git a/t/t4018/rust-trait.ctx b/t/t4018/rust-trait.ctx
new file mode 100644
index 0000000000..6af803db29
--- /dev/null
+++ b/t/t4018/rust-trait.ctx
@@ -0,0 +1 @@
+unsafe trait RIGHT<T> {
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Johannes Sixt]

Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
> Fix a regression in the test framework for userdiff added in
> bfa7d01413 (t4018: an infrastructure to test hunk headers,
> 2014-03-21).
> 
> The testing infrastructure added in that change went overboard with
> simplifying the tests, to the point where we lost test coverage.
> 
> Before that we'd been able to test the full context line, or ever
> since the feature was originally added in f258475a6e (Per-path
> attribute based hunk header selection., 2007-07-06).
> 
> After bfa7d01413 all we cared about was whether "RIGHT" appeared on
> the line. We thus lost the information about whether or not "RIGHT"
> was extracted from the line for the hunk header, or the line appeared
> in full (or other subset of the line).
> 
> Let's bring back coverage for that by adding corresponding *.ctx
> files, this has the added advantage that we're doing a "test_cmp", so
> when we have failures it's just a non-zero exit code from "grep",
> we'll actually have something meaningful in the "-v" output.
> 
> As we'll see in a follow-up commit this is an intermediate step
> towards even better test coverage. I'm structuring these tests in such
> a way as to benefit from the diff.colorMove detection.
> 
> The "sed -n -e" here was originally a single 's/^.*@@\( \|$\)//p'
> pattern, but the '\( \|$\)' part had portability issues on OSX and
> AIX.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   t/t4018-diff-funcname.sh                      |  7 +++---
>   t/t4018/README                                | 22 +++++++++----------
>   t/t4018/README.ctx                            |  1 +
>   t/t4018/bash-arithmetic-function.ctx          |  1 +
>   t/t4018/bash-bashism-style-compact.ctx        |  1 +
>   [...and so on...]

This is what I meant by "without burdening test writers with lots of
subtleties".

I'm not a friend of this change :-(

I think you are going overboard with required test precision. To have 
useful tests for userdiff patterns that demonstrate its features, 
authors should write *many* tests. The right balance should be on the 
coverage of userdiff pattern features, not on the subtle details of each 
and everyone of it. Requiring that many additional context files makes 
it *really hard* to comply.

-- Hannes

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 15 2021, Johannes Sixt wrote:

> Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
>> Fix a regression in the test framework for userdiff added in
>> bfa7d01413 (t4018: an infrastructure to test hunk headers,
>> 2014-03-21).
>> The testing infrastructure added in that change went overboard with
>> simplifying the tests, to the point where we lost test coverage.
>> Before that we'd been able to test the full context line, or ever
>> since the feature was originally added in f258475a6e (Per-path
>> attribute based hunk header selection., 2007-07-06).
>> After bfa7d01413 all we cared about was whether "RIGHT" appeared on
>> the line. We thus lost the information about whether or not "RIGHT"
>> was extracted from the line for the hunk header, or the line appeared
>> in full (or other subset of the line).
>> Let's bring back coverage for that by adding corresponding *.ctx
>> files, this has the added advantage that we're doing a "test_cmp", so
>> when we have failures it's just a non-zero exit code from "grep",
>> we'll actually have something meaningful in the "-v" output.
>> As we'll see in a follow-up commit this is an intermediate step
>> towards even better test coverage. I'm structuring these tests in such
>> a way as to benefit from the diff.colorMove detection.
>> The "sed -n -e" here was originally a single 's/^.*@@\( \|$\)//p'
>> pattern, but the '\( \|$\)' part had portability issues on OSX and
>> AIX.
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>>   t/t4018-diff-funcname.sh                      |  7 +++---
>>   t/t4018/README                                | 22 +++++++++----------
>>   t/t4018/README.ctx                            |  1 +
>>   t/t4018/bash-arithmetic-function.ctx          |  1 +
>>   t/t4018/bash-bashism-style-compact.ctx        |  1 +
>>   [...and so on...]
>
> This is what I meant by "without burdening test writers with lots of
> subtleties".
>
> I'm not a friend of this change :-(
>
> I think you are going overboard with required test precision. To have
> useful tests for userdiff patterns that demonstrate its features, 
> authors should write *many* tests. The right balance should be on the
> coverage of userdiff pattern features, not on the subtle details of
> each and everyone of it. Requiring that many additional context files
> makes it *really hard* to comply.

I agree that this change sucks when viewed in isolation. I think it has
value as part of the larger series, since (as noted in the commit
message) the point here is to allow the reviewer to see the the test
content & semantics aren't different.

I suppose I could do some version of squashing this and 12/27, but then
I'd be introducing the full testing of the context line at the same
time.

By doing it this way I split the change in test semantics from the test
structure change itself in 12/27.

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

>>   t/t4018-diff-funcname.sh                      |  7 +++---
>>   t/t4018/README                                | 22 +++++++++----------
>>   t/t4018/README.ctx                            |  1 +
>>   t/t4018/bash-arithmetic-function.ctx          |  1 +
>>   t/t4018/bash-bashism-style-compact.ctx        |  1 +
>>   [...and so on...]
>
> This is what I meant by "without burdening test writers with lots of
> subtleties".
>
> I'm not a friend of this change :-(
>
> I think you are going overboard with required test precision. To have
> useful tests for userdiff patterns that demonstrate its features, 
> authors should write *many* tests. The right balance should be on the
> coverage of userdiff pattern features, not on the subtle details of
> each and everyone of it. Requiring that many additional context files
> makes it *really hard* to comply.

Yeah, the first time I saw the t4018 test framework appeared in my
tree, I truly appreciated its simplicity, how the test input file is
self-documenting and self-contained, with the clever use of "RIGHT",
"broken" and "ChangeMe" magic tokens, admired the cleverness of the
approach, and wished I was clever enough to invent that pattern to
apply to other tests myself.

A little new for each and every test for the miniscule gain of
checking which part of the function header line is extracted feels a
bit too much noise and rubs my sense of aesthetics, spoiled by the
existing t4018 tests, the wrong way.

This is a rough sketch of a different approach aiming for the same.
I converted only a few files, but I hope that this is enough to
illustrate the idea.

 t/t4018-diff-funcname.sh         | 17 ++++++++++++++---
 t/t4018/README                   |  9 ++++++---
 t/t4018/bash-arithmetic-function |  3 +++
 3 files changed, 23 insertions(+), 6 deletions(-)

diff --git c/t/t4018-diff-funcname.sh w/t/t4018-diff-funcname.sh
index 9675bc17db..dd79c99fc5 100755
--- c/t/t4018-diff-funcname.sh
+++ w/t/t4018-diff-funcname.sh
@@ -107,10 +107,21 @@ do
 	else
 		result=success
 	fi
-	test_expect_$result "hunk header: $i" "
+
+	test_expect_$result "hunk header: $i" '
+		HEAD=$(sed -n \
+			-e "s/^.*HEADER.*|\(.*\)right\(.*\)|.*/ \1RIGHT\2/p" "$i") &&
+
 		git diff -U1 $i >actual &&
-		grep '@@ .* @@.*RIGHT' actual
-	"
+
+		sed -ne "s/^@@[^@]*@@//p" actual |
+		if test -n "$HEAD"
+		then
+			grep -F "$HEAD"
+		else
+			grep "^.*RIGHT"
+		fi
+	'
 done
 
 test_done
diff --git c/t/t4018/README w/t/t4018/README
index 283e01cca1..bc3c11e083 100644
--- c/t/t4018/README
+++ w/t/t4018/README
@@ -10,9 +10,12 @@ The text that must appear in the hunk header must contain the word
 To mark a test case that highlights a malfunction, insert the word
 BROKEN in all lower-case somewhere in the file.
 
-This text is a bit twisted and out of order, but it is itself a
-test case for the default hunk header pattern. Know what you are doing
-if you change it.
+This text is a bit twisted and out of order, but it is itself a test
+case for the default hunk header pattern. Know what you are doing if
+you change it.  You can optionally force exactly what should be on
+the hunk header line by enclosing the expect text by vertical bars
+(but downcasing the word right in it) and place it on a line with
+HEADER on it, like |How to write right test cases|.
 
 BTW, this tests that the head line goes to the hunk header, not the line
 of equal signs.
diff --git c/t/t4018/bash-arithmetic-function w/t/t4018/bash-arithmetic-function
index c0b276cb50..935f18d96d 100644
--- c/t/t4018/bash-arithmetic-function
+++ w/t/t4018/bash-arithmetic-function
@@ -2,3 +2,6 @@ RIGHT() ((
 
     ChangeMe = "$x" + "$y"
 ))
+
+
+# HEADER |right()|

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Johannes Sixt]

Am 16.02.21 um 19:32 schrieb Junio C Hamano:
> Johannes Sixt <j6t@kdbg.org> writes:
> 
>>>    t/t4018-diff-funcname.sh                      |  7 +++---
>>>    t/t4018/README                                | 22 +++++++++----------
>>>    t/t4018/README.ctx                            |  1 +
>>>    t/t4018/bash-arithmetic-function.ctx          |  1 +
>>>    t/t4018/bash-bashism-style-compact.ctx        |  1 +
>>>    [...and so on...]
>>
>> This is what I meant by "without burdening test writers with lots of
>> subtleties".
>>
>> I'm not a friend of this change :-(
>>
>> I think you are going overboard with required test precision. To have
>> useful tests for userdiff patterns that demonstrate its features,
>> authors should write *many* tests. The right balance should be on the
>> coverage of userdiff pattern features, not on the subtle details of
>> each and everyone of it. Requiring that many additional context files
>> makes it *really hard* to comply.
> 
> Yeah, the first time I saw the t4018 test framework appeared in my
> tree, I truly appreciated its simplicity, how the test input file is
> self-documenting and self-contained, with the clever use of "RIGHT",
> "broken" and "ChangeMe" magic tokens, admired the cleverness of the
> approach, and wished I was clever enough to invent that pattern to
> apply to other tests myself.
> 
> A little new for each and every test for the miniscule gain of
> checking which part of the function header line is extracted feels a
> bit too much noise and rubs my sense of aesthetics, spoiled by the
> existing t4018 tests, the wrong way.
> 
> This is a rough sketch of a different approach aiming for the same.
> I converted only a few files, but I hope that this is enough to
> illustrate the idea.
> 
>   t/t4018-diff-funcname.sh         | 17 ++++++++++++++---
>   t/t4018/README                   |  9 ++++++---
>   t/t4018/bash-arithmetic-function |  3 +++
>   3 files changed, 23 insertions(+), 6 deletions(-)
> 
> diff --git c/t/t4018-diff-funcname.sh w/t/t4018-diff-funcname.sh
> index 9675bc17db..dd79c99fc5 100755
> --- c/t/t4018-diff-funcname.sh
> +++ w/t/t4018-diff-funcname.sh
> @@ -107,10 +107,21 @@ do
>   	else
>   		result=success
>   	fi
> -	test_expect_$result "hunk header: $i" "
> +
> +	test_expect_$result "hunk header: $i" '
> +		HEAD=$(sed -n \
> +			-e "s/^.*HEADER.*|\(.*\)right\(.*\)|.*/ \1RIGHT\2/p" "$i") &&
> +
>   		git diff -U1 $i >actual &&
> -		grep '@@ .* @@.*RIGHT' actual
> -	"
> +
> +		sed -ne "s/^@@[^@]*@@//p" actual |
> +		if test -n "$HEAD"
> +		then
> +			grep -F "$HEAD"
> +		else
> +			grep "^.*RIGHT"
> +		fi
> +	'
>   done
>   
>   test_done

> diff --git c/t/t4018/bash-arithmetic-function w/t/t4018/bash-arithmetic-function
> index c0b276cb50..935f18d96d 100644
> --- c/t/t4018/bash-arithmetic-function
> +++ w/t/t4018/bash-arithmetic-function
> @@ -2,3 +2,6 @@ RIGHT() ((
>   
>       ChangeMe = "$x" + "$y"
>   ))
> +
> +
> +# HEADER |right()|
> 

Clever! Opt-in for those who desire precise tests.

-- Hannes

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

On Tue, Feb 16 2021, Johannes Sixt wrote:

> Am 16.02.21 um 19:32 schrieb Junio C Hamano:
>> Johannes Sixt <j6t@kdbg.org> writes:
>> 
>>>>    t/t4018-diff-funcname.sh                      |  7 +++---
>>>>    t/t4018/README                                | 22 +++++++++----------
>>>>    t/t4018/README.ctx                            |  1 +
>>>>    t/t4018/bash-arithmetic-function.ctx          |  1 +
>>>>    t/t4018/bash-bashism-style-compact.ctx        |  1 +
>>>>    [...and so on...]
>>>
>>> This is what I meant by "without burdening test writers with lots of
>>> subtleties".
>>>
>>> I'm not a friend of this change :-(
>>>
>>> I think you are going overboard with required test precision. To have
>>> useful tests for userdiff patterns that demonstrate its features,
>>> authors should write *many* tests. The right balance should be on the
>>> coverage of userdiff pattern features, not on the subtle details of
>>> each and everyone of it. Requiring that many additional context files
>>> makes it *really hard* to comply.
>> Yeah, the first time I saw the t4018 test framework appeared in my
>> tree, I truly appreciated its simplicity, how the test input file is
>> self-documenting and self-contained, with the clever use of "RIGHT",
>> "broken" and "ChangeMe" magic tokens, admired the cleverness of the
>> approach, and wished I was clever enough to invent that pattern to
>> apply to other tests myself.
>> A little new for each and every test for the miniscule gain of
>> checking which part of the function header line is extracted feels a
>> bit too much noise and rubs my sense of aesthetics, spoiled by the
>> existing t4018 tests, the wrong way.
>> This is a rough sketch of a different approach aiming for the same.
>> I converted only a few files, but I hope that this is enough to
>> illustrate the idea.
>>   t/t4018-diff-funcname.sh         | 17 ++++++++++++++---
>>   t/t4018/README                   |  9 ++++++---
>>   t/t4018/bash-arithmetic-function |  3 +++
>>   3 files changed, 23 insertions(+), 6 deletions(-)
>> diff --git c/t/t4018-diff-funcname.sh w/t/t4018-diff-funcname.sh
>> index 9675bc17db..dd79c99fc5 100755
>> --- c/t/t4018-diff-funcname.sh
>> +++ w/t/t4018-diff-funcname.sh
>> @@ -107,10 +107,21 @@ do
>>   	else
>>   		result=success
>>   	fi
>> -	test_expect_$result "hunk header: $i" "
>> +
>> +	test_expect_$result "hunk header: $i" '
>> +		HEAD=$(sed -n \
>> +			-e "s/^.*HEADER.*|\(.*\)right\(.*\)|.*/ \1RIGHT\2/p" "$i") &&
>> +
>>   		git diff -U1 $i >actual &&
>> -		grep '@@ .* @@.*RIGHT' actual
>> -	"
>> +
>> +		sed -ne "s/^@@[^@]*@@//p" actual |
>> +		if test -n "$HEAD"
>> +		then
>> +			grep -F "$HEAD"
>> +		else
>> +			grep "^.*RIGHT"
>> +		fi
>> +	'
>>   done
>>     test_done
>
>> diff --git c/t/t4018/bash-arithmetic-function w/t/t4018/bash-arithmetic-function
>> index c0b276cb50..935f18d96d 100644
>> --- c/t/t4018/bash-arithmetic-function
>> +++ w/t/t4018/bash-arithmetic-function
>> @@ -2,3 +2,6 @@ RIGHT() ((
>>         ChangeMe = "$x" + "$y"
>>   ))
>> +
>> +
>> +# HEADER |right()|
>> 
>
> Clever! Opt-in for those who desire precise tests.

Tests aren't only for testing a subjective "good enough" in the
estimation of the author of the code in question, but also for others
who later touch the same area and want to avoid regressions.

Which is why I think it's an anti-pattern to use "grep SOME-SUBSTR" in
lieu of test_cmp if we can easily do the latter.

So e.g. in this case part of my motivation is that this is one of the
things I want to look at porting to some general PCREv2 powered backend
regex matching library once some of the pickaxe work I have pending
lands.

I'm very interested in whether such a port subtly breaks existing
semantics, and it's not useful if such regressions are hidden because
someone who wrote a userdiff rule didn't think they needed to care about
exact matching for their own purposes.

I find this notion that patch authors who we'd expect to hack userdiff.c
and somehow understand the arcane rules of the list form of
diff.<func>.xfunction (which wasn't even documented until this series),
and to carefully read t/t4018/README to see how their test is parsed,
would be discouraged by some pretty plain shell syntax to pass in two
strings to be implausible.

Whenever I've tried to hack up userdiff.c rules my first stumbling block
has been that if you fail the previous test gave you no output, because
it used grep instead of test_cmp.

So you'd need to monkeypatch it to dump the value, or know about "-d"
and trash directory inspection. So much for sparing prospective authors
from the intricacies of our sh-based tests.

The old test format also forces you to try to mix real examples of a
programming language with our need to shove an all-caps "RIGHT"
identifier somewhere on the hunk line.

In languages like Go, Perl or Elisp such an identifier is somewhere
between so strongly discouraged that you'd never see one in the wild, or
in others even a syntax error.

It seems much simpler to just ask the author to paste in their snippet,
test it, and then then paste what they got on the @@ line as another
parameter to their test.

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

>>> +		sed -ne "s/^@@[^@]*@@//p" actual |
>>> +		if test -n "$HEAD"
>>> +		then
>>> +			grep -F "$HEAD"
>>> +		else
>>> +			grep "^.*RIGHT"
>>> +		fi
>>> +	'
>>>   done
>>>     test_done
>>
>>> diff --git c/t/t4018/bash-arithmetic-function w/t/t4018/bash-arithmetic-function
>>> index c0b276cb50..935f18d96d 100644
>>> --- c/t/t4018/bash-arithmetic-function
>>> +++ w/t/t4018/bash-arithmetic-function
>>> @@ -2,3 +2,6 @@ RIGHT() ((
>>>         ChangeMe = "$x" + "$y"
>>>   ))
>>> +
>>> +
>>> +# HEADER |right()|
>>> 
>>
>> Clever! Opt-in for those who desire precise tests.
>
> Tests aren't only for testing a subjective "good enough" in the
> estimation of the author of the code in question, but also for others
> who later touch the same area and want to avoid regressions.
>
> Which is why I think it's an anti-pattern to use "grep SOME-SUBSTR" in
> lieu of test_cmp if we can easily do the latter.

Sounds good.  It shouldn't be too hard to satisfy both camps,
i.e. the quoted demonstrates one way to allow test writers to
give expectation in-place in the single test file, and replacing
how it uses "grep" to check the output with test_cmp or whatever
wouldn't make the resulting tests too hard to write and maintain.

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

On Wed, Feb 17 2021, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>>>> +		sed -ne "s/^@@[^@]*@@//p" actual |
>>>> +		if test -n "$HEAD"
>>>> +		then
>>>> +			grep -F "$HEAD"
>>>> +		else
>>>> +			grep "^.*RIGHT"
>>>> +		fi
>>>> +	'
>>>>   done
>>>>     test_done
>>>
>>>> diff --git c/t/t4018/bash-arithmetic-function w/t/t4018/bash-arithmetic-function
>>>> index c0b276cb50..935f18d96d 100644
>>>> --- c/t/t4018/bash-arithmetic-function
>>>> +++ w/t/t4018/bash-arithmetic-function
>>>> @@ -2,3 +2,6 @@ RIGHT() ((
>>>>         ChangeMe = "$x" + "$y"
>>>>   ))
>>>> +
>>>> +
>>>> +# HEADER |right()|
>>>> 
>>>
>>> Clever! Opt-in for those who desire precise tests.
>>
>> Tests aren't only for testing a subjective "good enough" in the
>> estimation of the author of the code in question, but also for others
>> who later touch the same area and want to avoid regressions.
>>
>> Which is why I think it's an anti-pattern to use "grep SOME-SUBSTR" in
>> lieu of test_cmp if we can easily do the latter.
>
> Sounds good.  It shouldn't be too hard to satisfy both camps,
> i.e. the quoted demonstrates one way to allow test writers to
> give expectation in-place in the single test file, and replacing
> how it uses "grep" to check the output with test_cmp or whatever
> wouldn't make the resulting tests too hard to write and maintain.

It doesn't satisfy both camps, because I'd like to convert all these
tests to test_cmp because for a subsequent refactoring of userdiff.c by
me or others I don't know in advance what might break, so I'd like to
assert the exact current behavior.

Whereas your patch provides a way to opt-in individual tests to a
test_cmp-alike, but leaves the rest at grepping for the "RIGHT"
substring. Failures in the tests who aren't opted-in will be hidden.

It also means that subsequent changes to the behavior in the form of
submitted patches won't be as self-documenting, e.g. I've wondered if we
could introduce a case to balance parens in this code (sometimes C
function declarations stretch across lines), and there's e.g. the
arbitrary limit of 80 bytes on the line (which to be fair, we don't
curretly have tests for).

Anyway, as noted in [1] I don't see how this custom format of grepping
stuff out of plain-text files is simpler, particularly when its behavior
would start to rely on other things like "# HEADER |right()|" whose
behavior is a function of what we grep/sed when/where in the logic
driving the tests.

But if you & Johannes S. disagree with that I don't really say a way
forward with this series. I think e.g. squashing 09/27 into the rest
would make things simpler/less verbose, but the end-state would still be
matching the full hunk line, and if that's not something that's wanted
in any shape or form as a default...

1. https://lore.kernel.org/git/87h7mba3h3.fsf@evledraar.gmail.com/

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Johannes Sixt]

Am 23.02.21 um 14:11 schrieb Ævar Arnfjörð Bjarmason:
> On Wed, Feb 17 2021, Junio C Hamano wrote:
>> Sounds good.  It shouldn't be too hard to satisfy both camps,
>> i.e. the quoted demonstrates one way to allow test writers to
>> give expectation in-place in the single test file, and replacing
>> how it uses "grep" to check the output with test_cmp or whatever
>> wouldn't make the resulting tests too hard to write and maintain.
> 
> It doesn't satisfy both camps, because I'd like to convert all these
> tests to test_cmp because for a subsequent refactoring of userdiff.c by
> me or others I don't know in advance what might break, so I'd like to
> assert the exact current behavior.
> 
> Whereas your patch provides a way to opt-in individual tests to a
> test_cmp-alike, but leaves the rest at grepping for the "RIGHT"
> substring. Failures in the tests who aren't opted-in will be hidden.
> 
> It also means that subsequent changes to the behavior in the form of
> submitted patches won't be as self-documenting, e.g. I've wondered if we
> could introduce a case to balance parens in this code (sometimes C
> function declarations stretch across lines), and there's e.g. the
> arbitrary limit of 80 bytes on the line (which to be fair, we don't
> curretly have tests for).
> 
> Anyway, as noted in [1] I don't see how this custom format of grepping
> stuff out of plain-text files is simpler, particularly when its behavior
> would start to rely on other things like "# HEADER |right()|" whose
> behavior is a function of what we grep/sed when/where in the logic
> driving the tests.
> 
> But if you & Johannes S. disagree with that I don't really say a way
> forward with this series. I think e.g. squashing 09/27 into the rest
> would make things simpler/less verbose, but the end-state would still be
> matching the full hunk line, and if that's not something that's wanted
> in any shape or form as a default...
> 
> 1. https://lore.kernel.org/git/87h7mba3h3.fsf@evledraar.gmail.com/

I could live with a version of Junio's suggestion that is not opt-in,
i.e., the checks are mandatory and exact. The important point is that
there is only one file per test case; that would still count as
"sufficiently simple" in my book.

-- Hannes

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

On Tue, Feb 23 2021, Johannes Sixt wrote:

> Am 23.02.21 um 14:11 schrieb Ævar Arnfjörð Bjarmason:
>> On Wed, Feb 17 2021, Junio C Hamano wrote:
>>> Sounds good.  It shouldn't be too hard to satisfy both camps,
>>> i.e. the quoted demonstrates one way to allow test writers to
>>> give expectation in-place in the single test file, and replacing
>>> how it uses "grep" to check the output with test_cmp or whatever
>>> wouldn't make the resulting tests too hard to write and maintain.
>> 
>> It doesn't satisfy both camps, because I'd like to convert all these
>> tests to test_cmp because for a subsequent refactoring of userdiff.c by
>> me or others I don't know in advance what might break, so I'd like to
>> assert the exact current behavior.
>> 
>> Whereas your patch provides a way to opt-in individual tests to a
>> test_cmp-alike, but leaves the rest at grepping for the "RIGHT"
>> substring. Failures in the tests who aren't opted-in will be hidden.
>> 
>> It also means that subsequent changes to the behavior in the form of
>> submitted patches won't be as self-documenting, e.g. I've wondered if we
>> could introduce a case to balance parens in this code (sometimes C
>> function declarations stretch across lines), and there's e.g. the
>> arbitrary limit of 80 bytes on the line (which to be fair, we don't
>> curretly have tests for).
>> 
>> Anyway, as noted in [1] I don't see how this custom format of grepping
>> stuff out of plain-text files is simpler, particularly when its behavior
>> would start to rely on other things like "# HEADER |right()|" whose
>> behavior is a function of what we grep/sed when/where in the logic
>> driving the tests.
>> 
>> But if you & Johannes S. disagree with that I don't really say a way
>> forward with this series. I think e.g. squashing 09/27 into the rest
>> would make things simpler/less verbose, but the end-state would still be
>> matching the full hunk line, and if that's not something that's wanted
>> in any shape or form as a default...
>> 
>> 1. https://lore.kernel.org/git/87h7mba3h3.fsf@evledraar.gmail.com/
>
> I could live with a version of Junio's suggestion that is not opt-in,
> i.e., the checks are mandatory and exact. The important point is that
> there is only one file per test case; that would still count as
> "sufficiently simple" in my book.

So something where the tests in 14-15,17,19/27 would need to go back to
having the config that's part of the test driven by setup in
t/t4018-diff-funcname.sh, and the contents of a test file being e.g. (to
take the test from 25/27):

    # HEADER |package main|
    package main

    import "fmt"
    // ChangeMe

?

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Johannes Sixt]

Am 24.02.21 um 12:12 schrieb Ævar Arnfjörð Bjarmason:
> 
> On Tue, Feb 23 2021, Johannes Sixt wrote:
>> I could live with a version of Junio's suggestion that is not opt-in,
>> i.e., the checks are mandatory and exact. The important point is that
>> there is only one file per test case; that would still count as
>> "sufficiently simple" in my book.
> 
> So something where the tests in 14-15,17,19/27 would need to go back to
> having the config that's part of the test driven by setup in
> t/t4018-diff-funcname.sh,

I'm afraid, I don't understand what you mean by this sentence. Do you
mean "not do the refactoring like in those patches"? Yes, that would
likely be unnecessary churn.

> and the contents of a test file being e.g. (to
> take the test from 25/27):
> 
>     # HEADER |package main|
>     package main
> 
>     import "fmt"
>     // ChangeMe
> 
> ?
> 

Yes. Except that I would move the "expected text" part to the end of the
file so that there is no danger that it is mistaken as a hunk header (in
case that a pattern is so loose that it matches that line).

-- Hannes

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> +		sed -ne "s/^@@[^@]*@@//p" actual |
> +		if test -n "$HEAD"
> +		then
> +			grep -F "$HEAD"

I think this should also use "-x" (match the whole line) if it is
portable enough; the option should be available in POSIX.1 but we
haven't used it anywhere in our test suite.

> +		else
> +			grep "^.*RIGHT"
> +		fi
> +	'
>  done

Re: [PATCH v2 09/27] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

On Wed, Feb 17 2021, Junio C Hamano wrote:

> Junio C Hamano <gitster@pobox.com> writes:
>
>> +		sed -ne "s/^@@[^@]*@@//p" actual |
>> +		if test -n "$HEAD"
>> +		then
>> +			grep -F "$HEAD"
>
> I think this should also use "-x" (match the whole line) if it is
> portable enough; the option should be available in POSIX.1 but we
> haven't used it anywhere in our test suite.

Isn't that just test_cmp with no debug output when it fails? Noted in
https://lore.kernel.org/git/87h7mba3h3.fsf@evledraar.gmail.com/

>> +		else
>> +			grep "^.*RIGHT"
>> +		fi
>> +	'
>>  done

[PATCH v2 10/27] blame tests: don't rely on t/t4018/ directory

[Ævar Arnfjörð Bjarmason]

Refactor a test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) so that the blame tests don't rely
on stealing the contents of "t/t4018/fortran-external-function".

I'm about to refactor that directory to delete that file, just moving
the relevant test file here inline is the easiest solution, and I
think also the most readable.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 29ce89090d..04a2c58594 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -482,12 +482,22 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 test_expect_success 'setup -L :funcname with userdiff driver' '
 	echo "fortran-* diff=fortran" >.gitattributes &&
 	fortran_file=fortran-external-function &&
-	orig_file="$TEST_DIRECTORY/t4018/$fortran_file" &&
-	cp "$orig_file" . &&
+	cat >$fortran_file <<-\EOF &&
+	function RIGHT(a, b) result(c)
+
+	integer, intent(in) :: ChangeMe
+	integer, intent(in) :: b
+	integer, intent(out) :: c
+
+	c = a+b
+
+	end function RIGHT
+	EOF
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
 	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$orig_file" >"$fortran_file" &&
+	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
+	mv "$fortran_file".tmp "$fortran_file" &&
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
 	git commit -m "change fortran file"
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 10/27] blame tests: don't rely on t/t4018/ directory

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> Refactor a test added in 9466e3809d (blame: enable funcname blaming
> with userdiff driver, 2020-11-01) so that the blame tests don't rely
> on stealing the contents of "t/t4018/fortran-external-function".

I like changes like this and wish they (I haven't finished reading
the 27 patches, but I am hoping that there may be more) were
separately available ;-)

[PATCH v2 11/27] blame tests: simplify userdiff driver test

[Ævar Arnfjörð Bjarmason]

Simplify the test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) to use the --author support recently
added in 999cfc4f45 (test-lib functions: add --author support to
test_commit, 2021-01-12).

We also did not need the full fortran-external-function content, let's
cut it down to just the important parts, and further modify it to
demonstrate that the fortran-specific userdiff function is in effect
by adding "WRONG" lines surrounding the "RIGHT" one.

The test also left behind a .gitattributes files, let's clean it up
with "test_when_finished".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 36 +++++++++++++++---------------------
 1 file changed, 15 insertions(+), 21 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 04a2c58594..4a86e0f349 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -479,32 +479,26 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 	check_count -f hello.c -L$n -L^:ma.. F 4 G 1 H 1
 '
 
-test_expect_success 'setup -L :funcname with userdiff driver' '
-	echo "fortran-* diff=fortran" >.gitattributes &&
-	fortran_file=fortran-external-function &&
-	cat >$fortran_file <<-\EOF &&
+test_expect_success 'blame -L :funcname with userdiff driver' '
+	cat >file.template <<-\EOF &&
+	def WRONG begin end
 	function RIGHT(a, b) result(c)
+	int WRONG(void) {}
 
 	integer, intent(in) :: ChangeMe
-	integer, intent(in) :: b
-	integer, intent(out) :: c
-
-	c = a+b
-
-	end function RIGHT
 	EOF
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
-	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
-	mv "$fortran_file".tmp "$fortran_file" &&
-	git add "$fortran_file" &&
-	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
-	git commit -m "change fortran file"
-'
 
-test_expect_success 'blame -L :funcname with userdiff driver' '
-	check_count -f fortran-external-function -L:RIGHT A 7 B 1
+	fortran_file=file.f03 &&
+	test_when_finished "rm .gitattributes" &&
+	echo "$fortran_file diff=fortran" >.gitattributes &&
+
+	test_commit --author "A <A@test.git>" \
+		"add" $fortran_file \
+		"$(cat file.template)" &&
+	test_commit --author "B <B@test.git>" \
+		"change" $fortran_file \
+		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
+	check_count -f $fortran_file -L:RIGHT A 3 B 1
 '
 
 test_expect_success 'setup incremental' '
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 11/27] blame tests: simplify userdiff driver test

[Johannes Sixt]

Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
> Simplify the test added in 9466e3809d (blame: enable funcname blaming
> with userdiff driver, 2020-11-01) to use the --author support recently
> added in 999cfc4f45 (test-lib functions: add --author support to
> test_commit, 2021-01-12).
> 
> We also did not need the full fortran-external-function content, let's
> cut it down to just the important parts, and further modify it to
> demonstrate that the fortran-specific userdiff function is in effect
> by adding "WRONG" lines surrounding the "RIGHT" one.
> 
> The test also left behind a .gitattributes files, let's clean it up
> with "test_when_finished".
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   t/annotate-tests.sh | 36 +++++++++++++++---------------------
>   1 file changed, 15 insertions(+), 21 deletions(-)
> 
> diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
> index 04a2c58594..4a86e0f349 100644
> --- a/t/annotate-tests.sh
> +++ b/t/annotate-tests.sh
> @@ -479,32 +479,26 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
>   	check_count -f hello.c -L$n -L^:ma.. F 4 G 1 H 1
>   '
>   
> -test_expect_success 'setup -L :funcname with userdiff driver' '
> -	echo "fortran-* diff=fortran" >.gitattributes &&
> -	fortran_file=fortran-external-function &&
> -	cat >$fortran_file <<-\EOF &&
> +test_expect_success 'blame -L :funcname with userdiff driver' '
> +	cat >file.template <<-\EOF &&
> +	def WRONG begin end
>   	function RIGHT(a, b) result(c)
> +	int WRONG(void) {}
>   
>   	integer, intent(in) :: ChangeMe
> -	integer, intent(in) :: b
> -	integer, intent(out) :: c
> -
> -	c = a+b
> -
> -	end function RIGHT
>   	EOF
> -	git add "$fortran_file" &&
> -	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
> -	git commit -m "add fortran file" &&
> -	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
> -	mv "$fortran_file".tmp "$fortran_file" &&
> -	git add "$fortran_file" &&
> -	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
> -	git commit -m "change fortran file"
> -'
>   
> -test_expect_success 'blame -L :funcname with userdiff driver' '
> -	check_count -f fortran-external-function -L:RIGHT A 7 B 1
> +	fortran_file=file.f03 &&
> +	test_when_finished "rm .gitattributes" &&
> +	echo "$fortran_file diff=fortran" >.gitattributes &&
> +
> +	test_commit --author "A <A@test.git>" \
> +		"add" $fortran_file \
> +		"$(cat file.template)" &&
> +	test_commit --author "B <B@test.git>" \
> +		"change" $fortran_file \
> +		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
> +	check_count -f $fortran_file -L:RIGHT A 3 B 1
>   '
>   
>   test_expect_success 'setup incremental' '
> 

I don't get the point. What do you need the tokens "WRONG" for when they 
are not checked anywhere? Instead of adding unrelated lines (that do not 
even look like Fortran), couldn't you just not remove some of the 
others? In particular, the last one that contains "RIGHT" as well may be 
useful to keep in order to show that the code is not confused by it.

Please place "$fortran_file" in dquotes on the check_count line.

-- Hannes

Re: [PATCH v2 11/27] blame tests: simplify userdiff driver test

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 15 2021, Johannes Sixt wrote:

> Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
>> Simplify the test added in 9466e3809d (blame: enable funcname blaming
>> with userdiff driver, 2020-11-01) to use the --author support recently
>> added in 999cfc4f45 (test-lib functions: add --author support to
>> test_commit, 2021-01-12).
>> We also did not need the full fortran-external-function content,
>> let's
>> cut it down to just the important parts, and further modify it to
>> demonstrate that the fortran-specific userdiff function is in effect
>> by adding "WRONG" lines surrounding the "RIGHT" one.
>> The test also left behind a .gitattributes files, let's clean it up
>> with "test_when_finished".
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>>   t/annotate-tests.sh | 36 +++++++++++++++---------------------
>>   1 file changed, 15 insertions(+), 21 deletions(-)
>> diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
>> index 04a2c58594..4a86e0f349 100644
>> --- a/t/annotate-tests.sh
>> +++ b/t/annotate-tests.sh
>> @@ -479,32 +479,26 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
>>   	check_count -f hello.c -L$n -L^:ma.. F 4 G 1 H 1
>>   '
>>   -test_expect_success 'setup -L :funcname with userdiff driver' '
>> -	echo "fortran-* diff=fortran" >.gitattributes &&
>> -	fortran_file=fortran-external-function &&
>> -	cat >$fortran_file <<-\EOF &&
>> +test_expect_success 'blame -L :funcname with userdiff driver' '
>> +	cat >file.template <<-\EOF &&
>> +	def WRONG begin end
>>   	function RIGHT(a, b) result(c)
>> +	int WRONG(void) {}
>>     	integer, intent(in) :: ChangeMe
>> -	integer, intent(in) :: b
>> -	integer, intent(out) :: c
>> -
>> -	c = a+b
>> -
>> -	end function RIGHT
>>   	EOF
>> -	git add "$fortran_file" &&
>> -	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
>> -	git commit -m "add fortran file" &&
>> -	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
>> -	mv "$fortran_file".tmp "$fortran_file" &&
>> -	git add "$fortran_file" &&
>> -	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
>> -	git commit -m "change fortran file"
>> -'
>>   -test_expect_success 'blame -L :funcname with userdiff driver' '
>> -	check_count -f fortran-external-function -L:RIGHT A 7 B 1
>> +	fortran_file=file.f03 &&
>> +	test_when_finished "rm .gitattributes" &&
>> +	echo "$fortran_file diff=fortran" >.gitattributes &&
>> +
>> +	test_commit --author "A <A@test.git>" \
>> +		"add" $fortran_file \
>> +		"$(cat file.template)" &&
>> +	test_commit --author "B <B@test.git>" \
>> +		"change" $fortran_file \
>> +		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
>> +	check_count -f $fortran_file -L:RIGHT A 3 B 1
>>   '
>>     test_expect_success 'setup incremental' '
>> 
>
> I don't get the point. What do you need the tokens "WRONG" for when
> they are not checked anywhere? Instead of adding unrelated lines (that
> do not even look like Fortran), couldn't you just not remove some of
> the others? In particular, the last one that contains "RIGHT" as well
> may be useful to keep in order to show that the code is not confused
> by it.

Isn't the point of the test to assert that we're using a userdiff driver
over the built-in xdiff rules here?

We can imagine that a change to its default heuristics would be to find
the first non-whitespace line, but it jumping over non-whitespace lines
in search of a fortran-looking line doesn't seem like it would ever
happen. Hence the WRONG lines.

> Please place "$fortran_file" in dquotes on the check_count line.

Why do we need to dquote a convenience variable defined in the test
itself that'll never contain spaces or other funny things we'd get if we
had $(pwd) or whatever in there? It wouldn't hurt, but maybe I'm missing
some reason for why it's necessary or desired here.

Re: [PATCH v2 11/27] blame tests: simplify userdiff driver test

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

>>> +	fortran_file=file.f03 &&
>>> +	test_when_finished "rm .gitattributes" &&
>>> +	echo "$fortran_file diff=fortran" >.gitattributes &&
>>> +
>>> +	test_commit --author "A <A@test.git>" \
>>> +		"add" $fortran_file \
>>> +		"$(cat file.template)" &&
>>> +	test_commit --author "B <B@test.git>" \
>>> +		"change" $fortran_file \
>>> +		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
>>> +	check_count -f $fortran_file -L:RIGHT A 3 B 1
> ...
>> Please place "$fortran_file" in dquotes on the check_count line.
>
> Why do we need to dquote a convenience variable defined in the test
> itself that'll never contain spaces or other funny things we'd get if we
> had $(pwd) or whatever in there? It wouldn't hurt, but maybe I'm missing
> some reason for why it's necessary or desired here.

Always dquoting when the code does not depend on splitting at IFS
whitespace reduces cognitive load.  The person who looks at the
variable reference, $fortran_file, has to wonder if the unquoted
form is used to take advantage of being split at IFS whitespaces, or
the test author saved two keystrokes because the filename does not
have any such whitespace, and go back to the assignment to check.

Re: [PATCH v2 11/27] blame tests: simplify userdiff driver test

[Johannes Sixt]

Am 17.02.21 um 02:33 schrieb Ævar Arnfjörð Bjarmason:
> 
> On Mon, Feb 15 2021, Johannes Sixt wrote:
> 
>> Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
>>> Simplify the test added in 9466e3809d (blame: enable funcname blaming
>>> with userdiff driver, 2020-11-01) to use the --author support recently
>>> added in 999cfc4f45 (test-lib functions: add --author support to
>>> test_commit, 2021-01-12).
>>> We also did not need the full fortran-external-function content,
>>> let's
>>> cut it down to just the important parts, and further modify it to
>>> demonstrate that the fortran-specific userdiff function is in effect
>>> by adding "WRONG" lines surrounding the "RIGHT" one.
>>> The test also left behind a .gitattributes files, let's clean it up
>>> with "test_when_finished".
>>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>>> ---
>>>    t/annotate-tests.sh | 36 +++++++++++++++---------------------
>>>    1 file changed, 15 insertions(+), 21 deletions(-)
>>> diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
>>> index 04a2c58594..4a86e0f349 100644
>>> --- a/t/annotate-tests.sh
>>> +++ b/t/annotate-tests.sh
>>> @@ -479,32 +479,26 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
>>>    	check_count -f hello.c -L$n -L^:ma.. F 4 G 1 H 1
>>>    '
>>>    -test_expect_success 'setup -L :funcname with userdiff driver' '
>>> -	echo "fortran-* diff=fortran" >.gitattributes &&
>>> -	fortran_file=fortran-external-function &&
>>> -	cat >$fortran_file <<-\EOF &&
>>> +test_expect_success 'blame -L :funcname with userdiff driver' '
>>> +	cat >file.template <<-\EOF &&
>>> +	def WRONG begin end
>>>    	function RIGHT(a, b) result(c)
>>> +	int WRONG(void) {}
>>>      	integer, intent(in) :: ChangeMe
>>> -	integer, intent(in) :: b
>>> -	integer, intent(out) :: c
>>> -
>>> -	c = a+b
>>> -
>>> -	end function RIGHT
>>>    	EOF
>>> -	git add "$fortran_file" &&
>>> -	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
>>> -	git commit -m "add fortran file" &&
>>> -	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
>>> -	mv "$fortran_file".tmp "$fortran_file" &&
>>> -	git add "$fortran_file" &&
>>> -	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
>>> -	git commit -m "change fortran file"
>>> -'
>>>    -test_expect_success 'blame -L :funcname with userdiff driver' '
>>> -	check_count -f fortran-external-function -L:RIGHT A 7 B 1
>>> +	fortran_file=file.f03 &&
>>> +	test_when_finished "rm .gitattributes" &&
>>> +	echo "$fortran_file diff=fortran" >.gitattributes &&
>>> +
>>> +	test_commit --author "A <A@test.git>" \
>>> +		"add" $fortran_file \
>>> +		"$(cat file.template)" &&
>>> +	test_commit --author "B <B@test.git>" \
>>> +		"change" $fortran_file \
>>> +		"$(cat file.template | sed -e s/ChangeMe/IWasChanged/)" &&
>>> +	check_count -f $fortran_file -L:RIGHT A 3 B 1
>>>    '
>>>      test_expect_success 'setup incremental' '
>>>
>>
>> I don't get the point. What do you need the tokens "WRONG" for when
>> they are not checked anywhere? Instead of adding unrelated lines (that
>> do not even look like Fortran), couldn't you just not remove some of
>> the others? In particular, the last one that contains "RIGHT" as well
>> may be useful to keep in order to show that the code is not confused
>> by it.
> 
> Isn't the point of the test to assert that we're using a userdiff driver
> over the built-in xdiff rules here?
> 
> We can imagine that a change to its default heuristics would be to find
> the first non-whitespace line, but it jumping over non-whitespace lines
> in search of a fortran-looking line doesn't seem like it would ever
> happen. Hence the WRONG lines.

Fair enough. A justification along these lines either as comment or in 
the commit message would have helped; the existing text in the commit 
message was not sufficient to get the point across. I was distracted by 
the capitalized "WRONG" token because we use a capitalized "RIGHT" token 
and check for it, so I expected that we also should check for "WRONG". 
Wouldn't it then just be sufficient to remove the blank lines from the 
exiting Fortran content?

>> Please place "$fortran_file" in dquotes on the check_count line.

I just noticed that there are more unquoted expansions above that line. 
(I'm just mentioning it for completeness.)

-- Hannes

[PATCH v2 12/27] userdiff tests: rewrite hunk header test infrastructure

[Ævar Arnfjörð Bjarmason]

Rewrite the hunk header test infrastructure introduced in
bfa7d01413 (t4018: an infrastructure to test hunk headers,
2014-03-21). See c228a5c077 (Merge branch 'js/userdiff-cc',
2014-03-31) for the whole series that commit was part of.

As noted in an earlier commit that change introduced the regression of
not testing for the full hunk line, but just whether "RIGHT" appeared
on it[1]. A preceding commit fixed that specific issue, but we were
still left with the inflexibility of the approach described in the
now-deleted t/t4018/README.

I.e. to add any sort of new tests that used the existing test data
we'd either need to add more files like the recently added (but now
deleted) *.ctx) files, using the filesystem as our test datastructure,
or introduce more parsing for the custom file format we were growing
here.

Let's instead just move this over to using a custom test
function. This makes it trivial to add new tests by adding new
optional parameters to the function. Let's still keep the relevant
files in the "t/t4018/" subdirectory instead of adding ~1.5k
lines (and growing) to "t/t4018-diff-funcname.sh"

If this diff is viewed with "--color-moved=plain" we can see that
there's no changes to the lines being moved into the new *.sh files,
i.e. all the deletions are moves. I'm just adding boilerplate around
those existing lines.

The one-off refactoring was performed by an ad-hoc shellscript [2].

1. https://lore.kernel.org/git/87wnvbbf2y.fsf@evledraar.gmail.com/
2.
	#!/bin/sh
	set -ex

	git rm README*
	for t in $(git ls-files ':!*.ctx')
	do
		lang=$(echo $t | sed 's/-.*//')
		desc=$(echo $t | sed -E 's/^[^-]*-//' | tr - " ")

		if ! test -e $lang.sh
		then
			cat >$lang.sh <<-EOF
			#!/bin/sh
			#
			# See ../t4018-diff-funcname.sh's test_diff_funcname()
			#

			EOF
		else
			echo >>$lang.sh
	        fi

		(
	            printf "test_diff_funcname '%s: %s' \\" "$lang" "$desc"
	            echo
	            printf "\t8<<%sEOF_HUNK 9<<%sEOF_TEST\n" '\' '\'
	            cat $t.ctx
	            printf "EOF_HUNK\n"
	            cat $t
	            printf "EOF_TEST\n"
		) >>$lang.sh

		chmod +x $lang.sh
		git add $lang.sh
	        git rm $t $t.ctx
	done

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh                      |  59 +++--
 t/t4018/README                                |  15 --
 t/t4018/README.ctx                            |   1 -
 t/t4018/bash-arithmetic-function              |   4 -
 t/t4018/bash-arithmetic-function.ctx          |   1 -
 t/t4018/bash-bashism-style-compact            |   6 -
 t/t4018/bash-bashism-style-compact.ctx        |   1 -
 t/t4018/bash-bashism-style-function           |   4 -
 t/t4018/bash-bashism-style-function.ctx       |   1 -
 t/t4018/bash-bashism-style-whitespace         |   4 -
 t/t4018/bash-bashism-style-whitespace.ctx     |   1 -
 t/t4018/bash-conditional-function             |   4 -
 t/t4018/bash-conditional-function.ctx         |   1 -
 t/t4018/bash-missing-parentheses              |   6 -
 t/t4018/bash-missing-parentheses.ctx          |   1 -
 t/t4018/bash-mixed-style-compact              |   4 -
 t/t4018/bash-mixed-style-compact.ctx          |   1 -
 t/t4018/bash-mixed-style-function             |   4 -
 t/t4018/bash-mixed-style-function.ctx         |   1 -
 t/t4018/bash-nested-functions                 |   6 -
 t/t4018/bash-nested-functions.ctx             |   1 -
 t/t4018/bash-other-characters                 |   4 -
 t/t4018/bash-other-characters.ctx             |   1 -
 t/t4018/bash-posix-style-compact              |   4 -
 t/t4018/bash-posix-style-compact.ctx          |   1 -
 t/t4018/bash-posix-style-function             |   4 -
 t/t4018/bash-posix-style-function.ctx         |   1 -
 t/t4018/bash-posix-style-whitespace           |   4 -
 t/t4018/bash-posix-style-whitespace.ctx       |   1 -
 t/t4018/bash-subshell-function                |   4 -
 t/t4018/bash-subshell-function.ctx            |   1 -
 t/t4018/bash-trailing-comment                 |   4 -
 t/t4018/bash-trailing-comment.ctx             |   1 -
 t/t4018/bash.sh                               | 160 ++++++++++++
 t/t4018/cpp-c++-function                      |   4 -
 t/t4018/cpp-c++-function.ctx                  |   1 -
 t/t4018/cpp-class-constructor                 |   4 -
 t/t4018/cpp-class-constructor-mem-init        |   5 -
 t/t4018/cpp-class-constructor-mem-init.ctx    |   1 -
 t/t4018/cpp-class-constructor.ctx             |   1 -
 t/t4018/cpp-class-definition                  |   4 -
 t/t4018/cpp-class-definition-derived          |   5 -
 t/t4018/cpp-class-definition-derived.ctx      |   1 -
 t/t4018/cpp-class-definition.ctx              |   1 -
 t/t4018/cpp-class-destructor                  |   4 -
 t/t4018/cpp-class-destructor.ctx              |   1 -
 t/t4018/cpp-function-returning-global-type    |   4 -
 .../cpp-function-returning-global-type.ctx    |   1 -
 t/t4018/cpp-function-returning-nested         |   5 -
 t/t4018/cpp-function-returning-nested.ctx     |   1 -
 t/t4018/cpp-function-returning-pointer        |   4 -
 t/t4018/cpp-function-returning-pointer.ctx    |   1 -
 t/t4018/cpp-function-returning-reference      |   4 -
 t/t4018/cpp-function-returning-reference.ctx  |   1 -
 t/t4018/cpp-gnu-style-function                |   5 -
 t/t4018/cpp-gnu-style-function.ctx            |   1 -
 t/t4018/cpp-namespace-definition              |   4 -
 t/t4018/cpp-namespace-definition.ctx          |   1 -
 t/t4018/cpp-operator-definition               |   4 -
 t/t4018/cpp-operator-definition.ctx           |   1 -
 t/t4018/cpp-skip-access-specifiers            |   8 -
 t/t4018/cpp-skip-access-specifiers.ctx        |   1 -
 t/t4018/cpp-skip-comment-block                |   9 -
 t/t4018/cpp-skip-comment-block.ctx            |   1 -
 t/t4018/cpp-skip-labels                       |   8 -
 t/t4018/cpp-skip-labels.ctx                   |   1 -
 t/t4018/cpp-struct-definition                 |   9 -
 t/t4018/cpp-struct-definition.ctx             |   1 -
 t/t4018/cpp-struct-single-line                |   7 -
 t/t4018/cpp-struct-single-line.ctx            |   1 -
 t/t4018/cpp-template-function-definition      |   4 -
 t/t4018/cpp-template-function-definition.ctx  |   1 -
 t/t4018/cpp-union-definition                  |   4 -
 t/t4018/cpp-union-definition.ctx              |   1 -
 t/t4018/cpp-void-c-function                   |   4 -
 t/t4018/cpp-void-c-function.ctx               |   1 -
 t/t4018/cpp.sh                                | 239 ++++++++++++++++++
 t/t4018/css-attribute-value-selector          |   4 -
 t/t4018/css-attribute-value-selector.ctx      |   1 -
 t/t4018/css-block-level-@-statements          |  10 -
 t/t4018/css-block-level-@-statements.ctx      |   1 -
 t/t4018/css-brace-in-col-1                    |   5 -
 t/t4018/css-brace-in-col-1.ctx                |   1 -
 t/t4018/css-class-selector                    |   4 -
 t/t4018/css-class-selector.ctx                |   1 -
 t/t4018/css-colon-eol                         |   4 -
 t/t4018/css-colon-eol.ctx                     |   1 -
 t/t4018/css-colon-selector                    |   5 -
 t/t4018/css-colon-selector.ctx                |   1 -
 t/t4018/css-common                            |   4 -
 t/t4018/css-common.ctx                        |   1 -
 t/t4018/css-id-selector                       |   4 -
 t/t4018/css-id-selector.ctx                   |   1 -
 t/t4018/css-long-selector-list                |   6 -
 t/t4018/css-long-selector-list.ctx            |   1 -
 t/t4018/css-prop-sans-indent                  |   5 -
 t/t4018/css-prop-sans-indent.ctx              |   1 -
 t/t4018/css-root-selector                     |   4 -
 t/t4018/css-root-selector.ctx                 |   1 -
 t/t4018/css-short-selector-list               |   4 -
 t/t4018/css-short-selector-list.ctx           |   1 -
 t/t4018/css-trailing-space                    |   5 -
 t/t4018/css-trailing-space.ctx                |   1 -
 t/t4018/css.sh                                | 146 +++++++++++
 t/t4018/custom1-pattern.ctx                   |   1 -
 t/t4018/{custom1-pattern => custom1.sh}       |  10 +
 t/t4018/custom2-match-to-end-of-line          |   8 -
 t/t4018/custom2-match-to-end-of-line.ctx      |   1 -
 t/t4018/custom2.sh                            |  18 ++
 t/t4018/custom3-alternation-in-pattern.ctx    |   1 -
 ...tom3-alternation-in-pattern => custom3.sh} |  10 +
 t/t4018/dts-labels                            |   9 -
 t/t4018/dts-labels.ctx                        |   1 -
 t/t4018/dts-node-unitless                     |   8 -
 t/t4018/dts-node-unitless.ctx                 |   1 -
 t/t4018/dts-nodes                             |   8 -
 t/t4018/dts-nodes-boolean-prop                |   9 -
 t/t4018/dts-nodes-boolean-prop.ctx            |   1 -
 t/t4018/dts-nodes-comment1                    |   8 -
 t/t4018/dts-nodes-comment1.ctx                |   1 -
 t/t4018/dts-nodes-comment2                    |   8 -
 t/t4018/dts-nodes-comment2.ctx                |   1 -
 t/t4018/dts-nodes-multiline-prop              |  13 -
 t/t4018/dts-nodes-multiline-prop.ctx          |   1 -
 t/t4018/dts-nodes.ctx                         |   1 -
 t/t4018/dts-reference                         |   9 -
 t/t4018/dts-reference.ctx                     |   1 -
 t/t4018/dts-root                              |   5 -
 t/t4018/dts-root-comment                      |   8 -
 t/t4018/dts-root-comment.ctx                  |   1 -
 t/t4018/dts-root.ctx                          |   1 -
 t/t4018/dts.sh                                | 149 +++++++++++
 t/t4018/elixir-do-not-pick-end                |   5 -
 t/t4018/elixir-do-not-pick-end.ctx            |   1 -
 t/t4018/elixir-ex-unit-test                   |   6 -
 t/t4018/elixir-ex-unit-test.ctx               |   1 -
 t/t4018/elixir-function                       |   5 -
 t/t4018/elixir-function.ctx                   |   1 -
 t/t4018/elixir-macro                          |   5 -
 t/t4018/elixir-macro.ctx                      |   1 -
 t/t4018/elixir-module                         |   9 -
 t/t4018/elixir-module-func                    |   8 -
 t/t4018/elixir-module-func.ctx                |   1 -
 t/t4018/elixir-module.ctx                     |   1 -
 t/t4018/elixir-nested-module                  |   9 -
 t/t4018/elixir-nested-module.ctx              |   1 -
 t/t4018/elixir-private-function               |   5 -
 t/t4018/elixir-private-function.ctx           |   1 -
 t/t4018/elixir-protocol                       |   6 -
 t/t4018/elixir-protocol-implementation        |   5 -
 t/t4018/elixir-protocol-implementation.ctx    |   1 -
 t/t4018/elixir-protocol.ctx                   |   1 -
 t/t4018/elixir.sh                             | 127 ++++++++++
 t/t4018/fortran-block-data                    |   5 -
 t/t4018/fortran-block-data.ctx                |   1 -
 t/t4018/fortran-comment                       |  13 -
 t/t4018/fortran-comment-keyword               |  14 -
 t/t4018/fortran-comment-keyword.ctx           |   1 -
 t/t4018/fortran-comment-legacy                |  13 -
 t/t4018/fortran-comment-legacy-star           |  13 -
 t/t4018/fortran-comment-legacy-star.ctx       |   1 -
 t/t4018/fortran-comment-legacy.ctx            |   1 -
 t/t4018/fortran-comment.ctx                   |   1 -
 t/t4018/fortran-external-function             |   9 -
 t/t4018/fortran-external-function.ctx         |   1 -
 t/t4018/fortran-external-subroutine           |   5 -
 t/t4018/fortran-external-subroutine.ctx       |   1 -
 t/t4018/fortran-module                        |   5 -
 t/t4018/fortran-module-procedure              |  13 -
 t/t4018/fortran-module-procedure.ctx          |   1 -
 t/t4018/fortran-module.ctx                    |   1 -
 t/t4018/fortran-program                       |   5 -
 t/t4018/fortran-program.ctx                   |   1 -
 t/t4018/fortran.sh                            | 159 ++++++++++++
 t/t4018/fountain-scene                        |   4 -
 t/t4018/fountain-scene.ctx                    |   1 -
 t/t4018/fountain.sh                           |  14 +
 t/t4018/golang-complex-function               |   8 -
 t/t4018/golang-complex-function.ctx           |   1 -
 t/t4018/golang-func                           |   4 -
 t/t4018/golang-func.ctx                       |   1 -
 t/t4018/golang-interface                      |   4 -
 t/t4018/golang-interface.ctx                  |   1 -
 t/t4018/golang-long-func                      |   5 -
 t/t4018/golang-long-func.ctx                  |   1 -
 t/t4018/golang-struct                         |   4 -
 t/t4018/golang-struct.ctx                     |   1 -
 t/t4018/golang.sh                             |  59 +++++
 t/t4018/java-class-member-function            |   8 -
 t/t4018/java-class-member-function.ctx        |   1 -
 t/t4018/java.sh                               |  18 ++
 t/t4018/markdown-heading-indented             |   6 -
 t/t4018/markdown-heading-indented.ctx         |   1 -
 t/t4018/markdown-heading-non-headings         |  17 --
 t/t4018/markdown-heading-non-headings.ctx     |   1 -
 t/t4018/markdown.sh                           |  39 +++
 t/t4018/matlab-class-definition               |   5 -
 t/t4018/matlab-class-definition.ctx           |   1 -
 t/t4018/matlab-function                       |   4 -
 t/t4018/matlab-function.ctx                   |   1 -
 t/t4018/matlab-octave-section-1               |   3 -
 t/t4018/matlab-octave-section-1.ctx           |   1 -
 t/t4018/matlab-octave-section-2               |   3 -
 t/t4018/matlab-octave-section-2.ctx           |   1 -
 t/t4018/matlab-section                        |   3 -
 t/t4018/matlab-section.ctx                    |   1 -
 t/t4018/matlab.sh                             |  52 ++++
 t/t4018/perl-skip-end-of-heredoc              |   8 -
 t/t4018/perl-skip-end-of-heredoc.ctx          |   1 -
 t/t4018/perl-skip-forward-decl                |  10 -
 t/t4018/perl-skip-forward-decl.ctx            |   1 -
 t/t4018/perl-skip-sub-in-pod                  |  18 --
 t/t4018/perl-skip-sub-in-pod.ctx              |   1 -
 t/t4018/perl-sub-definition                   |   4 -
 t/t4018/perl-sub-definition-kr-brace          |   4 -
 t/t4018/perl-sub-definition-kr-brace.ctx      |   1 -
 t/t4018/perl-sub-definition.ctx               |   1 -
 t/t4018/perl.sh                               |  78 ++++++
 t/t4018/php-abstract-class                    |   4 -
 t/t4018/php-abstract-class.ctx                |   1 -
 t/t4018/php-abstract-method                   |   7 -
 t/t4018/php-abstract-method.ctx               |   1 -
 t/t4018/php-class                             |   4 -
 t/t4018/php-class.ctx                         |   1 -
 t/t4018/php-final-class                       |   4 -
 t/t4018/php-final-class.ctx                   |   1 -
 t/t4018/php-final-method                      |   7 -
 t/t4018/php-final-method.ctx                  |   1 -
 t/t4018/php-function                          |   4 -
 t/t4018/php-function.ctx                      |   1 -
 t/t4018/php-interface                         |   4 -
 t/t4018/php-interface.ctx                     |   1 -
 t/t4018/php-method                            |   7 -
 t/t4018/php-method.ctx                        |   1 -
 t/t4018/php-trait                             |   7 -
 t/t4018/php-trait.ctx                         |   1 -
 t/t4018/php.sh                                | 106 ++++++++
 t/t4018/python-async-def                      |   4 -
 t/t4018/python-async-def.ctx                  |   1 -
 t/t4018/python-class                          |   4 -
 t/t4018/python-class.ctx                      |   1 -
 t/t4018/python-def                            |   4 -
 t/t4018/python-def.ctx                        |   1 -
 t/t4018/python-indented-async-def             |   7 -
 t/t4018/python-indented-async-def.ctx         |   1 -
 t/t4018/python-indented-class                 |   5 -
 t/t4018/python-indented-class.ctx             |   1 -
 t/t4018/python-indented-def                   |   7 -
 t/t4018/python-indented-def.ctx               |   1 -
 t/t4018/python.sh                             |  71 ++++++
 t/t4018/rust-fn                               |   5 -
 t/t4018/rust-fn.ctx                           |   1 -
 t/t4018/rust-impl                             |   5 -
 t/t4018/rust-impl.ctx                         |   1 -
 t/t4018/rust-macro-rules                      |   6 -
 t/t4018/rust-macro-rules.ctx                  |   1 -
 t/t4018/rust-struct                           |   5 -
 t/t4018/rust-struct.ctx                       |   1 -
 t/t4018/rust-trait                            |   5 -
 t/t4018/rust-trait.ctx                        |   1 -
 t/t4018/rust.sh                               |  60 +++++
 261 files changed, 1549 insertions(+), 879 deletions(-)
 delete mode 100644 t/t4018/README
 delete mode 100644 t/t4018/README.ctx
 delete mode 100644 t/t4018/bash-arithmetic-function
 delete mode 100644 t/t4018/bash-arithmetic-function.ctx
 delete mode 100644 t/t4018/bash-bashism-style-compact
 delete mode 100644 t/t4018/bash-bashism-style-compact.ctx
 delete mode 100644 t/t4018/bash-bashism-style-function
 delete mode 100644 t/t4018/bash-bashism-style-function.ctx
 delete mode 100644 t/t4018/bash-bashism-style-whitespace
 delete mode 100644 t/t4018/bash-bashism-style-whitespace.ctx
 delete mode 100644 t/t4018/bash-conditional-function
 delete mode 100644 t/t4018/bash-conditional-function.ctx
 delete mode 100644 t/t4018/bash-missing-parentheses
 delete mode 100644 t/t4018/bash-missing-parentheses.ctx
 delete mode 100644 t/t4018/bash-mixed-style-compact
 delete mode 100644 t/t4018/bash-mixed-style-compact.ctx
 delete mode 100644 t/t4018/bash-mixed-style-function
 delete mode 100644 t/t4018/bash-mixed-style-function.ctx
 delete mode 100644 t/t4018/bash-nested-functions
 delete mode 100644 t/t4018/bash-nested-functions.ctx
 delete mode 100644 t/t4018/bash-other-characters
 delete mode 100644 t/t4018/bash-other-characters.ctx
 delete mode 100644 t/t4018/bash-posix-style-compact
 delete mode 100644 t/t4018/bash-posix-style-compact.ctx
 delete mode 100644 t/t4018/bash-posix-style-function
 delete mode 100644 t/t4018/bash-posix-style-function.ctx
 delete mode 100644 t/t4018/bash-posix-style-whitespace
 delete mode 100644 t/t4018/bash-posix-style-whitespace.ctx
 delete mode 100644 t/t4018/bash-subshell-function
 delete mode 100644 t/t4018/bash-subshell-function.ctx
 delete mode 100644 t/t4018/bash-trailing-comment
 delete mode 100644 t/t4018/bash-trailing-comment.ctx
 create mode 100755 t/t4018/bash.sh
 delete mode 100644 t/t4018/cpp-c++-function
 delete mode 100644 t/t4018/cpp-c++-function.ctx
 delete mode 100644 t/t4018/cpp-class-constructor
 delete mode 100644 t/t4018/cpp-class-constructor-mem-init
 delete mode 100644 t/t4018/cpp-class-constructor-mem-init.ctx
 delete mode 100644 t/t4018/cpp-class-constructor.ctx
 delete mode 100644 t/t4018/cpp-class-definition
 delete mode 100644 t/t4018/cpp-class-definition-derived
 delete mode 100644 t/t4018/cpp-class-definition-derived.ctx
 delete mode 100644 t/t4018/cpp-class-definition.ctx
 delete mode 100644 t/t4018/cpp-class-destructor
 delete mode 100644 t/t4018/cpp-class-destructor.ctx
 delete mode 100644 t/t4018/cpp-function-returning-global-type
 delete mode 100644 t/t4018/cpp-function-returning-global-type.ctx
 delete mode 100644 t/t4018/cpp-function-returning-nested
 delete mode 100644 t/t4018/cpp-function-returning-nested.ctx
 delete mode 100644 t/t4018/cpp-function-returning-pointer
 delete mode 100644 t/t4018/cpp-function-returning-pointer.ctx
 delete mode 100644 t/t4018/cpp-function-returning-reference
 delete mode 100644 t/t4018/cpp-function-returning-reference.ctx
 delete mode 100644 t/t4018/cpp-gnu-style-function
 delete mode 100644 t/t4018/cpp-gnu-style-function.ctx
 delete mode 100644 t/t4018/cpp-namespace-definition
 delete mode 100644 t/t4018/cpp-namespace-definition.ctx
 delete mode 100644 t/t4018/cpp-operator-definition
 delete mode 100644 t/t4018/cpp-operator-definition.ctx
 delete mode 100644 t/t4018/cpp-skip-access-specifiers
 delete mode 100644 t/t4018/cpp-skip-access-specifiers.ctx
 delete mode 100644 t/t4018/cpp-skip-comment-block
 delete mode 100644 t/t4018/cpp-skip-comment-block.ctx
 delete mode 100644 t/t4018/cpp-skip-labels
 delete mode 100644 t/t4018/cpp-skip-labels.ctx
 delete mode 100644 t/t4018/cpp-struct-definition
 delete mode 100644 t/t4018/cpp-struct-definition.ctx
 delete mode 100644 t/t4018/cpp-struct-single-line
 delete mode 100644 t/t4018/cpp-struct-single-line.ctx
 delete mode 100644 t/t4018/cpp-template-function-definition
 delete mode 100644 t/t4018/cpp-template-function-definition.ctx
 delete mode 100644 t/t4018/cpp-union-definition
 delete mode 100644 t/t4018/cpp-union-definition.ctx
 delete mode 100644 t/t4018/cpp-void-c-function
 delete mode 100644 t/t4018/cpp-void-c-function.ctx
 create mode 100755 t/t4018/cpp.sh
 delete mode 100644 t/t4018/css-attribute-value-selector
 delete mode 100644 t/t4018/css-attribute-value-selector.ctx
 delete mode 100644 t/t4018/css-block-level-@-statements
 delete mode 100644 t/t4018/css-block-level-@-statements.ctx
 delete mode 100644 t/t4018/css-brace-in-col-1
 delete mode 100644 t/t4018/css-brace-in-col-1.ctx
 delete mode 100644 t/t4018/css-class-selector
 delete mode 100644 t/t4018/css-class-selector.ctx
 delete mode 100644 t/t4018/css-colon-eol
 delete mode 100644 t/t4018/css-colon-eol.ctx
 delete mode 100644 t/t4018/css-colon-selector
 delete mode 100644 t/t4018/css-colon-selector.ctx
 delete mode 100644 t/t4018/css-common
 delete mode 100644 t/t4018/css-common.ctx
 delete mode 100644 t/t4018/css-id-selector
 delete mode 100644 t/t4018/css-id-selector.ctx
 delete mode 100644 t/t4018/css-long-selector-list
 delete mode 100644 t/t4018/css-long-selector-list.ctx
 delete mode 100644 t/t4018/css-prop-sans-indent
 delete mode 100644 t/t4018/css-prop-sans-indent.ctx
 delete mode 100644 t/t4018/css-root-selector
 delete mode 100644 t/t4018/css-root-selector.ctx
 delete mode 100644 t/t4018/css-short-selector-list
 delete mode 100644 t/t4018/css-short-selector-list.ctx
 delete mode 100644 t/t4018/css-trailing-space
 delete mode 100644 t/t4018/css-trailing-space.ctx
 create mode 100755 t/t4018/css.sh
 delete mode 100644 t/t4018/custom1-pattern.ctx
 rename t/t4018/{custom1-pattern => custom1.sh} (71%)
 mode change 100644 => 100755
 delete mode 100644 t/t4018/custom2-match-to-end-of-line
 delete mode 100644 t/t4018/custom2-match-to-end-of-line.ctx
 create mode 100755 t/t4018/custom2.sh
 delete mode 100644 t/t4018/custom3-alternation-in-pattern.ctx
 rename t/t4018/{custom3-alternation-in-pattern => custom3.sh} (66%)
 mode change 100644 => 100755
 delete mode 100644 t/t4018/dts-labels
 delete mode 100644 t/t4018/dts-labels.ctx
 delete mode 100644 t/t4018/dts-node-unitless
 delete mode 100644 t/t4018/dts-node-unitless.ctx
 delete mode 100644 t/t4018/dts-nodes
 delete mode 100644 t/t4018/dts-nodes-boolean-prop
 delete mode 100644 t/t4018/dts-nodes-boolean-prop.ctx
 delete mode 100644 t/t4018/dts-nodes-comment1
 delete mode 100644 t/t4018/dts-nodes-comment1.ctx
 delete mode 100644 t/t4018/dts-nodes-comment2
 delete mode 100644 t/t4018/dts-nodes-comment2.ctx
 delete mode 100644 t/t4018/dts-nodes-multiline-prop
 delete mode 100644 t/t4018/dts-nodes-multiline-prop.ctx
 delete mode 100644 t/t4018/dts-nodes.ctx
 delete mode 100644 t/t4018/dts-reference
 delete mode 100644 t/t4018/dts-reference.ctx
 delete mode 100644 t/t4018/dts-root
 delete mode 100644 t/t4018/dts-root-comment
 delete mode 100644 t/t4018/dts-root-comment.ctx
 delete mode 100644 t/t4018/dts-root.ctx
 create mode 100755 t/t4018/dts.sh
 delete mode 100644 t/t4018/elixir-do-not-pick-end
 delete mode 100644 t/t4018/elixir-do-not-pick-end.ctx
 delete mode 100644 t/t4018/elixir-ex-unit-test
 delete mode 100644 t/t4018/elixir-ex-unit-test.ctx
 delete mode 100644 t/t4018/elixir-function
 delete mode 100644 t/t4018/elixir-function.ctx
 delete mode 100644 t/t4018/elixir-macro
 delete mode 100644 t/t4018/elixir-macro.ctx
 delete mode 100644 t/t4018/elixir-module
 delete mode 100644 t/t4018/elixir-module-func
 delete mode 100644 t/t4018/elixir-module-func.ctx
 delete mode 100644 t/t4018/elixir-module.ctx
 delete mode 100644 t/t4018/elixir-nested-module
 delete mode 100644 t/t4018/elixir-nested-module.ctx
 delete mode 100644 t/t4018/elixir-private-function
 delete mode 100644 t/t4018/elixir-private-function.ctx
 delete mode 100644 t/t4018/elixir-protocol
 delete mode 100644 t/t4018/elixir-protocol-implementation
 delete mode 100644 t/t4018/elixir-protocol-implementation.ctx
 delete mode 100644 t/t4018/elixir-protocol.ctx
 create mode 100755 t/t4018/elixir.sh
 delete mode 100644 t/t4018/fortran-block-data
 delete mode 100644 t/t4018/fortran-block-data.ctx
 delete mode 100644 t/t4018/fortran-comment
 delete mode 100644 t/t4018/fortran-comment-keyword
 delete mode 100644 t/t4018/fortran-comment-keyword.ctx
 delete mode 100644 t/t4018/fortran-comment-legacy
 delete mode 100644 t/t4018/fortran-comment-legacy-star
 delete mode 100644 t/t4018/fortran-comment-legacy-star.ctx
 delete mode 100644 t/t4018/fortran-comment-legacy.ctx
 delete mode 100644 t/t4018/fortran-comment.ctx
 delete mode 100644 t/t4018/fortran-external-function
 delete mode 100644 t/t4018/fortran-external-function.ctx
 delete mode 100644 t/t4018/fortran-external-subroutine
 delete mode 100644 t/t4018/fortran-external-subroutine.ctx
 delete mode 100644 t/t4018/fortran-module
 delete mode 100644 t/t4018/fortran-module-procedure
 delete mode 100644 t/t4018/fortran-module-procedure.ctx
 delete mode 100644 t/t4018/fortran-module.ctx
 delete mode 100644 t/t4018/fortran-program
 delete mode 100644 t/t4018/fortran-program.ctx
 create mode 100755 t/t4018/fortran.sh
 delete mode 100644 t/t4018/fountain-scene
 delete mode 100644 t/t4018/fountain-scene.ctx
 create mode 100755 t/t4018/fountain.sh
 delete mode 100644 t/t4018/golang-complex-function
 delete mode 100644 t/t4018/golang-complex-function.ctx
 delete mode 100644 t/t4018/golang-func
 delete mode 100644 t/t4018/golang-func.ctx
 delete mode 100644 t/t4018/golang-interface
 delete mode 100644 t/t4018/golang-interface.ctx
 delete mode 100644 t/t4018/golang-long-func
 delete mode 100644 t/t4018/golang-long-func.ctx
 delete mode 100644 t/t4018/golang-struct
 delete mode 100644 t/t4018/golang-struct.ctx
 create mode 100755 t/t4018/golang.sh
 delete mode 100644 t/t4018/java-class-member-function
 delete mode 100644 t/t4018/java-class-member-function.ctx
 create mode 100755 t/t4018/java.sh
 delete mode 100644 t/t4018/markdown-heading-indented
 delete mode 100644 t/t4018/markdown-heading-indented.ctx
 delete mode 100644 t/t4018/markdown-heading-non-headings
 delete mode 100644 t/t4018/markdown-heading-non-headings.ctx
 create mode 100755 t/t4018/markdown.sh
 delete mode 100644 t/t4018/matlab-class-definition
 delete mode 100644 t/t4018/matlab-class-definition.ctx
 delete mode 100644 t/t4018/matlab-function
 delete mode 100644 t/t4018/matlab-function.ctx
 delete mode 100644 t/t4018/matlab-octave-section-1
 delete mode 100644 t/t4018/matlab-octave-section-1.ctx
 delete mode 100644 t/t4018/matlab-octave-section-2
 delete mode 100644 t/t4018/matlab-octave-section-2.ctx
 delete mode 100644 t/t4018/matlab-section
 delete mode 100644 t/t4018/matlab-section.ctx
 create mode 100755 t/t4018/matlab.sh
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc.ctx
 delete mode 100644 t/t4018/perl-skip-forward-decl
 delete mode 100644 t/t4018/perl-skip-forward-decl.ctx
 delete mode 100644 t/t4018/perl-skip-sub-in-pod
 delete mode 100644 t/t4018/perl-skip-sub-in-pod.ctx
 delete mode 100644 t/t4018/perl-sub-definition
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace.ctx
 delete mode 100644 t/t4018/perl-sub-definition.ctx
 create mode 100755 t/t4018/perl.sh
 delete mode 100644 t/t4018/php-abstract-class
 delete mode 100644 t/t4018/php-abstract-class.ctx
 delete mode 100644 t/t4018/php-abstract-method
 delete mode 100644 t/t4018/php-abstract-method.ctx
 delete mode 100644 t/t4018/php-class
 delete mode 100644 t/t4018/php-class.ctx
 delete mode 100644 t/t4018/php-final-class
 delete mode 100644 t/t4018/php-final-class.ctx
 delete mode 100644 t/t4018/php-final-method
 delete mode 100644 t/t4018/php-final-method.ctx
 delete mode 100644 t/t4018/php-function
 delete mode 100644 t/t4018/php-function.ctx
 delete mode 100644 t/t4018/php-interface
 delete mode 100644 t/t4018/php-interface.ctx
 delete mode 100644 t/t4018/php-method
 delete mode 100644 t/t4018/php-method.ctx
 delete mode 100644 t/t4018/php-trait
 delete mode 100644 t/t4018/php-trait.ctx
 create mode 100755 t/t4018/php.sh
 delete mode 100644 t/t4018/python-async-def
 delete mode 100644 t/t4018/python-async-def.ctx
 delete mode 100644 t/t4018/python-class
 delete mode 100644 t/t4018/python-class.ctx
 delete mode 100644 t/t4018/python-def
 delete mode 100644 t/t4018/python-def.ctx
 delete mode 100644 t/t4018/python-indented-async-def
 delete mode 100644 t/t4018/python-indented-async-def.ctx
 delete mode 100644 t/t4018/python-indented-class
 delete mode 100644 t/t4018/python-indented-class.ctx
 delete mode 100644 t/t4018/python-indented-def
 delete mode 100644 t/t4018/python-indented-def.ctx
 create mode 100755 t/t4018/python.sh
 delete mode 100644 t/t4018/rust-fn
 delete mode 100644 t/t4018/rust-fn.ctx
 delete mode 100644 t/t4018/rust-impl
 delete mode 100644 t/t4018/rust-impl.ctx
 delete mode 100644 t/t4018/rust-macro-rules
 delete mode 100644 t/t4018/rust-macro-rules.ctx
 delete mode 100644 t/t4018/rust-struct
 delete mode 100644 t/t4018/rust-struct.ctx
 delete mode 100644 t/t4018/rust-trait
 delete mode 100644 t/t4018/rust-trait.ctx
 create mode 100755 t/t4018/rust.sh

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 3941316682..decf7961f9 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -60,33 +60,42 @@ test_expect_success 'last regexp must not be negated' '
 	test_i18ngrep ": Last expression must not be negated:" msg
 '
 
-test_expect_success 'setup hunk header tests' '
-	for i in $diffpatterns
-	do
-		echo "$i-* diff=$i"
-	done > .gitattributes &&
-
-	# add all test files to the index
-	(
-		cd "$TEST_DIRECTORY"/t4018 &&
-		git --git-dir="$TRASH_DIRECTORY/.git" add .
-	) &&
-
-	# place modified files in the worktree
-	for i in $(git ls-files)
-	do
-		sed -e "s/ChangeMe/IWasChanged/" <"$TEST_DIRECTORY/t4018/$i" >"$i" || return 1
-	done
-'
+test_diff_funcname () {
+	desc=$1
+	cat <&8 >arg.header &&
+	cat <&9 >arg.test &&
+	what=$(cat arg.what) &&
+
+	test_expect_success "setup: $desc" '
+		cp arg.test "$what" &&
+		cp arg.header expected &&
+		git add "$what" &&
+		sed -e "s/ChangeMe/IWasChanged/" <"$what" >tmp &&
+		mv tmp "$what"
+	' &&
+
+	test_expect_success "$desc" '
+		git diff -U1 "$what" >diff &&
+		sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <diff >actual &&
+		test_cmp expected actual
+	'
+}
 
-# check each individual file
-for i in $(git ls-files -- ':!*.ctx')
+for what in $diffpatterns
 do
-	test_expect_success "hunk header: $i" "
-		git diff -U1 $i >diff &&
-		sed -n -e 's/^.*@@$//p' -e 's/^.*@@ //p' <diff >ctx &&
-		test_cmp $i.ctx ctx
-	"
+	test="$TEST_DIRECTORY/t4018/$what.sh"
+	if ! test -e "$test"
+	then
+		test_expect_failure "$what: no tests" 'false'
+		continue
+	fi &&
+
+	test_expect_success "setup: hunk header for $what" '
+		echo "$what diff=$what" >.gitattributes &&
+		echo "$what" >arg.what
+	' &&
+
+	. "$test"
 done
 
 test_done
diff --git a/t/t4018/README b/t/t4018/README
deleted file mode 100644
index d0619f76d4..0000000000
--- a/t/t4018/README
+++ /dev/null
@@ -1,15 +0,0 @@
-How to write test cases
-=======================
-
-Create test cases called "LANG-whatever" in this directory, where
-"LANG" is e.g. the userdiff driver name, where "whatever" is a brief
-description of the test.
-
-Insert the word "ChangeMe" (exactly this form) at a distance of
-at least two lines from the line that must appear in the hunk header.
-
-The text that must appear in the hunk header must contains the word
-"RIGHT" by convention. The "LANG-whatever.ctx" file contains what we
-expect to appear in the hunk header. We munged away the starting "@@
-[...] @@" part of the line for ease of not having to hardcode the line
-numbers and offsets.
diff --git a/t/t4018/README.ctx b/t/t4018/README.ctx
deleted file mode 100644
index cd79384b04..0000000000
--- a/t/t4018/README.ctx
+++ /dev/null
@@ -1 +0,0 @@
-description of the test.
diff --git a/t/t4018/bash-arithmetic-function b/t/t4018/bash-arithmetic-function
deleted file mode 100644
index c0b276cb50..0000000000
--- a/t/t4018/bash-arithmetic-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() ((
-
-    ChangeMe = "$x" + "$y"
-))
diff --git a/t/t4018/bash-arithmetic-function.ctx b/t/t4018/bash-arithmetic-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-arithmetic-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-bashism-style-compact b/t/t4018/bash-bashism-style-compact
deleted file mode 100644
index 1ca3126f61..0000000000
--- a/t/t4018/bash-bashism-style-compact
+++ /dev/null
@@ -1,6 +0,0 @@
-function RIGHT {
-    function InvalidSyntax{
-        :
-        echo 'ChangeMe'
-    }
-}
diff --git a/t/t4018/bash-bashism-style-compact.ctx b/t/t4018/bash-bashism-style-compact.ctx
deleted file mode 100644
index 4f8eac48c6..0000000000
--- a/t/t4018/bash-bashism-style-compact.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT {
diff --git a/t/t4018/bash-bashism-style-function b/t/t4018/bash-bashism-style-function
deleted file mode 100644
index f1de4fa831..0000000000
--- a/t/t4018/bash-bashism-style-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT {
-    :
-    echo 'ChangeMe'
-}
diff --git a/t/t4018/bash-bashism-style-function.ctx b/t/t4018/bash-bashism-style-function.ctx
deleted file mode 100644
index 4f8eac48c6..0000000000
--- a/t/t4018/bash-bashism-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT {
diff --git a/t/t4018/bash-bashism-style-whitespace b/t/t4018/bash-bashism-style-whitespace
deleted file mode 100644
index ade85dd3a5..0000000000
--- a/t/t4018/bash-bashism-style-whitespace
+++ /dev/null
@@ -1,4 +0,0 @@
-	 function 	RIGHT 	( 	) 	{
-
-	    ChangeMe
-	 }
diff --git a/t/t4018/bash-bashism-style-whitespace.ctx b/t/t4018/bash-bashism-style-whitespace.ctx
deleted file mode 100644
index 35dbd0220e..0000000000
--- a/t/t4018/bash-bashism-style-whitespace.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function 	RIGHT 	( 	) 	{
diff --git a/t/t4018/bash-conditional-function b/t/t4018/bash-conditional-function
deleted file mode 100644
index c5949e829b..0000000000
--- a/t/t4018/bash-conditional-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() [[ \
-
-    "$a" > "$ChangeMe"
-]]
diff --git a/t/t4018/bash-conditional-function.ctx b/t/t4018/bash-conditional-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-conditional-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-missing-parentheses b/t/t4018/bash-missing-parentheses
deleted file mode 100644
index 8c8a05dd7a..0000000000
--- a/t/t4018/bash-missing-parentheses
+++ /dev/null
@@ -1,6 +0,0 @@
-function RIGHT {
-    functionInvalidSyntax {
-        :
-        echo 'ChangeMe'
-    }
-}
diff --git a/t/t4018/bash-missing-parentheses.ctx b/t/t4018/bash-missing-parentheses.ctx
deleted file mode 100644
index 4f8eac48c6..0000000000
--- a/t/t4018/bash-missing-parentheses.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT {
diff --git a/t/t4018/bash-mixed-style-compact b/t/t4018/bash-mixed-style-compact
deleted file mode 100644
index d9364cba67..0000000000
--- a/t/t4018/bash-mixed-style-compact
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT(){
-    :
-    echo 'ChangeMe'
-}
diff --git a/t/t4018/bash-mixed-style-compact.ctx b/t/t4018/bash-mixed-style-compact.ctx
deleted file mode 100644
index bba11074eb..0000000000
--- a/t/t4018/bash-mixed-style-compact.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT(){
diff --git a/t/t4018/bash-mixed-style-function b/t/t4018/bash-mixed-style-function
deleted file mode 100644
index 555f9b2466..0000000000
--- a/t/t4018/bash-mixed-style-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT() {
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-mixed-style-function.ctx b/t/t4018/bash-mixed-style-function.ctx
deleted file mode 100644
index 922b87a4aa..0000000000
--- a/t/t4018/bash-mixed-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT() {
diff --git a/t/t4018/bash-nested-functions b/t/t4018/bash-nested-functions
deleted file mode 100644
index 2c9237ead4..0000000000
--- a/t/t4018/bash-nested-functions
+++ /dev/null
@@ -1,6 +0,0 @@
-outer() {
-    RIGHT() {
-        :
-        echo 'ChangeMe'
-    }
-}
diff --git a/t/t4018/bash-nested-functions.ctx b/t/t4018/bash-nested-functions.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-nested-functions.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-other-characters b/t/t4018/bash-other-characters
deleted file mode 100644
index a3f390d525..0000000000
--- a/t/t4018/bash-other-characters
+++ /dev/null
@@ -1,4 +0,0 @@
-_RIGHT_0n() {
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-other-characters.ctx b/t/t4018/bash-other-characters.ctx
deleted file mode 100644
index 6a55317fdf..0000000000
--- a/t/t4018/bash-other-characters.ctx
+++ /dev/null
@@ -1 +0,0 @@
-_RIGHT_0n()
diff --git a/t/t4018/bash-posix-style-compact b/t/t4018/bash-posix-style-compact
deleted file mode 100644
index 045bd2029b..0000000000
--- a/t/t4018/bash-posix-style-compact
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT(){
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-posix-style-compact.ctx b/t/t4018/bash-posix-style-compact.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-posix-style-compact.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-posix-style-function b/t/t4018/bash-posix-style-function
deleted file mode 100644
index a4d144856e..0000000000
--- a/t/t4018/bash-posix-style-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() {
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-posix-style-function.ctx b/t/t4018/bash-posix-style-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-posix-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-posix-style-whitespace b/t/t4018/bash-posix-style-whitespace
deleted file mode 100644
index 4d984f0aa4..0000000000
--- a/t/t4018/bash-posix-style-whitespace
+++ /dev/null
@@ -1,4 +0,0 @@
-	 RIGHT 	( 	) 	{
-
-	    ChangeMe
-	 }
diff --git a/t/t4018/bash-posix-style-whitespace.ctx b/t/t4018/bash-posix-style-whitespace.ctx
deleted file mode 100644
index 28f8698e14..0000000000
--- a/t/t4018/bash-posix-style-whitespace.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT 	( 	)
diff --git a/t/t4018/bash-subshell-function b/t/t4018/bash-subshell-function
deleted file mode 100644
index 80baa09484..0000000000
--- a/t/t4018/bash-subshell-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() (
-
-    ChangeMe=2
-)
diff --git a/t/t4018/bash-subshell-function.ctx b/t/t4018/bash-subshell-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-subshell-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-trailing-comment b/t/t4018/bash-trailing-comment
deleted file mode 100644
index f1edbeda31..0000000000
--- a/t/t4018/bash-trailing-comment
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() { # Comment
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-trailing-comment.ctx b/t/t4018/bash-trailing-comment.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-trailing-comment.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash.sh b/t/t4018/bash.sh
new file mode 100755
index 0000000000..69144d9144
--- /dev/null
+++ b/t/t4018/bash.sh
@@ -0,0 +1,160 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'bash: arithmetic function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() ((
+
+    ChangeMe = "$x" + "$y"
+))
+EOF_TEST
+
+test_diff_funcname 'bash: bashism style compact' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT {
+EOF_HUNK
+function RIGHT {
+    function InvalidSyntax{
+        :
+        echo 'ChangeMe'
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'bash: bashism style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT {
+EOF_HUNK
+function RIGHT {
+    :
+    echo 'ChangeMe'
+}
+EOF_TEST
+
+test_diff_funcname 'bash: bashism style whitespace' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function 	RIGHT 	( 	) 	{
+EOF_HUNK
+	 function 	RIGHT 	( 	) 	{
+
+	    ChangeMe
+	 }
+EOF_TEST
+
+test_diff_funcname 'bash: conditional function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() [[ \
+
+    "$a" > "$ChangeMe"
+]]
+EOF_TEST
+
+test_diff_funcname 'bash: missing parentheses' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT {
+EOF_HUNK
+function RIGHT {
+    functionInvalidSyntax {
+        :
+        echo 'ChangeMe'
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'bash: mixed style compact' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT(){
+EOF_HUNK
+function RIGHT(){
+    :
+    echo 'ChangeMe'
+}
+EOF_TEST
+
+test_diff_funcname 'bash: mixed style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT() {
+EOF_HUNK
+function RIGHT() {
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: nested functions' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+outer() {
+    RIGHT() {
+        :
+        echo 'ChangeMe'
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'bash: other characters' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+_RIGHT_0n()
+EOF_HUNK
+_RIGHT_0n() {
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: posix style compact' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT(){
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: posix style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() {
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: posix style whitespace' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT 	( 	)
+EOF_HUNK
+	 RIGHT 	( 	) 	{
+
+	    ChangeMe
+	 }
+EOF_TEST
+
+test_diff_funcname 'bash: subshell function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() (
+
+    ChangeMe=2
+)
+EOF_TEST
+
+test_diff_funcname 'bash: trailing comment' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() { # Comment
+
+    ChangeMe
+}
+EOF_TEST
diff --git a/t/t4018/cpp-c++-function b/t/t4018/cpp-c++-function
deleted file mode 100644
index 9ee6bbef55..0000000000
--- a/t/t4018/cpp-c++-function
+++ /dev/null
@@ -1,4 +0,0 @@
-Item RIGHT::DoSomething( Args with_spaces )
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-c++-function.ctx b/t/t4018/cpp-c++-function.ctx
deleted file mode 100644
index 337b49dbb3..0000000000
--- a/t/t4018/cpp-c++-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Item RIGHT::DoSomething( Args with_spaces )
diff --git a/t/t4018/cpp-class-constructor b/t/t4018/cpp-class-constructor
deleted file mode 100644
index ec4f115c25..0000000000
--- a/t/t4018/cpp-class-constructor
+++ /dev/null
@@ -1,4 +0,0 @@
-Item::Item(int RIGHT)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-class-constructor-mem-init b/t/t4018/cpp-class-constructor-mem-init
deleted file mode 100644
index 49a69f37e1..0000000000
--- a/t/t4018/cpp-class-constructor-mem-init
+++ /dev/null
@@ -1,5 +0,0 @@
-Item::Item(int RIGHT) :
-	member(0)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-class-constructor-mem-init.ctx b/t/t4018/cpp-class-constructor-mem-init.ctx
deleted file mode 100644
index 6664b8b3d8..0000000000
--- a/t/t4018/cpp-class-constructor-mem-init.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Item::Item(int RIGHT) :
diff --git a/t/t4018/cpp-class-constructor.ctx b/t/t4018/cpp-class-constructor.ctx
deleted file mode 100644
index 2dcadfc0ba..0000000000
--- a/t/t4018/cpp-class-constructor.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Item::Item(int RIGHT)
diff --git a/t/t4018/cpp-class-definition b/t/t4018/cpp-class-definition
deleted file mode 100644
index 11b61da3b7..0000000000
--- a/t/t4018/cpp-class-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-class RIGHT
-{
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-class-definition-derived b/t/t4018/cpp-class-definition-derived
deleted file mode 100644
index 3b98cd09ab..0000000000
--- a/t/t4018/cpp-class-definition-derived
+++ /dev/null
@@ -1,5 +0,0 @@
-class RIGHT :
-	public Baseclass
-{
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-class-definition-derived.ctx b/t/t4018/cpp-class-definition-derived.ctx
deleted file mode 100644
index 146f0a7b7c..0000000000
--- a/t/t4018/cpp-class-definition-derived.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT :
diff --git a/t/t4018/cpp-class-definition.ctx b/t/t4018/cpp-class-definition.ctx
deleted file mode 100644
index 54bff816d6..0000000000
--- a/t/t4018/cpp-class-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT
diff --git a/t/t4018/cpp-class-destructor b/t/t4018/cpp-class-destructor
deleted file mode 100644
index 5487665096..0000000000
--- a/t/t4018/cpp-class-destructor
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT::~RIGHT()
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-class-destructor.ctx b/t/t4018/cpp-class-destructor.ctx
deleted file mode 100644
index 5390c17cf6..0000000000
--- a/t/t4018/cpp-class-destructor.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT::~RIGHT()
diff --git a/t/t4018/cpp-function-returning-global-type b/t/t4018/cpp-function-returning-global-type
deleted file mode 100644
index 1084d5990e..0000000000
--- a/t/t4018/cpp-function-returning-global-type
+++ /dev/null
@@ -1,4 +0,0 @@
-::Item get::it::RIGHT()
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-function-returning-global-type.ctx b/t/t4018/cpp-function-returning-global-type.ctx
deleted file mode 100644
index 4dcdde25f4..0000000000
--- a/t/t4018/cpp-function-returning-global-type.ctx
+++ /dev/null
@@ -1 +0,0 @@
-::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-nested b/t/t4018/cpp-function-returning-nested
deleted file mode 100644
index d9750aa61a..0000000000
--- a/t/t4018/cpp-function-returning-nested
+++ /dev/null
@@ -1,5 +0,0 @@
-get::Item get::it::RIGHT()
-{
-	ChangeMe;
-}
-
diff --git a/t/t4018/cpp-function-returning-nested.ctx b/t/t4018/cpp-function-returning-nested.ctx
deleted file mode 100644
index 6ef73c8368..0000000000
--- a/t/t4018/cpp-function-returning-nested.ctx
+++ /dev/null
@@ -1 +0,0 @@
-get::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-pointer b/t/t4018/cpp-function-returning-pointer
deleted file mode 100644
index ef15657ea8..0000000000
--- a/t/t4018/cpp-function-returning-pointer
+++ /dev/null
@@ -1,4 +0,0 @@
-const char *get_it_RIGHT(char *ptr)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-function-returning-pointer.ctx b/t/t4018/cpp-function-returning-pointer.ctx
deleted file mode 100644
index bb0acce5c7..0000000000
--- a/t/t4018/cpp-function-returning-pointer.ctx
+++ /dev/null
@@ -1 +0,0 @@
-const char *get_it_RIGHT(char *ptr)
diff --git a/t/t4018/cpp-function-returning-reference b/t/t4018/cpp-function-returning-reference
deleted file mode 100644
index 01b051df70..0000000000
--- a/t/t4018/cpp-function-returning-reference
+++ /dev/null
@@ -1,4 +0,0 @@
-string& get::it::RIGHT(char *ptr)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-function-returning-reference.ctx b/t/t4018/cpp-function-returning-reference.ctx
deleted file mode 100644
index 76afe381fd..0000000000
--- a/t/t4018/cpp-function-returning-reference.ctx
+++ /dev/null
@@ -1 +0,0 @@
-string& get::it::RIGHT(char *ptr)
diff --git a/t/t4018/cpp-gnu-style-function b/t/t4018/cpp-gnu-style-function
deleted file mode 100644
index 08c7c7565a..0000000000
--- a/t/t4018/cpp-gnu-style-function
+++ /dev/null
@@ -1,5 +0,0 @@
-const char *
-RIGHT(int arg)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-gnu-style-function.ctx b/t/t4018/cpp-gnu-style-function.ctx
deleted file mode 100644
index 1858287812..0000000000
--- a/t/t4018/cpp-gnu-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT(int arg)
diff --git a/t/t4018/cpp-namespace-definition b/t/t4018/cpp-namespace-definition
deleted file mode 100644
index 6749980241..0000000000
--- a/t/t4018/cpp-namespace-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-namespace RIGHT
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-namespace-definition.ctx b/t/t4018/cpp-namespace-definition.ctx
deleted file mode 100644
index 14c29c4638..0000000000
--- a/t/t4018/cpp-namespace-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-namespace RIGHT
diff --git a/t/t4018/cpp-operator-definition b/t/t4018/cpp-operator-definition
deleted file mode 100644
index 1acd827159..0000000000
--- a/t/t4018/cpp-operator-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-Value operator+(Value LEFT, Value RIGHT)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-operator-definition.ctx b/t/t4018/cpp-operator-definition.ctx
deleted file mode 100644
index 5b56778961..0000000000
--- a/t/t4018/cpp-operator-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Value operator+(Value LEFT, Value RIGHT)
diff --git a/t/t4018/cpp-skip-access-specifiers b/t/t4018/cpp-skip-access-specifiers
deleted file mode 100644
index 4d4a9dbb9d..0000000000
--- a/t/t4018/cpp-skip-access-specifiers
+++ /dev/null
@@ -1,8 +0,0 @@
-class RIGHT : public Baseclass
-{
-public:
-protected:
-private:
-	void DoSomething();
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-skip-access-specifiers.ctx b/t/t4018/cpp-skip-access-specifiers.ctx
deleted file mode 100644
index 075bcd883b..0000000000
--- a/t/t4018/cpp-skip-access-specifiers.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT : public Baseclass
diff --git a/t/t4018/cpp-skip-comment-block b/t/t4018/cpp-skip-comment-block
deleted file mode 100644
index 3800b9967a..0000000000
--- a/t/t4018/cpp-skip-comment-block
+++ /dev/null
@@ -1,9 +0,0 @@
-struct item RIGHT(int i)
-// Do not
-// pick up
-/* these
-** comments.
-*/
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-skip-comment-block.ctx b/t/t4018/cpp-skip-comment-block.ctx
deleted file mode 100644
index 656c59893d..0000000000
--- a/t/t4018/cpp-skip-comment-block.ctx
+++ /dev/null
@@ -1 +0,0 @@
-struct item RIGHT(int i)
diff --git a/t/t4018/cpp-skip-labels b/t/t4018/cpp-skip-labels
deleted file mode 100644
index b9c10aba22..0000000000
--- a/t/t4018/cpp-skip-labels
+++ /dev/null
@@ -1,8 +0,0 @@
-void RIGHT (void)
-{
-repeat:		// C++ comment
-next:		/* C comment */
-	do_something();
-
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-skip-labels.ctx b/t/t4018/cpp-skip-labels.ctx
deleted file mode 100644
index 6b0635f7f7..0000000000
--- a/t/t4018/cpp-skip-labels.ctx
+++ /dev/null
@@ -1 +0,0 @@
-void RIGHT (void)
diff --git a/t/t4018/cpp-struct-definition b/t/t4018/cpp-struct-definition
deleted file mode 100644
index 521c59fd15..0000000000
--- a/t/t4018/cpp-struct-definition
+++ /dev/null
@@ -1,9 +0,0 @@
-struct RIGHT {
-	unsigned
-	/* this bit field looks like a label and should not be picked up */
-		decoy_bitfield: 2,
-		more : 1;
-	int filler;
-
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-struct-definition.ctx b/t/t4018/cpp-struct-definition.ctx
deleted file mode 100644
index 48ed893279..0000000000
--- a/t/t4018/cpp-struct-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-struct RIGHT {
diff --git a/t/t4018/cpp-struct-single-line b/t/t4018/cpp-struct-single-line
deleted file mode 100644
index a0de5fb800..0000000000
--- a/t/t4018/cpp-struct-single-line
+++ /dev/null
@@ -1,7 +0,0 @@
-void wrong()
-{
-}
-
-struct RIGHT_iterator_tag {};
-
-int ChangeMe;
diff --git a/t/t4018/cpp-struct-single-line.ctx b/t/t4018/cpp-struct-single-line.ctx
deleted file mode 100644
index e3bc9d5017..0000000000
--- a/t/t4018/cpp-struct-single-line.ctx
+++ /dev/null
@@ -1 +0,0 @@
-struct RIGHT_iterator_tag {};
diff --git a/t/t4018/cpp-template-function-definition b/t/t4018/cpp-template-function-definition
deleted file mode 100644
index 0cdf5ba5bd..0000000000
--- a/t/t4018/cpp-template-function-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-template<class T> int RIGHT(T arg)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-template-function-definition.ctx b/t/t4018/cpp-template-function-definition.ctx
deleted file mode 100644
index c9da39cf65..0000000000
--- a/t/t4018/cpp-template-function-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-template<class T> int RIGHT(T arg)
diff --git a/t/t4018/cpp-union-definition b/t/t4018/cpp-union-definition
deleted file mode 100644
index 7ec94df697..0000000000
--- a/t/t4018/cpp-union-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-union RIGHT {
-	double v;
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-union-definition.ctx b/t/t4018/cpp-union-definition.ctx
deleted file mode 100644
index 2fc7b54fb8..0000000000
--- a/t/t4018/cpp-union-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-union RIGHT {
diff --git a/t/t4018/cpp-void-c-function b/t/t4018/cpp-void-c-function
deleted file mode 100644
index 153081e872..0000000000
--- a/t/t4018/cpp-void-c-function
+++ /dev/null
@@ -1,4 +0,0 @@
-void RIGHT (void)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-void-c-function.ctx b/t/t4018/cpp-void-c-function.ctx
deleted file mode 100644
index 6b0635f7f7..0000000000
--- a/t/t4018/cpp-void-c-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-void RIGHT (void)
diff --git a/t/t4018/cpp.sh b/t/t4018/cpp.sh
new file mode 100755
index 0000000000..185d40d5ef
--- /dev/null
+++ b/t/t4018/cpp.sh
@@ -0,0 +1,239 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'cpp: c++ function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Item RIGHT::DoSomething( Args with_spaces )
+EOF_HUNK
+Item RIGHT::DoSomething( Args with_spaces )
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: class constructor' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Item::Item(int RIGHT)
+EOF_HUNK
+Item::Item(int RIGHT)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: class constructor mem init' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Item::Item(int RIGHT) :
+EOF_HUNK
+Item::Item(int RIGHT) :
+	member(0)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: class definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT
+EOF_HUNK
+class RIGHT
+{
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: class definition derived' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT :
+EOF_HUNK
+class RIGHT :
+	public Baseclass
+{
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: class destructor' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT::~RIGHT()
+EOF_HUNK
+RIGHT::~RIGHT()
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning global type' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+::Item get::it::RIGHT()
+EOF_HUNK
+::Item get::it::RIGHT()
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning nested' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+get::Item get::it::RIGHT()
+EOF_HUNK
+get::Item get::it::RIGHT()
+{
+	ChangeMe;
+}
+
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning pointer' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+const char *get_it_RIGHT(char *ptr)
+EOF_HUNK
+const char *get_it_RIGHT(char *ptr)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning reference' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+string& get::it::RIGHT(char *ptr)
+EOF_HUNK
+string& get::it::RIGHT(char *ptr)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: gnu style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT(int arg)
+EOF_HUNK
+const char *
+RIGHT(int arg)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: namespace definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+namespace RIGHT
+EOF_HUNK
+namespace RIGHT
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: operator definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Value operator+(Value LEFT, Value RIGHT)
+EOF_HUNK
+Value operator+(Value LEFT, Value RIGHT)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: skip access specifiers' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT : public Baseclass
+EOF_HUNK
+class RIGHT : public Baseclass
+{
+public:
+protected:
+private:
+	void DoSomething();
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: skip comment block' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+struct item RIGHT(int i)
+EOF_HUNK
+struct item RIGHT(int i)
+// Do not
+// pick up
+/* these
+** comments.
+*/
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: skip labels' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+void RIGHT (void)
+EOF_HUNK
+void RIGHT (void)
+{
+repeat:		// C++ comment
+next:		/* C comment */
+	do_something();
+
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: struct definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+struct RIGHT {
+EOF_HUNK
+struct RIGHT {
+	unsigned
+	/* this bit field looks like a label and should not be picked up */
+		decoy_bitfield: 2,
+		more : 1;
+	int filler;
+
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: struct single line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+struct RIGHT_iterator_tag {};
+EOF_HUNK
+void wrong()
+{
+}
+
+struct RIGHT_iterator_tag {};
+
+int ChangeMe;
+EOF_TEST
+
+test_diff_funcname 'cpp: template function definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+template<class T> int RIGHT(T arg)
+EOF_HUNK
+template<class T> int RIGHT(T arg)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: union definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+union RIGHT {
+EOF_HUNK
+union RIGHT {
+	double v;
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: void c function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+void RIGHT (void)
+EOF_HUNK
+void RIGHT (void)
+{
+	ChangeMe;
+}
+EOF_TEST
diff --git a/t/t4018/css-attribute-value-selector b/t/t4018/css-attribute-value-selector
deleted file mode 100644
index 918256b20c..0000000000
--- a/t/t4018/css-attribute-value-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-[class*="RIGHT"] {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-attribute-value-selector.ctx b/t/t4018/css-attribute-value-selector.ctx
deleted file mode 100644
index 7f8956251c..0000000000
--- a/t/t4018/css-attribute-value-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-[class*="RIGHT"] {
diff --git a/t/t4018/css-block-level-@-statements b/t/t4018/css-block-level-@-statements
deleted file mode 100644
index d6755f2f3d..0000000000
--- a/t/t4018/css-block-level-@-statements
+++ /dev/null
@@ -1,10 +0,0 @@
-@keyframes RIGHT {
-    from {
-        background : #000;
-        border : 10px ChangeMe #C6C6C6;
-    }
-    to {
-        background : #fff;
-        border : 10px solid #C6C6C6;
-    }
-}
diff --git a/t/t4018/css-block-level-@-statements.ctx b/t/t4018/css-block-level-@-statements.ctx
deleted file mode 100644
index 7f5e90468c..0000000000
--- a/t/t4018/css-block-level-@-statements.ctx
+++ /dev/null
@@ -1 +0,0 @@
-@keyframes RIGHT {
diff --git a/t/t4018/css-brace-in-col-1 b/t/t4018/css-brace-in-col-1
deleted file mode 100644
index 7831577506..0000000000
--- a/t/t4018/css-brace-in-col-1
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT label.control-label
-{
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-brace-in-col-1.ctx b/t/t4018/css-brace-in-col-1.ctx
deleted file mode 100644
index 91a9105c6a..0000000000
--- a/t/t4018/css-brace-in-col-1.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT label.control-label
diff --git a/t/t4018/css-class-selector b/t/t4018/css-class-selector
deleted file mode 100644
index f790a0062f..0000000000
--- a/t/t4018/css-class-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-.RIGHT {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-class-selector.ctx b/t/t4018/css-class-selector.ctx
deleted file mode 100644
index ac7367d7f4..0000000000
--- a/t/t4018/css-class-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-.RIGHT {
diff --git a/t/t4018/css-colon-eol b/t/t4018/css-colon-eol
deleted file mode 100644
index 5a30553d29..0000000000
--- a/t/t4018/css-colon-eol
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT h1 {
-color:
-ChangeMe;
-}
diff --git a/t/t4018/css-colon-eol.ctx b/t/t4018/css-colon-eol.ctx
deleted file mode 100644
index b68493b9b0..0000000000
--- a/t/t4018/css-colon-eol.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT h1 {
diff --git a/t/t4018/css-colon-selector b/t/t4018/css-colon-selector
deleted file mode 100644
index c6d71fb42d..0000000000
--- a/t/t4018/css-colon-selector
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT a:hover {
-    margin-top:
-    10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-colon-selector.ctx b/t/t4018/css-colon-selector.ctx
deleted file mode 100644
index 00b1a5aefe..0000000000
--- a/t/t4018/css-colon-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT a:hover {
diff --git a/t/t4018/css-common b/t/t4018/css-common
deleted file mode 100644
index 84ed754b33..0000000000
--- a/t/t4018/css-common
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT label.control-label {
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-common.ctx b/t/t4018/css-common.ctx
deleted file mode 100644
index 43686b4081..0000000000
--- a/t/t4018/css-common.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT label.control-label {
diff --git a/t/t4018/css-id-selector b/t/t4018/css-id-selector
deleted file mode 100644
index 17c5111052..0000000000
--- a/t/t4018/css-id-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-#RIGHT {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-id-selector.ctx b/t/t4018/css-id-selector.ctx
deleted file mode 100644
index ce19f6d8dc..0000000000
--- a/t/t4018/css-id-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-#RIGHT {
diff --git a/t/t4018/css-long-selector-list b/t/t4018/css-long-selector-list
deleted file mode 100644
index 7ccd25d9ed..0000000000
--- a/t/t4018/css-long-selector-list
+++ /dev/null
@@ -1,6 +0,0 @@
-p.header,
-label.control-label,
-div ul#RIGHT {
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-long-selector-list.ctx b/t/t4018/css-long-selector-list.ctx
deleted file mode 100644
index bc8d0fb62c..0000000000
--- a/t/t4018/css-long-selector-list.ctx
+++ /dev/null
@@ -1 +0,0 @@
-div ul#RIGHT {
diff --git a/t/t4018/css-prop-sans-indent b/t/t4018/css-prop-sans-indent
deleted file mode 100644
index a9e3c86b3c..0000000000
--- a/t/t4018/css-prop-sans-indent
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT, label.control-label {
-margin-top: 10px!important;
-padding: 0;
-border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-prop-sans-indent.ctx b/t/t4018/css-prop-sans-indent.ctx
deleted file mode 100644
index cc880b2f44..0000000000
--- a/t/t4018/css-prop-sans-indent.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT, label.control-label {
diff --git a/t/t4018/css-root-selector b/t/t4018/css-root-selector
deleted file mode 100644
index 22b958e369..0000000000
--- a/t/t4018/css-root-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-:RIGHT {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-root-selector.ctx b/t/t4018/css-root-selector.ctx
deleted file mode 100644
index 3010cded2a..0000000000
--- a/t/t4018/css-root-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-:RIGHT {
diff --git a/t/t4018/css-short-selector-list b/t/t4018/css-short-selector-list
deleted file mode 100644
index 6a0bdee336..0000000000
--- a/t/t4018/css-short-selector-list
+++ /dev/null
@@ -1,4 +0,0 @@
-label.control, div ul#RIGHT {
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-short-selector-list.ctx b/t/t4018/css-short-selector-list.ctx
deleted file mode 100644
index 9e5d87d126..0000000000
--- a/t/t4018/css-short-selector-list.ctx
+++ /dev/null
@@ -1 +0,0 @@
-label.control, div ul#RIGHT {
diff --git a/t/t4018/css-trailing-space b/t/t4018/css-trailing-space
deleted file mode 100644
index 32b5606c70..0000000000
--- a/t/t4018/css-trailing-space
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT label.control-label {
-    margin:10px;   
-    padding:10px;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-trailing-space.ctx b/t/t4018/css-trailing-space.ctx
deleted file mode 100644
index 43686b4081..0000000000
--- a/t/t4018/css-trailing-space.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT label.control-label {
diff --git a/t/t4018/css.sh b/t/t4018/css.sh
new file mode 100755
index 0000000000..106a3de242
--- /dev/null
+++ b/t/t4018/css.sh
@@ -0,0 +1,146 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'css: attribute value selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+[class*="RIGHT"] {
+EOF_HUNK
+[class*="RIGHT"] {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: block level @ statements' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+@keyframes RIGHT {
+EOF_HUNK
+@keyframes RIGHT {
+    from {
+        background : #000;
+        border : 10px ChangeMe #C6C6C6;
+    }
+    to {
+        background : #fff;
+        border : 10px solid #C6C6C6;
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'css: brace in col 1' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT label.control-label
+EOF_HUNK
+RIGHT label.control-label
+{
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: class selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+.RIGHT {
+EOF_HUNK
+.RIGHT {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: colon eol' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT h1 {
+EOF_HUNK
+RIGHT h1 {
+color:
+ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'css: colon selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT a:hover {
+EOF_HUNK
+RIGHT a:hover {
+    margin-top:
+    10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: common' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT label.control-label {
+EOF_HUNK
+RIGHT label.control-label {
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: id selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+#RIGHT {
+EOF_HUNK
+#RIGHT {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: long selector list' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+div ul#RIGHT {
+EOF_HUNK
+p.header,
+label.control-label,
+div ul#RIGHT {
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: prop sans indent' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT, label.control-label {
+EOF_HUNK
+RIGHT, label.control-label {
+margin-top: 10px!important;
+padding: 0;
+border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: root selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+:RIGHT {
+EOF_HUNK
+:RIGHT {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: short selector list' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+label.control, div ul#RIGHT {
+EOF_HUNK
+label.control, div ul#RIGHT {
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: trailing space' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT label.control-label {
+EOF_HUNK
+RIGHT label.control-label {
+    margin:10px;   
+    padding:10px;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
diff --git a/t/t4018/custom1-pattern.ctx b/t/t4018/custom1-pattern.ctx
deleted file mode 100644
index d1609cc9a6..0000000000
--- a/t/t4018/custom1-pattern.ctx
+++ /dev/null
@@ -1 +0,0 @@
-int special, RIGHT;
diff --git a/t/t4018/custom1-pattern b/t/t4018/custom1.sh
old mode 100644
new mode 100755
similarity index 71%
rename from t/t4018/custom1-pattern
rename to t/t4018/custom1.sh
index e8fd59f884..f8bbccadb4
--- a/t/t4018/custom1-pattern
+++ b/t/t4018/custom1.sh
@@ -1,3 +1,12 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom1: pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+int special, RIGHT;
+EOF_HUNK
 public class Beer
 {
 	int special, RIGHT;
@@ -15,3 +24,4 @@ public class Beer
 			+ "99 bottles of beer on the wall.\n");
 	}
 }
+EOF_TEST
diff --git a/t/t4018/custom2-match-to-end-of-line b/t/t4018/custom2-match-to-end-of-line
deleted file mode 100644
index f88ac318b7..0000000000
--- a/t/t4018/custom2-match-to-end-of-line
+++ /dev/null
@@ -1,8 +0,0 @@
-public class RIGHT_Beer
-{
-	int special;
-	public static void main(String args[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
diff --git a/t/t4018/custom2-match-to-end-of-line.ctx b/t/t4018/custom2-match-to-end-of-line.ctx
deleted file mode 100644
index 8294c6e49b..0000000000
--- a/t/t4018/custom2-match-to-end-of-line.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT_Beer
diff --git a/t/t4018/custom2.sh b/t/t4018/custom2.sh
new file mode 100755
index 0000000000..c68421f788
--- /dev/null
+++ b/t/t4018/custom2.sh
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom2: match to end of line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT_Beer
+EOF_HUNK
+public class RIGHT_Beer
+{
+	int special;
+	public static void main(String args[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
diff --git a/t/t4018/custom3-alternation-in-pattern.ctx b/t/t4018/custom3-alternation-in-pattern.ctx
deleted file mode 100644
index 2125474b68..0000000000
--- a/t/t4018/custom3-alternation-in-pattern.ctx
+++ /dev/null
@@ -1 +0,0 @@
-public static void main(String RIGHT[])
diff --git a/t/t4018/custom3-alternation-in-pattern b/t/t4018/custom3.sh
old mode 100644
new mode 100755
similarity index 66%
rename from t/t4018/custom3-alternation-in-pattern
rename to t/t4018/custom3.sh
index 5f3769c64f..07c5c134ff
--- a/t/t4018/custom3-alternation-in-pattern
+++ b/t/t4018/custom3.sh
@@ -1,3 +1,12 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom3: alternation in pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
 public class Beer
 {
 	int special;
@@ -15,3 +24,4 @@ public class Beer
 			+ "99 bottles of beer on the wall.\n");
 	}
 }
+EOF_TEST
diff --git a/t/t4018/dts-labels b/t/t4018/dts-labels
deleted file mode 100644
index b21ef8737b..0000000000
--- a/t/t4018/dts-labels
+++ /dev/null
@@ -1,9 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		label2: RIGHT {
-			vendor,some-property;
-
-			ChangeMe = <0x45-30>;
-		};
-	};
-};
diff --git a/t/t4018/dts-labels.ctx b/t/t4018/dts-labels.ctx
deleted file mode 100644
index 48d9373cab..0000000000
--- a/t/t4018/dts-labels.ctx
+++ /dev/null
@@ -1 +0,0 @@
-label2: RIGHT {
diff --git a/t/t4018/dts-node-unitless b/t/t4018/dts-node-unitless
deleted file mode 100644
index c5287d9141..0000000000
--- a/t/t4018/dts-node-unitless
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1 {
-		RIGHT {
-			prop-array = <1>, <4>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-node-unitless.ctx b/t/t4018/dts-node-unitless.ctx
deleted file mode 100644
index 82c8683fa1..0000000000
--- a/t/t4018/dts-node-unitless.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT {
diff --git a/t/t4018/dts-nodes b/t/t4018/dts-nodes
deleted file mode 100644
index 5a4334bb16..0000000000
--- a/t/t4018/dts-nodes
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 {
-			#size-cells = <1>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-boolean-prop b/t/t4018/dts-nodes-boolean-prop
deleted file mode 100644
index afc6b5b404..0000000000
--- a/t/t4018/dts-nodes-boolean-prop
+++ /dev/null
@@ -1,9 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 {
-			boolean-prop1;
-
-			ChangeMe;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-boolean-prop.ctx b/t/t4018/dts-nodes-boolean-prop.ctx
deleted file mode 100644
index 3a0232d55d..0000000000
--- a/t/t4018/dts-nodes-boolean-prop.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes-comment1 b/t/t4018/dts-nodes-comment1
deleted file mode 100644
index 559dfce9b3..0000000000
--- a/t/t4018/dts-nodes-comment1
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 /* &a comment */ {
-			#size-cells = <1>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-comment1.ctx b/t/t4018/dts-nodes-comment1.ctx
deleted file mode 100644
index ec364600b1..0000000000
--- a/t/t4018/dts-nodes-comment1.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 /* &a comment */ {
diff --git a/t/t4018/dts-nodes-comment2 b/t/t4018/dts-nodes-comment2
deleted file mode 100644
index 27e9718b31..0000000000
--- a/t/t4018/dts-nodes-comment2
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 { /* a trailing comment */ 
-			#size-cells = <1>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-comment2.ctx b/t/t4018/dts-nodes-comment2.ctx
deleted file mode 100644
index 75f0d75258..0000000000
--- a/t/t4018/dts-nodes-comment2.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 { /* a trailing comment */
diff --git a/t/t4018/dts-nodes-multiline-prop b/t/t4018/dts-nodes-multiline-prop
deleted file mode 100644
index 072d58b69d..0000000000
--- a/t/t4018/dts-nodes-multiline-prop
+++ /dev/null
@@ -1,13 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 {
-			multilineprop = <3>,
-					<4>,
-					<5>,
-					<6>,
-					<7>;
-
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-multiline-prop.ctx b/t/t4018/dts-nodes-multiline-prop.ctx
deleted file mode 100644
index 3a0232d55d..0000000000
--- a/t/t4018/dts-nodes-multiline-prop.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes.ctx b/t/t4018/dts-nodes.ctx
deleted file mode 100644
index 3a0232d55d..0000000000
--- a/t/t4018/dts-nodes.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-reference b/t/t4018/dts-reference
deleted file mode 100644
index 8f0c87d863..0000000000
--- a/t/t4018/dts-reference
+++ /dev/null
@@ -1,9 +0,0 @@
-&label_1 {
-	TEST = <455>;
-};
-
-&RIGHT {
-	vendor,some-property;
-
-	ChangeMe = <0x45-30>;
-};
diff --git a/t/t4018/dts-reference.ctx b/t/t4018/dts-reference.ctx
deleted file mode 100644
index c1e13409ee..0000000000
--- a/t/t4018/dts-reference.ctx
+++ /dev/null
@@ -1 +0,0 @@
-&RIGHT {
diff --git a/t/t4018/dts-root b/t/t4018/dts-root
deleted file mode 100644
index 4353b8220c..0000000000
--- a/t/t4018/dts-root
+++ /dev/null
@@ -1,5 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
-	#size-cells = <1>;
-
-	ChangeMe = <0xffeedd00>;
-};
diff --git a/t/t4018/dts-root-comment b/t/t4018/dts-root-comment
deleted file mode 100644
index 333a625c70..0000000000
--- a/t/t4018/dts-root-comment
+++ /dev/null
@@ -1,8 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
-	#size-cells = <1>;
-
-	/* This comment should be ignored */
-
-	some-property = <40+2>;
-	ChangeMe = <0xffeedd00>;
-};
diff --git a/t/t4018/dts-root-comment.ctx b/t/t4018/dts-root-comment.ctx
deleted file mode 100644
index 656053dd42..0000000000
--- a/t/t4018/dts-root-comment.ctx
+++ /dev/null
@@ -1 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/dts-root.ctx b/t/t4018/dts-root.ctx
deleted file mode 100644
index 656053dd42..0000000000
--- a/t/t4018/dts-root.ctx
+++ /dev/null
@@ -1 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/dts.sh b/t/t4018/dts.sh
new file mode 100755
index 0000000000..304c131d86
--- /dev/null
+++ b/t/t4018/dts.sh
@@ -0,0 +1,149 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'dts: labels' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+label2: RIGHT {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		label2: RIGHT {
+			vendor,some-property;
+
+			ChangeMe = <0x45-30>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: node unitless' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT {
+EOF_HUNK
+/ {
+	label_1: node1 {
+		RIGHT {
+			prop-array = <1>, <4>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 {
+			#size-cells = <1>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes boolean prop' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 {
+			boolean-prop1;
+
+			ChangeMe;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes comment1' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 /* &a comment */ {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 /* &a comment */ {
+			#size-cells = <1>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes comment2' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 { /* a trailing comment */
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 { /* a trailing comment */ 
+			#size-cells = <1>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes multiline prop' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 {
+			multilineprop = <3>,
+					<4>,
+					<5>,
+					<6>,
+					<7>;
+
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: reference' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+&RIGHT {
+EOF_HUNK
+&label_1 {
+	TEST = <455>;
+};
+
+&RIGHT {
+	vendor,some-property;
+
+	ChangeMe = <0x45-30>;
+};
+EOF_TEST
+
+test_diff_funcname 'dts: root' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+EOF_HUNK
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+	#size-cells = <1>;
+
+	ChangeMe = <0xffeedd00>;
+};
+EOF_TEST
+
+test_diff_funcname 'dts: root comment' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+EOF_HUNK
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+	#size-cells = <1>;
+
+	/* This comment should be ignored */
+
+	some-property = <40+2>;
+	ChangeMe = <0xffeedd00>;
+};
+EOF_TEST
diff --git a/t/t4018/elixir-do-not-pick-end b/t/t4018/elixir-do-not-pick-end
deleted file mode 100644
index fae08ba7e8..0000000000
--- a/t/t4018/elixir-do-not-pick-end
+++ /dev/null
@@ -1,5 +0,0 @@
-defmodule RIGHT do
-end
-#
-#
-# ChangeMe; do not pick up 'end' line
diff --git a/t/t4018/elixir-do-not-pick-end.ctx b/t/t4018/elixir-do-not-pick-end.ctx
deleted file mode 100644
index 8f28a7a689..0000000000
--- a/t/t4018/elixir-do-not-pick-end.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmodule RIGHT do
diff --git a/t/t4018/elixir-ex-unit-test b/t/t4018/elixir-ex-unit-test
deleted file mode 100644
index 0560a2b697..0000000000
--- a/t/t4018/elixir-ex-unit-test
+++ /dev/null
@@ -1,6 +0,0 @@
-defmodule Test do
-  test "RIGHT" do
-    assert true == true
-    assert ChangeMe
-  end
-end
diff --git a/t/t4018/elixir-ex-unit-test.ctx b/t/t4018/elixir-ex-unit-test.ctx
deleted file mode 100644
index a55e3de2cc..0000000000
--- a/t/t4018/elixir-ex-unit-test.ctx
+++ /dev/null
@@ -1 +0,0 @@
-test "RIGHT" do
diff --git a/t/t4018/elixir-function b/t/t4018/elixir-function
deleted file mode 100644
index d452f495a7..0000000000
--- a/t/t4018/elixir-function
+++ /dev/null
@@ -1,5 +0,0 @@
-def function(RIGHT, arg) do
-  # comment
-  # comment
-  ChangeMe
-end
diff --git a/t/t4018/elixir-function.ctx b/t/t4018/elixir-function.ctx
deleted file mode 100644
index 62aee9c8b1..0000000000
--- a/t/t4018/elixir-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def function(RIGHT, arg) do
diff --git a/t/t4018/elixir-macro b/t/t4018/elixir-macro
deleted file mode 100644
index 4f925e9ad4..0000000000
--- a/t/t4018/elixir-macro
+++ /dev/null
@@ -1,5 +0,0 @@
-defmacro foo(RIGHT) do
-  # Code
-  # Code
-  ChangeMe
-end
diff --git a/t/t4018/elixir-macro.ctx b/t/t4018/elixir-macro.ctx
deleted file mode 100644
index fc1d3b85e8..0000000000
--- a/t/t4018/elixir-macro.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmacro foo(RIGHT) do
diff --git a/t/t4018/elixir-module b/t/t4018/elixir-module
deleted file mode 100644
index 91a4e7aa20..0000000000
--- a/t/t4018/elixir-module
+++ /dev/null
@@ -1,9 +0,0 @@
-defmodule RIGHT do
-  @moduledoc """
-  Foo bar
-  """
-
-  def ChangeMe(a) where is_map(a) do
-    a
-  end
-end
diff --git a/t/t4018/elixir-module-func b/t/t4018/elixir-module-func
deleted file mode 100644
index c9910d0675..0000000000
--- a/t/t4018/elixir-module-func
+++ /dev/null
@@ -1,8 +0,0 @@
-defmodule Foo do
-  def fun(RIGHT) do
-     # Code
-     # Code
-     # Code
-     ChangeMe
-  end
-end
diff --git a/t/t4018/elixir-module-func.ctx b/t/t4018/elixir-module-func.ctx
deleted file mode 100644
index 8239214386..0000000000
--- a/t/t4018/elixir-module-func.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def fun(RIGHT) do
diff --git a/t/t4018/elixir-module.ctx b/t/t4018/elixir-module.ctx
deleted file mode 100644
index 8f28a7a689..0000000000
--- a/t/t4018/elixir-module.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmodule RIGHT do
diff --git a/t/t4018/elixir-nested-module b/t/t4018/elixir-nested-module
deleted file mode 100644
index 771ebc5c42..0000000000
--- a/t/t4018/elixir-nested-module
+++ /dev/null
@@ -1,9 +0,0 @@
-defmodule MyApp.RIGHT do
-  @moduledoc """
-  Foo bar
-  """
-
-  def ChangeMe(a) where is_map(a) do
-    a
-  end
-end
diff --git a/t/t4018/elixir-nested-module.ctx b/t/t4018/elixir-nested-module.ctx
deleted file mode 100644
index 3ffbdd18b1..0000000000
--- a/t/t4018/elixir-nested-module.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmodule MyApp.RIGHT do
diff --git a/t/t4018/elixir-private-function b/t/t4018/elixir-private-function
deleted file mode 100644
index 1aabe33b7a..0000000000
--- a/t/t4018/elixir-private-function
+++ /dev/null
@@ -1,5 +0,0 @@
-defp function(RIGHT, arg) do
-  # comment
-  # comment
-  ChangeMe
-end
diff --git a/t/t4018/elixir-private-function.ctx b/t/t4018/elixir-private-function.ctx
deleted file mode 100644
index 1c4eba44f7..0000000000
--- a/t/t4018/elixir-private-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defp function(RIGHT, arg) do
diff --git a/t/t4018/elixir-protocol b/t/t4018/elixir-protocol
deleted file mode 100644
index 7d9173691e..0000000000
--- a/t/t4018/elixir-protocol
+++ /dev/null
@@ -1,6 +0,0 @@
-defprotocol RIGHT do
-  @doc """
-  Calculates the size (and not the length!) of a data structure
-  """
-  def size(data, ChangeMe)
-end
diff --git a/t/t4018/elixir-protocol-implementation b/t/t4018/elixir-protocol-implementation
deleted file mode 100644
index f9234bbfc4..0000000000
--- a/t/t4018/elixir-protocol-implementation
+++ /dev/null
@@ -1,5 +0,0 @@
-defimpl RIGHT do
-  # Docs
-  # Docs
-  def foo(ChangeMe), do: :ok
-end
diff --git a/t/t4018/elixir-protocol-implementation.ctx b/t/t4018/elixir-protocol-implementation.ctx
deleted file mode 100644
index efb758aea6..0000000000
--- a/t/t4018/elixir-protocol-implementation.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defimpl RIGHT do
diff --git a/t/t4018/elixir-protocol.ctx b/t/t4018/elixir-protocol.ctx
deleted file mode 100644
index d0204e9f14..0000000000
--- a/t/t4018/elixir-protocol.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defprotocol RIGHT do
diff --git a/t/t4018/elixir.sh b/t/t4018/elixir.sh
new file mode 100755
index 0000000000..6b09df9520
--- /dev/null
+++ b/t/t4018/elixir.sh
@@ -0,0 +1,127 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'elixir: do not pick end' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmodule RIGHT do
+EOF_HUNK
+defmodule RIGHT do
+end
+#
+#
+# ChangeMe; do not pick up 'end' line
+EOF_TEST
+
+test_diff_funcname 'elixir: ex unit test' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+test "RIGHT" do
+EOF_HUNK
+defmodule Test do
+  test "RIGHT" do
+    assert true == true
+    assert ChangeMe
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def function(RIGHT, arg) do
+EOF_HUNK
+def function(RIGHT, arg) do
+  # comment
+  # comment
+  ChangeMe
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: macro' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmacro foo(RIGHT) do
+EOF_HUNK
+defmacro foo(RIGHT) do
+  # Code
+  # Code
+  ChangeMe
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: module' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmodule RIGHT do
+EOF_HUNK
+defmodule RIGHT do
+  @moduledoc """
+  Foo bar
+  """
+
+  def ChangeMe(a) where is_map(a) do
+    a
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: module func' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def fun(RIGHT) do
+EOF_HUNK
+defmodule Foo do
+  def fun(RIGHT) do
+     # Code
+     # Code
+     # Code
+     ChangeMe
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: nested module' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmodule MyApp.RIGHT do
+EOF_HUNK
+defmodule MyApp.RIGHT do
+  @moduledoc """
+  Foo bar
+  """
+
+  def ChangeMe(a) where is_map(a) do
+    a
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: private function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defp function(RIGHT, arg) do
+EOF_HUNK
+defp function(RIGHT, arg) do
+  # comment
+  # comment
+  ChangeMe
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: protocol' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defprotocol RIGHT do
+EOF_HUNK
+defprotocol RIGHT do
+  @doc """
+  Calculates the size (and not the length!) of a data structure
+  """
+  def size(data, ChangeMe)
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: protocol implementation' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defimpl RIGHT do
+EOF_HUNK
+defimpl RIGHT do
+  # Docs
+  # Docs
+  def foo(ChangeMe), do: :ok
+end
+EOF_TEST
diff --git a/t/t4018/fortran-block-data b/t/t4018/fortran-block-data
deleted file mode 100644
index 63d4e21d0a..0000000000
--- a/t/t4018/fortran-block-data
+++ /dev/null
@@ -1,5 +0,0 @@
-       BLOCK DATA RIGHT
-       
-       COMMON /B/ C, ChangeMe
-       DATA C, ChangeMe  / 2.0, 6.0 / 
-       END 
diff --git a/t/t4018/fortran-block-data.ctx b/t/t4018/fortran-block-data.ctx
deleted file mode 100644
index c3db084ccc..0000000000
--- a/t/t4018/fortran-block-data.ctx
+++ /dev/null
@@ -1 +0,0 @@
-BLOCK DATA RIGHT
diff --git a/t/t4018/fortran-comment b/t/t4018/fortran-comment
deleted file mode 100644
index 7b10d17658..0000000000
--- a/t/t4018/fortran-comment
+++ /dev/null
@@ -1,13 +0,0 @@
-      module a
-
-      contains
-
-      ! subroutine wrong
-      subroutine RIGHT
-      ! subroutine wrong
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-keyword b/t/t4018/fortran-comment-keyword
deleted file mode 100644
index e9206a5379..0000000000
--- a/t/t4018/fortran-comment-keyword
+++ /dev/null
@@ -1,14 +0,0 @@
-      module a
-
-      contains
-
-      subroutine RIGHT (funcA, funcB)
-
-      real funcA  ! grid function a
-      real funcB  ! grid function b
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-keyword.ctx b/t/t4018/fortran-comment-keyword.ctx
deleted file mode 100644
index 0b9220b355..0000000000
--- a/t/t4018/fortran-comment-keyword.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT (funcA, funcB)
diff --git a/t/t4018/fortran-comment-legacy b/t/t4018/fortran-comment-legacy
deleted file mode 100644
index 53cd062c1e..0000000000
--- a/t/t4018/fortran-comment-legacy
+++ /dev/null
@@ -1,13 +0,0 @@
-      module a
-
-      contains
-
-C subroutine wrong
-      subroutine RIGHT
-C subroutine wrong
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-legacy-star b/t/t4018/fortran-comment-legacy-star
deleted file mode 100644
index 2cbcdc3d8a..0000000000
--- a/t/t4018/fortran-comment-legacy-star
+++ /dev/null
@@ -1,13 +0,0 @@
-      module a
-
-      contains
-
-* subroutine wrong
-      subroutine RIGHT
-* subroutine wrong
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-legacy-star.ctx b/t/t4018/fortran-comment-legacy-star.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-comment-legacy-star.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-comment-legacy.ctx b/t/t4018/fortran-comment-legacy.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-comment-legacy.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-comment.ctx b/t/t4018/fortran-comment.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-comment.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-external-function b/t/t4018/fortran-external-function
deleted file mode 100644
index 5a2d85d3aa..0000000000
--- a/t/t4018/fortran-external-function
+++ /dev/null
@@ -1,9 +0,0 @@
-function RIGHT(a, b) result(c)
-
-integer, intent(in) :: ChangeMe
-integer, intent(in) :: b
-integer, intent(out) :: c
-
-c = a+b
-
-end function RIGHT
diff --git a/t/t4018/fortran-external-function.ctx b/t/t4018/fortran-external-function.ctx
deleted file mode 100644
index 56ec4d8eca..0000000000
--- a/t/t4018/fortran-external-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT(a, b) result(c)
diff --git a/t/t4018/fortran-external-subroutine b/t/t4018/fortran-external-subroutine
deleted file mode 100644
index 4ce85fea13..0000000000
--- a/t/t4018/fortran-external-subroutine
+++ /dev/null
@@ -1,5 +0,0 @@
-subroutine RIGHT
-
-real ChangeMe
-
-end subroutine RIGHT
diff --git a/t/t4018/fortran-external-subroutine.ctx b/t/t4018/fortran-external-subroutine.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-external-subroutine.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-module b/t/t4018/fortran-module
deleted file mode 100644
index c4b737dac3..0000000000
--- a/t/t4018/fortran-module
+++ /dev/null
@@ -1,5 +0,0 @@
-module RIGHT
-
-use ChangeMe
-
-end module RIGHT
diff --git a/t/t4018/fortran-module-procedure b/t/t4018/fortran-module-procedure
deleted file mode 100644
index 1ce6d854c2..0000000000
--- a/t/t4018/fortran-module-procedure
+++ /dev/null
@@ -1,13 +0,0 @@
- module RIGHT
-
-   implicit none
-   private
-
-   interface letters  ! generic interface
-      module procedure aaaa, &
-                       bbbb, &
-                       ChangeMe, &
-                       dddd
-   end interface
-   
-end module RIGHT
diff --git a/t/t4018/fortran-module-procedure.ctx b/t/t4018/fortran-module-procedure.ctx
deleted file mode 100644
index 4f5ff2e4b8..0000000000
--- a/t/t4018/fortran-module-procedure.ctx
+++ /dev/null
@@ -1 +0,0 @@
-module RIGHT
diff --git a/t/t4018/fortran-module.ctx b/t/t4018/fortran-module.ctx
deleted file mode 100644
index 4f5ff2e4b8..0000000000
--- a/t/t4018/fortran-module.ctx
+++ /dev/null
@@ -1 +0,0 @@
-module RIGHT
diff --git a/t/t4018/fortran-program b/t/t4018/fortran-program
deleted file mode 100644
index 4616895e4b..0000000000
--- a/t/t4018/fortran-program
+++ /dev/null
@@ -1,5 +0,0 @@
-program RIGHT
-
-call ChangeMe
-
-end program RIGHT
diff --git a/t/t4018/fortran-program.ctx b/t/t4018/fortran-program.ctx
deleted file mode 100644
index c4e844df30..0000000000
--- a/t/t4018/fortran-program.ctx
+++ /dev/null
@@ -1 +0,0 @@
-program RIGHT
diff --git a/t/t4018/fortran.sh b/t/t4018/fortran.sh
new file mode 100755
index 0000000000..7b0c6789d3
--- /dev/null
+++ b/t/t4018/fortran.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'fortran: block data' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+BLOCK DATA RIGHT
+EOF_HUNK
+       BLOCK DATA RIGHT
+       
+       COMMON /B/ C, ChangeMe
+       DATA C, ChangeMe  / 2.0, 6.0 / 
+       END 
+EOF_TEST
+
+test_diff_funcname 'fortran: comment' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+      module a
+
+      contains
+
+      ! subroutine wrong
+      subroutine RIGHT
+      ! subroutine wrong
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: comment keyword' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT (funcA, funcB)
+EOF_HUNK
+      module a
+
+      contains
+
+      subroutine RIGHT (funcA, funcB)
+
+      real funcA  ! grid function a
+      real funcB  ! grid function b
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: comment legacy' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+      module a
+
+      contains
+
+C subroutine wrong
+      subroutine RIGHT
+C subroutine wrong
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: comment legacy star' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+      module a
+
+      contains
+
+* subroutine wrong
+      subroutine RIGHT
+* subroutine wrong
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: external function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT(a, b) result(c)
+EOF_HUNK
+function RIGHT(a, b) result(c)
+
+integer, intent(in) :: ChangeMe
+integer, intent(in) :: b
+integer, intent(out) :: c
+
+c = a+b
+
+end function RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: external subroutine' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+subroutine RIGHT
+
+real ChangeMe
+
+end subroutine RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: module' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+module RIGHT
+EOF_HUNK
+module RIGHT
+
+use ChangeMe
+
+end module RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: module procedure' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+module RIGHT
+EOF_HUNK
+ module RIGHT
+
+   implicit none
+   private
+
+   interface letters  ! generic interface
+      module procedure aaaa, &
+                       bbbb, &
+                       ChangeMe, &
+                       dddd
+   end interface
+   
+end module RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: program' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+program RIGHT
+EOF_HUNK
+program RIGHT
+
+call ChangeMe
+
+end program RIGHT
+EOF_TEST
diff --git a/t/t4018/fountain-scene b/t/t4018/fountain-scene
deleted file mode 100644
index 6b3257d680..0000000000
--- a/t/t4018/fountain-scene
+++ /dev/null
@@ -1,4 +0,0 @@
-EXT. STREET RIGHT OUTSIDE - DAY
-
-CHARACTER
-You didn't say the magic phrase, "ChangeMe".
diff --git a/t/t4018/fountain-scene.ctx b/t/t4018/fountain-scene.ctx
deleted file mode 100644
index bf10171418..0000000000
--- a/t/t4018/fountain-scene.ctx
+++ /dev/null
@@ -1 +0,0 @@
-EXT. STREET RIGHT OUTSIDE - DAY
diff --git a/t/t4018/fountain.sh b/t/t4018/fountain.sh
new file mode 100755
index 0000000000..02b44d6a3f
--- /dev/null
+++ b/t/t4018/fountain.sh
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'fountain: scene' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+EXT. STREET RIGHT OUTSIDE - DAY
+EOF_HUNK
+EXT. STREET RIGHT OUTSIDE - DAY
+
+CHARACTER
+You didn't say the magic phrase, "ChangeMe".
+EOF_TEST
diff --git a/t/t4018/golang-complex-function b/t/t4018/golang-complex-function
deleted file mode 100644
index e057dcefed..0000000000
--- a/t/t4018/golang-complex-function
+++ /dev/null
@@ -1,8 +0,0 @@
-type Test struct {
-	a Type
-}
-
-func (t *Test) RIGHT(a Type) (Type, error) {
-	t.a = a
-	return ChangeMe, nil
-}
diff --git a/t/t4018/golang-complex-function.ctx b/t/t4018/golang-complex-function.ctx
deleted file mode 100644
index 8e8d5582ff..0000000000
--- a/t/t4018/golang-complex-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-func (t *Test) RIGHT(a Type) (Type, error) {
diff --git a/t/t4018/golang-func b/t/t4018/golang-func
deleted file mode 100644
index 8e9c9ac7c3..0000000000
--- a/t/t4018/golang-func
+++ /dev/null
@@ -1,4 +0,0 @@
-func RIGHT() {
-	a := 5
-	b := ChangeMe
-}
diff --git a/t/t4018/golang-func.ctx b/t/t4018/golang-func.ctx
deleted file mode 100644
index 88bc823813..0000000000
--- a/t/t4018/golang-func.ctx
+++ /dev/null
@@ -1 +0,0 @@
-func RIGHT() {
diff --git a/t/t4018/golang-interface b/t/t4018/golang-interface
deleted file mode 100644
index 553bedec96..0000000000
--- a/t/t4018/golang-interface
+++ /dev/null
@@ -1,4 +0,0 @@
-type RIGHT interface {
-	a() Type
-	b() ChangeMe
-}
diff --git a/t/t4018/golang-interface.ctx b/t/t4018/golang-interface.ctx
deleted file mode 100644
index 2d07f5a383..0000000000
--- a/t/t4018/golang-interface.ctx
+++ /dev/null
@@ -1 +0,0 @@
-type RIGHT interface {
diff --git a/t/t4018/golang-long-func b/t/t4018/golang-long-func
deleted file mode 100644
index ac3a77b5c4..0000000000
--- a/t/t4018/golang-long-func
+++ /dev/null
@@ -1,5 +0,0 @@
-func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
-	anotherLongVariableName AnotherLongType) {
-	a := 5
-	b := ChangeMe
-}
diff --git a/t/t4018/golang-long-func.ctx b/t/t4018/golang-long-func.ctx
deleted file mode 100644
index 25635e712e..0000000000
--- a/t/t4018/golang-long-func.ctx
+++ /dev/null
@@ -1 +0,0 @@
-func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
diff --git a/t/t4018/golang-struct b/t/t4018/golang-struct
deleted file mode 100644
index 5deda77fee..0000000000
--- a/t/t4018/golang-struct
+++ /dev/null
@@ -1,4 +0,0 @@
-type RIGHT struct {
-	a Type
-	b ChangeMe
-}
diff --git a/t/t4018/golang-struct.ctx b/t/t4018/golang-struct.ctx
deleted file mode 100644
index 8a1240699d..0000000000
--- a/t/t4018/golang-struct.ctx
+++ /dev/null
@@ -1 +0,0 @@
-type RIGHT struct {
diff --git a/t/t4018/golang.sh b/t/t4018/golang.sh
new file mode 100755
index 0000000000..bf22f58c12
--- /dev/null
+++ b/t/t4018/golang.sh
@@ -0,0 +1,59 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'golang: complex function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+func (t *Test) RIGHT(a Type) (Type, error) {
+EOF_HUNK
+type Test struct {
+	a Type
+}
+
+func (t *Test) RIGHT(a Type) (Type, error) {
+	t.a = a
+	return ChangeMe, nil
+}
+EOF_TEST
+
+test_diff_funcname 'golang: func' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+func RIGHT() {
+EOF_HUNK
+func RIGHT() {
+	a := 5
+	b := ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'golang: interface' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+type RIGHT interface {
+EOF_HUNK
+type RIGHT interface {
+	a() Type
+	b() ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'golang: long func' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
+EOF_HUNK
+func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
+	anotherLongVariableName AnotherLongType) {
+	a := 5
+	b := ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'golang: struct' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+type RIGHT struct {
+EOF_HUNK
+type RIGHT struct {
+	a Type
+	b ChangeMe
+}
+EOF_TEST
diff --git a/t/t4018/java-class-member-function b/t/t4018/java-class-member-function
deleted file mode 100644
index 298bc7a71b..0000000000
--- a/t/t4018/java-class-member-function
+++ /dev/null
@@ -1,8 +0,0 @@
-public class Beer
-{
-	int special;
-	public static void main(String RIGHT[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
diff --git a/t/t4018/java-class-member-function.ctx b/t/t4018/java-class-member-function.ctx
deleted file mode 100644
index 2125474b68..0000000000
--- a/t/t4018/java-class-member-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-public static void main(String RIGHT[])
diff --git a/t/t4018/java.sh b/t/t4018/java.sh
new file mode 100755
index 0000000000..c89cf2f9d8
--- /dev/null
+++ b/t/t4018/java.sh
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'java: class member function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
+public class Beer
+{
+	int special;
+	public static void main(String RIGHT[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
diff --git a/t/t4018/markdown-heading-indented b/t/t4018/markdown-heading-indented
deleted file mode 100644
index 1991c2bd45..0000000000
--- a/t/t4018/markdown-heading-indented
+++ /dev/null
@@ -1,6 +0,0 @@
-Indented headings are allowed, as long as the indent is no more than 3 spaces.
-
-   ### RIGHT
-
-- something
-- ChangeMe
diff --git a/t/t4018/markdown-heading-indented.ctx b/t/t4018/markdown-heading-indented.ctx
deleted file mode 100644
index 5938336743..0000000000
--- a/t/t4018/markdown-heading-indented.ctx
+++ /dev/null
@@ -1 +0,0 @@
-   ### RIGHT
diff --git a/t/t4018/markdown-heading-non-headings b/t/t4018/markdown-heading-non-headings
deleted file mode 100644
index c479c1a3f1..0000000000
--- a/t/t4018/markdown-heading-non-headings
+++ /dev/null
@@ -1,17 +0,0 @@
-Headings can be right next to other lines of the file:
-# RIGHT
-Indents of four or more spaces make a code block:
-
-    # code comment, not heading
-
-If there's no space after the final hash, it's not a heading:
-
-#hashtag
-
-Sequences of more than 6 hashes don't make a heading:
-
-####### over-enthusiastic heading
-
-So the detected heading should be right up at the start of this file.
-
-ChangeMe
diff --git a/t/t4018/markdown-heading-non-headings.ctx b/t/t4018/markdown-heading-non-headings.ctx
deleted file mode 100644
index 7e2165be6e..0000000000
--- a/t/t4018/markdown-heading-non-headings.ctx
+++ /dev/null
@@ -1 +0,0 @@
-# RIGHT
diff --git a/t/t4018/markdown.sh b/t/t4018/markdown.sh
new file mode 100755
index 0000000000..3e1c79b139
--- /dev/null
+++ b/t/t4018/markdown.sh
@@ -0,0 +1,39 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'markdown: heading indented' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+   ### RIGHT
+EOF_HUNK
+Indented headings are allowed, as long as the indent is no more than 3 spaces.
+
+   ### RIGHT
+
+- something
+- ChangeMe
+EOF_TEST
+
+test_diff_funcname 'markdown: heading non headings' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+# RIGHT
+EOF_HUNK
+Headings can be right next to other lines of the file:
+# RIGHT
+Indents of four or more spaces make a code block:
+
+    # code comment, not heading
+
+If there's no space after the final hash, it's not a heading:
+
+#hashtag
+
+Sequences of more than 6 hashes don't make a heading:
+
+####### over-enthusiastic heading
+
+So the detected heading should be right up at the start of this file.
+
+ChangeMe
+EOF_TEST
diff --git a/t/t4018/matlab-class-definition b/t/t4018/matlab-class-definition
deleted file mode 100644
index 84daedfb4e..0000000000
--- a/t/t4018/matlab-class-definition
+++ /dev/null
@@ -1,5 +0,0 @@
-classdef RIGHT
-    properties
-        ChangeMe
-    end
-end
diff --git a/t/t4018/matlab-class-definition.ctx b/t/t4018/matlab-class-definition.ctx
deleted file mode 100644
index 5dd5b45628..0000000000
--- a/t/t4018/matlab-class-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-classdef RIGHT
diff --git a/t/t4018/matlab-function b/t/t4018/matlab-function
deleted file mode 100644
index 897a9b13ff..0000000000
--- a/t/t4018/matlab-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function y = RIGHT()
-x = 5;
-y = ChangeMe + x;
-end
diff --git a/t/t4018/matlab-function.ctx b/t/t4018/matlab-function.ctx
deleted file mode 100644
index 72d2350b13..0000000000
--- a/t/t4018/matlab-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function y = RIGHT()
diff --git a/t/t4018/matlab-octave-section-1 b/t/t4018/matlab-octave-section-1
deleted file mode 100644
index 3bb6c4670e..0000000000
--- a/t/t4018/matlab-octave-section-1
+++ /dev/null
@@ -1,3 +0,0 @@
-%%% RIGHT section
-# this is octave script
-ChangeMe = 1;
diff --git a/t/t4018/matlab-octave-section-1.ctx b/t/t4018/matlab-octave-section-1.ctx
deleted file mode 100644
index ca9b349f94..0000000000
--- a/t/t4018/matlab-octave-section-1.ctx
+++ /dev/null
@@ -1 +0,0 @@
-%%% RIGHT section
diff --git a/t/t4018/matlab-octave-section-2 b/t/t4018/matlab-octave-section-2
deleted file mode 100644
index ab2980f7f2..0000000000
--- a/t/t4018/matlab-octave-section-2
+++ /dev/null
@@ -1,3 +0,0 @@
-## RIGHT section
-# this is octave script
-ChangeMe = 1;
diff --git a/t/t4018/matlab-octave-section-2.ctx b/t/t4018/matlab-octave-section-2.ctx
deleted file mode 100644
index 5cbb77faf5..0000000000
--- a/t/t4018/matlab-octave-section-2.ctx
+++ /dev/null
@@ -1 +0,0 @@
-## RIGHT section
diff --git a/t/t4018/matlab-section b/t/t4018/matlab-section
deleted file mode 100644
index 5ea59a5de0..0000000000
--- a/t/t4018/matlab-section
+++ /dev/null
@@ -1,3 +0,0 @@
-%% RIGHT section
-% this is understood by both matlab and octave
-ChangeMe = 1;
diff --git a/t/t4018/matlab-section.ctx b/t/t4018/matlab-section.ctx
deleted file mode 100644
index e83fee6f4d..0000000000
--- a/t/t4018/matlab-section.ctx
+++ /dev/null
@@ -1 +0,0 @@
-%% RIGHT section
diff --git a/t/t4018/matlab.sh b/t/t4018/matlab.sh
new file mode 100755
index 0000000000..f62289148e
--- /dev/null
+++ b/t/t4018/matlab.sh
@@ -0,0 +1,52 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'matlab: class definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+classdef RIGHT
+EOF_HUNK
+classdef RIGHT
+    properties
+        ChangeMe
+    end
+end
+EOF_TEST
+
+test_diff_funcname 'matlab: function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function y = RIGHT()
+EOF_HUNK
+function y = RIGHT()
+x = 5;
+y = ChangeMe + x;
+end
+EOF_TEST
+
+test_diff_funcname 'matlab: octave section 1' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+%%% RIGHT section
+EOF_HUNK
+%%% RIGHT section
+# this is octave script
+ChangeMe = 1;
+EOF_TEST
+
+test_diff_funcname 'matlab: octave section 2' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+## RIGHT section
+EOF_HUNK
+## RIGHT section
+# this is octave script
+ChangeMe = 1;
+EOF_TEST
+
+test_diff_funcname 'matlab: section' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+%% RIGHT section
+EOF_HUNK
+%% RIGHT section
+% this is understood by both matlab and octave
+ChangeMe = 1;
+EOF_TEST
diff --git a/t/t4018/perl-skip-end-of-heredoc b/t/t4018/perl-skip-end-of-heredoc
deleted file mode 100644
index c22d39b256..0000000000
--- a/t/t4018/perl-skip-end-of-heredoc
+++ /dev/null
@@ -1,8 +0,0 @@
-sub RIGHTwithheredocument {
-	print <<"EOF"
-decoy here-doc
-EOF
-	# some lines of context
-	# to pad it out
-	print "ChangeMe\n";
-}
diff --git a/t/t4018/perl-skip-end-of-heredoc.ctx b/t/t4018/perl-skip-end-of-heredoc.ctx
deleted file mode 100644
index c15f4b78bd..0000000000
--- a/t/t4018/perl-skip-end-of-heredoc.ctx
+++ /dev/null
@@ -1 +0,0 @@
-sub RIGHTwithheredocument {
diff --git a/t/t4018/perl-skip-forward-decl b/t/t4018/perl-skip-forward-decl
deleted file mode 100644
index a98cb8bdad..0000000000
--- a/t/t4018/perl-skip-forward-decl
+++ /dev/null
@@ -1,10 +0,0 @@
-package RIGHT;
-
-use strict;
-use warnings;
-use parent qw(Exporter);
-our @EXPORT_OK = qw(round finalround);
-
-sub other; # forward declaration
-
-# ChangeMe
diff --git a/t/t4018/perl-skip-forward-decl.ctx b/t/t4018/perl-skip-forward-decl.ctx
deleted file mode 100644
index e0c51599ad..0000000000
--- a/t/t4018/perl-skip-forward-decl.ctx
+++ /dev/null
@@ -1 +0,0 @@
-package RIGHT;
diff --git a/t/t4018/perl-skip-sub-in-pod b/t/t4018/perl-skip-sub-in-pod
deleted file mode 100644
index e39f02462e..0000000000
--- a/t/t4018/perl-skip-sub-in-pod
+++ /dev/null
@@ -1,18 +0,0 @@
-=head1 NAME
-
-Beer - subroutine to output fragment of a drinking song
-
-=head1 SYNOPSIS_RIGHT
-
-	use Beer qw(round finalround);
-
-	sub song {
-		for (my $i = 99; $i > 0; $i--) {
-			round $i;
-		}
-		finalround;
-	}
-
-	ChangeMe;
-
-=cut
diff --git a/t/t4018/perl-skip-sub-in-pod.ctx b/t/t4018/perl-skip-sub-in-pod.ctx
deleted file mode 100644
index abddd76655..0000000000
--- a/t/t4018/perl-skip-sub-in-pod.ctx
+++ /dev/null
@@ -1 +0,0 @@
-=head1 SYNOPSIS_RIGHT
diff --git a/t/t4018/perl-sub-definition b/t/t4018/perl-sub-definition
deleted file mode 100644
index a507d1f645..0000000000
--- a/t/t4018/perl-sub-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-sub RIGHT {
-	my ($n) = @_;
-	print "ChangeMe";
-}
diff --git a/t/t4018/perl-sub-definition-kr-brace b/t/t4018/perl-sub-definition-kr-brace
deleted file mode 100644
index 330b3df114..0000000000
--- a/t/t4018/perl-sub-definition-kr-brace
+++ /dev/null
@@ -1,4 +0,0 @@
-sub RIGHT
-{
-	print "ChangeMe\n";
-}
diff --git a/t/t4018/perl-sub-definition-kr-brace.ctx b/t/t4018/perl-sub-definition-kr-brace.ctx
deleted file mode 100644
index 7e5aee5cde..0000000000
--- a/t/t4018/perl-sub-definition-kr-brace.ctx
+++ /dev/null
@@ -1 +0,0 @@
-sub RIGHT
diff --git a/t/t4018/perl-sub-definition.ctx b/t/t4018/perl-sub-definition.ctx
deleted file mode 100644
index d49a63598e..0000000000
--- a/t/t4018/perl-sub-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-sub RIGHT {
diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
new file mode 100755
index 0000000000..ac8fff7417
--- /dev/null
+++ b/t/t4018/perl.sh
@@ -0,0 +1,78 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'perl: skip end of heredoc' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHTwithheredocument {
+EOF_HUNK
+sub RIGHTwithheredocument {
+	print <<"EOF"
+decoy here-doc
+EOF
+	# some lines of context
+	# to pad it out
+	print "ChangeMe\n";
+}
+EOF_TEST
+
+test_diff_funcname 'perl: skip forward decl' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+package RIGHT;
+EOF_HUNK
+package RIGHT;
+
+use strict;
+use warnings;
+use parent qw(Exporter);
+our @EXPORT_OK = qw(round finalround);
+
+sub other; # forward declaration
+
+# ChangeMe
+EOF_TEST
+
+test_diff_funcname 'perl: skip sub in pod' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+=head1 SYNOPSIS_RIGHT
+EOF_HUNK
+=head1 NAME
+
+Beer - subroutine to output fragment of a drinking song
+
+=head1 SYNOPSIS_RIGHT
+
+	use Beer qw(round finalround);
+
+	sub song {
+		for (my $i = 99; $i > 0; $i--) {
+			round $i;
+		}
+		finalround;
+	}
+
+	ChangeMe;
+
+=cut
+EOF_TEST
+
+test_diff_funcname 'perl: sub definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHT {
+EOF_HUNK
+sub RIGHT {
+	my ($n) = @_;
+	print "ChangeMe";
+}
+EOF_TEST
+
+test_diff_funcname 'perl: sub definition kr brace' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHT
+EOF_HUNK
+sub RIGHT
+{
+	print "ChangeMe\n";
+}
+EOF_TEST
diff --git a/t/t4018/php-abstract-class b/t/t4018/php-abstract-class
deleted file mode 100644
index 5213e12494..0000000000
--- a/t/t4018/php-abstract-class
+++ /dev/null
@@ -1,4 +0,0 @@
-abstract class RIGHT
-{
-    const FOO = 'ChangeMe';
-}
diff --git a/t/t4018/php-abstract-class.ctx b/t/t4018/php-abstract-class.ctx
deleted file mode 100644
index f572d2129b..0000000000
--- a/t/t4018/php-abstract-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-abstract class RIGHT
diff --git a/t/t4018/php-abstract-method b/t/t4018/php-abstract-method
deleted file mode 100644
index ce215df75a..0000000000
--- a/t/t4018/php-abstract-method
+++ /dev/null
@@ -1,7 +0,0 @@
-abstract class Klass
-{
-    abstract public function RIGHT(): ?string
-    {
-        return 'ChangeMe';
-    }
-}
diff --git a/t/t4018/php-abstract-method.ctx b/t/t4018/php-abstract-method.ctx
deleted file mode 100644
index 14cb6df42e..0000000000
--- a/t/t4018/php-abstract-method.ctx
+++ /dev/null
@@ -1 +0,0 @@
-abstract public function RIGHT(): ?string
diff --git a/t/t4018/php-class b/t/t4018/php-class
deleted file mode 100644
index 7785b6303c..0000000000
--- a/t/t4018/php-class
+++ /dev/null
@@ -1,4 +0,0 @@
-class RIGHT
-{
-    const FOO = 'ChangeMe';
-}
diff --git a/t/t4018/php-class.ctx b/t/t4018/php-class.ctx
deleted file mode 100644
index 54bff816d6..0000000000
--- a/t/t4018/php-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT
diff --git a/t/t4018/php-final-class b/t/t4018/php-final-class
deleted file mode 100644
index 69f5710552..0000000000
--- a/t/t4018/php-final-class
+++ /dev/null
@@ -1,4 +0,0 @@
-final class RIGHT
-{
-    const FOO = 'ChangeMe';
-}
diff --git a/t/t4018/php-final-class.ctx b/t/t4018/php-final-class.ctx
deleted file mode 100644
index 4d59fb749b..0000000000
--- a/t/t4018/php-final-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-final class RIGHT
diff --git a/t/t4018/php-final-method b/t/t4018/php-final-method
deleted file mode 100644
index 537fb8ad9a..0000000000
--- a/t/t4018/php-final-method
+++ /dev/null
@@ -1,7 +0,0 @@
-class Klass
-{
-    final public function RIGHT(): string
-    {
-        return 'ChangeMe';
-    }
-}
diff --git a/t/t4018/php-final-method.ctx b/t/t4018/php-final-method.ctx
deleted file mode 100644
index b7da8f8082..0000000000
--- a/t/t4018/php-final-method.ctx
+++ /dev/null
@@ -1 +0,0 @@
-final public function RIGHT(): string
diff --git a/t/t4018/php-function b/t/t4018/php-function
deleted file mode 100644
index 35717c51c3..0000000000
--- a/t/t4018/php-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT()
-{
-    return 'ChangeMe';
-}
diff --git a/t/t4018/php-function.ctx b/t/t4018/php-function.ctx
deleted file mode 100644
index c5f3e55302..0000000000
--- a/t/t4018/php-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT()
diff --git a/t/t4018/php-interface b/t/t4018/php-interface
deleted file mode 100644
index 86b49ad5d9..0000000000
--- a/t/t4018/php-interface
+++ /dev/null
@@ -1,4 +0,0 @@
-interface RIGHT
-{
-    public function foo($ChangeMe);
-}
diff --git a/t/t4018/php-interface.ctx b/t/t4018/php-interface.ctx
deleted file mode 100644
index a45fa0532a..0000000000
--- a/t/t4018/php-interface.ctx
+++ /dev/null
@@ -1 +0,0 @@
-interface RIGHT
diff --git a/t/t4018/php-method b/t/t4018/php-method
deleted file mode 100644
index 03af1a6d9d..0000000000
--- a/t/t4018/php-method
+++ /dev/null
@@ -1,7 +0,0 @@
-class Klass
-{
-    public static function RIGHT()
-    {
-        return 'ChangeMe';
-    }
-}
diff --git a/t/t4018/php-method.ctx b/t/t4018/php-method.ctx
deleted file mode 100644
index eb1659ff9f..0000000000
--- a/t/t4018/php-method.ctx
+++ /dev/null
@@ -1 +0,0 @@
-public static function RIGHT()
diff --git a/t/t4018/php-trait b/t/t4018/php-trait
deleted file mode 100644
index 65b8c82a61..0000000000
--- a/t/t4018/php-trait
+++ /dev/null
@@ -1,7 +0,0 @@
-trait RIGHT
-{
-    public function foo($ChangeMe)
-    {
-        return 'foo';
-    }
-}
diff --git a/t/t4018/php-trait.ctx b/t/t4018/php-trait.ctx
deleted file mode 100644
index 57aa4c6267..0000000000
--- a/t/t4018/php-trait.ctx
+++ /dev/null
@@ -1 +0,0 @@
-trait RIGHT
diff --git a/t/t4018/php.sh b/t/t4018/php.sh
new file mode 100755
index 0000000000..e0ccb2277b
--- /dev/null
+++ b/t/t4018/php.sh
@@ -0,0 +1,106 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'php: abstract class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+abstract class RIGHT
+EOF_HUNK
+abstract class RIGHT
+{
+    const FOO = 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: abstract method' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+abstract public function RIGHT(): ?string
+EOF_HUNK
+abstract class Klass
+{
+    abstract public function RIGHT(): ?string
+    {
+        return 'ChangeMe';
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'php: class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT
+EOF_HUNK
+class RIGHT
+{
+    const FOO = 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: final class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+final class RIGHT
+EOF_HUNK
+final class RIGHT
+{
+    const FOO = 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: final method' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+final public function RIGHT(): string
+EOF_HUNK
+class Klass
+{
+    final public function RIGHT(): string
+    {
+        return 'ChangeMe';
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'php: function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT()
+EOF_HUNK
+function RIGHT()
+{
+    return 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: interface' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+interface RIGHT
+EOF_HUNK
+interface RIGHT
+{
+    public function foo($ChangeMe);
+}
+EOF_TEST
+
+test_diff_funcname 'php: method' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static function RIGHT()
+EOF_HUNK
+class Klass
+{
+    public static function RIGHT()
+    {
+        return 'ChangeMe';
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'php: trait' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+trait RIGHT
+EOF_HUNK
+trait RIGHT
+{
+    public function foo($ChangeMe)
+    {
+        return 'foo';
+    }
+}
+EOF_TEST
diff --git a/t/t4018/python-async-def b/t/t4018/python-async-def
deleted file mode 100644
index 87640e03d2..0000000000
--- a/t/t4018/python-async-def
+++ /dev/null
@@ -1,4 +0,0 @@
-async def RIGHT(pi: int = 3.14):
-    while True:
-        break
-    return ChangeMe()
diff --git a/t/t4018/python-async-def.ctx b/t/t4018/python-async-def.ctx
deleted file mode 100644
index 468c548bbe..0000000000
--- a/t/t4018/python-async-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-async def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-class b/t/t4018/python-class
deleted file mode 100644
index ba9e741430..0000000000
--- a/t/t4018/python-class
+++ /dev/null
@@ -1,4 +0,0 @@
-class RIGHT(int, str):
-    # comment
-    # another comment
-    # ChangeMe
diff --git a/t/t4018/python-class.ctx b/t/t4018/python-class.ctx
deleted file mode 100644
index a40b755e29..0000000000
--- a/t/t4018/python-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT(int, str):
diff --git a/t/t4018/python-def b/t/t4018/python-def
deleted file mode 100644
index e50b31b0ad..0000000000
--- a/t/t4018/python-def
+++ /dev/null
@@ -1,4 +0,0 @@
-def RIGHT(pi: int = 3.14):
-    while True:
-        break
-    return ChangeMe()
diff --git a/t/t4018/python-def.ctx b/t/t4018/python-def.ctx
deleted file mode 100644
index a1a9cbad63..0000000000
--- a/t/t4018/python-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-indented-async-def b/t/t4018/python-indented-async-def
deleted file mode 100644
index f5d03258af..0000000000
--- a/t/t4018/python-indented-async-def
+++ /dev/null
@@ -1,7 +0,0 @@
-class Foo:
-    async def RIGHT(self, x: int):
-        return [
-            1,
-            2,
-            ChangeMe,
-        ]
diff --git a/t/t4018/python-indented-async-def.ctx b/t/t4018/python-indented-async-def.ctx
deleted file mode 100644
index d393620a1e..0000000000
--- a/t/t4018/python-indented-async-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-async def RIGHT(self, x: int):
diff --git a/t/t4018/python-indented-class b/t/t4018/python-indented-class
deleted file mode 100644
index 19b4f35c4c..0000000000
--- a/t/t4018/python-indented-class
+++ /dev/null
@@ -1,5 +0,0 @@
-if TYPE_CHECKING:
-    class RIGHT:
-        # comment
-        # another comment
-        # ChangeMe
diff --git a/t/t4018/python-indented-class.ctx b/t/t4018/python-indented-class.ctx
deleted file mode 100644
index 0881c84dba..0000000000
--- a/t/t4018/python-indented-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT:
diff --git a/t/t4018/python-indented-def b/t/t4018/python-indented-def
deleted file mode 100644
index 208fbadd2b..0000000000
--- a/t/t4018/python-indented-def
+++ /dev/null
@@ -1,7 +0,0 @@
-class Foo:
-    def RIGHT(self, x: int):
-        return [
-            1,
-            2,
-            ChangeMe,
-        ]
diff --git a/t/t4018/python-indented-def.ctx b/t/t4018/python-indented-def.ctx
deleted file mode 100644
index 6e5a44b391..0000000000
--- a/t/t4018/python-indented-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def RIGHT(self, x: int):
diff --git a/t/t4018/python.sh b/t/t4018/python.sh
new file mode 100755
index 0000000000..ecb5736d57
--- /dev/null
+++ b/t/t4018/python.sh
@@ -0,0 +1,71 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'python: async def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+async def RIGHT(pi: int = 3.14):
+EOF_HUNK
+async def RIGHT(pi: int = 3.14):
+    while True:
+        break
+    return ChangeMe()
+EOF_TEST
+
+test_diff_funcname 'python: class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT(int, str):
+EOF_HUNK
+class RIGHT(int, str):
+    # comment
+    # another comment
+    # ChangeMe
+EOF_TEST
+
+test_diff_funcname 'python: def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def RIGHT(pi: int = 3.14):
+EOF_HUNK
+def RIGHT(pi: int = 3.14):
+    while True:
+        break
+    return ChangeMe()
+EOF_TEST
+
+test_diff_funcname 'python: indented async def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+async def RIGHT(self, x: int):
+EOF_HUNK
+class Foo:
+    async def RIGHT(self, x: int):
+        return [
+            1,
+            2,
+            ChangeMe,
+        ]
+EOF_TEST
+
+test_diff_funcname 'python: indented class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT:
+EOF_HUNK
+if TYPE_CHECKING:
+    class RIGHT:
+        # comment
+        # another comment
+        # ChangeMe
+EOF_TEST
+
+test_diff_funcname 'python: indented def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def RIGHT(self, x: int):
+EOF_HUNK
+class Foo:
+    def RIGHT(self, x: int):
+        return [
+            1,
+            2,
+            ChangeMe,
+        ]
+EOF_TEST
diff --git a/t/t4018/rust-fn b/t/t4018/rust-fn
deleted file mode 100644
index cbe02155f1..0000000000
--- a/t/t4018/rust-fn
+++ /dev/null
@@ -1,5 +0,0 @@
-pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
-    let _ = x;
-    // a comment
-    let a = ChangeMe;
-}
diff --git a/t/t4018/rust-fn.ctx b/t/t4018/rust-fn.ctx
deleted file mode 100644
index baa37cf253..0000000000
--- a/t/t4018/rust-fn.ctx
+++ /dev/null
@@ -1 +0,0 @@
-pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
diff --git a/t/t4018/rust-impl b/t/t4018/rust-impl
deleted file mode 100644
index 09df3cd93b..0000000000
--- a/t/t4018/rust-impl
+++ /dev/null
@@ -1,5 +0,0 @@
-impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
-
-    pub fn ChangeMe(&self) -> () {
-    }
-}
diff --git a/t/t4018/rust-impl.ctx b/t/t4018/rust-impl.ctx
deleted file mode 100644
index 5344c35f3f..0000000000
--- a/t/t4018/rust-impl.ctx
+++ /dev/null
@@ -1 +0,0 @@
-impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
diff --git a/t/t4018/rust-macro-rules b/t/t4018/rust-macro-rules
deleted file mode 100644
index ec610c5b62..0000000000
--- a/t/t4018/rust-macro-rules
+++ /dev/null
@@ -1,6 +0,0 @@
-macro_rules! RIGHT {
-    () => {
-        // a comment
-        let x = ChangeMe;
-    };
-}
diff --git a/t/t4018/rust-macro-rules.ctx b/t/t4018/rust-macro-rules.ctx
deleted file mode 100644
index 7520463aa0..0000000000
--- a/t/t4018/rust-macro-rules.ctx
+++ /dev/null
@@ -1 +0,0 @@
-macro_rules! RIGHT {
diff --git a/t/t4018/rust-struct b/t/t4018/rust-struct
deleted file mode 100644
index 76aff1c0d8..0000000000
--- a/t/t4018/rust-struct
+++ /dev/null
@@ -1,5 +0,0 @@
-#[derive(Debug)]
-pub(super) struct RIGHT<'a> {
-    name: &'a str,
-    age: ChangeMe,
-}
diff --git a/t/t4018/rust-struct.ctx b/t/t4018/rust-struct.ctx
deleted file mode 100644
index c1e09dc808..0000000000
--- a/t/t4018/rust-struct.ctx
+++ /dev/null
@@ -1 +0,0 @@
-pub(super) struct RIGHT<'a> {
diff --git a/t/t4018/rust-trait b/t/t4018/rust-trait
deleted file mode 100644
index ea397f09ed..0000000000
--- a/t/t4018/rust-trait
+++ /dev/null
@@ -1,5 +0,0 @@
-unsafe trait RIGHT<T> {
-    fn len(&self) -> u32;
-    fn ChangeMe(&self, n: u32) -> T;
-    fn iter<F>(&self, f: F) where F: Fn(T);
-}
diff --git a/t/t4018/rust-trait.ctx b/t/t4018/rust-trait.ctx
deleted file mode 100644
index 6af803db29..0000000000
--- a/t/t4018/rust-trait.ctx
+++ /dev/null
@@ -1 +0,0 @@
-unsafe trait RIGHT<T> {
diff --git a/t/t4018/rust.sh b/t/t4018/rust.sh
new file mode 100755
index 0000000000..ba018c6b95
--- /dev/null
+++ b/t/t4018/rust.sh
@@ -0,0 +1,60 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'rust: fn' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
+EOF_HUNK
+pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
+    let _ = x;
+    // a comment
+    let a = ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'rust: impl' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
+EOF_HUNK
+impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
+
+    pub fn ChangeMe(&self) -> () {
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'rust: macro rules' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+macro_rules! RIGHT {
+EOF_HUNK
+macro_rules! RIGHT {
+    () => {
+        // a comment
+        let x = ChangeMe;
+    };
+}
+EOF_TEST
+
+test_diff_funcname 'rust: struct' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+pub(super) struct RIGHT<'a> {
+EOF_HUNK
+#[derive(Debug)]
+pub(super) struct RIGHT<'a> {
+    name: &'a str,
+    age: ChangeMe,
+}
+EOF_TEST
+
+test_diff_funcname 'rust: trait' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+unsafe trait RIGHT<T> {
+EOF_HUNK
+unsafe trait RIGHT<T> {
+    fn len(&self) -> u32;
+    fn ChangeMe(&self, n: u32) -> T;
+    fn iter<F>(&self, f: F) where F: Fn(T);
+}
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 12/27] userdiff tests: rewrite hunk header test infrastructure

[Johannes Sixt]

Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
> Rewrite the hunk header test infrastructure introduced in
> bfa7d01413 (t4018: an infrastructure to test hunk headers,
> 2014-03-21). See c228a5c077 (Merge branch 'js/userdiff-cc',
> 2014-03-31) for the whole series that commit was part of.
> 
> As noted in an earlier commit that change introduced the regression of
> not testing for the full hunk line, but just whether "RIGHT" appeared
> on it[1]. A preceding commit fixed that specific issue, but we were
> still left with the inflexibility of the approach described in the
> now-deleted t/t4018/README.
> 
> I.e. to add any sort of new tests that used the existing test data
> we'd either need to add more files like the recently added (but now
> deleted) *.ctx) files, using the filesystem as our test datastructure,
> or introduce more parsing for the custom file format we were growing
> here.
> 
> Let's instead just move this over to using a custom test
> function. This makes it trivial to add new tests by adding new
> optional parameters to the function. Let's still keep the relevant
> files in the "t/t4018/" subdirectory instead of adding ~1.5k
> lines (and growing) to "t/t4018-diff-funcname.sh"
> 
> If this diff is viewed with "--color-moved=plain" we can see that
> there's no changes to the lines being moved into the new *.sh files,
> i.e. all the deletions are moves. I'm just adding boilerplate around
> those existing lines.
> 
> The one-off refactoring was performed by an ad-hoc shellscript [2].
> 
> 1. https://lore.kernel.org/git/87wnvbbf2y.fsf@evledraar.gmail.com/
> 2.
> 	#!/bin/sh
> 	set -ex
> 
> 	git rm README*
> 	for t in $(git ls-files ':!*.ctx')
> 	do
> 		lang=$(echo $t | sed 's/-.*//')
> 		desc=$(echo $t | sed -E 's/^[^-]*-//' | tr - " ")
> 
> 		if ! test -e $lang.sh
> 		then
> 			cat >$lang.sh <<-EOF
> 			#!/bin/sh
> 			#
> 			# See ../t4018-diff-funcname.sh's test_diff_funcname()
> 			#
> 
> 			EOF
> 		else
> 			echo >>$lang.sh
> 	        fi
> 
> 		(
> 	            printf "test_diff_funcname '%s: %s' \\" "$lang" "$desc"
> 	            echo
> 	            printf "\t8<<%sEOF_HUNK 9<<%sEOF_TEST\n" '\' '\'
> 	            cat $t.ctx
> 	            printf "EOF_HUNK\n"
> 	            cat $t
> 	            printf "EOF_TEST\n"
> 		) >>$lang.sh
> 
> 		chmod +x $lang.sh
> 		git add $lang.sh
> 	        git rm $t $t.ctx
> 	done
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---

> diff --git a/t/t4018/bash.sh b/t/t4018/bash.sh
> new file mode 100755
> index 0000000000..69144d9144
> --- /dev/null
> +++ b/t/t4018/bash.sh
> @@ -0,0 +1,160 @@
> +#!/bin/sh
> +#
> +# See ../t4018-diff-funcname.sh's test_diff_funcname()
> +#
> +
> +test_diff_funcname 'bash: arithmetic function' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +RIGHT()
> +EOF_HUNK
> +RIGHT() ((
> +
> +    ChangeMe = "$x" + "$y"
> +))
> +EOF_TEST
> +
> +test_diff_funcname 'bash: bashism style compact' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +function RIGHT {
> +EOF_HUNK
> +function RIGHT {
> +    function InvalidSyntax{
> +        :
> +        echo 'ChangeMe'
> +    }
> +}
> +EOF_TEST
> +
> +test_diff_funcname 'bash: bashism style function' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +function RIGHT {
> +EOF_HUNK
> +function RIGHT {
> +    :
> +    echo 'ChangeMe'
> +}
> +EOF_TEST
> [...]

That is not my dream of "simple". But I'm not a userdiff author anymore, 
so...

I don't know, yet, where this is heading to what the advantage is. At 
any rate, "trivial to add new tests" was also the case when each test 
case was in its own file. Without the boilerplate!

-- Hannes

Re: [PATCH v2 12/27] userdiff tests: rewrite hunk header test infrastructure

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 15 2021, Johannes Sixt wrote:

> Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
>> Rewrite the hunk header test infrastructure introduced in
>> bfa7d01413 (t4018: an infrastructure to test hunk headers,
>> 2014-03-21). See c228a5c077 (Merge branch 'js/userdiff-cc',
>> 2014-03-31) for the whole series that commit was part of.
>> As noted in an earlier commit that change introduced the regression
>> of
>> not testing for the full hunk line, but just whether "RIGHT" appeared
>> on it[1]. A preceding commit fixed that specific issue, but we were
>> still left with the inflexibility of the approach described in the
>> now-deleted t/t4018/README.
>> I.e. to add any sort of new tests that used the existing test data
>> we'd either need to add more files like the recently added (but now
>> deleted) *.ctx) files, using the filesystem as our test datastructure,
>> or introduce more parsing for the custom file format we were growing
>> here.
>> Let's instead just move this over to using a custom test
>> function. This makes it trivial to add new tests by adding new
>> optional parameters to the function. Let's still keep the relevant
>> files in the "t/t4018/" subdirectory instead of adding ~1.5k
>> lines (and growing) to "t/t4018-diff-funcname.sh"
>> If this diff is viewed with "--color-moved=plain" we can see that
>> there's no changes to the lines being moved into the new *.sh files,
>> i.e. all the deletions are moves. I'm just adding boilerplate around
>> those existing lines.
>> The one-off refactoring was performed by an ad-hoc shellscript [2].
>> 1. https://lore.kernel.org/git/87wnvbbf2y.fsf@evledraar.gmail.com/
>> 2.
>> 	#!/bin/sh
>> 	set -ex
>> 	git rm README*
>> 	for t in $(git ls-files ':!*.ctx')
>> 	do
>> 		lang=$(echo $t | sed 's/-.*//')
>> 		desc=$(echo $t | sed -E 's/^[^-]*-//' | tr - " ")
>> 		if ! test -e $lang.sh
>> 		then
>> 			cat >$lang.sh <<-EOF
>> 			#!/bin/sh
>> 			#
>> 			# See ../t4018-diff-funcname.sh's test_diff_funcname()
>> 			#
>> 			EOF
>> 		else
>> 			echo >>$lang.sh
>> 	        fi
>> 		(
>> 	            printf "test_diff_funcname '%s: %s' \\" "$lang" "$desc"
>> 	            echo
>> 	            printf "\t8<<%sEOF_HUNK 9<<%sEOF_TEST\n" '\' '\'
>> 	            cat $t.ctx
>> 	            printf "EOF_HUNK\n"
>> 	            cat $t
>> 	            printf "EOF_TEST\n"
>> 		) >>$lang.sh
>> 		chmod +x $lang.sh
>> 		git add $lang.sh
>> 	        git rm $t $t.ctx
>> 	done
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>
>> diff --git a/t/t4018/bash.sh b/t/t4018/bash.sh
>> new file mode 100755
>> index 0000000000..69144d9144
>> --- /dev/null
>> +++ b/t/t4018/bash.sh
>> @@ -0,0 +1,160 @@
>> +#!/bin/sh
>> +#
>> +# See ../t4018-diff-funcname.sh's test_diff_funcname()
>> +#
>> +
>> +test_diff_funcname 'bash: arithmetic function' \
>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>> +RIGHT()
>> +EOF_HUNK
>> +RIGHT() ((
>> +
>> +    ChangeMe = "$x" + "$y"
>> +))
>> +EOF_TEST
>> +
>> +test_diff_funcname 'bash: bashism style compact' \
>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>> +function RIGHT {
>> +EOF_HUNK
>> +function RIGHT {
>> +    function InvalidSyntax{
>> +        :
>> +        echo 'ChangeMe'
>> +    }
>> +}
>> +EOF_TEST
>> +
>> +test_diff_funcname 'bash: bashism style function' \
>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>> +function RIGHT {
>> +EOF_HUNK
>> +function RIGHT {
>> +    :
>> +    echo 'ChangeMe'
>> +}
>> +EOF_TEST
>> [...]
>
> That is not my dream of "simple". But I'm not a userdiff author
> anymore, so...
>
> I don't know, yet, where this is heading to what the advantage is. At
> any rate,[...]

I originally started writing this because I noticed I could break the
userdiff.c patterns and still have all tests pass, i.e. if you screw up
the capture grouping you can go from:
    
    @@ -2,3 +2,3 @@ function        RIGHT   (       )       {
    @@ -2,3 +2,3 @@ RIGHT   (       )

to:
    
    @@ -2,3 +2,3 @@ function        RIGHT   (       )       {
    @@ -2,3 +2,3 @@          RIGHT  (       )
    
And we wouldn't care because we just "grep 'RIGHT'". In this case we
really care about the difference between "^[ \t]*(.*)$" and a broken
"^([ \t]*.*)$" so not having the tests structurally hide the difference
makes sense.

> [...] "trivial to add new tests" was also the case when each test case
> was in its own file[...]

"trivial to add new tests by adding new optional parameters to the
function". I.e. aside from the s/grep/test_cmp/ change in 09/27 the
existing tests were OK if you wanted to test exactly what they expected,
and no more.

I think it just makes sense to have a test helper function instead and
little bit of boilerplate, as seen e.g. in 14/27 and later in the series
we can add new test modes and set per-test config without needing the
top-level dispatch loop to be aware of it.

> [...] Without the boilerplate!

I realize that's a matter of taste, i.e. when to come up with some
custom format v.s. writng a function.

FWIW as someone who didn't author the format I've come across it N times
over the years and each time ended up being more confused than when
reading any custom test function we have.

For those you can usually just look at the definition/arguments, whereas
this always required a careful read of t4018-diff-funcname.sh.

I also find it easier to have one ~160 line file in my editor than ~150
lines spread over 15 files, as in the recent addition of bash support in
2ff6c34612 (userdiff: support Bash, 2020-10-22).

It also depends on how you're counting boilerplate, if you're looking at
it as a patch on the ML it would be ~160 lines of bash.sh, v.s. ~150
lines of the same content, if we're counting the boilerplate diff of 6
lines for every new file :)

Re: [PATCH v2 12/27] userdiff tests: rewrite hunk header test infrastructure

[Johannes Sixt]

Am 15.02.21 um 21:06 schrieb Ævar Arnfjörð Bjarmason:
> On Mon, Feb 15 2021, Johannes Sixt wrote:
>> Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
>>> +test_diff_funcname 'bash: bashism style compact' \
>>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>>> +function RIGHT {
>>> +EOF_HUNK
>>> +function RIGHT {
>>> +    function InvalidSyntax{
>>> +        :
>>> +        echo 'ChangeMe'
>>> +    }
>>> +}
>>> +EOF_TEST
>>> +
>>> +test_diff_funcname 'bash: bashism style function' \
>>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>>> +function RIGHT {
>>> +EOF_HUNK
>>> +function RIGHT {
>>> +    :
>>> +    echo 'ChangeMe'
>>> +}
>>> +EOF_TEST
>>> [...]
>>
>> That is not my dream of "simple". But I'm not a userdiff author
>> anymore, so...
>>
>> I don't know, yet, where this is heading to what the advantage is. At
>> any rate,[...]
> 
> I originally started writing this because I noticed I could break the
> userdiff.c patterns and still have all tests pass, i.e. if you screw up
> the capture grouping you can go from:
>      
>      @@ -2,3 +2,3 @@ function        RIGHT   (       )       {
>      @@ -2,3 +2,3 @@ RIGHT   (       )
> 
> to:
>      
>      @@ -2,3 +2,3 @@ function        RIGHT   (       )       {
>      @@ -2,3 +2,3 @@          RIGHT  (       )
>      
> And we wouldn't care because we just "grep 'RIGHT'". In this case we
> really care about the difference between "^[ \t]*(.*)$" and a broken
> "^([ \t]*.*)$" so not having the tests structurally hide the difference
> makes sense.

But why care? The second version is not wrong. If it's not pretty, the 
pattern author will notice soon enough. I regard avoiding the regression 
you cite less important than the simplicity of adding new test cases.

> 
>> [...] "trivial to add new tests" was also the case when each test case
>> was in its own file[...]
> 
> "trivial to add new tests by adding new optional parameters to the
> function". I.e. aside from the s/grep/test_cmp/ change in 09/27 the
> existing tests were OK if you wanted to test exactly what they expected,
> and no more.

I did not understand what you meant with "new optional parameters" and I 
still do not and why it is desirable.

> 
> I think it just makes sense to have a test helper function instead and
> little bit of boilerplate, as seen e.g. in 14/27 and later in the series
> we can add new test modes and set per-test config without needing the
> top-level dispatch loop to be aware of it.
> 
>> [...] Without the boilerplate!
> 
> I realize that's a matter of taste, i.e. when to come up with some
> custom format v.s. writng a function.
> 
> FWIW as someone who didn't author the format I've come across it N times
> over the years and each time ended up being more confused than when
> reading any custom test function we have.
> 
> For those you can usually just look at the definition/arguments, whereas
> this always required a careful read of t4018-diff-funcname.sh.
> 
> I also find it easier to have one ~160 line file in my editor than ~150
> lines spread over 15 files, as in the recent addition of bash support in
> 2ff6c34612 (userdiff: support Bash, 2020-10-22).

I wouldn't mind having all test cases in a single file. But then it 
should be one file in the language that is being tested, not shell 
script with foreign text between the lines. Adding test cases should be 
easy for authors; they should not have to be proficient in shell 
scripting (and knwo about 130 lines of CodingGuidelines).

-- Hannes

Re: [PATCH v2 12/27] userdiff tests: rewrite hunk header test infrastructure

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> Rewrite the hunk header test infrastructure introduced in
> bfa7d01413 (t4018: an infrastructure to test hunk headers,
> 2014-03-21).

Is this "oops, we added something unacceptably ugly in 09/27 and
this step fixes it"?  Please don't.

[PATCH v2 13/27] userdiff tests: do config teardown in test_diff_funcname()

[Ævar Arnfjörð Bjarmason]

Do a teardown of any custom "diff.<what>.x?funcname" config after a
test_diff_funcname() test runs. Nothing currently uses this, but a
follow-up commit will start setting custom config before certain
tests. Centralizing this teardown makes the tests simpler.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index decf7961f9..4cb0b7ba2b 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -78,6 +78,13 @@ test_diff_funcname () {
 		git diff -U1 "$what" >diff &&
 		sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <diff >actual &&
 		test_cmp expected actual
+	' &&
+
+	test_expect_success "teardown: $desc" '
+		# In case any custom config was set immediately before
+		# the test itself in the test file
+		test_unconfig "diff.$what.funcname" &&
+		test_unconfig "diff.$what.xfuncname"
 	'
 }
 
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 14/27] userdiff tests: move custom patterns into one test file

[Ævar Arnfjörð Bjarmason]

In a preceding commit the test infrastructure got rewritten so
"t/t4018/" are now normal test files which can do things like set
config, so let's make it responsible for setting up and tearing down
the config for its tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 16 +-------
 t/t4018/custom.sh        | 79 ++++++++++++++++++++++++++++++++++++++++
 t/t4018/custom1.sh       | 27 --------------
 t/t4018/custom2.sh       | 18 ---------
 t/t4018/custom3.sh       | 27 --------------
 5 files changed, 80 insertions(+), 87 deletions(-)
 create mode 100755 t/t4018/custom.sh
 delete mode 100755 t/t4018/custom1.sh
 delete mode 100755 t/t4018/custom2.sh
 delete mode 100755 t/t4018/custom3.sh

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 4cb0b7ba2b..d80a2ad4a4 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -11,18 +11,6 @@ test_expect_success 'setup' '
 	builtin_drivers=$(test-tool userdiff list-builtin-drivers) &&
 	test -n "$builtin_drivers" &&
 
-	# a non-trivial custom pattern
-	git config diff.custom1.funcname "!static
-!String
-[^ 	].*s.*" &&
-
-	# a custom pattern which matches to end of line
-	git config diff.custom2.funcname "......Beer\$" &&
-
-	# alternation in pattern
-	git config diff.custom3.funcname "Beer$" &&
-	git config diff.custom3.xfuncname "^[ 	]*((public|static).*)$" &&
-
 	# for regexp compilation tests
 	echo A >A.java &&
 	echo B >B.java
@@ -30,9 +18,7 @@ test_expect_success 'setup' '
 
 diffpatterns="
 	$builtin_drivers
-	custom1
-	custom2
-	custom3
+	custom
 "
 
 for p in $diffpatterns
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
new file mode 100755
index 0000000000..59d855c01c
--- /dev/null
+++ b/t/t4018/custom.sh
@@ -0,0 +1,79 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_expect_success 'custom: setup non-trivial custom' '
+	git config diff.custom.funcname "!static
+!String
+[^ 	].*s.*"
+'
+
+test_diff_funcname 'custom: non-trivial custom pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+int special, RIGHT;
+EOF_HUNK
+public class Beer
+{
+	int special, RIGHT;
+	public static void main(String args[])
+	{
+		String s=" ";
+		for(int x = 99; x > 0; x--)
+		{
+			System.out.print(x + " bottles of beer on the wall "
+				+ x + " bottles of beer\n" // ChangeMe
+				+ "Take one down, pass it around, " + (x - 1)
+				+ " bottles of beer on the wall.\n");
+		}
+		System.out.print("Go to the store, buy some more,\n"
+			+ "99 bottles of beer on the wall.\n");
+	}
+}
+EOF_TEST
+
+test_expect_success 'custom: setup match to end of line' '
+	git config diff.custom.funcname "......Beer\$"
+'
+
+test_diff_funcname 'custom: match to end of line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT_Beer
+EOF_HUNK
+public class RIGHT_Beer
+{
+	int special;
+	public static void main(String args[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
+
+test_expect_success 'custom: setup alternation in pattern' '
+	git config diff.custom.funcname "Beer$" &&
+	git config diff.custom.xfuncname "^[ 	]*((public|static).*)$"
+'
+
+test_diff_funcname 'custom: alternation in pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
+public class Beer
+{
+	int special;
+	public static void main(String RIGHT[])
+	{
+		String s=" ";
+		for(int x = 99; x > 0; x--)
+		{
+			System.out.print(x + " bottles of beer on the wall "
+				+ x + " bottles of beer\n" // ChangeMe
+				+ "Take one down, pass it around, " + (x - 1)
+				+ " bottles of beer on the wall.\n");
+		}
+		System.out.print("Go to the store, buy some more,\n"
+			+ "99 bottles of beer on the wall.\n");
+	}
+}
+EOF_TEST
diff --git a/t/t4018/custom1.sh b/t/t4018/custom1.sh
deleted file mode 100755
index f8bbccadb4..0000000000
--- a/t/t4018/custom1.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom1: pattern' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-int special, RIGHT;
-EOF_HUNK
-public class Beer
-{
-	int special, RIGHT;
-	public static void main(String args[])
-	{
-		String s=" ";
-		for(int x = 99; x > 0; x--)
-		{
-			System.out.print(x + " bottles of beer on the wall "
-				+ x + " bottles of beer\n" // ChangeMe
-				+ "Take one down, pass it around, " + (x - 1)
-				+ " bottles of beer on the wall.\n");
-		}
-		System.out.print("Go to the store, buy some more,\n"
-			+ "99 bottles of beer on the wall.\n");
-	}
-}
-EOF_TEST
diff --git a/t/t4018/custom2.sh b/t/t4018/custom2.sh
deleted file mode 100755
index c68421f788..0000000000
--- a/t/t4018/custom2.sh
+++ /dev/null
@@ -1,18 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom2: match to end of line' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-RIGHT_Beer
-EOF_HUNK
-public class RIGHT_Beer
-{
-	int special;
-	public static void main(String args[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
-EOF_TEST
diff --git a/t/t4018/custom3.sh b/t/t4018/custom3.sh
deleted file mode 100755
index 07c5c134ff..0000000000
--- a/t/t4018/custom3.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom3: alternation in pattern' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-public static void main(String RIGHT[])
-EOF_HUNK
-public class Beer
-{
-	int special;
-	public static void main(String RIGHT[])
-	{
-		String s=" ";
-		for(int x = 99; x > 0; x--)
-		{
-			System.out.print(x + " bottles of beer on the wall "
-				+ x + " bottles of beer\n" // ChangeMe
-				+ "Take one down, pass it around, " + (x - 1)
-				+ " bottles of beer on the wall.\n");
-		}
-		System.out.print("Go to the store, buy some more,\n"
-			+ "99 bottles of beer on the wall.\n");
-	}
-}
-EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 15/27] userdiff tests: remove hack for "RIGHT" token

[Ævar Arnfjörð Bjarmason]

Now that the "RIGHT" token isn't how we select the desired hunk header
line in the test anymore we can revert a hack added in
f1b75fbaf1 (t4018: convert custom pattern test to the new
infrastructure, 2014-03-21) and go back to the regular expression we
were testing before that change.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/custom.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 59d855c01c..b208a771d2 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -33,14 +33,14 @@ public class Beer
 EOF_TEST
 
 test_expect_success 'custom: setup match to end of line' '
-	git config diff.custom.funcname "......Beer\$"
+	git config diff.custom.funcname "Beer\$"
 '
 
 test_diff_funcname 'custom: match to end of line' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-RIGHT_Beer
+Beer
 EOF_HUNK
-public class RIGHT_Beer
+public class Beer
 {
 	int special;
 	public static void main(String args[])
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 16/27] userdiff tests: do not do compile tests on "custom" pattern

[Ævar Arnfjörð Bjarmason]

Since f1b75fbaf1 (t4018: convert custom pattern test to the new
infrastructure, 2014-03-21) we have been doing the basic sanity check
of whether patterns in userdiff.c compile on the "custom" patterns.

That we were doing this was an emergent effect of that change and an
earlier refactoring in bfa7d01413 (t4018: an infrastructure to test
hunk headers, 2014-03-21).

This was never intended by the test added in
e3bf5e43fd (t4018-diff-funcname: test syntax of builtin xfuncname
patterns, 2008-09-22), nor is there any point in doing this. We'll
error out in the custom.sh test itself if those patterns don't
compile.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index d80a2ad4a4..3ba9d657b1 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -16,12 +16,7 @@ test_expect_success 'setup' '
 	echo B >B.java
 '
 
-diffpatterns="
-	$builtin_drivers
-	custom
-"
-
-for p in $diffpatterns
+for p in $builtin_drivers
 do
 	test_expect_success "builtin $p pattern compiles" '
 		echo "*.java diff=$p" >.gitattributes &&
@@ -74,7 +69,7 @@ test_diff_funcname () {
 	'
 }
 
-for what in $diffpatterns
+for what in $builtin_drivers custom
 do
 	test="$TEST_DIRECTORY/t4018/$what.sh"
 	if ! test -e "$test"
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 17/27] userdiff tests + docs: document & test "diff.<driver>.x?funcname"

[Ævar Arnfjörð Bjarmason]

Add the missing documentation for "diff.<driver>.funcname" and test
for how it and "diff.<driver>.xfuncname" interact.

Between the introduction of the "diff.<driver>.xfuncname" form in
45d9414fa5 (diff.*.xfuncname which uses "extended" regex's for hunk
header selection, 2008-09-18) and when this documentation was written
in 90b94c26f7 (Documentation: Add diff.<driver>.* to config,
2011-04-07) we forgot to document the existence of
"diff.<driver>.funcname".

Let's make a mention of it here, we could also partially revert the
former commit and discuss the more verbose form in gitattributes(5),
but let's stop short of that. It makes sense to guide users towards
ERE over BRE whenever possible.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/config/diff.txt | 11 +++++++++++
 t/t4018/custom.sh             | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 45 insertions(+)

diff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt
index 2d3331f55c..6f39ef1da9 100644
--- a/Documentation/config/diff.txt
+++ b/Documentation/config/diff.txt
@@ -153,10 +153,21 @@ diff.<driver>.command::
 	The custom diff driver command.  See linkgit:gitattributes[5]
 	for details.
 
+diff.<driver>.funcname::
 diff.<driver>.xfuncname::
 	The regular expression that the diff driver should use to
 	recognize the hunk header.  A built-in pattern may also be used.
 	See linkgit:gitattributes[5] for details.
++
+When provided as `diff.<driver>.funcname` the regular expression is
+interpreted as a basic regular expression. With
+`diff.<driver>.xfuncname` it's interpreted as an extended regular
+expression.
++
+The `*.funcname` and `*.xfuncname` variables behave as if though they
+were one configuration variable for the purposes of what value
+eventually gets used. Setting `*.funcname` will override an earlier
+`*.xfuncname` and vice-versa.
 
 diff.<driver>.binary::
 	Set this option to true to make the diff driver treat files as
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index b208a771d2..72d38dad68 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -77,3 +77,37 @@ public class Beer
 	}
 }
 EOF_TEST
+
+test_expect_success 'custom: setup config precedence' '
+	git config diff.custom.funcname "foo" &&
+	git config diff.custom.xfuncname "bar"
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+bar
+EOF_HUNK
+foo
+bar
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: setup config precedence' '
+	git config diff.custom.xfuncname "bar" &&
+	git config diff.custom.funcname "foo"
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+foo
+EOF_HUNK
+foo
+bar
+
+ChangeMe
+
+baz
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 18/27] gitattributes doc: reword discussion of built-in userdiff patterns

[Ævar Arnfjörð Bjarmason]

Reword the discussion of the built-in userdiff patterns to make it
more natural to precede it with a discussion about the semantics of
pattern matching, instead of assuming that it follows right after the
"diff.tex.xfuncname" example which now immediately precedes it. This
will make a follow-up commit smaller.

Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/gitattributes.txt | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index e84e104f93..62c1147ba9 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -794,11 +794,17 @@ backslashes; the pattern above picks a line that begins with a
 backslash, and zero or more occurrences of `sub` followed by
 `section` followed by open brace, to the end of line.
 
-There are a few built-in patterns to make this easier, and `tex`
-is one of them, so you do not have to write the above in your
-configuration file (you still need to enable this with the
-attribute mechanism, via `.gitattributes`).  The following built in
-patterns are available:
+There are built-in patterns shipped as part of git itself. A more
+advanced version of the `tex` pattern discussed above is one of them.
+
+For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
+configuration file as discussed above, but if present, it will
+override a built-in pattern.
+
+Nevertheless, you need to enable built-in patterns via .gitattributes`
+for the pattern to take effect.
+
+The following built-in patterns are available:
 
 - `ada` suitable for source code in the Ada language.
 
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 18/27] gitattributes doc: reword discussion of built-in userdiff patterns

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> Reword the discussion of the built-in userdiff patterns to make it
> more natural to precede it with a discussion about the semantics of
> pattern matching, instead of assuming that it follows right after the
> "diff.tex.xfuncname" example which now immediately precedes it. This
> will make a follow-up commit smaller.
>
> Helped-by: Eric Sunshine <sunshine@sunshineco.com>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  Documentation/gitattributes.txt | 16 +++++++++++-----
>  1 file changed, 11 insertions(+), 5 deletions(-)

Good changes.  The current text, which dates back to ae7aa499
(Document custom hunk header selection, 2007-07-08), may have been
adequate back when there were only "java" and "tex" built-in
patterns but not anymore.

I may (or may not) give the series a second pass to collect
independently salvageable bits, and if I did so, this would
be one of those that would be gleaned.

Thanks.

> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
> index e84e104f93..62c1147ba9 100644
> --- a/Documentation/gitattributes.txt
> +++ b/Documentation/gitattributes.txt
> @@ -794,11 +794,17 @@ backslashes; the pattern above picks a line that begins with a
>  backslash, and zero or more occurrences of `sub` followed by
>  `section` followed by open brace, to the end of line.
>  
> -There are a few built-in patterns to make this easier, and `tex`
> -is one of them, so you do not have to write the above in your
> -configuration file (you still need to enable this with the
> -attribute mechanism, via `.gitattributes`).  The following built in
> -patterns are available:
> +There are built-in patterns shipped as part of git itself. A more
> +advanced version of the `tex` pattern discussed above is one of them.
> +
> +For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
> +configuration file as discussed above, but if present, it will
> +override a built-in pattern.
> +
> +Nevertheless, you need to enable built-in patterns via .gitattributes`
> +for the pattern to take effect.
> +
> +The following built-in patterns are available:
>  
>  - `ada` suitable for source code in the Ada language.

[PATCH v2 19/27] gitattributes doc: document multi-line userdiff patterns

[Ævar Arnfjörð Bjarmason]

Document the multi-line userdiff patterns and how their matching and
the negation syntax works.

These patterns have been supported since f258475a6e (Per-path
attribute based hunk header selection., 2007-07-06), and have had
their current semantics ever since 3d8dccd74a (diff: fix "multiple
regexp" semantics to find hunk header comment, 2008-09-20).

But we had no documentation for them, let's fix that, and also add
tests showing how some of the things being discussed here work.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/gitattributes.txt | 33 +++++++++++++++++++---
 t/t4018/custom.sh               | 50 +++++++++++++++++++++++++++++++++
 t/t4018/perl.sh                 | 16 +++++++++++
 3 files changed, 95 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index 62c1147ba9..b51d2c86e0 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -794,12 +794,37 @@ backslashes; the pattern above picks a line that begins with a
 backslash, and zero or more occurrences of `sub` followed by
 `section` followed by open brace, to the end of line.
 
-There are built-in patterns shipped as part of git itself. A more
-advanced version of the `tex` pattern discussed above is one of them.
+Multiple patterns can be supplied by listing them one per line
+separated by `\n`. They will be matched one at a time from left to
+right. Do not supply a trailing "\n" for the last pattern. E.g.:
+
+------------------------
+[diff "perl"]
+	xfuncname = "!^=head\n^[^ ]+.*"
+------------------------
+
+Patterns in in a list of multiple that begin with "!" are negated. A
+matching negated pattern will cause the matched line to be
+skipped. Use it to skip a later pattern that would otherwise match. It
+is an error if one or more negated patterns aren't followed by a
+non-negated pattern.
+
+To match a literal "!" at the start of a line, use some other regex
+construct that will match a literal "!" without "!" being the first
+character on that line, such as "[!]".
+
+If the pattern contains a `$1` capture it will be used instead of the
+entire matching line (`$0`) to display the hunk header. This can be
+used e.g. to strip whitespace from the beginning of the line, or to
+only display the function name as part of a longer function
+definition.
+
+There are built-in patterns shipped as part of git itself, see the
+full listing below.
 
 For built-in patterns, you do not need `diff.<lang>.xfuncname` in your
-configuration file as discussed above, but if present, it will
-override a built-in pattern.
+configuration file. If present, it will override a built-in pattern,
+as shown in the `diff.perl.xfuncname` example above.
 
 Nevertheless, you need to enable built-in patterns via .gitattributes`
 for the pattern to take effect.
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 72d38dad68..30df13d8b2 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -111,3 +111,53 @@ ChangeMe
 
 baz
 EOF_TEST
+
+test_expect_success 'custom: setup negation syntax, ! is magic' '
+	git config diff.custom.xfuncname "!negation
+line"
+'
+
+test_diff_funcname 'custom: negation syntax, ! is magic' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+line
+EOF_HUNK
+line
+!negation
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: setup negation syntax, use [!] to override ! magic' '
+	git config diff.custom.xfuncname "[!]negation
+line"
+'
+
+test_diff_funcname 'custom: negation syntax, use [!] to override ! magic' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+!negation
+EOF_HUNK
+line
+!negation
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: setup captures in multiple patterns' '
+	git config diff.custom.xfuncname "!^=head
+^format ([^ ]+)
+^sub ([^;]+)"
+'
+
+test_diff_funcname 'custom: captures in multiple patterns' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+foo
+EOF_HUNK
+sub foo;
+=head1
+ChangeMe
+
+EOF_TEST
diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
index ac8fff7417..2952483a2c 100755
--- a/t/t4018/perl.sh
+++ b/t/t4018/perl.sh
@@ -76,3 +76,19 @@ sub RIGHT
 	print "ChangeMe\n";
 }
 EOF_TEST
+
+
+test_expect_success 'custom: setup config overrides built-in patterns' '
+	git config diff.perl.xfuncname "!^=head
+^[^ ]+.*"
+'
+
+test_diff_funcname 'custom: config overrides built-in patterns' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub foo;
+EOF_HUNK
+sub foo;
+=head1
+ChangeMe
+
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 19/27] gitattributes doc: document multi-line userdiff patterns

[Johannes Sixt]

Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
> diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
> index 72d38dad68..30df13d8b2 100755
> --- a/t/t4018/custom.sh
> +++ b/t/t4018/custom.sh
> @@ -111,3 +111,53 @@ ChangeMe
>   
>   baz
>   EOF_TEST
> +
> +test_expect_success 'custom: setup negation syntax, ! is magic' '
> +	git config diff.custom.xfuncname "!negation
> +line"
> +'
> +
> +test_diff_funcname 'custom: negation syntax, ! is magic' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +line
> +EOF_HUNK
> +line
> +!negation
> +
> +ChangeMe
> +
> +baz
> +EOF_TEST
> +
> +test_expect_success 'custom: setup negation syntax, use [!] to override ! magic' '
> +	git config diff.custom.xfuncname "[!]negation
> +line"
> +'
> +
> +test_diff_funcname 'custom: negation syntax, use [!] to override ! magic' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +!negation
> +EOF_HUNK
> +line
> +!negation
> +
> +ChangeMe
> +
> +baz
> +EOF_TEST
> +
> +test_expect_success 'custom: setup captures in multiple patterns' '
> +	git config diff.custom.xfuncname "!^=head
> +^format ([^ ]+)
> +^sub ([^;]+)"
> +'
> +
> +test_diff_funcname 'custom: captures in multiple patterns' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +foo
> +EOF_HUNK
> +sub foo;
> +=head1
> +ChangeMe
> +
> +EOF_TEST

This test would not catch a regression. You must leave a line between 
the candidate-that-must-not-be, =head1, and ChangeMe, otherwise, =head1 
is never tested against the negation pattern.

Or did you change the diff invocation in an earlier patch such that it 
does not emit context lines?

> diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
> index ac8fff7417..2952483a2c 100755
> --- a/t/t4018/perl.sh
> +++ b/t/t4018/perl.sh
> @@ -76,3 +76,19 @@ sub RIGHT
>   	print "ChangeMe\n";
>   }
>   EOF_TEST
> +
> +
> +test_expect_success 'custom: setup config overrides built-in patterns' '
> +	git config diff.perl.xfuncname "!^=head
> +^[^ ]+.*"
> +'
> +
> +test_diff_funcname 'custom: config overrides built-in patterns' \
> +	8<<\EOF_HUNK 9<<\EOF_TEST
> +sub foo;
> +EOF_HUNK
> +sub foo;
> +=head1
> +ChangeMe
> +
> +EOF_TEST

Same here.

-- Hannes

Re: [PATCH v2 19/27] gitattributes doc: document multi-line userdiff patterns

[Ævar Arnfjörð Bjarmason]

On Mon, Feb 15 2021, Johannes Sixt wrote:

> Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
>> diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
>> index 72d38dad68..30df13d8b2 100755
>> --- a/t/t4018/custom.sh
>> +++ b/t/t4018/custom.sh
>> @@ -111,3 +111,53 @@ ChangeMe
>>     baz
>>   EOF_TEST
>> +
>> +test_expect_success 'custom: setup negation syntax, ! is magic' '
>> +	git config diff.custom.xfuncname "!negation
>> +line"
>> +'
>> +
>> +test_diff_funcname 'custom: negation syntax, ! is magic' \
>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>> +line
>> +EOF_HUNK
>> +line
>> +!negation
>> +
>> +ChangeMe
>> +
>> +baz
>> +EOF_TEST
>> +
>> +test_expect_success 'custom: setup negation syntax, use [!] to override ! magic' '
>> +	git config diff.custom.xfuncname "[!]negation
>> +line"
>> +'
>> +
>> +test_diff_funcname 'custom: negation syntax, use [!] to override ! magic' \
>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>> +!negation
>> +EOF_HUNK
>> +line
>> +!negation
>> +
>> +ChangeMe
>> +
>> +baz
>> +EOF_TEST
>> +
>> +test_expect_success 'custom: setup captures in multiple patterns' '
>> +	git config diff.custom.xfuncname "!^=head
>> +^format ([^ ]+)
>> +^sub ([^;]+)"
>> +'
>> +
>> +test_diff_funcname 'custom: captures in multiple patterns' \
>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>> +foo
>> +EOF_HUNK
>> +sub foo;
>> +=head1
>> +ChangeMe
>> +
>> +EOF_TEST
>
> This test would not catch a regression. You must leave a line between
> the candidate-that-must-not-be, =head1, and ChangeMe, otherwise,
> =head1 is never tested against the negation pattern.
>
> Or did you change the diff invocation in an earlier patch such that it
> does not emit context lines?
>
>> diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
>> index ac8fff7417..2952483a2c 100755
>> --- a/t/t4018/perl.sh
>> +++ b/t/t4018/perl.sh
>> @@ -76,3 +76,19 @@ sub RIGHT
>>   	print "ChangeMe\n";
>>   }
>>   EOF_TEST
>> +
>> +
>> +test_expect_success 'custom: setup config overrides built-in patterns' '
>> +	git config diff.perl.xfuncname "!^=head
>> +^[^ ]+.*"
>> +'
>> +
>> +test_diff_funcname 'custom: config overrides built-in patterns' \
>> +	8<<\EOF_HUNK 9<<\EOF_TEST
>> +sub foo;
>> +EOF_HUNK
>> +sub foo;
>> +=head1
>> +ChangeMe
>> +
>> +EOF_TEST
>
> Same here.

Well spotted. Both of these test work for their advertised
purpose. I.e. "we can override the perl built-in config" and "capture
count gets reset", but you're right that the =head1 negation rule is
redundant at this point in the series.

In 23/27 we start testing with -U0, at which point it starts
working. But of course a better fix is to just add an empty line here...

Re: [PATCH v2 19/27] gitattributes doc: document multi-line userdiff patterns

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> -There are built-in patterns shipped as part of git itself. A more
> -advanced version of the `tex` pattern discussed above is one of them.
> +Multiple patterns can be supplied by listing them one per line
> +separated by `\n`. They will be matched one at a time from left to
> +right. Do not supply a trailing "\n" for the last pattern. E.g.:

I've always thought that this was modelled after "grep --file=FILE",
where patterns are listed one per line, so "from left to right"
looked a bit funny, at least to me.

If the "do not supply" is because "it would result in giving an
empty string '' as the final pattern, which may not be what you
want", then it is a good idea to spell it out.

> +------------------------
> +[diff "perl"]
> +	xfuncname = "!^=head\n^[^ ]+.*"
> +------------------------
> +
> +Patterns in in a list of multiple that begin with "!" are negated. A

"in in"???

> +matching negated pattern will cause the matched line to be
> +skipped. Use it to skip a later pattern that would otherwise match. It
> +is an error if one or more negated patterns aren't followed by a
> +non-negated pattern.
> +
> +To match a literal "!" at the start of a line, use some other regex
> +construct that will match a literal "!" without "!" being the first
> +character on that line, such as "[!]".
> +
> +If the pattern contains a `$1` capture it will be used instead of the
> +entire matching line (`$0`) to display the hunk header. This can be
> +used e.g. to strip whitespace from the beginning of the line, or to
> +only display the function name as part of a longer function
> +definition.
> +
> +There are built-in patterns shipped as part of git itself, see the
> +full listing below.

Other than the minor nits above, the new text is written very well.

[PATCH v2 20/27] userdiff tests: remove "funcname" from custom3 test

[Ævar Arnfjörð Bjarmason]

We can only have one "funcname" or "xfuncname", any later definition
overrides the earlier one, so this configuration wasn't doing
anything.

When this test was originally added in 3632cfc248 (Use compatibility
regex library for OSX/Darwin, 2008-09-07) we had no such definition of
two patters for this test. Back then this was setting the
"diff.java.funcname" configuration variable.

The stage for that second pattern being set got set later. In
45d9414fa5 (diff.*.xfuncname which uses "extended" regex's for hunk
header selection, 2008-09-18) the pattern got converted from
"funcname" to "xfuncname".

Soon after in b19d288b4d (t4018-diff-funcname: demonstrate end of line
funcname matching flaw, 2008-10-15) another test immediately preceding
this one got added, using "diff.java.funcname" for its configuration.

Then f792a0b88e (t4018 (funcname patterns): make configuration easier
to track, 2011-05-21) came along and codified this whole thing when
converting the two tests from "git config" to "test_config".

Since this was never the intent of the test let's just remove this,
the rationale in f792a0b88e for having some test for the clobbering
behavior makes sense, but I'll do that in another follow-up test, not
as a hard to read side-effect of this one.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/custom.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 30df13d8b2..886de9cddb 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -51,7 +51,6 @@ public class Beer
 EOF_TEST
 
 test_expect_success 'custom: setup alternation in pattern' '
-	git config diff.custom.funcname "Beer$" &&
 	git config diff.custom.xfuncname "^[ 	]*((public|static).*)$"
 '
 
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 21/27] userdiff tests: factor out test_diff_funcname() logic

[Ævar Arnfjörð Bjarmason]

Factor out logic in test_diff_funcname() into two helper functions,
these will be useful in a follow-up commit where we'll do this munging
in more than one place.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 3ba9d657b1..2efe4e5bdd 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -41,6 +41,17 @@ test_expect_success 'last regexp must not be negated' '
 	test_i18ngrep ": Last expression must not be negated:" msg
 '
 
+do_change_me () {
+	file=$1
+	sed -e "s/ChangeMe/IWasChanged/" <"$file" >tmp &&
+	mv tmp "$file"
+}
+
+last_diff_context_line () {
+	file=$1
+	sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <$file
+}
+
 test_diff_funcname () {
 	desc=$1
 	cat <&8 >arg.header &&
@@ -51,13 +62,12 @@ test_diff_funcname () {
 		cp arg.test "$what" &&
 		cp arg.header expected &&
 		git add "$what" &&
-		sed -e "s/ChangeMe/IWasChanged/" <"$what" >tmp &&
-		mv tmp "$what"
+		do_change_me "$what"
 	' &&
 
 	test_expect_success "$desc" '
 		git diff -U1 "$what" >diff &&
-		sed -n -e "s/^.*@@$//p" -e "s/^.*@@ //p" <diff >actual &&
+		last_diff_context_line diff >actual &&
 		test_cmp expected actual
 	' &&
 
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 22/27] userdiff tests: test hunk headers on accumulated files

[Ævar Arnfjörð Bjarmason]

The existing tests in "t/t4018/" are unrealistic in that they're all
setting up small few-line isolated test cases with one thing we could
match as a hunk header, right above the one change in the file.

Expand those tests by accumulating changes within the same file type
in the "test_diff_funcname" function. So e.g. for "bash" we'll end up
a "bash.acc" file with 15 s/ChangeMe/IWasChanged/ changes.

This stress tests whether the hunk header selection will "jump across"
to an earlier change because the match for that is greedier.

As it turns out we had one false positive in "t/t4018/cpp.sh" and
"t4018/matlab.sh" because of how the tests were structured, we must
always give the "ChangeMe" line at least one line of separation from
the header, since it was at the end of those tests we'd select the
"wrong" header. Let's adjust the spacing to compensate.

So in the end we found nothing of interest here, regardless, I think
it is useful to continue to test in this mode. It's likely to aid in
finding bugs in combinations of our positive and negative matching as
we add more built-in patterns.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 19 +++++++++++++++++++
 t/t4018/cpp.sh           |  1 +
 t/t4018/matlab.sh        |  3 +++
 3 files changed, 23 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 2efe4e5bdd..8b4500037f 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -65,12 +65,26 @@ test_diff_funcname () {
 		do_change_me "$what"
 	' &&
 
+	test_expect_success "setup: $desc (accumulated)" '
+		cat arg.test >>arg.tests &&
+		cp arg.tests "$what".acc &&
+		git add "$what".acc &&
+		do_change_me "$what".acc
+	' &&
+
 	test_expect_success "$desc" '
 		git diff -U1 "$what" >diff &&
 		last_diff_context_line diff >actual &&
 		test_cmp expected actual
 	' &&
 
+	test_expect_success "$desc (accumulated)" '
+		git diff -U1 "$what".acc >diff &&
+		last_diff_context_line diff >actual.lines &&
+		tail -n 1 actual.lines >actual &&
+		test_cmp expected actual
+	' &&
+
 	test_expect_success "teardown: $desc" '
 		# In case any custom config was set immediately before
 		# the test itself in the test file
@@ -93,6 +107,11 @@ do
 		echo "$what" >arg.what
 	' &&
 
+	test_expect_success "setup: hunk header for $what (accumulated)" '
+		>arg.tests &&
+		echo "$what.acc diff=$what" >>.gitattributes
+	' &&
+
 	. "$test"
 done
 
diff --git a/t/t4018/cpp.sh b/t/t4018/cpp.sh
index 185d40d5ef..e0ab749316 100755
--- a/t/t4018/cpp.sh
+++ b/t/t4018/cpp.sh
@@ -206,6 +206,7 @@ void wrong()
 struct RIGHT_iterator_tag {};
 
 int ChangeMe;
+
 EOF_TEST
 
 test_diff_funcname 'cpp: template function definition' \
diff --git a/t/t4018/matlab.sh b/t/t4018/matlab.sh
index f62289148e..fba410e6f5 100755
--- a/t/t4018/matlab.sh
+++ b/t/t4018/matlab.sh
@@ -31,6 +31,7 @@ EOF_HUNK
 %%% RIGHT section
 # this is octave script
 ChangeMe = 1;
+
 EOF_TEST
 
 test_diff_funcname 'matlab: octave section 2' \
@@ -40,6 +41,7 @@ EOF_HUNK
 ## RIGHT section
 # this is octave script
 ChangeMe = 1;
+
 EOF_TEST
 
 test_diff_funcname 'matlab: section' \
@@ -49,4 +51,5 @@ EOF_HUNK
 %% RIGHT section
 % this is understood by both matlab and octave
 ChangeMe = 1;
+
 EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 22/27] userdiff tests: test hunk headers on accumulated files

[Johannes Sixt]

Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
> The existing tests in "t/t4018/" are unrealistic in that they're all
> setting up small few-line isolated test cases with one thing we could
> match as a hunk header, right above the one change in the file.
> 
> Expand those tests by accumulating changes within the same file type
> in the "test_diff_funcname" function. So e.g. for "bash" we'll end up
> a "bash.acc" file with 15 s/ChangeMe/IWasChanged/ changes.
> 
> This stress tests whether the hunk header selection will "jump across"
> to an earlier change because the match for that is greedier.
> 
> As it turns out we had one false positive in "t/t4018/cpp.sh" and
> "t4018/matlab.sh" because of how the tests were structured, we must
> always give the "ChangeMe" line at least one line of separation from
> the header, since it was at the end of those tests we'd select the
> "wrong" header. Let's adjust the spacing to compensate.

I can't wrap my head around this. The ChangeMe lines are one line away 
from the RIGHT lines. Why would we need a line below ChangeMe?

Or is this just a fall-out from this change itself? Then I'd appreciate 
to describe it as "This change now triggers false positives in... 
because... Let's adjust the spacing compensate.".

> 
> So in the end we found nothing of interest here, regardless, I think
> it is useful to continue to test in this mode. It's likely to aid in
> finding bugs in combinations of our positive and negative matching as
> we add more built-in patterns.
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   t/t4018-diff-funcname.sh | 19 +++++++++++++++++++
>   t/t4018/cpp.sh           |  1 +
>   t/t4018/matlab.sh        |  3 +++
>   3 files changed, 23 insertions(+)
> 
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index 2efe4e5bdd..8b4500037f 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -65,12 +65,26 @@ test_diff_funcname () {
>   		do_change_me "$what"
>   	' &&
>   
> +	test_expect_success "setup: $desc (accumulated)" '
> +		cat arg.test >>arg.tests &&
> +		cp arg.tests "$what".acc &&
> +		git add "$what".acc &&
> +		do_change_me "$what".acc
> +	' &&
> +
>   	test_expect_success "$desc" '
>   		git diff -U1 "$what" >diff &&
>   		last_diff_context_line diff >actual &&
>   		test_cmp expected actual
>   	' &&
>   
> +	test_expect_success "$desc (accumulated)" '
> +		git diff -U1 "$what".acc >diff &&
> +		last_diff_context_line diff >actual.lines &&
> +		tail -n 1 actual.lines >actual &&
> +		test_cmp expected actual
> +	' &&
> +
>   	test_expect_success "teardown: $desc" '
>   		# In case any custom config was set immediately before
>   		# the test itself in the test file
> @@ -93,6 +107,11 @@ do
>   		echo "$what" >arg.what
>   	' &&
>   
> +	test_expect_success "setup: hunk header for $what (accumulated)" '
> +		>arg.tests &&
> +		echo "$what.acc diff=$what" >>.gitattributes
> +	' &&
> +
>   	. "$test"
>   done
>   
> diff --git a/t/t4018/cpp.sh b/t/t4018/cpp.sh
> index 185d40d5ef..e0ab749316 100755
> --- a/t/t4018/cpp.sh
> +++ b/t/t4018/cpp.sh
> @@ -206,6 +206,7 @@ void wrong()
>   struct RIGHT_iterator_tag {};
>   
>   int ChangeMe;
> +
>   EOF_TEST
>   
>   test_diff_funcname 'cpp: template function definition' \
> diff --git a/t/t4018/matlab.sh b/t/t4018/matlab.sh
> index f62289148e..fba410e6f5 100755
> --- a/t/t4018/matlab.sh
> +++ b/t/t4018/matlab.sh
> @@ -31,6 +31,7 @@ EOF_HUNK
>   %%% RIGHT section
>   # this is octave script
>   ChangeMe = 1;
> +
>   EOF_TEST
>   
>   test_diff_funcname 'matlab: octave section 2' \
> @@ -40,6 +41,7 @@ EOF_HUNK
>   ## RIGHT section
>   # this is octave script
>   ChangeMe = 1;
> +
>   EOF_TEST
>   
>   test_diff_funcname 'matlab: section' \
> @@ -49,4 +51,5 @@ EOF_HUNK
>   %% RIGHT section
>   % this is understood by both matlab and octave
>   ChangeMe = 1;
> +
>   EOF_TEST
> 

-- Hannes

[PATCH v2 23/27] userdiff tests: test hunk header selection with -U0

[Ævar Arnfjörð Bjarmason]

The userdiff tests have used a custom -U1 context since
f12c66b9bb (userdiff/perl: anchor "sub" and "package" patterns on the
left, 2011-05-21). Changing it to -U0 doesn't change the results for
any of the tests, except one.

Let's test for this case explicitly. I.e. that we go "beyond" the
selected context to find our hunk header. In many cases the desired
hunk header is part of the diff itself under -U1.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 13 +++++++++++++
 t/t4018/custom.sh        |  1 +
 2 files changed, 14 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 8b4500037f..d41aed9ba2 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -57,6 +57,7 @@ test_diff_funcname () {
 	cat <&8 >arg.header &&
 	cat <&9 >arg.test &&
 	what=$(cat arg.what) &&
+	arg_diff_U0=$2 &&
 
 	test_expect_success "setup: $desc" '
 		cp arg.test "$what" &&
@@ -78,6 +79,18 @@ test_diff_funcname () {
 		test_cmp expected actual
 	' &&
 
+	test_expect_success "$desc -U0" '
+		git diff -U0 "$what" >diff &&
+		last_diff_context_line diff >actual &&
+		if test -n "$arg_diff_U0"
+		then
+			echo "$arg_diff_U0" >new-expected &&
+			test_cmp new-expected actual
+		else
+			test_cmp expected actual
+		fi
+	' &&
+
 	test_expect_success "$desc (accumulated)" '
 		git diff -U1 "$what".acc >diff &&
 		last_diff_context_line diff >actual.lines &&
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 886de9cddb..a090f7bfc2 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -10,6 +10,7 @@ test_expect_success 'custom: setup non-trivial custom' '
 '
 
 test_diff_funcname 'custom: non-trivial custom pattern' \
+	'System.out.print(x + " bottles of beer on the wall "' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
 int special, RIGHT;
 EOF_HUNK
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH v2 23/27] userdiff tests: test hunk header selection with -U0

[Johannes Sixt]

Am 15.02.21 um 16:44 schrieb Ævar Arnfjörð Bjarmason:
> The userdiff tests have used a custom -U1 context since
> f12c66b9bb (userdiff/perl: anchor "sub" and "package" patterns on the
> left, 2011-05-21). Changing it to -U0 doesn't change the results for
> any of the tests, except one.
> 
> Let's test for this case explicitly. I.e. that we go "beyond" the
> selected context to find our hunk header. In many cases the desired
> hunk header is part of the diff itself under -U1.

Is this intended as a sanity check of test cases? (Otherwise, I see only 
that we are running two diffs instead of just one against every test 
case, but not which problem this is solving.)

> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>   t/t4018-diff-funcname.sh | 13 +++++++++++++
>   t/t4018/custom.sh        |  1 +
>   2 files changed, 14 insertions(+)
> 
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> index 8b4500037f..d41aed9ba2 100755
> --- a/t/t4018-diff-funcname.sh
> +++ b/t/t4018-diff-funcname.sh
> @@ -57,6 +57,7 @@ test_diff_funcname () {
>   	cat <&8 >arg.header &&
>   	cat <&9 >arg.test &&
>   	what=$(cat arg.what) &&
> +	arg_diff_U0=$2 &&
>   
>   	test_expect_success "setup: $desc" '
>   		cp arg.test "$what" &&
> @@ -78,6 +79,18 @@ test_diff_funcname () {
>   		test_cmp expected actual
>   	' &&
>   
> +	test_expect_success "$desc -U0" '
> +		git diff -U0 "$what" >diff &&
> +		last_diff_context_line diff >actual &&
> +		if test -n "$arg_diff_U0"
> +		then
> +			echo "$arg_diff_U0" >new-expected &&
> +			test_cmp new-expected actual
> +		else
> +			test_cmp expected actual
> +		fi
> +	' &&
> +
>   	test_expect_success "$desc (accumulated)" '
>   		git diff -U1 "$what".acc >diff &&
>   		last_diff_context_line diff >actual.lines &&
> diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
> index 886de9cddb..a090f7bfc2 100755
> --- a/t/t4018/custom.sh
> +++ b/t/t4018/custom.sh
> @@ -10,6 +10,7 @@ test_expect_success 'custom: setup non-trivial custom' '
>   '
>   
>   test_diff_funcname 'custom: non-trivial custom pattern' \
> +	'System.out.print(x + " bottles of beer on the wall "' \
>   	8<<\EOF_HUNK 9<<\EOF_TEST
>   int special, RIGHT;
>   EOF_HUNK
> 

-- Hannes

[PATCH v2 24/27] userdiff tests: assert empty hunk header context on -U<large>

[Ævar Arnfjörð Bjarmason]

Assert the existing behavior that under -U<large> we'll show no hunk
header context, where <large> takes us past the potential hunk header
we'd have extracted. I'm just picking a number over nine thousand as a
really large number we're unlikely to exceed in these tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index d41aed9ba2..80f35c5e16 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -98,6 +98,14 @@ test_diff_funcname () {
 		test_cmp expected actual
 	' &&
 
+	test_expect_success "$desc -U9001 (accumulated)" '
+		git diff -U9001 "$what".acc >diff &&
+		last_diff_context_line diff >actual.lines &&
+		tail -n 1 actual.lines >actual &&
+		echo >blank &&
+		test_cmp blank actual
+	' &&
+
 	test_expect_success "teardown: $desc" '
 		# In case any custom config was set immediately before
 		# the test itself in the test file
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 25/27] userdiff: match "package" in diff=golang

[Ævar Arnfjörð Bjarmason]

Improve the "golang" built-in pattern to match "package" lines, as
they weren't matched before changing e.g. the imports would commonly
result in an empty hunk header, now we'll instead show the package
name.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/golang.sh | 10 ++++++++++
 userdiff.c        |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/t/t4018/golang.sh b/t/t4018/golang.sh
index bf22f58c12..cdf9d6f8aa 100755
--- a/t/t4018/golang.sh
+++ b/t/t4018/golang.sh
@@ -3,6 +3,16 @@
 # See ../t4018-diff-funcname.sh's test_diff_funcname()
 #
 
+test_diff_funcname 'golang: package' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+package main
+EOF_HUNK
+package main
+
+import "fmt"
+// ChangeMe
+EOF_TEST
+
 test_diff_funcname 'golang: complex function' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
 func (t *Test) RIGHT(a Type) (Type, error) {
diff --git a/userdiff.c b/userdiff.c
index 55f4f769bd..f975aac8fe 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -128,6 +128,8 @@ IPATTERN("fountain",
 	 /* -- */
 	 "[^ \t-]+"),
 PATTERNS("golang",
+	 /* Packages */
+	 "^[ \t]*(package[ \t]*(.*))\n"
 	 /* Functions */
 	 "^[ \t]*(func[ \t]*.*(\\{[ \t]*)?)\n"
 	 /* Structs and interfaces */
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 26/27] userdiff tests: add basic test for ada

[Ævar Arnfjörð Bjarmason]

Add test for the ada userdiff pattern added in e90d065e64 (Add
userdiff patterns for Ada, 2012-09-16).

I don't know the ada language itself, I just stole a couple of
examples of code that used tokens we're matching[1][2]. Both test
examples stress our negative and positive matching rules.

1. https://rosettacode.org/wiki/99_bottles_of_beer#Ada
2. https://en.wikibooks.org/wiki/Ada_Programming/Tasking
---
 t/t4018/ada.sh | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
 create mode 100755 t/t4018/ada.sh

diff --git a/t/t4018/ada.sh b/t/t4018/ada.sh
new file mode 100755
index 0000000000..45fc2c7a3b
--- /dev/null
+++ b/t/t4018/ada.sh
@@ -0,0 +1,37 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'ada: "procedure" over "with"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+procedure Bottles is
+EOF_HUNK
+with Ada.Text_Io; use Ada.Text_Io;
+ procedure Bottles is
+ begin
+    for X in reverse 1..99 loop
+       Put_Line(Integer'Image(X) & " bottles of beer on the wall");
+       Put_Line(Integer'Image(X) & " bottles of beer"); -- ChangeMe
+       Put_Line("Take one down, pass it around");
+       Put_Line(Integer'Image(X - 1) & " bottles of beer on the wall");
+       New_Line;
+    end loop;
+ end Bottles;
+EOF_TEST
+
+test_diff_funcname 'ada: "task" over "procedure"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+task body Check_CPU is
+EOF_HUNK
+procedure Housekeeping is
+  task Check_CPU;
+  task Backup_Disk;
+
+  task body Check_CPU is
+    -- Comment for spacing with
+    -- the above "task" for -U1
+    ChangeMe
+  end Check_CPU;
+end Housekeeping;
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH v2 27/27] userdiff tests: add basic test for ruby

[Ævar Arnfjörð Bjarmason]

Add a test for the Ruby pattern added way back in ad8c1d9260 (diff:
add ruby funcname pattern, 2008-07-31).

The "One/Two" picking demonstrates existing behavior, and a general
case where we may not do what the user expects since we're not aware
of the indentation level.

The code is modified from the Ruby code we have in-tree at
Documentation/asciidoctor-extensions.rb.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/ruby.sh | 58 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100755 t/t4018/ruby.sh

diff --git a/t/t4018/ruby.sh b/t/t4018/ruby.sh
new file mode 100755
index 0000000000..1e9bfef863
--- /dev/null
+++ b/t/t4018/ruby.sh
@@ -0,0 +1,58 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'ruby: "def" over "class/module"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def process(parent)
+EOF_HUNK
+require 'asciidoctor'
+
+module Git
+  module Documentation
+    class SomeClass
+      use_some
+
+      def process(parent)
+        puts("hello")
+	puts(ChangeMe)
+      end
+    end
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'ruby: "class" over "class/module"' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class Two
+EOF_HUNK
+module Git
+  module Documentation
+    class One
+    end
+
+    class Two
+      # Spacing for -U1
+      ChangeMe
+    end
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'ruby: picks first "class/module/def" before changed context' \
+	"class Two" \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class One
+EOF_HUNK
+module Git
+  module Documentation
+    class One
+    end
+
+    class Two
+      ChangeMe
+    end
+  end
+end
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 01/20] userdiff: refactor away the parse_bool() function

[Ævar Arnfjörð Bjarmason]

Since 6680a0874f (drop odd return value semantics from
userdiff_config, 2012-02-07) we have not cared about the return values
of parse_tristate() or git_config_bool() v.s. falling through in
userdiff_config(), so let's do so in those cases to make the code
easier to read.

Having a wrapper function for git_config_bool() dates back to
d9bae1a178 (diff: cache textconv output, 2010-04-01) and
122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05), both of
which predated the change in 6680a0874f which made their return values
redundant.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index 3f81a2261c..c147bcbb17 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -275,19 +275,12 @@ static int parse_funcname(struct userdiff_funcname *f, const char *k,
 	return 0;
 }
 
-static int parse_tristate(int *b, const char *k, const char *v)
+static void parse_tristate(int *b, const char *k, const char *v)
 {
 	if (v && !strcasecmp(v, "auto"))
 		*b = -1;
 	else
 		*b = git_config_bool(k, v);
-	return 0;
-}
-
-static int parse_bool(int *b, const char *k, const char *v)
-{
-	*b = git_config_bool(k, v);
-	return 0;
 }
 
 int userdiff_config(const char *k, const char *v)
@@ -312,16 +305,17 @@ int userdiff_config(const char *k, const char *v)
 		return parse_funcname(&drv->funcname, k, v, 0);
 	if (!strcmp(type, "xfuncname"))
 		return parse_funcname(&drv->funcname, k, v, REG_EXTENDED);
-	if (!strcmp(type, "binary"))
-		return parse_tristate(&drv->binary, k, v);
 	if (!strcmp(type, "command"))
 		return git_config_string(&drv->external, k, v);
 	if (!strcmp(type, "textconv"))
 		return git_config_string(&drv->textconv, k, v);
-	if (!strcmp(type, "cachetextconv"))
-		return parse_bool(&drv->textconv_want_cache, k, v);
 	if (!strcmp(type, "wordregex"))
 		return git_config_string(&drv->word_regex, k, v);
+	/* Don't care about the parse errors for these, fallthrough */
+	if (!strcmp(type, "cachetextconv"))
+		drv->textconv_want_cache = git_config_bool(k, v);
+	if (!strcmp(type, "binary"))
+		parse_tristate(&drv->binary, k, v);
 
 	return 0;
 }
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 02/20] userdiff: re-order builtin drivers in alphabetical order

[Ævar Arnfjörð Bjarmason]

Address some old code smell and move around the built-in userdiff
definitions so they're both in alphabetical order, and now in the same
order they appear in the gitattributes(5) documentation.

The two started drifting in be58e70dba (diff: unify external diff and
funcname parsing code, 2008-10-05), and then even further in
80c49c3de2 (color-words: make regex configurable via attributes,
2009-01-17) when the "cpp" pattern was added.

There are no functional changes here, and as --color-moved will show
only moved existing lines.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 76 +++++++++++++++++++++++++++---------------------------
 1 file changed, 38 insertions(+), 38 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c147bcbb17..c92cbcc054 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -44,6 +44,44 @@ PATTERNS("bash",
 	 /* -- */
 	 /* Characters not in the default $IFS value */
 	 "[^ \t]+"),
+PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
+	 "[={}\"]|[^={}\" \t]+"),
+PATTERNS("cpp",
+	 /* Jump targets or access declarations */
+	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
+	 /* functions/methods, variables, and compounds at top level */
+	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
+PATTERNS("csharp",
+	 /* Keywords */
+	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
+	 /* Methods and constructors */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
+	 /* Properties */
+	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
+	 /* Type definitions */
+	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
+	 /* Namespace */
+	 "^[ \t]*(namespace[ \t]+.*)$",
+	 /* -- */
+	 "[a-zA-Z_][a-zA-Z0-9_]*"
+	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
+	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
+IPATTERN("css",
+	 "![:;][[:space:]]*$\n"
+	 "^[:[@.#]?[_a-z0-9].*$",
+	 /* -- */
+	 /*
+	  * This regex comes from W3C CSS specs. Should theoretically also
+	  * allow ISO 10646 characters U+00A0 and higher,
+	  * but they are not handled in this regex.
+	  */
+	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
+	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
+),
 PATTERNS("dts",
 	 "!;\n"
 	 "!=\n"
@@ -191,46 +229,8 @@ PATTERNS("rust",
 	 "[a-zA-Z_][a-zA-Z0-9_]*"
 	 "|[0-9][0-9_a-fA-Fiosuxz]*(\\.([0-9]*[eE][+-]?)?[0-9_fF]*)?"
 	 "|[-+*\\/<>%&^|=!:]=|<<=?|>>=?|&&|\\|\\||->|=>|\\.{2}=|\\.{3}|::"),
-PATTERNS("bibtex", "(@[a-zA-Z]{1,}[ \t]*\\{{0,1}[ \t]*[^ \t\"@',\\#}{~%]*).*$",
-	 "[={}\"]|[^={}\" \t]+"),
 PATTERNS("tex", "^(\\\\((sub)*section|chapter|part)\\*{0,1}\\{.*)$",
 	 "\\\\[a-zA-Z@]+|\\\\.|[a-zA-Z0-9\x80-\xff]+"),
-PATTERNS("cpp",
-	 /* Jump targets or access declarations */
-	 "!^[ \t]*[A-Za-z_][A-Za-z_0-9]*:[[:space:]]*($|/[/*])\n"
-	 /* functions/methods, variables, and compounds at top level */
-	 "^((::[[:space:]]*)?[A-Za-z_].*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lLuU]*"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->\\*?|\\.\\*"),
-PATTERNS("csharp",
-	 /* Keywords */
-	 "!^[ \t]*(do|while|for|if|else|instanceof|new|return|switch|case|throw|catch|using)\n"
-	 /* Methods and constructors */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe|async)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[<>@._[:alnum:]]+[ \t]*\\(.*\\))[ \t]*$\n"
-	 /* Properties */
-	 "^[ \t]*(((static|public|internal|private|protected|new|virtual|sealed|override|unsafe)[ \t]+)*[][<>@.~_[:alnum:]]+[ \t]+[@._[:alnum:]]+)[ \t]*$\n"
-	 /* Type definitions */
-	 "^[ \t]*(((static|public|internal|private|protected|new|unsafe|sealed|abstract|partial)[ \t]+)*(class|enum|interface|struct)[ \t]+.*)$\n"
-	 /* Namespace */
-	 "^[ \t]*(namespace[ \t]+.*)$",
-	 /* -- */
-	 "[a-zA-Z_][a-zA-Z0-9_]*"
-	 "|[-+0-9.e]+[fFlL]?|0[xXbB]?[0-9a-fA-F]+[lL]?"
-	 "|[-+*/<>%&^|=!]=|--|\\+\\+|<<=?|>>=?|&&|\\|\\||::|->"),
-IPATTERN("css",
-	 "![:;][[:space:]]*$\n"
-	 "^[:[@.#]?[_a-z0-9].*$",
-	 /* -- */
-	 /*
-	  * This regex comes from W3C CSS specs. Should theoretically also
-	  * allow ISO 10646 characters U+00A0 and higher,
-	  * but they are not handled in this regex.
-	  */
-	 "-?[_a-zA-Z][-_a-zA-Z0-9]*" /* identifiers */
-	 "|-?[0-9]+|\\#[0-9a-fA-F]+" /* numbers */
-),
 { "default", NULL, -1, { NULL, 0 } },
 };
 #undef PATTERNS
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 03/20] userdiff: add and use for_each_userdiff_driver()

[Ævar Arnfjörð Bjarmason]

Refactor the userdiff_find_by_namelen() function so that a new
for_each_userdiff_driver() API function does most of the work.

This will be useful for the same reason we've got other for_each_*()
API functions as part of various APIs, and will be used in a follow-up
commit.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 userdiff.c | 61 +++++++++++++++++++++++++++++++++++++++++++-----------
 userdiff.h | 15 ++++++++++++++
 2 files changed, 64 insertions(+), 12 deletions(-)

diff --git a/userdiff.c b/userdiff.c
index c92cbcc054..92b5a97e12 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -250,20 +250,32 @@ static struct userdiff_driver driver_false = {
 	{ NULL, 0 }
 };
 
-static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+struct for_each_userdiff_driver_cb {
+	const char *k;
+	size_t len;
+	struct userdiff_driver *driver;
+};
+
+static int userdiff_find_by_namelen_cb(struct userdiff_driver *driver,
+				       enum userdiff_driver_type type, void *priv)
 {
-	int i;
-	for (i = 0; i < ndrivers; i++) {
-		struct userdiff_driver *drv = drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
-	}
-	for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
-		struct userdiff_driver *drv = builtin_drivers + i;
-		if (!strncmp(drv->name, k, len) && !drv->name[len])
-			return drv;
+	struct for_each_userdiff_driver_cb *cb_data = priv;
+
+	if (!strncmp(driver->name, cb_data->k, cb_data->len) &&
+	    !driver->name[cb_data->len]) {
+		cb_data->driver = driver;
+		return -1; /* found it! */
 	}
-	return NULL;
+	return 0;
+}
+
+static struct userdiff_driver *userdiff_find_by_namelen(const char *k, size_t len)
+{
+	struct for_each_userdiff_driver_cb udcbdata = { .k = k, .len = len, .driver = NULL };
+
+	for_each_userdiff_driver(userdiff_find_by_namelen_cb,
+				 USERDIFF_DRIVER_TYPE_UNSPECIFIED, &udcbdata);
+	return udcbdata.driver;
 }
 
 static int parse_funcname(struct userdiff_funcname *f, const char *k,
@@ -364,3 +376,28 @@ struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 
 	return driver;
 }
+
+int for_each_userdiff_driver(each_userdiff_driver_fn fn,
+			     enum userdiff_driver_type type, void *cb_data)
+{
+	int i, ret;
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_CUSTOM)) {
+
+		for (i = 0; i < ndrivers; i++) {
+			struct userdiff_driver *drv = drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_CUSTOM, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	if (type & (USERDIFF_DRIVER_TYPE_UNSPECIFIED | USERDIFF_DRIVER_TYPE_BUILTIN)) {
+
+		for (i = 0; i < ARRAY_SIZE(builtin_drivers); i++) {
+			struct userdiff_driver *drv = builtin_drivers + i;
+			ret = fn(drv, USERDIFF_DRIVER_TYPE_BUILTIN, cb_data);
+			if (ret)
+				return ret;
+		}
+	}
+	return 0;
+}
diff --git a/userdiff.h b/userdiff.h
index 203057e13e..fe14014a77 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -21,6 +21,13 @@ struct userdiff_driver {
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
 };
+enum userdiff_driver_type {
+	USERDIFF_DRIVER_TYPE_UNSPECIFIED = 1<<0,
+	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<1,
+	USERDIFF_DRIVER_TYPE_CUSTOM = 1<<2,
+};
+typedef int (*each_userdiff_driver_fn)(struct userdiff_driver *,
+				       enum userdiff_driver_type, void *);
 
 int userdiff_config(const char *k, const char *v);
 struct userdiff_driver *userdiff_find_by_name(const char *name);
@@ -34,4 +41,12 @@ struct userdiff_driver *userdiff_find_by_path(struct index_state *istate,
 struct userdiff_driver *userdiff_get_textconv(struct repository *r,
 					      struct userdiff_driver *driver);
 
+/*
+ * Iterate over each driver of type userdiff_driver_type, or
+ * USERDIFF_DRIVER_TYPE_UNSPECIFIED for all of them. Return non-zero
+ * to exit from the loop.
+ */
+int for_each_userdiff_driver(each_userdiff_driver_fn,
+			     enum userdiff_driver_type, void *);
+
 #endif /* USERDIFF */
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 04/20] userdiff tests: explicitly test "default" pattern

[Ævar Arnfjörð Bjarmason]

Since 122aa6f9c0 (diff: introduce diff.<driver>.binary, 2008-10-05)
the internals of the userdiff.c code have understood a "default" name,
which is invoked as userdiff_find_by_name("default") and present in
the "builtin_drivers" struct. Let's test for this special case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 9675bc17db..cefe329aea 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -49,6 +49,7 @@ diffpatterns="
 	ruby
 	rust
 	tex
+	default
 	custom1
 	custom2
 	custom3
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 05/20] userdiff tests: list builtin drivers via test-tool

[Ævar Arnfjörð Bjarmason]

Change the userdiff test to list the builtin drivers via the
test-tool, using the new for_each_userdiff_driver() API function.

This gets rid of the need to modify this part of the test every time a
new pattern is added, see 2ff6c34612 (userdiff: support Bash,
2020-10-22) and 09dad9256a (userdiff: support Markdown, 2020-05-02)
for two recent examples.

I only need the "list-builtin-drivers "argument here, but let's add
"list-custom-drivers" and "list-drivers" too, just because it's easy.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile                 |  1 +
 t/helper/test-tool.c     |  1 +
 t/helper/test-tool.h     |  1 +
 t/helper/test-userdiff.c | 32 ++++++++++++++++++++++++++++++++
 t/t4018-diff-funcname.sh | 29 +++++------------------------
 5 files changed, 40 insertions(+), 24 deletions(-)
 create mode 100644 t/helper/test-userdiff.c

diff --git a/Makefile b/Makefile
index 5a239cac20..710a0deaed 100644
--- a/Makefile
+++ b/Makefile
@@ -741,6 +741,7 @@ TEST_BUILTINS_OBJS += test-submodule-nested-repo-config.o
 TEST_BUILTINS_OBJS += test-subprocess.o
 TEST_BUILTINS_OBJS += test-trace2.o
 TEST_BUILTINS_OBJS += test-urlmatch-normalization.o
+TEST_BUILTINS_OBJS += test-userdiff.o
 TEST_BUILTINS_OBJS += test-wildmatch.o
 TEST_BUILTINS_OBJS += test-windows-named-pipe.o
 TEST_BUILTINS_OBJS += test-write-cache.o
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index f97cd9f48a..dcb05ca6e5 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -71,6 +71,7 @@ static struct test_cmd cmds[] = {
 	{ "submodule-nested-repo-config", cmd__submodule_nested_repo_config },
 	{ "subprocess", cmd__subprocess },
 	{ "trace2", cmd__trace2 },
+	{ "userdiff", cmd__userdiff },
 	{ "urlmatch-normalization", cmd__urlmatch_normalization },
 	{ "xml-encode", cmd__xml_encode },
 	{ "wildmatch", cmd__wildmatch },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index 28072c0ad5..589f2e8ac6 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -61,6 +61,7 @@ int cmd__submodule_config(int argc, const char **argv);
 int cmd__submodule_nested_repo_config(int argc, const char **argv);
 int cmd__subprocess(int argc, const char **argv);
 int cmd__trace2(int argc, const char **argv);
+int cmd__userdiff(int argc, const char **argv);
 int cmd__urlmatch_normalization(int argc, const char **argv);
 int cmd__xml_encode(int argc, const char **argv);
 int cmd__wildmatch(int argc, const char **argv);
diff --git a/t/helper/test-userdiff.c b/t/helper/test-userdiff.c
new file mode 100644
index 0000000000..80de456287
--- /dev/null
+++ b/t/helper/test-userdiff.c
@@ -0,0 +1,32 @@
+#include "test-tool.h"
+#include "cache.h"
+#include "userdiff.h"
+
+static int userdiff_list_builtin_drivers_cb(struct userdiff_driver *driver,
+					    enum userdiff_driver_type type,
+					    void *priv)
+{
+	puts(driver->name);
+	return 0;
+}
+
+static int list_what(enum userdiff_driver_type type)
+{
+	return for_each_userdiff_driver(userdiff_list_builtin_drivers_cb, type,
+					NULL);
+}
+
+int cmd__userdiff(int argc, const char **argv)
+{
+	if (argc != 2)
+		return 1;
+
+	if (!strcmp(argv[1], "list-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_UNSPECIFIED);
+	else if (!strcmp(argv[1], "list-builtin-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_BUILTIN);
+	else if (!strcmp(argv[1], "list-custom-drivers"))
+		return list_what(USERDIFF_DRIVER_TYPE_CUSTOM);
+	else
+		return error("unknown argument %s", argv[1]);
+}
diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index cefe329aea..11ac648451 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -8,6 +8,10 @@ test_description='Test custom diff function name patterns'
 . ./test-lib.sh
 
 test_expect_success 'setup' '
+	test-tool userdiff list-builtin-drivers >builtin-drivers &&
+	test_file_not_empty builtin-drivers &&
+	builtin_drivers=$(cat builtin-drivers) &&
+
 	# a non-trivial custom pattern
 	git config diff.custom1.funcname "!static
 !String
@@ -26,30 +30,7 @@ test_expect_success 'setup' '
 '
 
 diffpatterns="
-	ada
-	bash
-	bibtex
-	cpp
-	csharp
-	css
-	dts
-	elixir
-	fortran
-	fountain
-	golang
-	html
-	java
-	markdown
-	matlab
-	objc
-	pascal
-	perl
-	php
-	python
-	ruby
-	rust
-	tex
-	default
+	$builtin_drivers
 	custom1
 	custom2
 	custom3
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH 05/20] userdiff tests: list builtin drivers via test-tool

[Eric Sunshine]

On Sun, Feb 14, 2021 at 7:56 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> Change the userdiff test to list the builtin drivers via the
> test-tool, using the new for_each_userdiff_driver() API function.
> [...]
> I only need the "list-builtin-drivers "argument here, but let's add
> "list-custom-drivers" and "list-drivers" too, just because it's easy.
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
> diff --git a/t/helper/test-userdiff.c b/t/helper/test-userdiff.c
> @@ -0,0 +1,32 @@
> +static int userdiff_list_builtin_drivers_cb(struct userdiff_driver *driver,
> +                                           enum userdiff_driver_type type,
> +                                           void *priv)
> +{
> +       puts(driver->name);
> +       return 0;
> +}

Nit: The word "builtin" in the name of this callback function made me
search the patch for its cousin function which would be used for
custom drivers. It was only after reading on that I discovered that
this one function is used for both builtin and custom drivers. (Caught
me off guard, but not itself worth a re-roll.)

> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> @@ -8,6 +8,10 @@ test_description='Test custom diff function name patterns'
>  test_expect_success 'setup' '
> +       test-tool userdiff list-builtin-drivers >builtin-drivers &&
> +       test_file_not_empty builtin-drivers &&
> +       builtin_drivers=$(cat builtin-drivers) &&

Nit: I get why you are using test_file_not_empty() for its diagnostic
value, but it does confuse the reader a bit to see the file
"builtin-drivers" created for no particularly good reason. This could
easily have been:

    builtin_drivers=$(test-tool userdiff list-builtin-drivers) &&
    test -n "$builtin_drivers" &&

Subjective and not worth a re-roll.

[PATCH 06/20] userdiff: remove support for "broken" tests

[Ævar Arnfjörð Bjarmason]

There have been no "broken" tests since 75c3b6b2e8 (userdiff: improve
Fortran xfuncname regex, 2020-08-12). Let's remove the test support
for them, this is in preparation for a more general refactoring of the
tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 8 +-------
 t/t4018/README           | 3 ---
 2 files changed, 1 insertion(+), 10 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 11ac648451..5fb5b0a651 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -83,13 +83,7 @@ test_expect_success 'setup hunk header tests' '
 # check each individual file
 for i in $(git ls-files)
 do
-	if grep broken "$i" >/dev/null 2>&1
-	then
-		result=failure
-	else
-		result=success
-	fi
-	test_expect_$result "hunk header: $i" "
+	test_expect_success "hunk header: $i" "
 		git diff -U1 $i >actual &&
 		grep '@@ .* @@.*RIGHT' actual
 	"
diff --git a/t/t4018/README b/t/t4018/README
index 283e01cca1..2d25b2b4fc 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -7,9 +7,6 @@ at least two lines from the line that must appear in the hunk header.
 The text that must appear in the hunk header must contain the word
 "right", but in all upper-case, like in the title above.
 
-To mark a test case that highlights a malfunction, insert the word
-BROKEN in all lower-case somewhere in the file.
-
 This text is a bit twisted and out of order, but it is itself a
 test case for the default hunk header pattern. Know what you are doing
 if you change it.
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 07/20] userdiff tests: match full hunk headers

[Ævar Arnfjörð Bjarmason]

Fix a regression in the test framework for userdiff added in
bfa7d01413 (t4018: an infrastructure to test hunk headers,
2014-03-21).

The testing infrastructure added in that change went overboard with
simplifying the tests, to the point where we lost test coverage.

Before that we'd been able to test the full context line, or ever
since the feature was originally added in f258475a6e (Per-path
attribute based hunk header selection., 2007-07-06).

After bfa7d01413 all we cared about was whether "RIGHT" appeared on
the line. We thus lost the information about whether or not "RIGHT"
was extracted from the line for the hunk header, or the line appeared
in full (or other subset of the line).

Let's bring back coverage for that by adding corresponding *.ctx
files, this has the added advantage that we're doing a "test_cmp", so
when we have failures it's just a non-zero exit code from "grep",
we'll actually have something meaningful in the "-v" output.

As we'll see in a follow-up commit this is an intermediate step
towards even better test coverage. I'm structuring these tests in such
a way as to benefit from the diff.colorMove detection.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh                      |  7 +++---
 t/t4018/README                                | 22 +++++++++----------
 t/t4018/README.ctx                            |  1 +
 t/t4018/bash-arithmetic-function.ctx          |  1 +
 t/t4018/bash-bashism-style-compact.ctx        |  1 +
 t/t4018/bash-bashism-style-function.ctx       |  1 +
 t/t4018/bash-bashism-style-whitespace.ctx     |  1 +
 t/t4018/bash-conditional-function.ctx         |  1 +
 t/t4018/bash-missing-parentheses.ctx          |  1 +
 t/t4018/bash-mixed-style-compact.ctx          |  1 +
 t/t4018/bash-mixed-style-function.ctx         |  1 +
 t/t4018/bash-nested-functions.ctx             |  1 +
 t/t4018/bash-other-characters.ctx             |  1 +
 t/t4018/bash-posix-style-compact.ctx          |  1 +
 t/t4018/bash-posix-style-function.ctx         |  1 +
 t/t4018/bash-posix-style-whitespace.ctx       |  1 +
 t/t4018/bash-subshell-function.ctx            |  1 +
 t/t4018/bash-trailing-comment.ctx             |  1 +
 t/t4018/cpp-c++-function.ctx                  |  1 +
 t/t4018/cpp-class-constructor-mem-init.ctx    |  1 +
 t/t4018/cpp-class-constructor.ctx             |  1 +
 t/t4018/cpp-class-definition-derived.ctx      |  1 +
 t/t4018/cpp-class-definition.ctx              |  1 +
 t/t4018/cpp-class-destructor.ctx              |  1 +
 .../cpp-function-returning-global-type.ctx    |  1 +
 t/t4018/cpp-function-returning-nested.ctx     |  1 +
 t/t4018/cpp-function-returning-pointer.ctx    |  1 +
 t/t4018/cpp-function-returning-reference.ctx  |  1 +
 t/t4018/cpp-gnu-style-function.ctx            |  1 +
 t/t4018/cpp-namespace-definition.ctx          |  1 +
 t/t4018/cpp-operator-definition.ctx           |  1 +
 t/t4018/cpp-skip-access-specifiers.ctx        |  1 +
 t/t4018/cpp-skip-comment-block.ctx            |  1 +
 t/t4018/cpp-skip-labels.ctx                   |  1 +
 t/t4018/cpp-struct-definition.ctx             |  1 +
 t/t4018/cpp-struct-single-line.ctx            |  1 +
 t/t4018/cpp-template-function-definition.ctx  |  1 +
 t/t4018/cpp-union-definition.ctx              |  1 +
 t/t4018/cpp-void-c-function.ctx               |  1 +
 t/t4018/css-attribute-value-selector.ctx      |  1 +
 t/t4018/css-block-level-@-statements.ctx      |  1 +
 t/t4018/css-brace-in-col-1.ctx                |  1 +
 t/t4018/css-class-selector.ctx                |  1 +
 t/t4018/css-colon-eol.ctx                     |  1 +
 t/t4018/css-colon-selector.ctx                |  1 +
 t/t4018/css-common.ctx                        |  1 +
 t/t4018/css-id-selector.ctx                   |  1 +
 t/t4018/css-long-selector-list.ctx            |  1 +
 t/t4018/css-prop-sans-indent.ctx              |  1 +
 t/t4018/css-root-selector.ctx                 |  1 +
 t/t4018/css-short-selector-list.ctx           |  1 +
 t/t4018/css-trailing-space.ctx                |  1 +
 t/t4018/custom1-pattern.ctx                   |  1 +
 t/t4018/custom2-match-to-end-of-line.ctx      |  1 +
 t/t4018/custom3-alternation-in-pattern.ctx    |  1 +
 t/t4018/dts-labels.ctx                        |  1 +
 t/t4018/dts-node-unitless.ctx                 |  1 +
 t/t4018/dts-nodes-boolean-prop.ctx            |  1 +
 t/t4018/dts-nodes-comment1.ctx                |  1 +
 t/t4018/dts-nodes-comment2.ctx                |  1 +
 t/t4018/dts-nodes-multiline-prop.ctx          |  1 +
 t/t4018/dts-nodes.ctx                         |  1 +
 t/t4018/dts-reference.ctx                     |  1 +
 t/t4018/dts-root-comment.ctx                  |  1 +
 t/t4018/dts-root.ctx                          |  1 +
 t/t4018/elixir-do-not-pick-end.ctx            |  1 +
 t/t4018/elixir-ex-unit-test.ctx               |  1 +
 t/t4018/elixir-function.ctx                   |  1 +
 t/t4018/elixir-macro.ctx                      |  1 +
 t/t4018/elixir-module-func.ctx                |  1 +
 t/t4018/elixir-module.ctx                     |  1 +
 t/t4018/elixir-nested-module.ctx              |  1 +
 t/t4018/elixir-private-function.ctx           |  1 +
 t/t4018/elixir-protocol-implementation.ctx    |  1 +
 t/t4018/elixir-protocol.ctx                   |  1 +
 t/t4018/fortran-block-data.ctx                |  1 +
 t/t4018/fortran-comment-keyword.ctx           |  1 +
 t/t4018/fortran-comment-legacy-star.ctx       |  1 +
 t/t4018/fortran-comment-legacy.ctx            |  1 +
 t/t4018/fortran-comment.ctx                   |  1 +
 t/t4018/fortran-external-function.ctx         |  1 +
 t/t4018/fortran-external-subroutine.ctx       |  1 +
 t/t4018/fortran-module-procedure.ctx          |  1 +
 t/t4018/fortran-module.ctx                    |  1 +
 t/t4018/fortran-program.ctx                   |  1 +
 t/t4018/fountain-scene.ctx                    |  1 +
 t/t4018/golang-complex-function.ctx           |  1 +
 t/t4018/golang-func.ctx                       |  1 +
 t/t4018/golang-interface.ctx                  |  1 +
 t/t4018/golang-long-func.ctx                  |  1 +
 t/t4018/golang-struct.ctx                     |  1 +
 t/t4018/java-class-member-function.ctx        |  1 +
 t/t4018/markdown-heading-indented.ctx         |  1 +
 t/t4018/markdown-heading-non-headings.ctx     |  1 +
 t/t4018/matlab-class-definition.ctx           |  1 +
 t/t4018/matlab-function.ctx                   |  1 +
 t/t4018/matlab-octave-section-1.ctx           |  1 +
 t/t4018/matlab-octave-section-2.ctx           |  1 +
 t/t4018/matlab-section.ctx                    |  1 +
 t/t4018/perl-skip-end-of-heredoc.ctx          |  1 +
 t/t4018/perl-skip-forward-decl.ctx            |  1 +
 t/t4018/perl-skip-sub-in-pod.ctx              |  1 +
 t/t4018/perl-sub-definition-kr-brace.ctx      |  1 +
 t/t4018/perl-sub-definition.ctx               |  1 +
 t/t4018/php-abstract-class.ctx                |  1 +
 t/t4018/php-abstract-method.ctx               |  1 +
 t/t4018/php-class.ctx                         |  1 +
 t/t4018/php-final-class.ctx                   |  1 +
 t/t4018/php-final-method.ctx                  |  1 +
 t/t4018/php-function.ctx                      |  1 +
 t/t4018/php-interface.ctx                     |  1 +
 t/t4018/php-method.ctx                        |  1 +
 t/t4018/php-trait.ctx                         |  1 +
 t/t4018/python-async-def.ctx                  |  1 +
 t/t4018/python-class.ctx                      |  1 +
 t/t4018/python-def.ctx                        |  1 +
 t/t4018/python-indented-async-def.ctx         |  1 +
 t/t4018/python-indented-class.ctx             |  1 +
 t/t4018/python-indented-def.ctx               |  1 +
 t/t4018/rust-fn.ctx                           |  1 +
 t/t4018/rust-impl.ctx                         |  1 +
 t/t4018/rust-macro-rules.ctx                  |  1 +
 t/t4018/rust-struct.ctx                       |  1 +
 t/t4018/rust-trait.ctx                        |  1 +
 124 files changed, 137 insertions(+), 14 deletions(-)
 create mode 100644 t/t4018/README.ctx
 create mode 100644 t/t4018/bash-arithmetic-function.ctx
 create mode 100644 t/t4018/bash-bashism-style-compact.ctx
 create mode 100644 t/t4018/bash-bashism-style-function.ctx
 create mode 100644 t/t4018/bash-bashism-style-whitespace.ctx
 create mode 100644 t/t4018/bash-conditional-function.ctx
 create mode 100644 t/t4018/bash-missing-parentheses.ctx
 create mode 100644 t/t4018/bash-mixed-style-compact.ctx
 create mode 100644 t/t4018/bash-mixed-style-function.ctx
 create mode 100644 t/t4018/bash-nested-functions.ctx
 create mode 100644 t/t4018/bash-other-characters.ctx
 create mode 100644 t/t4018/bash-posix-style-compact.ctx
 create mode 100644 t/t4018/bash-posix-style-function.ctx
 create mode 100644 t/t4018/bash-posix-style-whitespace.ctx
 create mode 100644 t/t4018/bash-subshell-function.ctx
 create mode 100644 t/t4018/bash-trailing-comment.ctx
 create mode 100644 t/t4018/cpp-c++-function.ctx
 create mode 100644 t/t4018/cpp-class-constructor-mem-init.ctx
 create mode 100644 t/t4018/cpp-class-constructor.ctx
 create mode 100644 t/t4018/cpp-class-definition-derived.ctx
 create mode 100644 t/t4018/cpp-class-definition.ctx
 create mode 100644 t/t4018/cpp-class-destructor.ctx
 create mode 100644 t/t4018/cpp-function-returning-global-type.ctx
 create mode 100644 t/t4018/cpp-function-returning-nested.ctx
 create mode 100644 t/t4018/cpp-function-returning-pointer.ctx
 create mode 100644 t/t4018/cpp-function-returning-reference.ctx
 create mode 100644 t/t4018/cpp-gnu-style-function.ctx
 create mode 100644 t/t4018/cpp-namespace-definition.ctx
 create mode 100644 t/t4018/cpp-operator-definition.ctx
 create mode 100644 t/t4018/cpp-skip-access-specifiers.ctx
 create mode 100644 t/t4018/cpp-skip-comment-block.ctx
 create mode 100644 t/t4018/cpp-skip-labels.ctx
 create mode 100644 t/t4018/cpp-struct-definition.ctx
 create mode 100644 t/t4018/cpp-struct-single-line.ctx
 create mode 100644 t/t4018/cpp-template-function-definition.ctx
 create mode 100644 t/t4018/cpp-union-definition.ctx
 create mode 100644 t/t4018/cpp-void-c-function.ctx
 create mode 100644 t/t4018/css-attribute-value-selector.ctx
 create mode 100644 t/t4018/css-block-level-@-statements.ctx
 create mode 100644 t/t4018/css-brace-in-col-1.ctx
 create mode 100644 t/t4018/css-class-selector.ctx
 create mode 100644 t/t4018/css-colon-eol.ctx
 create mode 100644 t/t4018/css-colon-selector.ctx
 create mode 100644 t/t4018/css-common.ctx
 create mode 100644 t/t4018/css-id-selector.ctx
 create mode 100644 t/t4018/css-long-selector-list.ctx
 create mode 100644 t/t4018/css-prop-sans-indent.ctx
 create mode 100644 t/t4018/css-root-selector.ctx
 create mode 100644 t/t4018/css-short-selector-list.ctx
 create mode 100644 t/t4018/css-trailing-space.ctx
 create mode 100644 t/t4018/custom1-pattern.ctx
 create mode 100644 t/t4018/custom2-match-to-end-of-line.ctx
 create mode 100644 t/t4018/custom3-alternation-in-pattern.ctx
 create mode 100644 t/t4018/dts-labels.ctx
 create mode 100644 t/t4018/dts-node-unitless.ctx
 create mode 100644 t/t4018/dts-nodes-boolean-prop.ctx
 create mode 100644 t/t4018/dts-nodes-comment1.ctx
 create mode 100644 t/t4018/dts-nodes-comment2.ctx
 create mode 100644 t/t4018/dts-nodes-multiline-prop.ctx
 create mode 100644 t/t4018/dts-nodes.ctx
 create mode 100644 t/t4018/dts-reference.ctx
 create mode 100644 t/t4018/dts-root-comment.ctx
 create mode 100644 t/t4018/dts-root.ctx
 create mode 100644 t/t4018/elixir-do-not-pick-end.ctx
 create mode 100644 t/t4018/elixir-ex-unit-test.ctx
 create mode 100644 t/t4018/elixir-function.ctx
 create mode 100644 t/t4018/elixir-macro.ctx
 create mode 100644 t/t4018/elixir-module-func.ctx
 create mode 100644 t/t4018/elixir-module.ctx
 create mode 100644 t/t4018/elixir-nested-module.ctx
 create mode 100644 t/t4018/elixir-private-function.ctx
 create mode 100644 t/t4018/elixir-protocol-implementation.ctx
 create mode 100644 t/t4018/elixir-protocol.ctx
 create mode 100644 t/t4018/fortran-block-data.ctx
 create mode 100644 t/t4018/fortran-comment-keyword.ctx
 create mode 100644 t/t4018/fortran-comment-legacy-star.ctx
 create mode 100644 t/t4018/fortran-comment-legacy.ctx
 create mode 100644 t/t4018/fortran-comment.ctx
 create mode 100644 t/t4018/fortran-external-function.ctx
 create mode 100644 t/t4018/fortran-external-subroutine.ctx
 create mode 100644 t/t4018/fortran-module-procedure.ctx
 create mode 100644 t/t4018/fortran-module.ctx
 create mode 100644 t/t4018/fortran-program.ctx
 create mode 100644 t/t4018/fountain-scene.ctx
 create mode 100644 t/t4018/golang-complex-function.ctx
 create mode 100644 t/t4018/golang-func.ctx
 create mode 100644 t/t4018/golang-interface.ctx
 create mode 100644 t/t4018/golang-long-func.ctx
 create mode 100644 t/t4018/golang-struct.ctx
 create mode 100644 t/t4018/java-class-member-function.ctx
 create mode 100644 t/t4018/markdown-heading-indented.ctx
 create mode 100644 t/t4018/markdown-heading-non-headings.ctx
 create mode 100644 t/t4018/matlab-class-definition.ctx
 create mode 100644 t/t4018/matlab-function.ctx
 create mode 100644 t/t4018/matlab-octave-section-1.ctx
 create mode 100644 t/t4018/matlab-octave-section-2.ctx
 create mode 100644 t/t4018/matlab-section.ctx
 create mode 100644 t/t4018/perl-skip-end-of-heredoc.ctx
 create mode 100644 t/t4018/perl-skip-forward-decl.ctx
 create mode 100644 t/t4018/perl-skip-sub-in-pod.ctx
 create mode 100644 t/t4018/perl-sub-definition-kr-brace.ctx
 create mode 100644 t/t4018/perl-sub-definition.ctx
 create mode 100644 t/t4018/php-abstract-class.ctx
 create mode 100644 t/t4018/php-abstract-method.ctx
 create mode 100644 t/t4018/php-class.ctx
 create mode 100644 t/t4018/php-final-class.ctx
 create mode 100644 t/t4018/php-final-method.ctx
 create mode 100644 t/t4018/php-function.ctx
 create mode 100644 t/t4018/php-interface.ctx
 create mode 100644 t/t4018/php-method.ctx
 create mode 100644 t/t4018/php-trait.ctx
 create mode 100644 t/t4018/python-async-def.ctx
 create mode 100644 t/t4018/python-class.ctx
 create mode 100644 t/t4018/python-def.ctx
 create mode 100644 t/t4018/python-indented-async-def.ctx
 create mode 100644 t/t4018/python-indented-class.ctx
 create mode 100644 t/t4018/python-indented-def.ctx
 create mode 100644 t/t4018/rust-fn.ctx
 create mode 100644 t/t4018/rust-impl.ctx
 create mode 100644 t/t4018/rust-macro-rules.ctx
 create mode 100644 t/t4018/rust-struct.ctx
 create mode 100644 t/t4018/rust-trait.ctx

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 5fb5b0a651..d32c38ad1a 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -81,11 +81,12 @@ test_expect_success 'setup hunk header tests' '
 '
 
 # check each individual file
-for i in $(git ls-files)
+for i in $(git ls-files -- ':!*.ctx')
 do
 	test_expect_success "hunk header: $i" "
-		git diff -U1 $i >actual &&
-		grep '@@ .* @@.*RIGHT' actual
+		git diff -U1 $i >diff &&
+		sed -n -e 's/^.*@@\( \|$\)//p' <diff >ctx &&
+		test_cmp $i.ctx ctx
 	"
 done
 
diff --git a/t/t4018/README b/t/t4018/README
index 2d25b2b4fc..2cccf8c950 100644
--- a/t/t4018/README
+++ b/t/t4018/README
@@ -1,15 +1,15 @@
-How to write RIGHT test cases
-=============================
+How to write test cases
+=======================
+
+Create test cases called "LANG-whatever" in this directory, where
+"LANG" is e.g. the userdiff driver name, where "whatever" is a brief
+description of the test.
 
 Insert the word "ChangeMe" (exactly this form) at a distance of
 at least two lines from the line that must appear in the hunk header.
 
-The text that must appear in the hunk header must contain the word
-"right", but in all upper-case, like in the title above.
-
-This text is a bit twisted and out of order, but it is itself a
-test case for the default hunk header pattern. Know what you are doing
-if you change it.
-
-BTW, this tests that the head line goes to the hunk header, not the line
-of equal signs.
+The text that must appear in the hunk header must contains the word
+"RIGHT" by convention. The "LANG-whatever.ctx" file contains what we
+expect to appear in the hunk header. We munged the start of the line
+to "@@ [...] @@" for ease of not having to hardcode the line numbers
+and offsets.
diff --git a/t/t4018/README.ctx b/t/t4018/README.ctx
new file mode 100644
index 0000000000..cd79384b04
--- /dev/null
+++ b/t/t4018/README.ctx
@@ -0,0 +1 @@
+description of the test.
diff --git a/t/t4018/bash-arithmetic-function.ctx b/t/t4018/bash-arithmetic-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-arithmetic-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-bashism-style-compact.ctx b/t/t4018/bash-bashism-style-compact.ctx
new file mode 100644
index 0000000000..4f8eac48c6
--- /dev/null
+++ b/t/t4018/bash-bashism-style-compact.ctx
@@ -0,0 +1 @@
+function RIGHT {
diff --git a/t/t4018/bash-bashism-style-function.ctx b/t/t4018/bash-bashism-style-function.ctx
new file mode 100644
index 0000000000..4f8eac48c6
--- /dev/null
+++ b/t/t4018/bash-bashism-style-function.ctx
@@ -0,0 +1 @@
+function RIGHT {
diff --git a/t/t4018/bash-bashism-style-whitespace.ctx b/t/t4018/bash-bashism-style-whitespace.ctx
new file mode 100644
index 0000000000..35dbd0220e
--- /dev/null
+++ b/t/t4018/bash-bashism-style-whitespace.ctx
@@ -0,0 +1 @@
+function 	RIGHT 	( 	) 	{
diff --git a/t/t4018/bash-conditional-function.ctx b/t/t4018/bash-conditional-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-conditional-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-missing-parentheses.ctx b/t/t4018/bash-missing-parentheses.ctx
new file mode 100644
index 0000000000..4f8eac48c6
--- /dev/null
+++ b/t/t4018/bash-missing-parentheses.ctx
@@ -0,0 +1 @@
+function RIGHT {
diff --git a/t/t4018/bash-mixed-style-compact.ctx b/t/t4018/bash-mixed-style-compact.ctx
new file mode 100644
index 0000000000..bba11074eb
--- /dev/null
+++ b/t/t4018/bash-mixed-style-compact.ctx
@@ -0,0 +1 @@
+function RIGHT(){
diff --git a/t/t4018/bash-mixed-style-function.ctx b/t/t4018/bash-mixed-style-function.ctx
new file mode 100644
index 0000000000..922b87a4aa
--- /dev/null
+++ b/t/t4018/bash-mixed-style-function.ctx
@@ -0,0 +1 @@
+function RIGHT() {
diff --git a/t/t4018/bash-nested-functions.ctx b/t/t4018/bash-nested-functions.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-nested-functions.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-other-characters.ctx b/t/t4018/bash-other-characters.ctx
new file mode 100644
index 0000000000..6a55317fdf
--- /dev/null
+++ b/t/t4018/bash-other-characters.ctx
@@ -0,0 +1 @@
+_RIGHT_0n()
diff --git a/t/t4018/bash-posix-style-compact.ctx b/t/t4018/bash-posix-style-compact.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-posix-style-compact.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-posix-style-function.ctx b/t/t4018/bash-posix-style-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-posix-style-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-posix-style-whitespace.ctx b/t/t4018/bash-posix-style-whitespace.ctx
new file mode 100644
index 0000000000..28f8698e14
--- /dev/null
+++ b/t/t4018/bash-posix-style-whitespace.ctx
@@ -0,0 +1 @@
+RIGHT 	( 	)
diff --git a/t/t4018/bash-subshell-function.ctx b/t/t4018/bash-subshell-function.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-subshell-function.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/bash-trailing-comment.ctx b/t/t4018/bash-trailing-comment.ctx
new file mode 100644
index 0000000000..811eac7d2f
--- /dev/null
+++ b/t/t4018/bash-trailing-comment.ctx
@@ -0,0 +1 @@
+RIGHT()
diff --git a/t/t4018/cpp-c++-function.ctx b/t/t4018/cpp-c++-function.ctx
new file mode 100644
index 0000000000..337b49dbb3
--- /dev/null
+++ b/t/t4018/cpp-c++-function.ctx
@@ -0,0 +1 @@
+Item RIGHT::DoSomething( Args with_spaces )
diff --git a/t/t4018/cpp-class-constructor-mem-init.ctx b/t/t4018/cpp-class-constructor-mem-init.ctx
new file mode 100644
index 0000000000..6664b8b3d8
--- /dev/null
+++ b/t/t4018/cpp-class-constructor-mem-init.ctx
@@ -0,0 +1 @@
+Item::Item(int RIGHT) :
diff --git a/t/t4018/cpp-class-constructor.ctx b/t/t4018/cpp-class-constructor.ctx
new file mode 100644
index 0000000000..2dcadfc0ba
--- /dev/null
+++ b/t/t4018/cpp-class-constructor.ctx
@@ -0,0 +1 @@
+Item::Item(int RIGHT)
diff --git a/t/t4018/cpp-class-definition-derived.ctx b/t/t4018/cpp-class-definition-derived.ctx
new file mode 100644
index 0000000000..146f0a7b7c
--- /dev/null
+++ b/t/t4018/cpp-class-definition-derived.ctx
@@ -0,0 +1 @@
+class RIGHT :
diff --git a/t/t4018/cpp-class-definition.ctx b/t/t4018/cpp-class-definition.ctx
new file mode 100644
index 0000000000..54bff816d6
--- /dev/null
+++ b/t/t4018/cpp-class-definition.ctx
@@ -0,0 +1 @@
+class RIGHT
diff --git a/t/t4018/cpp-class-destructor.ctx b/t/t4018/cpp-class-destructor.ctx
new file mode 100644
index 0000000000..5390c17cf6
--- /dev/null
+++ b/t/t4018/cpp-class-destructor.ctx
@@ -0,0 +1 @@
+RIGHT::~RIGHT()
diff --git a/t/t4018/cpp-function-returning-global-type.ctx b/t/t4018/cpp-function-returning-global-type.ctx
new file mode 100644
index 0000000000..4dcdde25f4
--- /dev/null
+++ b/t/t4018/cpp-function-returning-global-type.ctx
@@ -0,0 +1 @@
+::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-nested.ctx b/t/t4018/cpp-function-returning-nested.ctx
new file mode 100644
index 0000000000..6ef73c8368
--- /dev/null
+++ b/t/t4018/cpp-function-returning-nested.ctx
@@ -0,0 +1 @@
+get::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-pointer.ctx b/t/t4018/cpp-function-returning-pointer.ctx
new file mode 100644
index 0000000000..bb0acce5c7
--- /dev/null
+++ b/t/t4018/cpp-function-returning-pointer.ctx
@@ -0,0 +1 @@
+const char *get_it_RIGHT(char *ptr)
diff --git a/t/t4018/cpp-function-returning-reference.ctx b/t/t4018/cpp-function-returning-reference.ctx
new file mode 100644
index 0000000000..76afe381fd
--- /dev/null
+++ b/t/t4018/cpp-function-returning-reference.ctx
@@ -0,0 +1 @@
+string& get::it::RIGHT(char *ptr)
diff --git a/t/t4018/cpp-gnu-style-function.ctx b/t/t4018/cpp-gnu-style-function.ctx
new file mode 100644
index 0000000000..1858287812
--- /dev/null
+++ b/t/t4018/cpp-gnu-style-function.ctx
@@ -0,0 +1 @@
+RIGHT(int arg)
diff --git a/t/t4018/cpp-namespace-definition.ctx b/t/t4018/cpp-namespace-definition.ctx
new file mode 100644
index 0000000000..14c29c4638
--- /dev/null
+++ b/t/t4018/cpp-namespace-definition.ctx
@@ -0,0 +1 @@
+namespace RIGHT
diff --git a/t/t4018/cpp-operator-definition.ctx b/t/t4018/cpp-operator-definition.ctx
new file mode 100644
index 0000000000..5b56778961
--- /dev/null
+++ b/t/t4018/cpp-operator-definition.ctx
@@ -0,0 +1 @@
+Value operator+(Value LEFT, Value RIGHT)
diff --git a/t/t4018/cpp-skip-access-specifiers.ctx b/t/t4018/cpp-skip-access-specifiers.ctx
new file mode 100644
index 0000000000..075bcd883b
--- /dev/null
+++ b/t/t4018/cpp-skip-access-specifiers.ctx
@@ -0,0 +1 @@
+class RIGHT : public Baseclass
diff --git a/t/t4018/cpp-skip-comment-block.ctx b/t/t4018/cpp-skip-comment-block.ctx
new file mode 100644
index 0000000000..656c59893d
--- /dev/null
+++ b/t/t4018/cpp-skip-comment-block.ctx
@@ -0,0 +1 @@
+struct item RIGHT(int i)
diff --git a/t/t4018/cpp-skip-labels.ctx b/t/t4018/cpp-skip-labels.ctx
new file mode 100644
index 0000000000..6b0635f7f7
--- /dev/null
+++ b/t/t4018/cpp-skip-labels.ctx
@@ -0,0 +1 @@
+void RIGHT (void)
diff --git a/t/t4018/cpp-struct-definition.ctx b/t/t4018/cpp-struct-definition.ctx
new file mode 100644
index 0000000000..48ed893279
--- /dev/null
+++ b/t/t4018/cpp-struct-definition.ctx
@@ -0,0 +1 @@
+struct RIGHT {
diff --git a/t/t4018/cpp-struct-single-line.ctx b/t/t4018/cpp-struct-single-line.ctx
new file mode 100644
index 0000000000..e3bc9d5017
--- /dev/null
+++ b/t/t4018/cpp-struct-single-line.ctx
@@ -0,0 +1 @@
+struct RIGHT_iterator_tag {};
diff --git a/t/t4018/cpp-template-function-definition.ctx b/t/t4018/cpp-template-function-definition.ctx
new file mode 100644
index 0000000000..c9da39cf65
--- /dev/null
+++ b/t/t4018/cpp-template-function-definition.ctx
@@ -0,0 +1 @@
+template<class T> int RIGHT(T arg)
diff --git a/t/t4018/cpp-union-definition.ctx b/t/t4018/cpp-union-definition.ctx
new file mode 100644
index 0000000000..2fc7b54fb8
--- /dev/null
+++ b/t/t4018/cpp-union-definition.ctx
@@ -0,0 +1 @@
+union RIGHT {
diff --git a/t/t4018/cpp-void-c-function.ctx b/t/t4018/cpp-void-c-function.ctx
new file mode 100644
index 0000000000..6b0635f7f7
--- /dev/null
+++ b/t/t4018/cpp-void-c-function.ctx
@@ -0,0 +1 @@
+void RIGHT (void)
diff --git a/t/t4018/css-attribute-value-selector.ctx b/t/t4018/css-attribute-value-selector.ctx
new file mode 100644
index 0000000000..7f8956251c
--- /dev/null
+++ b/t/t4018/css-attribute-value-selector.ctx
@@ -0,0 +1 @@
+[class*="RIGHT"] {
diff --git a/t/t4018/css-block-level-@-statements.ctx b/t/t4018/css-block-level-@-statements.ctx
new file mode 100644
index 0000000000..7f5e90468c
--- /dev/null
+++ b/t/t4018/css-block-level-@-statements.ctx
@@ -0,0 +1 @@
+@keyframes RIGHT {
diff --git a/t/t4018/css-brace-in-col-1.ctx b/t/t4018/css-brace-in-col-1.ctx
new file mode 100644
index 0000000000..91a9105c6a
--- /dev/null
+++ b/t/t4018/css-brace-in-col-1.ctx
@@ -0,0 +1 @@
+RIGHT label.control-label
diff --git a/t/t4018/css-class-selector.ctx b/t/t4018/css-class-selector.ctx
new file mode 100644
index 0000000000..ac7367d7f4
--- /dev/null
+++ b/t/t4018/css-class-selector.ctx
@@ -0,0 +1 @@
+.RIGHT {
diff --git a/t/t4018/css-colon-eol.ctx b/t/t4018/css-colon-eol.ctx
new file mode 100644
index 0000000000..b68493b9b0
--- /dev/null
+++ b/t/t4018/css-colon-eol.ctx
@@ -0,0 +1 @@
+RIGHT h1 {
diff --git a/t/t4018/css-colon-selector.ctx b/t/t4018/css-colon-selector.ctx
new file mode 100644
index 0000000000..00b1a5aefe
--- /dev/null
+++ b/t/t4018/css-colon-selector.ctx
@@ -0,0 +1 @@
+RIGHT a:hover {
diff --git a/t/t4018/css-common.ctx b/t/t4018/css-common.ctx
new file mode 100644
index 0000000000..43686b4081
--- /dev/null
+++ b/t/t4018/css-common.ctx
@@ -0,0 +1 @@
+RIGHT label.control-label {
diff --git a/t/t4018/css-id-selector.ctx b/t/t4018/css-id-selector.ctx
new file mode 100644
index 0000000000..ce19f6d8dc
--- /dev/null
+++ b/t/t4018/css-id-selector.ctx
@@ -0,0 +1 @@
+#RIGHT {
diff --git a/t/t4018/css-long-selector-list.ctx b/t/t4018/css-long-selector-list.ctx
new file mode 100644
index 0000000000..bc8d0fb62c
--- /dev/null
+++ b/t/t4018/css-long-selector-list.ctx
@@ -0,0 +1 @@
+div ul#RIGHT {
diff --git a/t/t4018/css-prop-sans-indent.ctx b/t/t4018/css-prop-sans-indent.ctx
new file mode 100644
index 0000000000..cc880b2f44
--- /dev/null
+++ b/t/t4018/css-prop-sans-indent.ctx
@@ -0,0 +1 @@
+RIGHT, label.control-label {
diff --git a/t/t4018/css-root-selector.ctx b/t/t4018/css-root-selector.ctx
new file mode 100644
index 0000000000..3010cded2a
--- /dev/null
+++ b/t/t4018/css-root-selector.ctx
@@ -0,0 +1 @@
+:RIGHT {
diff --git a/t/t4018/css-short-selector-list.ctx b/t/t4018/css-short-selector-list.ctx
new file mode 100644
index 0000000000..9e5d87d126
--- /dev/null
+++ b/t/t4018/css-short-selector-list.ctx
@@ -0,0 +1 @@
+label.control, div ul#RIGHT {
diff --git a/t/t4018/css-trailing-space.ctx b/t/t4018/css-trailing-space.ctx
new file mode 100644
index 0000000000..43686b4081
--- /dev/null
+++ b/t/t4018/css-trailing-space.ctx
@@ -0,0 +1 @@
+RIGHT label.control-label {
diff --git a/t/t4018/custom1-pattern.ctx b/t/t4018/custom1-pattern.ctx
new file mode 100644
index 0000000000..d1609cc9a6
--- /dev/null
+++ b/t/t4018/custom1-pattern.ctx
@@ -0,0 +1 @@
+int special, RIGHT;
diff --git a/t/t4018/custom2-match-to-end-of-line.ctx b/t/t4018/custom2-match-to-end-of-line.ctx
new file mode 100644
index 0000000000..8294c6e49b
--- /dev/null
+++ b/t/t4018/custom2-match-to-end-of-line.ctx
@@ -0,0 +1 @@
+RIGHT_Beer
diff --git a/t/t4018/custom3-alternation-in-pattern.ctx b/t/t4018/custom3-alternation-in-pattern.ctx
new file mode 100644
index 0000000000..2125474b68
--- /dev/null
+++ b/t/t4018/custom3-alternation-in-pattern.ctx
@@ -0,0 +1 @@
+public static void main(String RIGHT[])
diff --git a/t/t4018/dts-labels.ctx b/t/t4018/dts-labels.ctx
new file mode 100644
index 0000000000..48d9373cab
--- /dev/null
+++ b/t/t4018/dts-labels.ctx
@@ -0,0 +1 @@
+label2: RIGHT {
diff --git a/t/t4018/dts-node-unitless.ctx b/t/t4018/dts-node-unitless.ctx
new file mode 100644
index 0000000000..82c8683fa1
--- /dev/null
+++ b/t/t4018/dts-node-unitless.ctx
@@ -0,0 +1 @@
+RIGHT {
diff --git a/t/t4018/dts-nodes-boolean-prop.ctx b/t/t4018/dts-nodes-boolean-prop.ctx
new file mode 100644
index 0000000000..3a0232d55d
--- /dev/null
+++ b/t/t4018/dts-nodes-boolean-prop.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes-comment1.ctx b/t/t4018/dts-nodes-comment1.ctx
new file mode 100644
index 0000000000..ec364600b1
--- /dev/null
+++ b/t/t4018/dts-nodes-comment1.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 /* &a comment */ {
diff --git a/t/t4018/dts-nodes-comment2.ctx b/t/t4018/dts-nodes-comment2.ctx
new file mode 100644
index 0000000000..75f0d75258
--- /dev/null
+++ b/t/t4018/dts-nodes-comment2.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 { /* a trailing comment */
diff --git a/t/t4018/dts-nodes-multiline-prop.ctx b/t/t4018/dts-nodes-multiline-prop.ctx
new file mode 100644
index 0000000000..3a0232d55d
--- /dev/null
+++ b/t/t4018/dts-nodes-multiline-prop.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes.ctx b/t/t4018/dts-nodes.ctx
new file mode 100644
index 0000000000..3a0232d55d
--- /dev/null
+++ b/t/t4018/dts-nodes.ctx
@@ -0,0 +1 @@
+RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-reference.ctx b/t/t4018/dts-reference.ctx
new file mode 100644
index 0000000000..c1e13409ee
--- /dev/null
+++ b/t/t4018/dts-reference.ctx
@@ -0,0 +1 @@
+&RIGHT {
diff --git a/t/t4018/dts-root-comment.ctx b/t/t4018/dts-root-comment.ctx
new file mode 100644
index 0000000000..656053dd42
--- /dev/null
+++ b/t/t4018/dts-root-comment.ctx
@@ -0,0 +1 @@
+/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/dts-root.ctx b/t/t4018/dts-root.ctx
new file mode 100644
index 0000000000..656053dd42
--- /dev/null
+++ b/t/t4018/dts-root.ctx
@@ -0,0 +1 @@
+/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/elixir-do-not-pick-end.ctx b/t/t4018/elixir-do-not-pick-end.ctx
new file mode 100644
index 0000000000..8f28a7a689
--- /dev/null
+++ b/t/t4018/elixir-do-not-pick-end.ctx
@@ -0,0 +1 @@
+defmodule RIGHT do
diff --git a/t/t4018/elixir-ex-unit-test.ctx b/t/t4018/elixir-ex-unit-test.ctx
new file mode 100644
index 0000000000..a55e3de2cc
--- /dev/null
+++ b/t/t4018/elixir-ex-unit-test.ctx
@@ -0,0 +1 @@
+test "RIGHT" do
diff --git a/t/t4018/elixir-function.ctx b/t/t4018/elixir-function.ctx
new file mode 100644
index 0000000000..62aee9c8b1
--- /dev/null
+++ b/t/t4018/elixir-function.ctx
@@ -0,0 +1 @@
+def function(RIGHT, arg) do
diff --git a/t/t4018/elixir-macro.ctx b/t/t4018/elixir-macro.ctx
new file mode 100644
index 0000000000..fc1d3b85e8
--- /dev/null
+++ b/t/t4018/elixir-macro.ctx
@@ -0,0 +1 @@
+defmacro foo(RIGHT) do
diff --git a/t/t4018/elixir-module-func.ctx b/t/t4018/elixir-module-func.ctx
new file mode 100644
index 0000000000..8239214386
--- /dev/null
+++ b/t/t4018/elixir-module-func.ctx
@@ -0,0 +1 @@
+def fun(RIGHT) do
diff --git a/t/t4018/elixir-module.ctx b/t/t4018/elixir-module.ctx
new file mode 100644
index 0000000000..8f28a7a689
--- /dev/null
+++ b/t/t4018/elixir-module.ctx
@@ -0,0 +1 @@
+defmodule RIGHT do
diff --git a/t/t4018/elixir-nested-module.ctx b/t/t4018/elixir-nested-module.ctx
new file mode 100644
index 0000000000..3ffbdd18b1
--- /dev/null
+++ b/t/t4018/elixir-nested-module.ctx
@@ -0,0 +1 @@
+defmodule MyApp.RIGHT do
diff --git a/t/t4018/elixir-private-function.ctx b/t/t4018/elixir-private-function.ctx
new file mode 100644
index 0000000000..1c4eba44f7
--- /dev/null
+++ b/t/t4018/elixir-private-function.ctx
@@ -0,0 +1 @@
+defp function(RIGHT, arg) do
diff --git a/t/t4018/elixir-protocol-implementation.ctx b/t/t4018/elixir-protocol-implementation.ctx
new file mode 100644
index 0000000000..efb758aea6
--- /dev/null
+++ b/t/t4018/elixir-protocol-implementation.ctx
@@ -0,0 +1 @@
+defimpl RIGHT do
diff --git a/t/t4018/elixir-protocol.ctx b/t/t4018/elixir-protocol.ctx
new file mode 100644
index 0000000000..d0204e9f14
--- /dev/null
+++ b/t/t4018/elixir-protocol.ctx
@@ -0,0 +1 @@
+defprotocol RIGHT do
diff --git a/t/t4018/fortran-block-data.ctx b/t/t4018/fortran-block-data.ctx
new file mode 100644
index 0000000000..c3db084ccc
--- /dev/null
+++ b/t/t4018/fortran-block-data.ctx
@@ -0,0 +1 @@
+BLOCK DATA RIGHT
diff --git a/t/t4018/fortran-comment-keyword.ctx b/t/t4018/fortran-comment-keyword.ctx
new file mode 100644
index 0000000000..0b9220b355
--- /dev/null
+++ b/t/t4018/fortran-comment-keyword.ctx
@@ -0,0 +1 @@
+subroutine RIGHT (funcA, funcB)
diff --git a/t/t4018/fortran-comment-legacy-star.ctx b/t/t4018/fortran-comment-legacy-star.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-comment-legacy-star.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-comment-legacy.ctx b/t/t4018/fortran-comment-legacy.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-comment-legacy.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-comment.ctx b/t/t4018/fortran-comment.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-comment.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-external-function.ctx b/t/t4018/fortran-external-function.ctx
new file mode 100644
index 0000000000..56ec4d8eca
--- /dev/null
+++ b/t/t4018/fortran-external-function.ctx
@@ -0,0 +1 @@
+function RIGHT(a, b) result(c)
diff --git a/t/t4018/fortran-external-subroutine.ctx b/t/t4018/fortran-external-subroutine.ctx
new file mode 100644
index 0000000000..6a34203f80
--- /dev/null
+++ b/t/t4018/fortran-external-subroutine.ctx
@@ -0,0 +1 @@
+subroutine RIGHT
diff --git a/t/t4018/fortran-module-procedure.ctx b/t/t4018/fortran-module-procedure.ctx
new file mode 100644
index 0000000000..4f5ff2e4b8
--- /dev/null
+++ b/t/t4018/fortran-module-procedure.ctx
@@ -0,0 +1 @@
+module RIGHT
diff --git a/t/t4018/fortran-module.ctx b/t/t4018/fortran-module.ctx
new file mode 100644
index 0000000000..4f5ff2e4b8
--- /dev/null
+++ b/t/t4018/fortran-module.ctx
@@ -0,0 +1 @@
+module RIGHT
diff --git a/t/t4018/fortran-program.ctx b/t/t4018/fortran-program.ctx
new file mode 100644
index 0000000000..c4e844df30
--- /dev/null
+++ b/t/t4018/fortran-program.ctx
@@ -0,0 +1 @@
+program RIGHT
diff --git a/t/t4018/fountain-scene.ctx b/t/t4018/fountain-scene.ctx
new file mode 100644
index 0000000000..bf10171418
--- /dev/null
+++ b/t/t4018/fountain-scene.ctx
@@ -0,0 +1 @@
+EXT. STREET RIGHT OUTSIDE - DAY
diff --git a/t/t4018/golang-complex-function.ctx b/t/t4018/golang-complex-function.ctx
new file mode 100644
index 0000000000..8e8d5582ff
--- /dev/null
+++ b/t/t4018/golang-complex-function.ctx
@@ -0,0 +1 @@
+func (t *Test) RIGHT(a Type) (Type, error) {
diff --git a/t/t4018/golang-func.ctx b/t/t4018/golang-func.ctx
new file mode 100644
index 0000000000..88bc823813
--- /dev/null
+++ b/t/t4018/golang-func.ctx
@@ -0,0 +1 @@
+func RIGHT() {
diff --git a/t/t4018/golang-interface.ctx b/t/t4018/golang-interface.ctx
new file mode 100644
index 0000000000..2d07f5a383
--- /dev/null
+++ b/t/t4018/golang-interface.ctx
@@ -0,0 +1 @@
+type RIGHT interface {
diff --git a/t/t4018/golang-long-func.ctx b/t/t4018/golang-long-func.ctx
new file mode 100644
index 0000000000..25635e712e
--- /dev/null
+++ b/t/t4018/golang-long-func.ctx
@@ -0,0 +1 @@
+func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
diff --git a/t/t4018/golang-struct.ctx b/t/t4018/golang-struct.ctx
new file mode 100644
index 0000000000..8a1240699d
--- /dev/null
+++ b/t/t4018/golang-struct.ctx
@@ -0,0 +1 @@
+type RIGHT struct {
diff --git a/t/t4018/java-class-member-function.ctx b/t/t4018/java-class-member-function.ctx
new file mode 100644
index 0000000000..2125474b68
--- /dev/null
+++ b/t/t4018/java-class-member-function.ctx
@@ -0,0 +1 @@
+public static void main(String RIGHT[])
diff --git a/t/t4018/markdown-heading-indented.ctx b/t/t4018/markdown-heading-indented.ctx
new file mode 100644
index 0000000000..5938336743
--- /dev/null
+++ b/t/t4018/markdown-heading-indented.ctx
@@ -0,0 +1 @@
+   ### RIGHT
diff --git a/t/t4018/markdown-heading-non-headings.ctx b/t/t4018/markdown-heading-non-headings.ctx
new file mode 100644
index 0000000000..7e2165be6e
--- /dev/null
+++ b/t/t4018/markdown-heading-non-headings.ctx
@@ -0,0 +1 @@
+# RIGHT
diff --git a/t/t4018/matlab-class-definition.ctx b/t/t4018/matlab-class-definition.ctx
new file mode 100644
index 0000000000..5dd5b45628
--- /dev/null
+++ b/t/t4018/matlab-class-definition.ctx
@@ -0,0 +1 @@
+classdef RIGHT
diff --git a/t/t4018/matlab-function.ctx b/t/t4018/matlab-function.ctx
new file mode 100644
index 0000000000..72d2350b13
--- /dev/null
+++ b/t/t4018/matlab-function.ctx
@@ -0,0 +1 @@
+function y = RIGHT()
diff --git a/t/t4018/matlab-octave-section-1.ctx b/t/t4018/matlab-octave-section-1.ctx
new file mode 100644
index 0000000000..ca9b349f94
--- /dev/null
+++ b/t/t4018/matlab-octave-section-1.ctx
@@ -0,0 +1 @@
+%%% RIGHT section
diff --git a/t/t4018/matlab-octave-section-2.ctx b/t/t4018/matlab-octave-section-2.ctx
new file mode 100644
index 0000000000..5cbb77faf5
--- /dev/null
+++ b/t/t4018/matlab-octave-section-2.ctx
@@ -0,0 +1 @@
+## RIGHT section
diff --git a/t/t4018/matlab-section.ctx b/t/t4018/matlab-section.ctx
new file mode 100644
index 0000000000..e83fee6f4d
--- /dev/null
+++ b/t/t4018/matlab-section.ctx
@@ -0,0 +1 @@
+%% RIGHT section
diff --git a/t/t4018/perl-skip-end-of-heredoc.ctx b/t/t4018/perl-skip-end-of-heredoc.ctx
new file mode 100644
index 0000000000..c15f4b78bd
--- /dev/null
+++ b/t/t4018/perl-skip-end-of-heredoc.ctx
@@ -0,0 +1 @@
+sub RIGHTwithheredocument {
diff --git a/t/t4018/perl-skip-forward-decl.ctx b/t/t4018/perl-skip-forward-decl.ctx
new file mode 100644
index 0000000000..e0c51599ad
--- /dev/null
+++ b/t/t4018/perl-skip-forward-decl.ctx
@@ -0,0 +1 @@
+package RIGHT;
diff --git a/t/t4018/perl-skip-sub-in-pod.ctx b/t/t4018/perl-skip-sub-in-pod.ctx
new file mode 100644
index 0000000000..abddd76655
--- /dev/null
+++ b/t/t4018/perl-skip-sub-in-pod.ctx
@@ -0,0 +1 @@
+=head1 SYNOPSIS_RIGHT
diff --git a/t/t4018/perl-sub-definition-kr-brace.ctx b/t/t4018/perl-sub-definition-kr-brace.ctx
new file mode 100644
index 0000000000..7e5aee5cde
--- /dev/null
+++ b/t/t4018/perl-sub-definition-kr-brace.ctx
@@ -0,0 +1 @@
+sub RIGHT
diff --git a/t/t4018/perl-sub-definition.ctx b/t/t4018/perl-sub-definition.ctx
new file mode 100644
index 0000000000..d49a63598e
--- /dev/null
+++ b/t/t4018/perl-sub-definition.ctx
@@ -0,0 +1 @@
+sub RIGHT {
diff --git a/t/t4018/php-abstract-class.ctx b/t/t4018/php-abstract-class.ctx
new file mode 100644
index 0000000000..f572d2129b
--- /dev/null
+++ b/t/t4018/php-abstract-class.ctx
@@ -0,0 +1 @@
+abstract class RIGHT
diff --git a/t/t4018/php-abstract-method.ctx b/t/t4018/php-abstract-method.ctx
new file mode 100644
index 0000000000..14cb6df42e
--- /dev/null
+++ b/t/t4018/php-abstract-method.ctx
@@ -0,0 +1 @@
+abstract public function RIGHT(): ?string
diff --git a/t/t4018/php-class.ctx b/t/t4018/php-class.ctx
new file mode 100644
index 0000000000..54bff816d6
--- /dev/null
+++ b/t/t4018/php-class.ctx
@@ -0,0 +1 @@
+class RIGHT
diff --git a/t/t4018/php-final-class.ctx b/t/t4018/php-final-class.ctx
new file mode 100644
index 0000000000..4d59fb749b
--- /dev/null
+++ b/t/t4018/php-final-class.ctx
@@ -0,0 +1 @@
+final class RIGHT
diff --git a/t/t4018/php-final-method.ctx b/t/t4018/php-final-method.ctx
new file mode 100644
index 0000000000..b7da8f8082
--- /dev/null
+++ b/t/t4018/php-final-method.ctx
@@ -0,0 +1 @@
+final public function RIGHT(): string
diff --git a/t/t4018/php-function.ctx b/t/t4018/php-function.ctx
new file mode 100644
index 0000000000..c5f3e55302
--- /dev/null
+++ b/t/t4018/php-function.ctx
@@ -0,0 +1 @@
+function RIGHT()
diff --git a/t/t4018/php-interface.ctx b/t/t4018/php-interface.ctx
new file mode 100644
index 0000000000..a45fa0532a
--- /dev/null
+++ b/t/t4018/php-interface.ctx
@@ -0,0 +1 @@
+interface RIGHT
diff --git a/t/t4018/php-method.ctx b/t/t4018/php-method.ctx
new file mode 100644
index 0000000000..eb1659ff9f
--- /dev/null
+++ b/t/t4018/php-method.ctx
@@ -0,0 +1 @@
+public static function RIGHT()
diff --git a/t/t4018/php-trait.ctx b/t/t4018/php-trait.ctx
new file mode 100644
index 0000000000..57aa4c6267
--- /dev/null
+++ b/t/t4018/php-trait.ctx
@@ -0,0 +1 @@
+trait RIGHT
diff --git a/t/t4018/python-async-def.ctx b/t/t4018/python-async-def.ctx
new file mode 100644
index 0000000000..468c548bbe
--- /dev/null
+++ b/t/t4018/python-async-def.ctx
@@ -0,0 +1 @@
+async def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-class.ctx b/t/t4018/python-class.ctx
new file mode 100644
index 0000000000..a40b755e29
--- /dev/null
+++ b/t/t4018/python-class.ctx
@@ -0,0 +1 @@
+class RIGHT(int, str):
diff --git a/t/t4018/python-def.ctx b/t/t4018/python-def.ctx
new file mode 100644
index 0000000000..a1a9cbad63
--- /dev/null
+++ b/t/t4018/python-def.ctx
@@ -0,0 +1 @@
+def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-indented-async-def.ctx b/t/t4018/python-indented-async-def.ctx
new file mode 100644
index 0000000000..d393620a1e
--- /dev/null
+++ b/t/t4018/python-indented-async-def.ctx
@@ -0,0 +1 @@
+async def RIGHT(self, x: int):
diff --git a/t/t4018/python-indented-class.ctx b/t/t4018/python-indented-class.ctx
new file mode 100644
index 0000000000..0881c84dba
--- /dev/null
+++ b/t/t4018/python-indented-class.ctx
@@ -0,0 +1 @@
+class RIGHT:
diff --git a/t/t4018/python-indented-def.ctx b/t/t4018/python-indented-def.ctx
new file mode 100644
index 0000000000..6e5a44b391
--- /dev/null
+++ b/t/t4018/python-indented-def.ctx
@@ -0,0 +1 @@
+def RIGHT(self, x: int):
diff --git a/t/t4018/rust-fn.ctx b/t/t4018/rust-fn.ctx
new file mode 100644
index 0000000000..baa37cf253
--- /dev/null
+++ b/t/t4018/rust-fn.ctx
@@ -0,0 +1 @@
+pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
diff --git a/t/t4018/rust-impl.ctx b/t/t4018/rust-impl.ctx
new file mode 100644
index 0000000000..5344c35f3f
--- /dev/null
+++ b/t/t4018/rust-impl.ctx
@@ -0,0 +1 @@
+impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
diff --git a/t/t4018/rust-macro-rules.ctx b/t/t4018/rust-macro-rules.ctx
new file mode 100644
index 0000000000..7520463aa0
--- /dev/null
+++ b/t/t4018/rust-macro-rules.ctx
@@ -0,0 +1 @@
+macro_rules! RIGHT {
diff --git a/t/t4018/rust-struct.ctx b/t/t4018/rust-struct.ctx
new file mode 100644
index 0000000000..c1e09dc808
--- /dev/null
+++ b/t/t4018/rust-struct.ctx
@@ -0,0 +1 @@
+pub(super) struct RIGHT<'a> {
diff --git a/t/t4018/rust-trait.ctx b/t/t4018/rust-trait.ctx
new file mode 100644
index 0000000000..6af803db29
--- /dev/null
+++ b/t/t4018/rust-trait.ctx
@@ -0,0 +1 @@
+unsafe trait RIGHT<T> {
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH 07/20] userdiff tests: match full hunk headers

[Eric Sunshine]

On Sun, Feb 14, 2021 at 7:56 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> [...]
> Let's bring back coverage for that by adding corresponding *.ctx
> files, this has the added advantage that we're doing a "test_cmp", so
> when we have failures it's just a non-zero exit code from "grep",
> we'll actually have something meaningful in the "-v" output.
> [...]
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
> diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
> @@ -81,11 +81,12 @@ test_expect_success 'setup hunk header tests' '
> +for i in $(git ls-files -- ':!*.ctx')
>  do
>         test_expect_success "hunk header: $i" "
> +               git diff -U1 $i >diff &&
> +               sed -n -e 's/^.*@@\( \|$\)//p' <diff >ctx &&
> +               test_cmp $i.ctx ctx
>         "
>  done

If I'm reading this correctly, you're simply stripping off all the
leading `@@ blah @@` stuff...

> diff --git a/t/t4018/README b/t/t4018/README
> @@ -1,15 +1,15 @@
> +The text that must appear in the hunk header must contains the word
> +"RIGHT" by convention. The "LANG-whatever.ctx" file contains what we
> +expect to appear in the hunk header. We munged the start of the line
> +to "@@ [...] @@" for ease of not having to hardcode the line numbers
> +and offsets.

...which makes me wonder what this "munging to `@@ [...] @@`" is about.

Is this documentation update wrong or am I misunderstanding?

[PATCH 08/20] userdiff tests: rewrite hunk header test infrastructure

[Ævar Arnfjörð Bjarmason]

Rewrite the hunk header test infrastructure introduced in
bfa7d01413 (t4018: an infrastructure to test hunk headers,
2014-03-21). See c228a5c077 (Merge branch 'js/userdiff-cc',
2014-03-31) for the whole series that commit was part of.

As noted in an earlier commit that change introduced the regression of
not testing for the full hunk line, but just whether "RIGHT" appeared
on it[1]. A preceding commit fixed that specific issue, but we were
still left with the inflexibility of the approach described in the
now-deleted t/t4018/README.

I.e. to add any sort of new tests that used the existing test data
we'd either need to add more files like the recently added (but now
deleted) *.ctx) files, using the filesystem as our test datastructure,
or introduce more parsing for the custom file format we were growing
here.

Let's instead just move this over to using a custom test
function. This makes it trivial to add new tests by adding new
optional parameters to the function. Let's still keep the relevant
files in the "t/t4018/" subdirectory instead of adding ~1.5k
lines (and growing) to "t/t4018-diff-funcname.sh"

If this diff is viewed with "--color-moved=plain" we can see that
there's no changes to the lines being moved into the new *.sh files,
i.e. all the deletions are moves. I'm just adding boilerplate around
those existing lines.

The one-off refactoring was performed by an ad-hoc shellscript [2].

1. https://lore.kernel.org/git/87wnvbbf2y.fsf@evledraar.gmail.com/
2.
	#!/bin/sh
	set -ex

	git rm README*
	for t in $(git ls-files ':!*.ctx')
	do
		lang=$(echo $t | sed 's/-.*//')
		desc=$(echo $t | sed -E 's/^[^-]*-//' | tr - " ")

		if ! test -e $lang.sh
		then
			cat >$lang.sh <<-EOF
			#!/bin/sh
			#
			# See ../t4018-diff-funcname.sh's test_diff_funcname()
			#

			EOF
		else
			echo >>$lang.sh
	        fi

		(
	            printf "test_diff_funcname '%s: %s' \\" "$lang" "$desc"
	            echo
	            printf "\t8<<%sEOF_HUNK 9<<%sEOF_TEST\n" '\' '\'
	            cat $t.ctx
	            printf "EOF_HUNK\n"
	            cat $t
	            printf "EOF_TEST\n"
		) >>$lang.sh

		chmod +x $lang.sh
		git add $lang.sh
	        git rm $t $t.ctx
	done

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh                      |  59 +++--
 t/t4018/README                                |  15 --
 t/t4018/README.ctx                            |   1 -
 t/t4018/bash-arithmetic-function              |   4 -
 t/t4018/bash-arithmetic-function.ctx          |   1 -
 t/t4018/bash-bashism-style-compact            |   6 -
 t/t4018/bash-bashism-style-compact.ctx        |   1 -
 t/t4018/bash-bashism-style-function           |   4 -
 t/t4018/bash-bashism-style-function.ctx       |   1 -
 t/t4018/bash-bashism-style-whitespace         |   4 -
 t/t4018/bash-bashism-style-whitespace.ctx     |   1 -
 t/t4018/bash-conditional-function             |   4 -
 t/t4018/bash-conditional-function.ctx         |   1 -
 t/t4018/bash-missing-parentheses              |   6 -
 t/t4018/bash-missing-parentheses.ctx          |   1 -
 t/t4018/bash-mixed-style-compact              |   4 -
 t/t4018/bash-mixed-style-compact.ctx          |   1 -
 t/t4018/bash-mixed-style-function             |   4 -
 t/t4018/bash-mixed-style-function.ctx         |   1 -
 t/t4018/bash-nested-functions                 |   6 -
 t/t4018/bash-nested-functions.ctx             |   1 -
 t/t4018/bash-other-characters                 |   4 -
 t/t4018/bash-other-characters.ctx             |   1 -
 t/t4018/bash-posix-style-compact              |   4 -
 t/t4018/bash-posix-style-compact.ctx          |   1 -
 t/t4018/bash-posix-style-function             |   4 -
 t/t4018/bash-posix-style-function.ctx         |   1 -
 t/t4018/bash-posix-style-whitespace           |   4 -
 t/t4018/bash-posix-style-whitespace.ctx       |   1 -
 t/t4018/bash-subshell-function                |   4 -
 t/t4018/bash-subshell-function.ctx            |   1 -
 t/t4018/bash-trailing-comment                 |   4 -
 t/t4018/bash-trailing-comment.ctx             |   1 -
 t/t4018/bash.sh                               | 160 ++++++++++++
 t/t4018/cpp-c++-function                      |   4 -
 t/t4018/cpp-c++-function.ctx                  |   1 -
 t/t4018/cpp-class-constructor                 |   4 -
 t/t4018/cpp-class-constructor-mem-init        |   5 -
 t/t4018/cpp-class-constructor-mem-init.ctx    |   1 -
 t/t4018/cpp-class-constructor.ctx             |   1 -
 t/t4018/cpp-class-definition                  |   4 -
 t/t4018/cpp-class-definition-derived          |   5 -
 t/t4018/cpp-class-definition-derived.ctx      |   1 -
 t/t4018/cpp-class-definition.ctx              |   1 -
 t/t4018/cpp-class-destructor                  |   4 -
 t/t4018/cpp-class-destructor.ctx              |   1 -
 t/t4018/cpp-function-returning-global-type    |   4 -
 .../cpp-function-returning-global-type.ctx    |   1 -
 t/t4018/cpp-function-returning-nested         |   5 -
 t/t4018/cpp-function-returning-nested.ctx     |   1 -
 t/t4018/cpp-function-returning-pointer        |   4 -
 t/t4018/cpp-function-returning-pointer.ctx    |   1 -
 t/t4018/cpp-function-returning-reference      |   4 -
 t/t4018/cpp-function-returning-reference.ctx  |   1 -
 t/t4018/cpp-gnu-style-function                |   5 -
 t/t4018/cpp-gnu-style-function.ctx            |   1 -
 t/t4018/cpp-namespace-definition              |   4 -
 t/t4018/cpp-namespace-definition.ctx          |   1 -
 t/t4018/cpp-operator-definition               |   4 -
 t/t4018/cpp-operator-definition.ctx           |   1 -
 t/t4018/cpp-skip-access-specifiers            |   8 -
 t/t4018/cpp-skip-access-specifiers.ctx        |   1 -
 t/t4018/cpp-skip-comment-block                |   9 -
 t/t4018/cpp-skip-comment-block.ctx            |   1 -
 t/t4018/cpp-skip-labels                       |   8 -
 t/t4018/cpp-skip-labels.ctx                   |   1 -
 t/t4018/cpp-struct-definition                 |   9 -
 t/t4018/cpp-struct-definition.ctx             |   1 -
 t/t4018/cpp-struct-single-line                |   7 -
 t/t4018/cpp-struct-single-line.ctx            |   1 -
 t/t4018/cpp-template-function-definition      |   4 -
 t/t4018/cpp-template-function-definition.ctx  |   1 -
 t/t4018/cpp-union-definition                  |   4 -
 t/t4018/cpp-union-definition.ctx              |   1 -
 t/t4018/cpp-void-c-function                   |   4 -
 t/t4018/cpp-void-c-function.ctx               |   1 -
 t/t4018/cpp.sh                                | 239 ++++++++++++++++++
 t/t4018/css-attribute-value-selector          |   4 -
 t/t4018/css-attribute-value-selector.ctx      |   1 -
 t/t4018/css-block-level-@-statements          |  10 -
 t/t4018/css-block-level-@-statements.ctx      |   1 -
 t/t4018/css-brace-in-col-1                    |   5 -
 t/t4018/css-brace-in-col-1.ctx                |   1 -
 t/t4018/css-class-selector                    |   4 -
 t/t4018/css-class-selector.ctx                |   1 -
 t/t4018/css-colon-eol                         |   4 -
 t/t4018/css-colon-eol.ctx                     |   1 -
 t/t4018/css-colon-selector                    |   5 -
 t/t4018/css-colon-selector.ctx                |   1 -
 t/t4018/css-common                            |   4 -
 t/t4018/css-common.ctx                        |   1 -
 t/t4018/css-id-selector                       |   4 -
 t/t4018/css-id-selector.ctx                   |   1 -
 t/t4018/css-long-selector-list                |   6 -
 t/t4018/css-long-selector-list.ctx            |   1 -
 t/t4018/css-prop-sans-indent                  |   5 -
 t/t4018/css-prop-sans-indent.ctx              |   1 -
 t/t4018/css-root-selector                     |   4 -
 t/t4018/css-root-selector.ctx                 |   1 -
 t/t4018/css-short-selector-list               |   4 -
 t/t4018/css-short-selector-list.ctx           |   1 -
 t/t4018/css-trailing-space                    |   5 -
 t/t4018/css-trailing-space.ctx                |   1 -
 t/t4018/css.sh                                | 146 +++++++++++
 t/t4018/custom1-pattern.ctx                   |   1 -
 t/t4018/{custom1-pattern => custom1.sh}       |  10 +
 t/t4018/custom2-match-to-end-of-line          |   8 -
 t/t4018/custom2-match-to-end-of-line.ctx      |   1 -
 t/t4018/custom2.sh                            |  18 ++
 t/t4018/custom3-alternation-in-pattern.ctx    |   1 -
 ...tom3-alternation-in-pattern => custom3.sh} |  10 +
 t/t4018/dts-labels                            |   9 -
 t/t4018/dts-labels.ctx                        |   1 -
 t/t4018/dts-node-unitless                     |   8 -
 t/t4018/dts-node-unitless.ctx                 |   1 -
 t/t4018/dts-nodes                             |   8 -
 t/t4018/dts-nodes-boolean-prop                |   9 -
 t/t4018/dts-nodes-boolean-prop.ctx            |   1 -
 t/t4018/dts-nodes-comment1                    |   8 -
 t/t4018/dts-nodes-comment1.ctx                |   1 -
 t/t4018/dts-nodes-comment2                    |   8 -
 t/t4018/dts-nodes-comment2.ctx                |   1 -
 t/t4018/dts-nodes-multiline-prop              |  13 -
 t/t4018/dts-nodes-multiline-prop.ctx          |   1 -
 t/t4018/dts-nodes.ctx                         |   1 -
 t/t4018/dts-reference                         |   9 -
 t/t4018/dts-reference.ctx                     |   1 -
 t/t4018/dts-root                              |   5 -
 t/t4018/dts-root-comment                      |   8 -
 t/t4018/dts-root-comment.ctx                  |   1 -
 t/t4018/dts-root.ctx                          |   1 -
 t/t4018/dts.sh                                | 149 +++++++++++
 t/t4018/elixir-do-not-pick-end                |   5 -
 t/t4018/elixir-do-not-pick-end.ctx            |   1 -
 t/t4018/elixir-ex-unit-test                   |   6 -
 t/t4018/elixir-ex-unit-test.ctx               |   1 -
 t/t4018/elixir-function                       |   5 -
 t/t4018/elixir-function.ctx                   |   1 -
 t/t4018/elixir-macro                          |   5 -
 t/t4018/elixir-macro.ctx                      |   1 -
 t/t4018/elixir-module                         |   9 -
 t/t4018/elixir-module-func                    |   8 -
 t/t4018/elixir-module-func.ctx                |   1 -
 t/t4018/elixir-module.ctx                     |   1 -
 t/t4018/elixir-nested-module                  |   9 -
 t/t4018/elixir-nested-module.ctx              |   1 -
 t/t4018/elixir-private-function               |   5 -
 t/t4018/elixir-private-function.ctx           |   1 -
 t/t4018/elixir-protocol                       |   6 -
 t/t4018/elixir-protocol-implementation        |   5 -
 t/t4018/elixir-protocol-implementation.ctx    |   1 -
 t/t4018/elixir-protocol.ctx                   |   1 -
 t/t4018/elixir.sh                             | 127 ++++++++++
 t/t4018/fortran-block-data                    |   5 -
 t/t4018/fortran-block-data.ctx                |   1 -
 t/t4018/fortran-comment                       |  13 -
 t/t4018/fortran-comment-keyword               |  14 -
 t/t4018/fortran-comment-keyword.ctx           |   1 -
 t/t4018/fortran-comment-legacy                |  13 -
 t/t4018/fortran-comment-legacy-star           |  13 -
 t/t4018/fortran-comment-legacy-star.ctx       |   1 -
 t/t4018/fortran-comment-legacy.ctx            |   1 -
 t/t4018/fortran-comment.ctx                   |   1 -
 t/t4018/fortran-external-function             |   9 -
 t/t4018/fortran-external-function.ctx         |   1 -
 t/t4018/fortran-external-subroutine           |   5 -
 t/t4018/fortran-external-subroutine.ctx       |   1 -
 t/t4018/fortran-module                        |   5 -
 t/t4018/fortran-module-procedure              |  13 -
 t/t4018/fortran-module-procedure.ctx          |   1 -
 t/t4018/fortran-module.ctx                    |   1 -
 t/t4018/fortran-program                       |   5 -
 t/t4018/fortran-program.ctx                   |   1 -
 t/t4018/fortran.sh                            | 159 ++++++++++++
 t/t4018/fountain-scene                        |   4 -
 t/t4018/fountain-scene.ctx                    |   1 -
 t/t4018/fountain.sh                           |  14 +
 t/t4018/golang-complex-function               |   8 -
 t/t4018/golang-complex-function.ctx           |   1 -
 t/t4018/golang-func                           |   4 -
 t/t4018/golang-func.ctx                       |   1 -
 t/t4018/golang-interface                      |   4 -
 t/t4018/golang-interface.ctx                  |   1 -
 t/t4018/golang-long-func                      |   5 -
 t/t4018/golang-long-func.ctx                  |   1 -
 t/t4018/golang-struct                         |   4 -
 t/t4018/golang-struct.ctx                     |   1 -
 t/t4018/golang.sh                             |  59 +++++
 t/t4018/java-class-member-function            |   8 -
 t/t4018/java-class-member-function.ctx        |   1 -
 t/t4018/java.sh                               |  18 ++
 t/t4018/markdown-heading-indented             |   6 -
 t/t4018/markdown-heading-indented.ctx         |   1 -
 t/t4018/markdown-heading-non-headings         |  17 --
 t/t4018/markdown-heading-non-headings.ctx     |   1 -
 t/t4018/markdown.sh                           |  39 +++
 t/t4018/matlab-class-definition               |   5 -
 t/t4018/matlab-class-definition.ctx           |   1 -
 t/t4018/matlab-function                       |   4 -
 t/t4018/matlab-function.ctx                   |   1 -
 t/t4018/matlab-octave-section-1               |   3 -
 t/t4018/matlab-octave-section-1.ctx           |   1 -
 t/t4018/matlab-octave-section-2               |   3 -
 t/t4018/matlab-octave-section-2.ctx           |   1 -
 t/t4018/matlab-section                        |   3 -
 t/t4018/matlab-section.ctx                    |   1 -
 t/t4018/matlab.sh                             |  52 ++++
 t/t4018/perl-skip-end-of-heredoc              |   8 -
 t/t4018/perl-skip-end-of-heredoc.ctx          |   1 -
 t/t4018/perl-skip-forward-decl                |  10 -
 t/t4018/perl-skip-forward-decl.ctx            |   1 -
 t/t4018/perl-skip-sub-in-pod                  |  18 --
 t/t4018/perl-skip-sub-in-pod.ctx              |   1 -
 t/t4018/perl-sub-definition                   |   4 -
 t/t4018/perl-sub-definition-kr-brace          |   4 -
 t/t4018/perl-sub-definition-kr-brace.ctx      |   1 -
 t/t4018/perl-sub-definition.ctx               |   1 -
 t/t4018/perl.sh                               |  78 ++++++
 t/t4018/php-abstract-class                    |   4 -
 t/t4018/php-abstract-class.ctx                |   1 -
 t/t4018/php-abstract-method                   |   7 -
 t/t4018/php-abstract-method.ctx               |   1 -
 t/t4018/php-class                             |   4 -
 t/t4018/php-class.ctx                         |   1 -
 t/t4018/php-final-class                       |   4 -
 t/t4018/php-final-class.ctx                   |   1 -
 t/t4018/php-final-method                      |   7 -
 t/t4018/php-final-method.ctx                  |   1 -
 t/t4018/php-function                          |   4 -
 t/t4018/php-function.ctx                      |   1 -
 t/t4018/php-interface                         |   4 -
 t/t4018/php-interface.ctx                     |   1 -
 t/t4018/php-method                            |   7 -
 t/t4018/php-method.ctx                        |   1 -
 t/t4018/php-trait                             |   7 -
 t/t4018/php-trait.ctx                         |   1 -
 t/t4018/php.sh                                | 106 ++++++++
 t/t4018/python-async-def                      |   4 -
 t/t4018/python-async-def.ctx                  |   1 -
 t/t4018/python-class                          |   4 -
 t/t4018/python-class.ctx                      |   1 -
 t/t4018/python-def                            |   4 -
 t/t4018/python-def.ctx                        |   1 -
 t/t4018/python-indented-async-def             |   7 -
 t/t4018/python-indented-async-def.ctx         |   1 -
 t/t4018/python-indented-class                 |   5 -
 t/t4018/python-indented-class.ctx             |   1 -
 t/t4018/python-indented-def                   |   7 -
 t/t4018/python-indented-def.ctx               |   1 -
 t/t4018/python.sh                             |  71 ++++++
 t/t4018/rust-fn                               |   5 -
 t/t4018/rust-fn.ctx                           |   1 -
 t/t4018/rust-impl                             |   5 -
 t/t4018/rust-impl.ctx                         |   1 -
 t/t4018/rust-macro-rules                      |   6 -
 t/t4018/rust-macro-rules.ctx                  |   1 -
 t/t4018/rust-struct                           |   5 -
 t/t4018/rust-struct.ctx                       |   1 -
 t/t4018/rust-trait                            |   5 -
 t/t4018/rust-trait.ctx                        |   1 -
 t/t4018/rust.sh                               |  60 +++++
 261 files changed, 1549 insertions(+), 879 deletions(-)
 delete mode 100644 t/t4018/README
 delete mode 100644 t/t4018/README.ctx
 delete mode 100644 t/t4018/bash-arithmetic-function
 delete mode 100644 t/t4018/bash-arithmetic-function.ctx
 delete mode 100644 t/t4018/bash-bashism-style-compact
 delete mode 100644 t/t4018/bash-bashism-style-compact.ctx
 delete mode 100644 t/t4018/bash-bashism-style-function
 delete mode 100644 t/t4018/bash-bashism-style-function.ctx
 delete mode 100644 t/t4018/bash-bashism-style-whitespace
 delete mode 100644 t/t4018/bash-bashism-style-whitespace.ctx
 delete mode 100644 t/t4018/bash-conditional-function
 delete mode 100644 t/t4018/bash-conditional-function.ctx
 delete mode 100644 t/t4018/bash-missing-parentheses
 delete mode 100644 t/t4018/bash-missing-parentheses.ctx
 delete mode 100644 t/t4018/bash-mixed-style-compact
 delete mode 100644 t/t4018/bash-mixed-style-compact.ctx
 delete mode 100644 t/t4018/bash-mixed-style-function
 delete mode 100644 t/t4018/bash-mixed-style-function.ctx
 delete mode 100644 t/t4018/bash-nested-functions
 delete mode 100644 t/t4018/bash-nested-functions.ctx
 delete mode 100644 t/t4018/bash-other-characters
 delete mode 100644 t/t4018/bash-other-characters.ctx
 delete mode 100644 t/t4018/bash-posix-style-compact
 delete mode 100644 t/t4018/bash-posix-style-compact.ctx
 delete mode 100644 t/t4018/bash-posix-style-function
 delete mode 100644 t/t4018/bash-posix-style-function.ctx
 delete mode 100644 t/t4018/bash-posix-style-whitespace
 delete mode 100644 t/t4018/bash-posix-style-whitespace.ctx
 delete mode 100644 t/t4018/bash-subshell-function
 delete mode 100644 t/t4018/bash-subshell-function.ctx
 delete mode 100644 t/t4018/bash-trailing-comment
 delete mode 100644 t/t4018/bash-trailing-comment.ctx
 create mode 100755 t/t4018/bash.sh
 delete mode 100644 t/t4018/cpp-c++-function
 delete mode 100644 t/t4018/cpp-c++-function.ctx
 delete mode 100644 t/t4018/cpp-class-constructor
 delete mode 100644 t/t4018/cpp-class-constructor-mem-init
 delete mode 100644 t/t4018/cpp-class-constructor-mem-init.ctx
 delete mode 100644 t/t4018/cpp-class-constructor.ctx
 delete mode 100644 t/t4018/cpp-class-definition
 delete mode 100644 t/t4018/cpp-class-definition-derived
 delete mode 100644 t/t4018/cpp-class-definition-derived.ctx
 delete mode 100644 t/t4018/cpp-class-definition.ctx
 delete mode 100644 t/t4018/cpp-class-destructor
 delete mode 100644 t/t4018/cpp-class-destructor.ctx
 delete mode 100644 t/t4018/cpp-function-returning-global-type
 delete mode 100644 t/t4018/cpp-function-returning-global-type.ctx
 delete mode 100644 t/t4018/cpp-function-returning-nested
 delete mode 100644 t/t4018/cpp-function-returning-nested.ctx
 delete mode 100644 t/t4018/cpp-function-returning-pointer
 delete mode 100644 t/t4018/cpp-function-returning-pointer.ctx
 delete mode 100644 t/t4018/cpp-function-returning-reference
 delete mode 100644 t/t4018/cpp-function-returning-reference.ctx
 delete mode 100644 t/t4018/cpp-gnu-style-function
 delete mode 100644 t/t4018/cpp-gnu-style-function.ctx
 delete mode 100644 t/t4018/cpp-namespace-definition
 delete mode 100644 t/t4018/cpp-namespace-definition.ctx
 delete mode 100644 t/t4018/cpp-operator-definition
 delete mode 100644 t/t4018/cpp-operator-definition.ctx
 delete mode 100644 t/t4018/cpp-skip-access-specifiers
 delete mode 100644 t/t4018/cpp-skip-access-specifiers.ctx
 delete mode 100644 t/t4018/cpp-skip-comment-block
 delete mode 100644 t/t4018/cpp-skip-comment-block.ctx
 delete mode 100644 t/t4018/cpp-skip-labels
 delete mode 100644 t/t4018/cpp-skip-labels.ctx
 delete mode 100644 t/t4018/cpp-struct-definition
 delete mode 100644 t/t4018/cpp-struct-definition.ctx
 delete mode 100644 t/t4018/cpp-struct-single-line
 delete mode 100644 t/t4018/cpp-struct-single-line.ctx
 delete mode 100644 t/t4018/cpp-template-function-definition
 delete mode 100644 t/t4018/cpp-template-function-definition.ctx
 delete mode 100644 t/t4018/cpp-union-definition
 delete mode 100644 t/t4018/cpp-union-definition.ctx
 delete mode 100644 t/t4018/cpp-void-c-function
 delete mode 100644 t/t4018/cpp-void-c-function.ctx
 create mode 100755 t/t4018/cpp.sh
 delete mode 100644 t/t4018/css-attribute-value-selector
 delete mode 100644 t/t4018/css-attribute-value-selector.ctx
 delete mode 100644 t/t4018/css-block-level-@-statements
 delete mode 100644 t/t4018/css-block-level-@-statements.ctx
 delete mode 100644 t/t4018/css-brace-in-col-1
 delete mode 100644 t/t4018/css-brace-in-col-1.ctx
 delete mode 100644 t/t4018/css-class-selector
 delete mode 100644 t/t4018/css-class-selector.ctx
 delete mode 100644 t/t4018/css-colon-eol
 delete mode 100644 t/t4018/css-colon-eol.ctx
 delete mode 100644 t/t4018/css-colon-selector
 delete mode 100644 t/t4018/css-colon-selector.ctx
 delete mode 100644 t/t4018/css-common
 delete mode 100644 t/t4018/css-common.ctx
 delete mode 100644 t/t4018/css-id-selector
 delete mode 100644 t/t4018/css-id-selector.ctx
 delete mode 100644 t/t4018/css-long-selector-list
 delete mode 100644 t/t4018/css-long-selector-list.ctx
 delete mode 100644 t/t4018/css-prop-sans-indent
 delete mode 100644 t/t4018/css-prop-sans-indent.ctx
 delete mode 100644 t/t4018/css-root-selector
 delete mode 100644 t/t4018/css-root-selector.ctx
 delete mode 100644 t/t4018/css-short-selector-list
 delete mode 100644 t/t4018/css-short-selector-list.ctx
 delete mode 100644 t/t4018/css-trailing-space
 delete mode 100644 t/t4018/css-trailing-space.ctx
 create mode 100755 t/t4018/css.sh
 delete mode 100644 t/t4018/custom1-pattern.ctx
 rename t/t4018/{custom1-pattern => custom1.sh} (71%)
 mode change 100644 => 100755
 delete mode 100644 t/t4018/custom2-match-to-end-of-line
 delete mode 100644 t/t4018/custom2-match-to-end-of-line.ctx
 create mode 100755 t/t4018/custom2.sh
 delete mode 100644 t/t4018/custom3-alternation-in-pattern.ctx
 rename t/t4018/{custom3-alternation-in-pattern => custom3.sh} (66%)
 mode change 100644 => 100755
 delete mode 100644 t/t4018/dts-labels
 delete mode 100644 t/t4018/dts-labels.ctx
 delete mode 100644 t/t4018/dts-node-unitless
 delete mode 100644 t/t4018/dts-node-unitless.ctx
 delete mode 100644 t/t4018/dts-nodes
 delete mode 100644 t/t4018/dts-nodes-boolean-prop
 delete mode 100644 t/t4018/dts-nodes-boolean-prop.ctx
 delete mode 100644 t/t4018/dts-nodes-comment1
 delete mode 100644 t/t4018/dts-nodes-comment1.ctx
 delete mode 100644 t/t4018/dts-nodes-comment2
 delete mode 100644 t/t4018/dts-nodes-comment2.ctx
 delete mode 100644 t/t4018/dts-nodes-multiline-prop
 delete mode 100644 t/t4018/dts-nodes-multiline-prop.ctx
 delete mode 100644 t/t4018/dts-nodes.ctx
 delete mode 100644 t/t4018/dts-reference
 delete mode 100644 t/t4018/dts-reference.ctx
 delete mode 100644 t/t4018/dts-root
 delete mode 100644 t/t4018/dts-root-comment
 delete mode 100644 t/t4018/dts-root-comment.ctx
 delete mode 100644 t/t4018/dts-root.ctx
 create mode 100755 t/t4018/dts.sh
 delete mode 100644 t/t4018/elixir-do-not-pick-end
 delete mode 100644 t/t4018/elixir-do-not-pick-end.ctx
 delete mode 100644 t/t4018/elixir-ex-unit-test
 delete mode 100644 t/t4018/elixir-ex-unit-test.ctx
 delete mode 100644 t/t4018/elixir-function
 delete mode 100644 t/t4018/elixir-function.ctx
 delete mode 100644 t/t4018/elixir-macro
 delete mode 100644 t/t4018/elixir-macro.ctx
 delete mode 100644 t/t4018/elixir-module
 delete mode 100644 t/t4018/elixir-module-func
 delete mode 100644 t/t4018/elixir-module-func.ctx
 delete mode 100644 t/t4018/elixir-module.ctx
 delete mode 100644 t/t4018/elixir-nested-module
 delete mode 100644 t/t4018/elixir-nested-module.ctx
 delete mode 100644 t/t4018/elixir-private-function
 delete mode 100644 t/t4018/elixir-private-function.ctx
 delete mode 100644 t/t4018/elixir-protocol
 delete mode 100644 t/t4018/elixir-protocol-implementation
 delete mode 100644 t/t4018/elixir-protocol-implementation.ctx
 delete mode 100644 t/t4018/elixir-protocol.ctx
 create mode 100755 t/t4018/elixir.sh
 delete mode 100644 t/t4018/fortran-block-data
 delete mode 100644 t/t4018/fortran-block-data.ctx
 delete mode 100644 t/t4018/fortran-comment
 delete mode 100644 t/t4018/fortran-comment-keyword
 delete mode 100644 t/t4018/fortran-comment-keyword.ctx
 delete mode 100644 t/t4018/fortran-comment-legacy
 delete mode 100644 t/t4018/fortran-comment-legacy-star
 delete mode 100644 t/t4018/fortran-comment-legacy-star.ctx
 delete mode 100644 t/t4018/fortran-comment-legacy.ctx
 delete mode 100644 t/t4018/fortran-comment.ctx
 delete mode 100644 t/t4018/fortran-external-function
 delete mode 100644 t/t4018/fortran-external-function.ctx
 delete mode 100644 t/t4018/fortran-external-subroutine
 delete mode 100644 t/t4018/fortran-external-subroutine.ctx
 delete mode 100644 t/t4018/fortran-module
 delete mode 100644 t/t4018/fortran-module-procedure
 delete mode 100644 t/t4018/fortran-module-procedure.ctx
 delete mode 100644 t/t4018/fortran-module.ctx
 delete mode 100644 t/t4018/fortran-program
 delete mode 100644 t/t4018/fortran-program.ctx
 create mode 100755 t/t4018/fortran.sh
 delete mode 100644 t/t4018/fountain-scene
 delete mode 100644 t/t4018/fountain-scene.ctx
 create mode 100755 t/t4018/fountain.sh
 delete mode 100644 t/t4018/golang-complex-function
 delete mode 100644 t/t4018/golang-complex-function.ctx
 delete mode 100644 t/t4018/golang-func
 delete mode 100644 t/t4018/golang-func.ctx
 delete mode 100644 t/t4018/golang-interface
 delete mode 100644 t/t4018/golang-interface.ctx
 delete mode 100644 t/t4018/golang-long-func
 delete mode 100644 t/t4018/golang-long-func.ctx
 delete mode 100644 t/t4018/golang-struct
 delete mode 100644 t/t4018/golang-struct.ctx
 create mode 100755 t/t4018/golang.sh
 delete mode 100644 t/t4018/java-class-member-function
 delete mode 100644 t/t4018/java-class-member-function.ctx
 create mode 100755 t/t4018/java.sh
 delete mode 100644 t/t4018/markdown-heading-indented
 delete mode 100644 t/t4018/markdown-heading-indented.ctx
 delete mode 100644 t/t4018/markdown-heading-non-headings
 delete mode 100644 t/t4018/markdown-heading-non-headings.ctx
 create mode 100755 t/t4018/markdown.sh
 delete mode 100644 t/t4018/matlab-class-definition
 delete mode 100644 t/t4018/matlab-class-definition.ctx
 delete mode 100644 t/t4018/matlab-function
 delete mode 100644 t/t4018/matlab-function.ctx
 delete mode 100644 t/t4018/matlab-octave-section-1
 delete mode 100644 t/t4018/matlab-octave-section-1.ctx
 delete mode 100644 t/t4018/matlab-octave-section-2
 delete mode 100644 t/t4018/matlab-octave-section-2.ctx
 delete mode 100644 t/t4018/matlab-section
 delete mode 100644 t/t4018/matlab-section.ctx
 create mode 100755 t/t4018/matlab.sh
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc
 delete mode 100644 t/t4018/perl-skip-end-of-heredoc.ctx
 delete mode 100644 t/t4018/perl-skip-forward-decl
 delete mode 100644 t/t4018/perl-skip-forward-decl.ctx
 delete mode 100644 t/t4018/perl-skip-sub-in-pod
 delete mode 100644 t/t4018/perl-skip-sub-in-pod.ctx
 delete mode 100644 t/t4018/perl-sub-definition
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace
 delete mode 100644 t/t4018/perl-sub-definition-kr-brace.ctx
 delete mode 100644 t/t4018/perl-sub-definition.ctx
 create mode 100755 t/t4018/perl.sh
 delete mode 100644 t/t4018/php-abstract-class
 delete mode 100644 t/t4018/php-abstract-class.ctx
 delete mode 100644 t/t4018/php-abstract-method
 delete mode 100644 t/t4018/php-abstract-method.ctx
 delete mode 100644 t/t4018/php-class
 delete mode 100644 t/t4018/php-class.ctx
 delete mode 100644 t/t4018/php-final-class
 delete mode 100644 t/t4018/php-final-class.ctx
 delete mode 100644 t/t4018/php-final-method
 delete mode 100644 t/t4018/php-final-method.ctx
 delete mode 100644 t/t4018/php-function
 delete mode 100644 t/t4018/php-function.ctx
 delete mode 100644 t/t4018/php-interface
 delete mode 100644 t/t4018/php-interface.ctx
 delete mode 100644 t/t4018/php-method
 delete mode 100644 t/t4018/php-method.ctx
 delete mode 100644 t/t4018/php-trait
 delete mode 100644 t/t4018/php-trait.ctx
 create mode 100755 t/t4018/php.sh
 delete mode 100644 t/t4018/python-async-def
 delete mode 100644 t/t4018/python-async-def.ctx
 delete mode 100644 t/t4018/python-class
 delete mode 100644 t/t4018/python-class.ctx
 delete mode 100644 t/t4018/python-def
 delete mode 100644 t/t4018/python-def.ctx
 delete mode 100644 t/t4018/python-indented-async-def
 delete mode 100644 t/t4018/python-indented-async-def.ctx
 delete mode 100644 t/t4018/python-indented-class
 delete mode 100644 t/t4018/python-indented-class.ctx
 delete mode 100644 t/t4018/python-indented-def
 delete mode 100644 t/t4018/python-indented-def.ctx
 create mode 100755 t/t4018/python.sh
 delete mode 100644 t/t4018/rust-fn
 delete mode 100644 t/t4018/rust-fn.ctx
 delete mode 100644 t/t4018/rust-impl
 delete mode 100644 t/t4018/rust-impl.ctx
 delete mode 100644 t/t4018/rust-macro-rules
 delete mode 100644 t/t4018/rust-macro-rules.ctx
 delete mode 100644 t/t4018/rust-struct
 delete mode 100644 t/t4018/rust-struct.ctx
 delete mode 100644 t/t4018/rust-trait
 delete mode 100644 t/t4018/rust-trait.ctx
 create mode 100755 t/t4018/rust.sh

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index d32c38ad1a..3ff34c13d7 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -61,33 +61,42 @@ test_expect_success 'last regexp must not be negated' '
 	test_i18ngrep ": Last expression must not be negated:" msg
 '
 
-test_expect_success 'setup hunk header tests' '
-	for i in $diffpatterns
-	do
-		echo "$i-* diff=$i"
-	done > .gitattributes &&
-
-	# add all test files to the index
-	(
-		cd "$TEST_DIRECTORY"/t4018 &&
-		git --git-dir="$TRASH_DIRECTORY/.git" add .
-	) &&
-
-	# place modified files in the worktree
-	for i in $(git ls-files)
-	do
-		sed -e "s/ChangeMe/IWasChanged/" <"$TEST_DIRECTORY/t4018/$i" >"$i" || return 1
-	done
-'
+test_diff_funcname () {
+	desc=$1
+	cat <&8 >arg.header &&
+	cat <&9 >arg.test &&
+	what=$(cat arg.what) &&
+
+	test_expect_success "setup: $desc" '
+		cp arg.test "$what" &&
+		cp arg.header expected &&
+		git add "$what" &&
+		sed -e "s/ChangeMe/IWasChanged/" <"$what" >tmp &&
+		mv tmp "$what"
+	' &&
+
+	test_expect_success "$desc" '
+		git diff -U1 "$what" >diff &&
+		sed -n -e "s/^.*@@\( \|$\)//p" <diff >actual &&
+		test_cmp expected actual
+	'
+}
 
-# check each individual file
-for i in $(git ls-files -- ':!*.ctx')
+for what in $diffpatterns
 do
-	test_expect_success "hunk header: $i" "
-		git diff -U1 $i >diff &&
-		sed -n -e 's/^.*@@\( \|$\)//p' <diff >ctx &&
-		test_cmp $i.ctx ctx
-	"
+	test="$TEST_DIRECTORY/t4018/$what.sh"
+	if ! test -e "$test"
+	then
+		test_expect_failure "$what: no tests" 'false'
+		continue
+	fi &&
+
+	test_expect_success "setup: hunk header for $what" '
+		echo "$what diff=$what" >.gitattributes &&
+		echo "$what" >arg.what
+	' &&
+
+	. "$test"
 done
 
 test_done
diff --git a/t/t4018/README b/t/t4018/README
deleted file mode 100644
index 2cccf8c950..0000000000
--- a/t/t4018/README
+++ /dev/null
@@ -1,15 +0,0 @@
-How to write test cases
-=======================
-
-Create test cases called "LANG-whatever" in this directory, where
-"LANG" is e.g. the userdiff driver name, where "whatever" is a brief
-description of the test.
-
-Insert the word "ChangeMe" (exactly this form) at a distance of
-at least two lines from the line that must appear in the hunk header.
-
-The text that must appear in the hunk header must contains the word
-"RIGHT" by convention. The "LANG-whatever.ctx" file contains what we
-expect to appear in the hunk header. We munged the start of the line
-to "@@ [...] @@" for ease of not having to hardcode the line numbers
-and offsets.
diff --git a/t/t4018/README.ctx b/t/t4018/README.ctx
deleted file mode 100644
index cd79384b04..0000000000
--- a/t/t4018/README.ctx
+++ /dev/null
@@ -1 +0,0 @@
-description of the test.
diff --git a/t/t4018/bash-arithmetic-function b/t/t4018/bash-arithmetic-function
deleted file mode 100644
index c0b276cb50..0000000000
--- a/t/t4018/bash-arithmetic-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() ((
-
-    ChangeMe = "$x" + "$y"
-))
diff --git a/t/t4018/bash-arithmetic-function.ctx b/t/t4018/bash-arithmetic-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-arithmetic-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-bashism-style-compact b/t/t4018/bash-bashism-style-compact
deleted file mode 100644
index 1ca3126f61..0000000000
--- a/t/t4018/bash-bashism-style-compact
+++ /dev/null
@@ -1,6 +0,0 @@
-function RIGHT {
-    function InvalidSyntax{
-        :
-        echo 'ChangeMe'
-    }
-}
diff --git a/t/t4018/bash-bashism-style-compact.ctx b/t/t4018/bash-bashism-style-compact.ctx
deleted file mode 100644
index 4f8eac48c6..0000000000
--- a/t/t4018/bash-bashism-style-compact.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT {
diff --git a/t/t4018/bash-bashism-style-function b/t/t4018/bash-bashism-style-function
deleted file mode 100644
index f1de4fa831..0000000000
--- a/t/t4018/bash-bashism-style-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT {
-    :
-    echo 'ChangeMe'
-}
diff --git a/t/t4018/bash-bashism-style-function.ctx b/t/t4018/bash-bashism-style-function.ctx
deleted file mode 100644
index 4f8eac48c6..0000000000
--- a/t/t4018/bash-bashism-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT {
diff --git a/t/t4018/bash-bashism-style-whitespace b/t/t4018/bash-bashism-style-whitespace
deleted file mode 100644
index ade85dd3a5..0000000000
--- a/t/t4018/bash-bashism-style-whitespace
+++ /dev/null
@@ -1,4 +0,0 @@
-	 function 	RIGHT 	( 	) 	{
-
-	    ChangeMe
-	 }
diff --git a/t/t4018/bash-bashism-style-whitespace.ctx b/t/t4018/bash-bashism-style-whitespace.ctx
deleted file mode 100644
index 35dbd0220e..0000000000
--- a/t/t4018/bash-bashism-style-whitespace.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function 	RIGHT 	( 	) 	{
diff --git a/t/t4018/bash-conditional-function b/t/t4018/bash-conditional-function
deleted file mode 100644
index c5949e829b..0000000000
--- a/t/t4018/bash-conditional-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() [[ \
-
-    "$a" > "$ChangeMe"
-]]
diff --git a/t/t4018/bash-conditional-function.ctx b/t/t4018/bash-conditional-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-conditional-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-missing-parentheses b/t/t4018/bash-missing-parentheses
deleted file mode 100644
index 8c8a05dd7a..0000000000
--- a/t/t4018/bash-missing-parentheses
+++ /dev/null
@@ -1,6 +0,0 @@
-function RIGHT {
-    functionInvalidSyntax {
-        :
-        echo 'ChangeMe'
-    }
-}
diff --git a/t/t4018/bash-missing-parentheses.ctx b/t/t4018/bash-missing-parentheses.ctx
deleted file mode 100644
index 4f8eac48c6..0000000000
--- a/t/t4018/bash-missing-parentheses.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT {
diff --git a/t/t4018/bash-mixed-style-compact b/t/t4018/bash-mixed-style-compact
deleted file mode 100644
index d9364cba67..0000000000
--- a/t/t4018/bash-mixed-style-compact
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT(){
-    :
-    echo 'ChangeMe'
-}
diff --git a/t/t4018/bash-mixed-style-compact.ctx b/t/t4018/bash-mixed-style-compact.ctx
deleted file mode 100644
index bba11074eb..0000000000
--- a/t/t4018/bash-mixed-style-compact.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT(){
diff --git a/t/t4018/bash-mixed-style-function b/t/t4018/bash-mixed-style-function
deleted file mode 100644
index 555f9b2466..0000000000
--- a/t/t4018/bash-mixed-style-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT() {
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-mixed-style-function.ctx b/t/t4018/bash-mixed-style-function.ctx
deleted file mode 100644
index 922b87a4aa..0000000000
--- a/t/t4018/bash-mixed-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT() {
diff --git a/t/t4018/bash-nested-functions b/t/t4018/bash-nested-functions
deleted file mode 100644
index 2c9237ead4..0000000000
--- a/t/t4018/bash-nested-functions
+++ /dev/null
@@ -1,6 +0,0 @@
-outer() {
-    RIGHT() {
-        :
-        echo 'ChangeMe'
-    }
-}
diff --git a/t/t4018/bash-nested-functions.ctx b/t/t4018/bash-nested-functions.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-nested-functions.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-other-characters b/t/t4018/bash-other-characters
deleted file mode 100644
index a3f390d525..0000000000
--- a/t/t4018/bash-other-characters
+++ /dev/null
@@ -1,4 +0,0 @@
-_RIGHT_0n() {
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-other-characters.ctx b/t/t4018/bash-other-characters.ctx
deleted file mode 100644
index 6a55317fdf..0000000000
--- a/t/t4018/bash-other-characters.ctx
+++ /dev/null
@@ -1 +0,0 @@
-_RIGHT_0n()
diff --git a/t/t4018/bash-posix-style-compact b/t/t4018/bash-posix-style-compact
deleted file mode 100644
index 045bd2029b..0000000000
--- a/t/t4018/bash-posix-style-compact
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT(){
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-posix-style-compact.ctx b/t/t4018/bash-posix-style-compact.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-posix-style-compact.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-posix-style-function b/t/t4018/bash-posix-style-function
deleted file mode 100644
index a4d144856e..0000000000
--- a/t/t4018/bash-posix-style-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() {
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-posix-style-function.ctx b/t/t4018/bash-posix-style-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-posix-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-posix-style-whitespace b/t/t4018/bash-posix-style-whitespace
deleted file mode 100644
index 4d984f0aa4..0000000000
--- a/t/t4018/bash-posix-style-whitespace
+++ /dev/null
@@ -1,4 +0,0 @@
-	 RIGHT 	( 	) 	{
-
-	    ChangeMe
-	 }
diff --git a/t/t4018/bash-posix-style-whitespace.ctx b/t/t4018/bash-posix-style-whitespace.ctx
deleted file mode 100644
index 28f8698e14..0000000000
--- a/t/t4018/bash-posix-style-whitespace.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT 	( 	)
diff --git a/t/t4018/bash-subshell-function b/t/t4018/bash-subshell-function
deleted file mode 100644
index 80baa09484..0000000000
--- a/t/t4018/bash-subshell-function
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() (
-
-    ChangeMe=2
-)
diff --git a/t/t4018/bash-subshell-function.ctx b/t/t4018/bash-subshell-function.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-subshell-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash-trailing-comment b/t/t4018/bash-trailing-comment
deleted file mode 100644
index f1edbeda31..0000000000
--- a/t/t4018/bash-trailing-comment
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT() { # Comment
-
-    ChangeMe
-}
diff --git a/t/t4018/bash-trailing-comment.ctx b/t/t4018/bash-trailing-comment.ctx
deleted file mode 100644
index 811eac7d2f..0000000000
--- a/t/t4018/bash-trailing-comment.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT()
diff --git a/t/t4018/bash.sh b/t/t4018/bash.sh
new file mode 100755
index 0000000000..69144d9144
--- /dev/null
+++ b/t/t4018/bash.sh
@@ -0,0 +1,160 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'bash: arithmetic function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() ((
+
+    ChangeMe = "$x" + "$y"
+))
+EOF_TEST
+
+test_diff_funcname 'bash: bashism style compact' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT {
+EOF_HUNK
+function RIGHT {
+    function InvalidSyntax{
+        :
+        echo 'ChangeMe'
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'bash: bashism style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT {
+EOF_HUNK
+function RIGHT {
+    :
+    echo 'ChangeMe'
+}
+EOF_TEST
+
+test_diff_funcname 'bash: bashism style whitespace' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function 	RIGHT 	( 	) 	{
+EOF_HUNK
+	 function 	RIGHT 	( 	) 	{
+
+	    ChangeMe
+	 }
+EOF_TEST
+
+test_diff_funcname 'bash: conditional function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() [[ \
+
+    "$a" > "$ChangeMe"
+]]
+EOF_TEST
+
+test_diff_funcname 'bash: missing parentheses' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT {
+EOF_HUNK
+function RIGHT {
+    functionInvalidSyntax {
+        :
+        echo 'ChangeMe'
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'bash: mixed style compact' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT(){
+EOF_HUNK
+function RIGHT(){
+    :
+    echo 'ChangeMe'
+}
+EOF_TEST
+
+test_diff_funcname 'bash: mixed style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT() {
+EOF_HUNK
+function RIGHT() {
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: nested functions' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+outer() {
+    RIGHT() {
+        :
+        echo 'ChangeMe'
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'bash: other characters' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+_RIGHT_0n()
+EOF_HUNK
+_RIGHT_0n() {
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: posix style compact' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT(){
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: posix style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() {
+
+    ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'bash: posix style whitespace' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT 	( 	)
+EOF_HUNK
+	 RIGHT 	( 	) 	{
+
+	    ChangeMe
+	 }
+EOF_TEST
+
+test_diff_funcname 'bash: subshell function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() (
+
+    ChangeMe=2
+)
+EOF_TEST
+
+test_diff_funcname 'bash: trailing comment' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT()
+EOF_HUNK
+RIGHT() { # Comment
+
+    ChangeMe
+}
+EOF_TEST
diff --git a/t/t4018/cpp-c++-function b/t/t4018/cpp-c++-function
deleted file mode 100644
index 9ee6bbef55..0000000000
--- a/t/t4018/cpp-c++-function
+++ /dev/null
@@ -1,4 +0,0 @@
-Item RIGHT::DoSomething( Args with_spaces )
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-c++-function.ctx b/t/t4018/cpp-c++-function.ctx
deleted file mode 100644
index 337b49dbb3..0000000000
--- a/t/t4018/cpp-c++-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Item RIGHT::DoSomething( Args with_spaces )
diff --git a/t/t4018/cpp-class-constructor b/t/t4018/cpp-class-constructor
deleted file mode 100644
index ec4f115c25..0000000000
--- a/t/t4018/cpp-class-constructor
+++ /dev/null
@@ -1,4 +0,0 @@
-Item::Item(int RIGHT)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-class-constructor-mem-init b/t/t4018/cpp-class-constructor-mem-init
deleted file mode 100644
index 49a69f37e1..0000000000
--- a/t/t4018/cpp-class-constructor-mem-init
+++ /dev/null
@@ -1,5 +0,0 @@
-Item::Item(int RIGHT) :
-	member(0)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-class-constructor-mem-init.ctx b/t/t4018/cpp-class-constructor-mem-init.ctx
deleted file mode 100644
index 6664b8b3d8..0000000000
--- a/t/t4018/cpp-class-constructor-mem-init.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Item::Item(int RIGHT) :
diff --git a/t/t4018/cpp-class-constructor.ctx b/t/t4018/cpp-class-constructor.ctx
deleted file mode 100644
index 2dcadfc0ba..0000000000
--- a/t/t4018/cpp-class-constructor.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Item::Item(int RIGHT)
diff --git a/t/t4018/cpp-class-definition b/t/t4018/cpp-class-definition
deleted file mode 100644
index 11b61da3b7..0000000000
--- a/t/t4018/cpp-class-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-class RIGHT
-{
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-class-definition-derived b/t/t4018/cpp-class-definition-derived
deleted file mode 100644
index 3b98cd09ab..0000000000
--- a/t/t4018/cpp-class-definition-derived
+++ /dev/null
@@ -1,5 +0,0 @@
-class RIGHT :
-	public Baseclass
-{
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-class-definition-derived.ctx b/t/t4018/cpp-class-definition-derived.ctx
deleted file mode 100644
index 146f0a7b7c..0000000000
--- a/t/t4018/cpp-class-definition-derived.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT :
diff --git a/t/t4018/cpp-class-definition.ctx b/t/t4018/cpp-class-definition.ctx
deleted file mode 100644
index 54bff816d6..0000000000
--- a/t/t4018/cpp-class-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT
diff --git a/t/t4018/cpp-class-destructor b/t/t4018/cpp-class-destructor
deleted file mode 100644
index 5487665096..0000000000
--- a/t/t4018/cpp-class-destructor
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT::~RIGHT()
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-class-destructor.ctx b/t/t4018/cpp-class-destructor.ctx
deleted file mode 100644
index 5390c17cf6..0000000000
--- a/t/t4018/cpp-class-destructor.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT::~RIGHT()
diff --git a/t/t4018/cpp-function-returning-global-type b/t/t4018/cpp-function-returning-global-type
deleted file mode 100644
index 1084d5990e..0000000000
--- a/t/t4018/cpp-function-returning-global-type
+++ /dev/null
@@ -1,4 +0,0 @@
-::Item get::it::RIGHT()
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-function-returning-global-type.ctx b/t/t4018/cpp-function-returning-global-type.ctx
deleted file mode 100644
index 4dcdde25f4..0000000000
--- a/t/t4018/cpp-function-returning-global-type.ctx
+++ /dev/null
@@ -1 +0,0 @@
-::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-nested b/t/t4018/cpp-function-returning-nested
deleted file mode 100644
index d9750aa61a..0000000000
--- a/t/t4018/cpp-function-returning-nested
+++ /dev/null
@@ -1,5 +0,0 @@
-get::Item get::it::RIGHT()
-{
-	ChangeMe;
-}
-
diff --git a/t/t4018/cpp-function-returning-nested.ctx b/t/t4018/cpp-function-returning-nested.ctx
deleted file mode 100644
index 6ef73c8368..0000000000
--- a/t/t4018/cpp-function-returning-nested.ctx
+++ /dev/null
@@ -1 +0,0 @@
-get::Item get::it::RIGHT()
diff --git a/t/t4018/cpp-function-returning-pointer b/t/t4018/cpp-function-returning-pointer
deleted file mode 100644
index ef15657ea8..0000000000
--- a/t/t4018/cpp-function-returning-pointer
+++ /dev/null
@@ -1,4 +0,0 @@
-const char *get_it_RIGHT(char *ptr)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-function-returning-pointer.ctx b/t/t4018/cpp-function-returning-pointer.ctx
deleted file mode 100644
index bb0acce5c7..0000000000
--- a/t/t4018/cpp-function-returning-pointer.ctx
+++ /dev/null
@@ -1 +0,0 @@
-const char *get_it_RIGHT(char *ptr)
diff --git a/t/t4018/cpp-function-returning-reference b/t/t4018/cpp-function-returning-reference
deleted file mode 100644
index 01b051df70..0000000000
--- a/t/t4018/cpp-function-returning-reference
+++ /dev/null
@@ -1,4 +0,0 @@
-string& get::it::RIGHT(char *ptr)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-function-returning-reference.ctx b/t/t4018/cpp-function-returning-reference.ctx
deleted file mode 100644
index 76afe381fd..0000000000
--- a/t/t4018/cpp-function-returning-reference.ctx
+++ /dev/null
@@ -1 +0,0 @@
-string& get::it::RIGHT(char *ptr)
diff --git a/t/t4018/cpp-gnu-style-function b/t/t4018/cpp-gnu-style-function
deleted file mode 100644
index 08c7c7565a..0000000000
--- a/t/t4018/cpp-gnu-style-function
+++ /dev/null
@@ -1,5 +0,0 @@
-const char *
-RIGHT(int arg)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-gnu-style-function.ctx b/t/t4018/cpp-gnu-style-function.ctx
deleted file mode 100644
index 1858287812..0000000000
--- a/t/t4018/cpp-gnu-style-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT(int arg)
diff --git a/t/t4018/cpp-namespace-definition b/t/t4018/cpp-namespace-definition
deleted file mode 100644
index 6749980241..0000000000
--- a/t/t4018/cpp-namespace-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-namespace RIGHT
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-namespace-definition.ctx b/t/t4018/cpp-namespace-definition.ctx
deleted file mode 100644
index 14c29c4638..0000000000
--- a/t/t4018/cpp-namespace-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-namespace RIGHT
diff --git a/t/t4018/cpp-operator-definition b/t/t4018/cpp-operator-definition
deleted file mode 100644
index 1acd827159..0000000000
--- a/t/t4018/cpp-operator-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-Value operator+(Value LEFT, Value RIGHT)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-operator-definition.ctx b/t/t4018/cpp-operator-definition.ctx
deleted file mode 100644
index 5b56778961..0000000000
--- a/t/t4018/cpp-operator-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-Value operator+(Value LEFT, Value RIGHT)
diff --git a/t/t4018/cpp-skip-access-specifiers b/t/t4018/cpp-skip-access-specifiers
deleted file mode 100644
index 4d4a9dbb9d..0000000000
--- a/t/t4018/cpp-skip-access-specifiers
+++ /dev/null
@@ -1,8 +0,0 @@
-class RIGHT : public Baseclass
-{
-public:
-protected:
-private:
-	void DoSomething();
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-skip-access-specifiers.ctx b/t/t4018/cpp-skip-access-specifiers.ctx
deleted file mode 100644
index 075bcd883b..0000000000
--- a/t/t4018/cpp-skip-access-specifiers.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT : public Baseclass
diff --git a/t/t4018/cpp-skip-comment-block b/t/t4018/cpp-skip-comment-block
deleted file mode 100644
index 3800b9967a..0000000000
--- a/t/t4018/cpp-skip-comment-block
+++ /dev/null
@@ -1,9 +0,0 @@
-struct item RIGHT(int i)
-// Do not
-// pick up
-/* these
-** comments.
-*/
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-skip-comment-block.ctx b/t/t4018/cpp-skip-comment-block.ctx
deleted file mode 100644
index 656c59893d..0000000000
--- a/t/t4018/cpp-skip-comment-block.ctx
+++ /dev/null
@@ -1 +0,0 @@
-struct item RIGHT(int i)
diff --git a/t/t4018/cpp-skip-labels b/t/t4018/cpp-skip-labels
deleted file mode 100644
index b9c10aba22..0000000000
--- a/t/t4018/cpp-skip-labels
+++ /dev/null
@@ -1,8 +0,0 @@
-void RIGHT (void)
-{
-repeat:		// C++ comment
-next:		/* C comment */
-	do_something();
-
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-skip-labels.ctx b/t/t4018/cpp-skip-labels.ctx
deleted file mode 100644
index 6b0635f7f7..0000000000
--- a/t/t4018/cpp-skip-labels.ctx
+++ /dev/null
@@ -1 +0,0 @@
-void RIGHT (void)
diff --git a/t/t4018/cpp-struct-definition b/t/t4018/cpp-struct-definition
deleted file mode 100644
index 521c59fd15..0000000000
--- a/t/t4018/cpp-struct-definition
+++ /dev/null
@@ -1,9 +0,0 @@
-struct RIGHT {
-	unsigned
-	/* this bit field looks like a label and should not be picked up */
-		decoy_bitfield: 2,
-		more : 1;
-	int filler;
-
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-struct-definition.ctx b/t/t4018/cpp-struct-definition.ctx
deleted file mode 100644
index 48ed893279..0000000000
--- a/t/t4018/cpp-struct-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-struct RIGHT {
diff --git a/t/t4018/cpp-struct-single-line b/t/t4018/cpp-struct-single-line
deleted file mode 100644
index a0de5fb800..0000000000
--- a/t/t4018/cpp-struct-single-line
+++ /dev/null
@@ -1,7 +0,0 @@
-void wrong()
-{
-}
-
-struct RIGHT_iterator_tag {};
-
-int ChangeMe;
diff --git a/t/t4018/cpp-struct-single-line.ctx b/t/t4018/cpp-struct-single-line.ctx
deleted file mode 100644
index e3bc9d5017..0000000000
--- a/t/t4018/cpp-struct-single-line.ctx
+++ /dev/null
@@ -1 +0,0 @@
-struct RIGHT_iterator_tag {};
diff --git a/t/t4018/cpp-template-function-definition b/t/t4018/cpp-template-function-definition
deleted file mode 100644
index 0cdf5ba5bd..0000000000
--- a/t/t4018/cpp-template-function-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-template<class T> int RIGHT(T arg)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-template-function-definition.ctx b/t/t4018/cpp-template-function-definition.ctx
deleted file mode 100644
index c9da39cf65..0000000000
--- a/t/t4018/cpp-template-function-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-template<class T> int RIGHT(T arg)
diff --git a/t/t4018/cpp-union-definition b/t/t4018/cpp-union-definition
deleted file mode 100644
index 7ec94df697..0000000000
--- a/t/t4018/cpp-union-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-union RIGHT {
-	double v;
-	int ChangeMe;
-};
diff --git a/t/t4018/cpp-union-definition.ctx b/t/t4018/cpp-union-definition.ctx
deleted file mode 100644
index 2fc7b54fb8..0000000000
--- a/t/t4018/cpp-union-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-union RIGHT {
diff --git a/t/t4018/cpp-void-c-function b/t/t4018/cpp-void-c-function
deleted file mode 100644
index 153081e872..0000000000
--- a/t/t4018/cpp-void-c-function
+++ /dev/null
@@ -1,4 +0,0 @@
-void RIGHT (void)
-{
-	ChangeMe;
-}
diff --git a/t/t4018/cpp-void-c-function.ctx b/t/t4018/cpp-void-c-function.ctx
deleted file mode 100644
index 6b0635f7f7..0000000000
--- a/t/t4018/cpp-void-c-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-void RIGHT (void)
diff --git a/t/t4018/cpp.sh b/t/t4018/cpp.sh
new file mode 100755
index 0000000000..185d40d5ef
--- /dev/null
+++ b/t/t4018/cpp.sh
@@ -0,0 +1,239 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'cpp: c++ function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Item RIGHT::DoSomething( Args with_spaces )
+EOF_HUNK
+Item RIGHT::DoSomething( Args with_spaces )
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: class constructor' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Item::Item(int RIGHT)
+EOF_HUNK
+Item::Item(int RIGHT)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: class constructor mem init' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Item::Item(int RIGHT) :
+EOF_HUNK
+Item::Item(int RIGHT) :
+	member(0)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: class definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT
+EOF_HUNK
+class RIGHT
+{
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: class definition derived' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT :
+EOF_HUNK
+class RIGHT :
+	public Baseclass
+{
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: class destructor' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT::~RIGHT()
+EOF_HUNK
+RIGHT::~RIGHT()
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning global type' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+::Item get::it::RIGHT()
+EOF_HUNK
+::Item get::it::RIGHT()
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning nested' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+get::Item get::it::RIGHT()
+EOF_HUNK
+get::Item get::it::RIGHT()
+{
+	ChangeMe;
+}
+
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning pointer' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+const char *get_it_RIGHT(char *ptr)
+EOF_HUNK
+const char *get_it_RIGHT(char *ptr)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: function returning reference' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+string& get::it::RIGHT(char *ptr)
+EOF_HUNK
+string& get::it::RIGHT(char *ptr)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: gnu style function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT(int arg)
+EOF_HUNK
+const char *
+RIGHT(int arg)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: namespace definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+namespace RIGHT
+EOF_HUNK
+namespace RIGHT
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: operator definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+Value operator+(Value LEFT, Value RIGHT)
+EOF_HUNK
+Value operator+(Value LEFT, Value RIGHT)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: skip access specifiers' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT : public Baseclass
+EOF_HUNK
+class RIGHT : public Baseclass
+{
+public:
+protected:
+private:
+	void DoSomething();
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: skip comment block' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+struct item RIGHT(int i)
+EOF_HUNK
+struct item RIGHT(int i)
+// Do not
+// pick up
+/* these
+** comments.
+*/
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: skip labels' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+void RIGHT (void)
+EOF_HUNK
+void RIGHT (void)
+{
+repeat:		// C++ comment
+next:		/* C comment */
+	do_something();
+
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: struct definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+struct RIGHT {
+EOF_HUNK
+struct RIGHT {
+	unsigned
+	/* this bit field looks like a label and should not be picked up */
+		decoy_bitfield: 2,
+		more : 1;
+	int filler;
+
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: struct single line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+struct RIGHT_iterator_tag {};
+EOF_HUNK
+void wrong()
+{
+}
+
+struct RIGHT_iterator_tag {};
+
+int ChangeMe;
+EOF_TEST
+
+test_diff_funcname 'cpp: template function definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+template<class T> int RIGHT(T arg)
+EOF_HUNK
+template<class T> int RIGHT(T arg)
+{
+	ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'cpp: union definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+union RIGHT {
+EOF_HUNK
+union RIGHT {
+	double v;
+	int ChangeMe;
+};
+EOF_TEST
+
+test_diff_funcname 'cpp: void c function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+void RIGHT (void)
+EOF_HUNK
+void RIGHT (void)
+{
+	ChangeMe;
+}
+EOF_TEST
diff --git a/t/t4018/css-attribute-value-selector b/t/t4018/css-attribute-value-selector
deleted file mode 100644
index 918256b20c..0000000000
--- a/t/t4018/css-attribute-value-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-[class*="RIGHT"] {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-attribute-value-selector.ctx b/t/t4018/css-attribute-value-selector.ctx
deleted file mode 100644
index 7f8956251c..0000000000
--- a/t/t4018/css-attribute-value-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-[class*="RIGHT"] {
diff --git a/t/t4018/css-block-level-@-statements b/t/t4018/css-block-level-@-statements
deleted file mode 100644
index d6755f2f3d..0000000000
--- a/t/t4018/css-block-level-@-statements
+++ /dev/null
@@ -1,10 +0,0 @@
-@keyframes RIGHT {
-    from {
-        background : #000;
-        border : 10px ChangeMe #C6C6C6;
-    }
-    to {
-        background : #fff;
-        border : 10px solid #C6C6C6;
-    }
-}
diff --git a/t/t4018/css-block-level-@-statements.ctx b/t/t4018/css-block-level-@-statements.ctx
deleted file mode 100644
index 7f5e90468c..0000000000
--- a/t/t4018/css-block-level-@-statements.ctx
+++ /dev/null
@@ -1 +0,0 @@
-@keyframes RIGHT {
diff --git a/t/t4018/css-brace-in-col-1 b/t/t4018/css-brace-in-col-1
deleted file mode 100644
index 7831577506..0000000000
--- a/t/t4018/css-brace-in-col-1
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT label.control-label
-{
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-brace-in-col-1.ctx b/t/t4018/css-brace-in-col-1.ctx
deleted file mode 100644
index 91a9105c6a..0000000000
--- a/t/t4018/css-brace-in-col-1.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT label.control-label
diff --git a/t/t4018/css-class-selector b/t/t4018/css-class-selector
deleted file mode 100644
index f790a0062f..0000000000
--- a/t/t4018/css-class-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-.RIGHT {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-class-selector.ctx b/t/t4018/css-class-selector.ctx
deleted file mode 100644
index ac7367d7f4..0000000000
--- a/t/t4018/css-class-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-.RIGHT {
diff --git a/t/t4018/css-colon-eol b/t/t4018/css-colon-eol
deleted file mode 100644
index 5a30553d29..0000000000
--- a/t/t4018/css-colon-eol
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT h1 {
-color:
-ChangeMe;
-}
diff --git a/t/t4018/css-colon-eol.ctx b/t/t4018/css-colon-eol.ctx
deleted file mode 100644
index b68493b9b0..0000000000
--- a/t/t4018/css-colon-eol.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT h1 {
diff --git a/t/t4018/css-colon-selector b/t/t4018/css-colon-selector
deleted file mode 100644
index c6d71fb42d..0000000000
--- a/t/t4018/css-colon-selector
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT a:hover {
-    margin-top:
-    10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-colon-selector.ctx b/t/t4018/css-colon-selector.ctx
deleted file mode 100644
index 00b1a5aefe..0000000000
--- a/t/t4018/css-colon-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT a:hover {
diff --git a/t/t4018/css-common b/t/t4018/css-common
deleted file mode 100644
index 84ed754b33..0000000000
--- a/t/t4018/css-common
+++ /dev/null
@@ -1,4 +0,0 @@
-RIGHT label.control-label {
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-common.ctx b/t/t4018/css-common.ctx
deleted file mode 100644
index 43686b4081..0000000000
--- a/t/t4018/css-common.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT label.control-label {
diff --git a/t/t4018/css-id-selector b/t/t4018/css-id-selector
deleted file mode 100644
index 17c5111052..0000000000
--- a/t/t4018/css-id-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-#RIGHT {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-id-selector.ctx b/t/t4018/css-id-selector.ctx
deleted file mode 100644
index ce19f6d8dc..0000000000
--- a/t/t4018/css-id-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-#RIGHT {
diff --git a/t/t4018/css-long-selector-list b/t/t4018/css-long-selector-list
deleted file mode 100644
index 7ccd25d9ed..0000000000
--- a/t/t4018/css-long-selector-list
+++ /dev/null
@@ -1,6 +0,0 @@
-p.header,
-label.control-label,
-div ul#RIGHT {
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-long-selector-list.ctx b/t/t4018/css-long-selector-list.ctx
deleted file mode 100644
index bc8d0fb62c..0000000000
--- a/t/t4018/css-long-selector-list.ctx
+++ /dev/null
@@ -1 +0,0 @@
-div ul#RIGHT {
diff --git a/t/t4018/css-prop-sans-indent b/t/t4018/css-prop-sans-indent
deleted file mode 100644
index a9e3c86b3c..0000000000
--- a/t/t4018/css-prop-sans-indent
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT, label.control-label {
-margin-top: 10px!important;
-padding: 0;
-border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-prop-sans-indent.ctx b/t/t4018/css-prop-sans-indent.ctx
deleted file mode 100644
index cc880b2f44..0000000000
--- a/t/t4018/css-prop-sans-indent.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT, label.control-label {
diff --git a/t/t4018/css-root-selector b/t/t4018/css-root-selector
deleted file mode 100644
index 22b958e369..0000000000
--- a/t/t4018/css-root-selector
+++ /dev/null
@@ -1,4 +0,0 @@
-:RIGHT {
-    background : #000;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-root-selector.ctx b/t/t4018/css-root-selector.ctx
deleted file mode 100644
index 3010cded2a..0000000000
--- a/t/t4018/css-root-selector.ctx
+++ /dev/null
@@ -1 +0,0 @@
-:RIGHT {
diff --git a/t/t4018/css-short-selector-list b/t/t4018/css-short-selector-list
deleted file mode 100644
index 6a0bdee336..0000000000
--- a/t/t4018/css-short-selector-list
+++ /dev/null
@@ -1,4 +0,0 @@
-label.control, div ul#RIGHT {
-    margin-top: 10px!important;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-short-selector-list.ctx b/t/t4018/css-short-selector-list.ctx
deleted file mode 100644
index 9e5d87d126..0000000000
--- a/t/t4018/css-short-selector-list.ctx
+++ /dev/null
@@ -1 +0,0 @@
-label.control, div ul#RIGHT {
diff --git a/t/t4018/css-trailing-space b/t/t4018/css-trailing-space
deleted file mode 100644
index 32b5606c70..0000000000
--- a/t/t4018/css-trailing-space
+++ /dev/null
@@ -1,5 +0,0 @@
-RIGHT label.control-label {
-    margin:10px;   
-    padding:10px;
-    border : 10px ChangeMe #C6C6C6;
-}
diff --git a/t/t4018/css-trailing-space.ctx b/t/t4018/css-trailing-space.ctx
deleted file mode 100644
index 43686b4081..0000000000
--- a/t/t4018/css-trailing-space.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT label.control-label {
diff --git a/t/t4018/css.sh b/t/t4018/css.sh
new file mode 100755
index 0000000000..106a3de242
--- /dev/null
+++ b/t/t4018/css.sh
@@ -0,0 +1,146 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'css: attribute value selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+[class*="RIGHT"] {
+EOF_HUNK
+[class*="RIGHT"] {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: block level @ statements' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+@keyframes RIGHT {
+EOF_HUNK
+@keyframes RIGHT {
+    from {
+        background : #000;
+        border : 10px ChangeMe #C6C6C6;
+    }
+    to {
+        background : #fff;
+        border : 10px solid #C6C6C6;
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'css: brace in col 1' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT label.control-label
+EOF_HUNK
+RIGHT label.control-label
+{
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: class selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+.RIGHT {
+EOF_HUNK
+.RIGHT {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: colon eol' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT h1 {
+EOF_HUNK
+RIGHT h1 {
+color:
+ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'css: colon selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT a:hover {
+EOF_HUNK
+RIGHT a:hover {
+    margin-top:
+    10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: common' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT label.control-label {
+EOF_HUNK
+RIGHT label.control-label {
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: id selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+#RIGHT {
+EOF_HUNK
+#RIGHT {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: long selector list' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+div ul#RIGHT {
+EOF_HUNK
+p.header,
+label.control-label,
+div ul#RIGHT {
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: prop sans indent' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT, label.control-label {
+EOF_HUNK
+RIGHT, label.control-label {
+margin-top: 10px!important;
+padding: 0;
+border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: root selector' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+:RIGHT {
+EOF_HUNK
+:RIGHT {
+    background : #000;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: short selector list' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+label.control, div ul#RIGHT {
+EOF_HUNK
+label.control, div ul#RIGHT {
+    margin-top: 10px!important;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
+
+test_diff_funcname 'css: trailing space' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT label.control-label {
+EOF_HUNK
+RIGHT label.control-label {
+    margin:10px;   
+    padding:10px;
+    border : 10px ChangeMe #C6C6C6;
+}
+EOF_TEST
diff --git a/t/t4018/custom1-pattern.ctx b/t/t4018/custom1-pattern.ctx
deleted file mode 100644
index d1609cc9a6..0000000000
--- a/t/t4018/custom1-pattern.ctx
+++ /dev/null
@@ -1 +0,0 @@
-int special, RIGHT;
diff --git a/t/t4018/custom1-pattern b/t/t4018/custom1.sh
old mode 100644
new mode 100755
similarity index 71%
rename from t/t4018/custom1-pattern
rename to t/t4018/custom1.sh
index e8fd59f884..f8bbccadb4
--- a/t/t4018/custom1-pattern
+++ b/t/t4018/custom1.sh
@@ -1,3 +1,12 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom1: pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+int special, RIGHT;
+EOF_HUNK
 public class Beer
 {
 	int special, RIGHT;
@@ -15,3 +24,4 @@ public class Beer
 			+ "99 bottles of beer on the wall.\n");
 	}
 }
+EOF_TEST
diff --git a/t/t4018/custom2-match-to-end-of-line b/t/t4018/custom2-match-to-end-of-line
deleted file mode 100644
index f88ac318b7..0000000000
--- a/t/t4018/custom2-match-to-end-of-line
+++ /dev/null
@@ -1,8 +0,0 @@
-public class RIGHT_Beer
-{
-	int special;
-	public static void main(String args[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
diff --git a/t/t4018/custom2-match-to-end-of-line.ctx b/t/t4018/custom2-match-to-end-of-line.ctx
deleted file mode 100644
index 8294c6e49b..0000000000
--- a/t/t4018/custom2-match-to-end-of-line.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT_Beer
diff --git a/t/t4018/custom2.sh b/t/t4018/custom2.sh
new file mode 100755
index 0000000000..c68421f788
--- /dev/null
+++ b/t/t4018/custom2.sh
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom2: match to end of line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT_Beer
+EOF_HUNK
+public class RIGHT_Beer
+{
+	int special;
+	public static void main(String args[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
diff --git a/t/t4018/custom3-alternation-in-pattern.ctx b/t/t4018/custom3-alternation-in-pattern.ctx
deleted file mode 100644
index 2125474b68..0000000000
--- a/t/t4018/custom3-alternation-in-pattern.ctx
+++ /dev/null
@@ -1 +0,0 @@
-public static void main(String RIGHT[])
diff --git a/t/t4018/custom3-alternation-in-pattern b/t/t4018/custom3.sh
old mode 100644
new mode 100755
similarity index 66%
rename from t/t4018/custom3-alternation-in-pattern
rename to t/t4018/custom3.sh
index 5f3769c64f..07c5c134ff
--- a/t/t4018/custom3-alternation-in-pattern
+++ b/t/t4018/custom3.sh
@@ -1,3 +1,12 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'custom3: alternation in pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
 public class Beer
 {
 	int special;
@@ -15,3 +24,4 @@ public class Beer
 			+ "99 bottles of beer on the wall.\n");
 	}
 }
+EOF_TEST
diff --git a/t/t4018/dts-labels b/t/t4018/dts-labels
deleted file mode 100644
index b21ef8737b..0000000000
--- a/t/t4018/dts-labels
+++ /dev/null
@@ -1,9 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		label2: RIGHT {
-			vendor,some-property;
-
-			ChangeMe = <0x45-30>;
-		};
-	};
-};
diff --git a/t/t4018/dts-labels.ctx b/t/t4018/dts-labels.ctx
deleted file mode 100644
index 48d9373cab..0000000000
--- a/t/t4018/dts-labels.ctx
+++ /dev/null
@@ -1 +0,0 @@
-label2: RIGHT {
diff --git a/t/t4018/dts-node-unitless b/t/t4018/dts-node-unitless
deleted file mode 100644
index c5287d9141..0000000000
--- a/t/t4018/dts-node-unitless
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1 {
-		RIGHT {
-			prop-array = <1>, <4>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-node-unitless.ctx b/t/t4018/dts-node-unitless.ctx
deleted file mode 100644
index 82c8683fa1..0000000000
--- a/t/t4018/dts-node-unitless.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT {
diff --git a/t/t4018/dts-nodes b/t/t4018/dts-nodes
deleted file mode 100644
index 5a4334bb16..0000000000
--- a/t/t4018/dts-nodes
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 {
-			#size-cells = <1>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-boolean-prop b/t/t4018/dts-nodes-boolean-prop
deleted file mode 100644
index afc6b5b404..0000000000
--- a/t/t4018/dts-nodes-boolean-prop
+++ /dev/null
@@ -1,9 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 {
-			boolean-prop1;
-
-			ChangeMe;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-boolean-prop.ctx b/t/t4018/dts-nodes-boolean-prop.ctx
deleted file mode 100644
index 3a0232d55d..0000000000
--- a/t/t4018/dts-nodes-boolean-prop.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes-comment1 b/t/t4018/dts-nodes-comment1
deleted file mode 100644
index 559dfce9b3..0000000000
--- a/t/t4018/dts-nodes-comment1
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 /* &a comment */ {
-			#size-cells = <1>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-comment1.ctx b/t/t4018/dts-nodes-comment1.ctx
deleted file mode 100644
index ec364600b1..0000000000
--- a/t/t4018/dts-nodes-comment1.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 /* &a comment */ {
diff --git a/t/t4018/dts-nodes-comment2 b/t/t4018/dts-nodes-comment2
deleted file mode 100644
index 27e9718b31..0000000000
--- a/t/t4018/dts-nodes-comment2
+++ /dev/null
@@ -1,8 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 { /* a trailing comment */ 
-			#size-cells = <1>;
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-comment2.ctx b/t/t4018/dts-nodes-comment2.ctx
deleted file mode 100644
index 75f0d75258..0000000000
--- a/t/t4018/dts-nodes-comment2.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 { /* a trailing comment */
diff --git a/t/t4018/dts-nodes-multiline-prop b/t/t4018/dts-nodes-multiline-prop
deleted file mode 100644
index 072d58b69d..0000000000
--- a/t/t4018/dts-nodes-multiline-prop
+++ /dev/null
@@ -1,13 +0,0 @@
-/ {
-	label_1: node1@ff00 {
-		RIGHT@deadf00,4000 {
-			multilineprop = <3>,
-					<4>,
-					<5>,
-					<6>,
-					<7>;
-
-			ChangeMe = <0xffeedd00>;
-		};
-	};
-};
diff --git a/t/t4018/dts-nodes-multiline-prop.ctx b/t/t4018/dts-nodes-multiline-prop.ctx
deleted file mode 100644
index 3a0232d55d..0000000000
--- a/t/t4018/dts-nodes-multiline-prop.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-nodes.ctx b/t/t4018/dts-nodes.ctx
deleted file mode 100644
index 3a0232d55d..0000000000
--- a/t/t4018/dts-nodes.ctx
+++ /dev/null
@@ -1 +0,0 @@
-RIGHT@deadf00,4000 {
diff --git a/t/t4018/dts-reference b/t/t4018/dts-reference
deleted file mode 100644
index 8f0c87d863..0000000000
--- a/t/t4018/dts-reference
+++ /dev/null
@@ -1,9 +0,0 @@
-&label_1 {
-	TEST = <455>;
-};
-
-&RIGHT {
-	vendor,some-property;
-
-	ChangeMe = <0x45-30>;
-};
diff --git a/t/t4018/dts-reference.ctx b/t/t4018/dts-reference.ctx
deleted file mode 100644
index c1e13409ee..0000000000
--- a/t/t4018/dts-reference.ctx
+++ /dev/null
@@ -1 +0,0 @@
-&RIGHT {
diff --git a/t/t4018/dts-root b/t/t4018/dts-root
deleted file mode 100644
index 4353b8220c..0000000000
--- a/t/t4018/dts-root
+++ /dev/null
@@ -1,5 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
-	#size-cells = <1>;
-
-	ChangeMe = <0xffeedd00>;
-};
diff --git a/t/t4018/dts-root-comment b/t/t4018/dts-root-comment
deleted file mode 100644
index 333a625c70..0000000000
--- a/t/t4018/dts-root-comment
+++ /dev/null
@@ -1,8 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
-	#size-cells = <1>;
-
-	/* This comment should be ignored */
-
-	some-property = <40+2>;
-	ChangeMe = <0xffeedd00>;
-};
diff --git a/t/t4018/dts-root-comment.ctx b/t/t4018/dts-root-comment.ctx
deleted file mode 100644
index 656053dd42..0000000000
--- a/t/t4018/dts-root-comment.ctx
+++ /dev/null
@@ -1 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/dts-root.ctx b/t/t4018/dts-root.ctx
deleted file mode 100644
index 656053dd42..0000000000
--- a/t/t4018/dts-root.ctx
+++ /dev/null
@@ -1 +0,0 @@
-/ { RIGHT /* Technically just supposed to be a slash and brace */
diff --git a/t/t4018/dts.sh b/t/t4018/dts.sh
new file mode 100755
index 0000000000..304c131d86
--- /dev/null
+++ b/t/t4018/dts.sh
@@ -0,0 +1,149 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'dts: labels' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+label2: RIGHT {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		label2: RIGHT {
+			vendor,some-property;
+
+			ChangeMe = <0x45-30>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: node unitless' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT {
+EOF_HUNK
+/ {
+	label_1: node1 {
+		RIGHT {
+			prop-array = <1>, <4>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 {
+			#size-cells = <1>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes boolean prop' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 {
+			boolean-prop1;
+
+			ChangeMe;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes comment1' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 /* &a comment */ {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 /* &a comment */ {
+			#size-cells = <1>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes comment2' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 { /* a trailing comment */
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 { /* a trailing comment */ 
+			#size-cells = <1>;
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: nodes multiline prop' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT@deadf00,4000 {
+EOF_HUNK
+/ {
+	label_1: node1@ff00 {
+		RIGHT@deadf00,4000 {
+			multilineprop = <3>,
+					<4>,
+					<5>,
+					<6>,
+					<7>;
+
+			ChangeMe = <0xffeedd00>;
+		};
+	};
+};
+EOF_TEST
+
+test_diff_funcname 'dts: reference' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+&RIGHT {
+EOF_HUNK
+&label_1 {
+	TEST = <455>;
+};
+
+&RIGHT {
+	vendor,some-property;
+
+	ChangeMe = <0x45-30>;
+};
+EOF_TEST
+
+test_diff_funcname 'dts: root' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+EOF_HUNK
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+	#size-cells = <1>;
+
+	ChangeMe = <0xffeedd00>;
+};
+EOF_TEST
+
+test_diff_funcname 'dts: root comment' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+EOF_HUNK
+/ { RIGHT /* Technically just supposed to be a slash and brace */
+	#size-cells = <1>;
+
+	/* This comment should be ignored */
+
+	some-property = <40+2>;
+	ChangeMe = <0xffeedd00>;
+};
+EOF_TEST
diff --git a/t/t4018/elixir-do-not-pick-end b/t/t4018/elixir-do-not-pick-end
deleted file mode 100644
index fae08ba7e8..0000000000
--- a/t/t4018/elixir-do-not-pick-end
+++ /dev/null
@@ -1,5 +0,0 @@
-defmodule RIGHT do
-end
-#
-#
-# ChangeMe; do not pick up 'end' line
diff --git a/t/t4018/elixir-do-not-pick-end.ctx b/t/t4018/elixir-do-not-pick-end.ctx
deleted file mode 100644
index 8f28a7a689..0000000000
--- a/t/t4018/elixir-do-not-pick-end.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmodule RIGHT do
diff --git a/t/t4018/elixir-ex-unit-test b/t/t4018/elixir-ex-unit-test
deleted file mode 100644
index 0560a2b697..0000000000
--- a/t/t4018/elixir-ex-unit-test
+++ /dev/null
@@ -1,6 +0,0 @@
-defmodule Test do
-  test "RIGHT" do
-    assert true == true
-    assert ChangeMe
-  end
-end
diff --git a/t/t4018/elixir-ex-unit-test.ctx b/t/t4018/elixir-ex-unit-test.ctx
deleted file mode 100644
index a55e3de2cc..0000000000
--- a/t/t4018/elixir-ex-unit-test.ctx
+++ /dev/null
@@ -1 +0,0 @@
-test "RIGHT" do
diff --git a/t/t4018/elixir-function b/t/t4018/elixir-function
deleted file mode 100644
index d452f495a7..0000000000
--- a/t/t4018/elixir-function
+++ /dev/null
@@ -1,5 +0,0 @@
-def function(RIGHT, arg) do
-  # comment
-  # comment
-  ChangeMe
-end
diff --git a/t/t4018/elixir-function.ctx b/t/t4018/elixir-function.ctx
deleted file mode 100644
index 62aee9c8b1..0000000000
--- a/t/t4018/elixir-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def function(RIGHT, arg) do
diff --git a/t/t4018/elixir-macro b/t/t4018/elixir-macro
deleted file mode 100644
index 4f925e9ad4..0000000000
--- a/t/t4018/elixir-macro
+++ /dev/null
@@ -1,5 +0,0 @@
-defmacro foo(RIGHT) do
-  # Code
-  # Code
-  ChangeMe
-end
diff --git a/t/t4018/elixir-macro.ctx b/t/t4018/elixir-macro.ctx
deleted file mode 100644
index fc1d3b85e8..0000000000
--- a/t/t4018/elixir-macro.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmacro foo(RIGHT) do
diff --git a/t/t4018/elixir-module b/t/t4018/elixir-module
deleted file mode 100644
index 91a4e7aa20..0000000000
--- a/t/t4018/elixir-module
+++ /dev/null
@@ -1,9 +0,0 @@
-defmodule RIGHT do
-  @moduledoc """
-  Foo bar
-  """
-
-  def ChangeMe(a) where is_map(a) do
-    a
-  end
-end
diff --git a/t/t4018/elixir-module-func b/t/t4018/elixir-module-func
deleted file mode 100644
index c9910d0675..0000000000
--- a/t/t4018/elixir-module-func
+++ /dev/null
@@ -1,8 +0,0 @@
-defmodule Foo do
-  def fun(RIGHT) do
-     # Code
-     # Code
-     # Code
-     ChangeMe
-  end
-end
diff --git a/t/t4018/elixir-module-func.ctx b/t/t4018/elixir-module-func.ctx
deleted file mode 100644
index 8239214386..0000000000
--- a/t/t4018/elixir-module-func.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def fun(RIGHT) do
diff --git a/t/t4018/elixir-module.ctx b/t/t4018/elixir-module.ctx
deleted file mode 100644
index 8f28a7a689..0000000000
--- a/t/t4018/elixir-module.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmodule RIGHT do
diff --git a/t/t4018/elixir-nested-module b/t/t4018/elixir-nested-module
deleted file mode 100644
index 771ebc5c42..0000000000
--- a/t/t4018/elixir-nested-module
+++ /dev/null
@@ -1,9 +0,0 @@
-defmodule MyApp.RIGHT do
-  @moduledoc """
-  Foo bar
-  """
-
-  def ChangeMe(a) where is_map(a) do
-    a
-  end
-end
diff --git a/t/t4018/elixir-nested-module.ctx b/t/t4018/elixir-nested-module.ctx
deleted file mode 100644
index 3ffbdd18b1..0000000000
--- a/t/t4018/elixir-nested-module.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defmodule MyApp.RIGHT do
diff --git a/t/t4018/elixir-private-function b/t/t4018/elixir-private-function
deleted file mode 100644
index 1aabe33b7a..0000000000
--- a/t/t4018/elixir-private-function
+++ /dev/null
@@ -1,5 +0,0 @@
-defp function(RIGHT, arg) do
-  # comment
-  # comment
-  ChangeMe
-end
diff --git a/t/t4018/elixir-private-function.ctx b/t/t4018/elixir-private-function.ctx
deleted file mode 100644
index 1c4eba44f7..0000000000
--- a/t/t4018/elixir-private-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defp function(RIGHT, arg) do
diff --git a/t/t4018/elixir-protocol b/t/t4018/elixir-protocol
deleted file mode 100644
index 7d9173691e..0000000000
--- a/t/t4018/elixir-protocol
+++ /dev/null
@@ -1,6 +0,0 @@
-defprotocol RIGHT do
-  @doc """
-  Calculates the size (and not the length!) of a data structure
-  """
-  def size(data, ChangeMe)
-end
diff --git a/t/t4018/elixir-protocol-implementation b/t/t4018/elixir-protocol-implementation
deleted file mode 100644
index f9234bbfc4..0000000000
--- a/t/t4018/elixir-protocol-implementation
+++ /dev/null
@@ -1,5 +0,0 @@
-defimpl RIGHT do
-  # Docs
-  # Docs
-  def foo(ChangeMe), do: :ok
-end
diff --git a/t/t4018/elixir-protocol-implementation.ctx b/t/t4018/elixir-protocol-implementation.ctx
deleted file mode 100644
index efb758aea6..0000000000
--- a/t/t4018/elixir-protocol-implementation.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defimpl RIGHT do
diff --git a/t/t4018/elixir-protocol.ctx b/t/t4018/elixir-protocol.ctx
deleted file mode 100644
index d0204e9f14..0000000000
--- a/t/t4018/elixir-protocol.ctx
+++ /dev/null
@@ -1 +0,0 @@
-defprotocol RIGHT do
diff --git a/t/t4018/elixir.sh b/t/t4018/elixir.sh
new file mode 100755
index 0000000000..6b09df9520
--- /dev/null
+++ b/t/t4018/elixir.sh
@@ -0,0 +1,127 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'elixir: do not pick end' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmodule RIGHT do
+EOF_HUNK
+defmodule RIGHT do
+end
+#
+#
+# ChangeMe; do not pick up 'end' line
+EOF_TEST
+
+test_diff_funcname 'elixir: ex unit test' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+test "RIGHT" do
+EOF_HUNK
+defmodule Test do
+  test "RIGHT" do
+    assert true == true
+    assert ChangeMe
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def function(RIGHT, arg) do
+EOF_HUNK
+def function(RIGHT, arg) do
+  # comment
+  # comment
+  ChangeMe
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: macro' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmacro foo(RIGHT) do
+EOF_HUNK
+defmacro foo(RIGHT) do
+  # Code
+  # Code
+  ChangeMe
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: module' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmodule RIGHT do
+EOF_HUNK
+defmodule RIGHT do
+  @moduledoc """
+  Foo bar
+  """
+
+  def ChangeMe(a) where is_map(a) do
+    a
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: module func' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def fun(RIGHT) do
+EOF_HUNK
+defmodule Foo do
+  def fun(RIGHT) do
+     # Code
+     # Code
+     # Code
+     ChangeMe
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: nested module' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defmodule MyApp.RIGHT do
+EOF_HUNK
+defmodule MyApp.RIGHT do
+  @moduledoc """
+  Foo bar
+  """
+
+  def ChangeMe(a) where is_map(a) do
+    a
+  end
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: private function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defp function(RIGHT, arg) do
+EOF_HUNK
+defp function(RIGHT, arg) do
+  # comment
+  # comment
+  ChangeMe
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: protocol' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defprotocol RIGHT do
+EOF_HUNK
+defprotocol RIGHT do
+  @doc """
+  Calculates the size (and not the length!) of a data structure
+  """
+  def size(data, ChangeMe)
+end
+EOF_TEST
+
+test_diff_funcname 'elixir: protocol implementation' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+defimpl RIGHT do
+EOF_HUNK
+defimpl RIGHT do
+  # Docs
+  # Docs
+  def foo(ChangeMe), do: :ok
+end
+EOF_TEST
diff --git a/t/t4018/fortran-block-data b/t/t4018/fortran-block-data
deleted file mode 100644
index 63d4e21d0a..0000000000
--- a/t/t4018/fortran-block-data
+++ /dev/null
@@ -1,5 +0,0 @@
-       BLOCK DATA RIGHT
-       
-       COMMON /B/ C, ChangeMe
-       DATA C, ChangeMe  / 2.0, 6.0 / 
-       END 
diff --git a/t/t4018/fortran-block-data.ctx b/t/t4018/fortran-block-data.ctx
deleted file mode 100644
index c3db084ccc..0000000000
--- a/t/t4018/fortran-block-data.ctx
+++ /dev/null
@@ -1 +0,0 @@
-BLOCK DATA RIGHT
diff --git a/t/t4018/fortran-comment b/t/t4018/fortran-comment
deleted file mode 100644
index 7b10d17658..0000000000
--- a/t/t4018/fortran-comment
+++ /dev/null
@@ -1,13 +0,0 @@
-      module a
-
-      contains
-
-      ! subroutine wrong
-      subroutine RIGHT
-      ! subroutine wrong
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-keyword b/t/t4018/fortran-comment-keyword
deleted file mode 100644
index e9206a5379..0000000000
--- a/t/t4018/fortran-comment-keyword
+++ /dev/null
@@ -1,14 +0,0 @@
-      module a
-
-      contains
-
-      subroutine RIGHT (funcA, funcB)
-
-      real funcA  ! grid function a
-      real funcB  ! grid function b
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-keyword.ctx b/t/t4018/fortran-comment-keyword.ctx
deleted file mode 100644
index 0b9220b355..0000000000
--- a/t/t4018/fortran-comment-keyword.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT (funcA, funcB)
diff --git a/t/t4018/fortran-comment-legacy b/t/t4018/fortran-comment-legacy
deleted file mode 100644
index 53cd062c1e..0000000000
--- a/t/t4018/fortran-comment-legacy
+++ /dev/null
@@ -1,13 +0,0 @@
-      module a
-
-      contains
-
-C subroutine wrong
-      subroutine RIGHT
-C subroutine wrong
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-legacy-star b/t/t4018/fortran-comment-legacy-star
deleted file mode 100644
index 2cbcdc3d8a..0000000000
--- a/t/t4018/fortran-comment-legacy-star
+++ /dev/null
@@ -1,13 +0,0 @@
-      module a
-
-      contains
-
-* subroutine wrong
-      subroutine RIGHT
-* subroutine wrong
-
-      real ChangeMe
-
-      end subroutine RIGHT
-
-      end module a
diff --git a/t/t4018/fortran-comment-legacy-star.ctx b/t/t4018/fortran-comment-legacy-star.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-comment-legacy-star.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-comment-legacy.ctx b/t/t4018/fortran-comment-legacy.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-comment-legacy.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-comment.ctx b/t/t4018/fortran-comment.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-comment.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-external-function b/t/t4018/fortran-external-function
deleted file mode 100644
index 5a2d85d3aa..0000000000
--- a/t/t4018/fortran-external-function
+++ /dev/null
@@ -1,9 +0,0 @@
-function RIGHT(a, b) result(c)
-
-integer, intent(in) :: ChangeMe
-integer, intent(in) :: b
-integer, intent(out) :: c
-
-c = a+b
-
-end function RIGHT
diff --git a/t/t4018/fortran-external-function.ctx b/t/t4018/fortran-external-function.ctx
deleted file mode 100644
index 56ec4d8eca..0000000000
--- a/t/t4018/fortran-external-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT(a, b) result(c)
diff --git a/t/t4018/fortran-external-subroutine b/t/t4018/fortran-external-subroutine
deleted file mode 100644
index 4ce85fea13..0000000000
--- a/t/t4018/fortran-external-subroutine
+++ /dev/null
@@ -1,5 +0,0 @@
-subroutine RIGHT
-
-real ChangeMe
-
-end subroutine RIGHT
diff --git a/t/t4018/fortran-external-subroutine.ctx b/t/t4018/fortran-external-subroutine.ctx
deleted file mode 100644
index 6a34203f80..0000000000
--- a/t/t4018/fortran-external-subroutine.ctx
+++ /dev/null
@@ -1 +0,0 @@
-subroutine RIGHT
diff --git a/t/t4018/fortran-module b/t/t4018/fortran-module
deleted file mode 100644
index c4b737dac3..0000000000
--- a/t/t4018/fortran-module
+++ /dev/null
@@ -1,5 +0,0 @@
-module RIGHT
-
-use ChangeMe
-
-end module RIGHT
diff --git a/t/t4018/fortran-module-procedure b/t/t4018/fortran-module-procedure
deleted file mode 100644
index 1ce6d854c2..0000000000
--- a/t/t4018/fortran-module-procedure
+++ /dev/null
@@ -1,13 +0,0 @@
- module RIGHT
-
-   implicit none
-   private
-
-   interface letters  ! generic interface
-      module procedure aaaa, &
-                       bbbb, &
-                       ChangeMe, &
-                       dddd
-   end interface
-   
-end module RIGHT
diff --git a/t/t4018/fortran-module-procedure.ctx b/t/t4018/fortran-module-procedure.ctx
deleted file mode 100644
index 4f5ff2e4b8..0000000000
--- a/t/t4018/fortran-module-procedure.ctx
+++ /dev/null
@@ -1 +0,0 @@
-module RIGHT
diff --git a/t/t4018/fortran-module.ctx b/t/t4018/fortran-module.ctx
deleted file mode 100644
index 4f5ff2e4b8..0000000000
--- a/t/t4018/fortran-module.ctx
+++ /dev/null
@@ -1 +0,0 @@
-module RIGHT
diff --git a/t/t4018/fortran-program b/t/t4018/fortran-program
deleted file mode 100644
index 4616895e4b..0000000000
--- a/t/t4018/fortran-program
+++ /dev/null
@@ -1,5 +0,0 @@
-program RIGHT
-
-call ChangeMe
-
-end program RIGHT
diff --git a/t/t4018/fortran-program.ctx b/t/t4018/fortran-program.ctx
deleted file mode 100644
index c4e844df30..0000000000
--- a/t/t4018/fortran-program.ctx
+++ /dev/null
@@ -1 +0,0 @@
-program RIGHT
diff --git a/t/t4018/fortran.sh b/t/t4018/fortran.sh
new file mode 100755
index 0000000000..7b0c6789d3
--- /dev/null
+++ b/t/t4018/fortran.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'fortran: block data' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+BLOCK DATA RIGHT
+EOF_HUNK
+       BLOCK DATA RIGHT
+       
+       COMMON /B/ C, ChangeMe
+       DATA C, ChangeMe  / 2.0, 6.0 / 
+       END 
+EOF_TEST
+
+test_diff_funcname 'fortran: comment' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+      module a
+
+      contains
+
+      ! subroutine wrong
+      subroutine RIGHT
+      ! subroutine wrong
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: comment keyword' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT (funcA, funcB)
+EOF_HUNK
+      module a
+
+      contains
+
+      subroutine RIGHT (funcA, funcB)
+
+      real funcA  ! grid function a
+      real funcB  ! grid function b
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: comment legacy' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+      module a
+
+      contains
+
+C subroutine wrong
+      subroutine RIGHT
+C subroutine wrong
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: comment legacy star' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+      module a
+
+      contains
+
+* subroutine wrong
+      subroutine RIGHT
+* subroutine wrong
+
+      real ChangeMe
+
+      end subroutine RIGHT
+
+      end module a
+EOF_TEST
+
+test_diff_funcname 'fortran: external function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT(a, b) result(c)
+EOF_HUNK
+function RIGHT(a, b) result(c)
+
+integer, intent(in) :: ChangeMe
+integer, intent(in) :: b
+integer, intent(out) :: c
+
+c = a+b
+
+end function RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: external subroutine' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+subroutine RIGHT
+EOF_HUNK
+subroutine RIGHT
+
+real ChangeMe
+
+end subroutine RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: module' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+module RIGHT
+EOF_HUNK
+module RIGHT
+
+use ChangeMe
+
+end module RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: module procedure' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+module RIGHT
+EOF_HUNK
+ module RIGHT
+
+   implicit none
+   private
+
+   interface letters  ! generic interface
+      module procedure aaaa, &
+                       bbbb, &
+                       ChangeMe, &
+                       dddd
+   end interface
+   
+end module RIGHT
+EOF_TEST
+
+test_diff_funcname 'fortran: program' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+program RIGHT
+EOF_HUNK
+program RIGHT
+
+call ChangeMe
+
+end program RIGHT
+EOF_TEST
diff --git a/t/t4018/fountain-scene b/t/t4018/fountain-scene
deleted file mode 100644
index 6b3257d680..0000000000
--- a/t/t4018/fountain-scene
+++ /dev/null
@@ -1,4 +0,0 @@
-EXT. STREET RIGHT OUTSIDE - DAY
-
-CHARACTER
-You didn't say the magic phrase, "ChangeMe".
diff --git a/t/t4018/fountain-scene.ctx b/t/t4018/fountain-scene.ctx
deleted file mode 100644
index bf10171418..0000000000
--- a/t/t4018/fountain-scene.ctx
+++ /dev/null
@@ -1 +0,0 @@
-EXT. STREET RIGHT OUTSIDE - DAY
diff --git a/t/t4018/fountain.sh b/t/t4018/fountain.sh
new file mode 100755
index 0000000000..02b44d6a3f
--- /dev/null
+++ b/t/t4018/fountain.sh
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'fountain: scene' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+EXT. STREET RIGHT OUTSIDE - DAY
+EOF_HUNK
+EXT. STREET RIGHT OUTSIDE - DAY
+
+CHARACTER
+You didn't say the magic phrase, "ChangeMe".
+EOF_TEST
diff --git a/t/t4018/golang-complex-function b/t/t4018/golang-complex-function
deleted file mode 100644
index e057dcefed..0000000000
--- a/t/t4018/golang-complex-function
+++ /dev/null
@@ -1,8 +0,0 @@
-type Test struct {
-	a Type
-}
-
-func (t *Test) RIGHT(a Type) (Type, error) {
-	t.a = a
-	return ChangeMe, nil
-}
diff --git a/t/t4018/golang-complex-function.ctx b/t/t4018/golang-complex-function.ctx
deleted file mode 100644
index 8e8d5582ff..0000000000
--- a/t/t4018/golang-complex-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-func (t *Test) RIGHT(a Type) (Type, error) {
diff --git a/t/t4018/golang-func b/t/t4018/golang-func
deleted file mode 100644
index 8e9c9ac7c3..0000000000
--- a/t/t4018/golang-func
+++ /dev/null
@@ -1,4 +0,0 @@
-func RIGHT() {
-	a := 5
-	b := ChangeMe
-}
diff --git a/t/t4018/golang-func.ctx b/t/t4018/golang-func.ctx
deleted file mode 100644
index 88bc823813..0000000000
--- a/t/t4018/golang-func.ctx
+++ /dev/null
@@ -1 +0,0 @@
-func RIGHT() {
diff --git a/t/t4018/golang-interface b/t/t4018/golang-interface
deleted file mode 100644
index 553bedec96..0000000000
--- a/t/t4018/golang-interface
+++ /dev/null
@@ -1,4 +0,0 @@
-type RIGHT interface {
-	a() Type
-	b() ChangeMe
-}
diff --git a/t/t4018/golang-interface.ctx b/t/t4018/golang-interface.ctx
deleted file mode 100644
index 2d07f5a383..0000000000
--- a/t/t4018/golang-interface.ctx
+++ /dev/null
@@ -1 +0,0 @@
-type RIGHT interface {
diff --git a/t/t4018/golang-long-func b/t/t4018/golang-long-func
deleted file mode 100644
index ac3a77b5c4..0000000000
--- a/t/t4018/golang-long-func
+++ /dev/null
@@ -1,5 +0,0 @@
-func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
-	anotherLongVariableName AnotherLongType) {
-	a := 5
-	b := ChangeMe
-}
diff --git a/t/t4018/golang-long-func.ctx b/t/t4018/golang-long-func.ctx
deleted file mode 100644
index 25635e712e..0000000000
--- a/t/t4018/golang-long-func.ctx
+++ /dev/null
@@ -1 +0,0 @@
-func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
diff --git a/t/t4018/golang-struct b/t/t4018/golang-struct
deleted file mode 100644
index 5deda77fee..0000000000
--- a/t/t4018/golang-struct
+++ /dev/null
@@ -1,4 +0,0 @@
-type RIGHT struct {
-	a Type
-	b ChangeMe
-}
diff --git a/t/t4018/golang-struct.ctx b/t/t4018/golang-struct.ctx
deleted file mode 100644
index 8a1240699d..0000000000
--- a/t/t4018/golang-struct.ctx
+++ /dev/null
@@ -1 +0,0 @@
-type RIGHT struct {
diff --git a/t/t4018/golang.sh b/t/t4018/golang.sh
new file mode 100755
index 0000000000..bf22f58c12
--- /dev/null
+++ b/t/t4018/golang.sh
@@ -0,0 +1,59 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'golang: complex function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+func (t *Test) RIGHT(a Type) (Type, error) {
+EOF_HUNK
+type Test struct {
+	a Type
+}
+
+func (t *Test) RIGHT(a Type) (Type, error) {
+	t.a = a
+	return ChangeMe, nil
+}
+EOF_TEST
+
+test_diff_funcname 'golang: func' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+func RIGHT() {
+EOF_HUNK
+func RIGHT() {
+	a := 5
+	b := ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'golang: interface' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+type RIGHT interface {
+EOF_HUNK
+type RIGHT interface {
+	a() Type
+	b() ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'golang: long func' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
+EOF_HUNK
+func RIGHT(aVeryVeryVeryLongVariableName AVeryVeryVeryLongType,
+	anotherLongVariableName AnotherLongType) {
+	a := 5
+	b := ChangeMe
+}
+EOF_TEST
+
+test_diff_funcname 'golang: struct' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+type RIGHT struct {
+EOF_HUNK
+type RIGHT struct {
+	a Type
+	b ChangeMe
+}
+EOF_TEST
diff --git a/t/t4018/java-class-member-function b/t/t4018/java-class-member-function
deleted file mode 100644
index 298bc7a71b..0000000000
--- a/t/t4018/java-class-member-function
+++ /dev/null
@@ -1,8 +0,0 @@
-public class Beer
-{
-	int special;
-	public static void main(String RIGHT[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
diff --git a/t/t4018/java-class-member-function.ctx b/t/t4018/java-class-member-function.ctx
deleted file mode 100644
index 2125474b68..0000000000
--- a/t/t4018/java-class-member-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-public static void main(String RIGHT[])
diff --git a/t/t4018/java.sh b/t/t4018/java.sh
new file mode 100755
index 0000000000..c89cf2f9d8
--- /dev/null
+++ b/t/t4018/java.sh
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'java: class member function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
+public class Beer
+{
+	int special;
+	public static void main(String RIGHT[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
diff --git a/t/t4018/markdown-heading-indented b/t/t4018/markdown-heading-indented
deleted file mode 100644
index 1991c2bd45..0000000000
--- a/t/t4018/markdown-heading-indented
+++ /dev/null
@@ -1,6 +0,0 @@
-Indented headings are allowed, as long as the indent is no more than 3 spaces.
-
-   ### RIGHT
-
-- something
-- ChangeMe
diff --git a/t/t4018/markdown-heading-indented.ctx b/t/t4018/markdown-heading-indented.ctx
deleted file mode 100644
index 5938336743..0000000000
--- a/t/t4018/markdown-heading-indented.ctx
+++ /dev/null
@@ -1 +0,0 @@
-   ### RIGHT
diff --git a/t/t4018/markdown-heading-non-headings b/t/t4018/markdown-heading-non-headings
deleted file mode 100644
index c479c1a3f1..0000000000
--- a/t/t4018/markdown-heading-non-headings
+++ /dev/null
@@ -1,17 +0,0 @@
-Headings can be right next to other lines of the file:
-# RIGHT
-Indents of four or more spaces make a code block:
-
-    # code comment, not heading
-
-If there's no space after the final hash, it's not a heading:
-
-#hashtag
-
-Sequences of more than 6 hashes don't make a heading:
-
-####### over-enthusiastic heading
-
-So the detected heading should be right up at the start of this file.
-
-ChangeMe
diff --git a/t/t4018/markdown-heading-non-headings.ctx b/t/t4018/markdown-heading-non-headings.ctx
deleted file mode 100644
index 7e2165be6e..0000000000
--- a/t/t4018/markdown-heading-non-headings.ctx
+++ /dev/null
@@ -1 +0,0 @@
-# RIGHT
diff --git a/t/t4018/markdown.sh b/t/t4018/markdown.sh
new file mode 100755
index 0000000000..3e1c79b139
--- /dev/null
+++ b/t/t4018/markdown.sh
@@ -0,0 +1,39 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'markdown: heading indented' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+   ### RIGHT
+EOF_HUNK
+Indented headings are allowed, as long as the indent is no more than 3 spaces.
+
+   ### RIGHT
+
+- something
+- ChangeMe
+EOF_TEST
+
+test_diff_funcname 'markdown: heading non headings' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+# RIGHT
+EOF_HUNK
+Headings can be right next to other lines of the file:
+# RIGHT
+Indents of four or more spaces make a code block:
+
+    # code comment, not heading
+
+If there's no space after the final hash, it's not a heading:
+
+#hashtag
+
+Sequences of more than 6 hashes don't make a heading:
+
+####### over-enthusiastic heading
+
+So the detected heading should be right up at the start of this file.
+
+ChangeMe
+EOF_TEST
diff --git a/t/t4018/matlab-class-definition b/t/t4018/matlab-class-definition
deleted file mode 100644
index 84daedfb4e..0000000000
--- a/t/t4018/matlab-class-definition
+++ /dev/null
@@ -1,5 +0,0 @@
-classdef RIGHT
-    properties
-        ChangeMe
-    end
-end
diff --git a/t/t4018/matlab-class-definition.ctx b/t/t4018/matlab-class-definition.ctx
deleted file mode 100644
index 5dd5b45628..0000000000
--- a/t/t4018/matlab-class-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-classdef RIGHT
diff --git a/t/t4018/matlab-function b/t/t4018/matlab-function
deleted file mode 100644
index 897a9b13ff..0000000000
--- a/t/t4018/matlab-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function y = RIGHT()
-x = 5;
-y = ChangeMe + x;
-end
diff --git a/t/t4018/matlab-function.ctx b/t/t4018/matlab-function.ctx
deleted file mode 100644
index 72d2350b13..0000000000
--- a/t/t4018/matlab-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function y = RIGHT()
diff --git a/t/t4018/matlab-octave-section-1 b/t/t4018/matlab-octave-section-1
deleted file mode 100644
index 3bb6c4670e..0000000000
--- a/t/t4018/matlab-octave-section-1
+++ /dev/null
@@ -1,3 +0,0 @@
-%%% RIGHT section
-# this is octave script
-ChangeMe = 1;
diff --git a/t/t4018/matlab-octave-section-1.ctx b/t/t4018/matlab-octave-section-1.ctx
deleted file mode 100644
index ca9b349f94..0000000000
--- a/t/t4018/matlab-octave-section-1.ctx
+++ /dev/null
@@ -1 +0,0 @@
-%%% RIGHT section
diff --git a/t/t4018/matlab-octave-section-2 b/t/t4018/matlab-octave-section-2
deleted file mode 100644
index ab2980f7f2..0000000000
--- a/t/t4018/matlab-octave-section-2
+++ /dev/null
@@ -1,3 +0,0 @@
-## RIGHT section
-# this is octave script
-ChangeMe = 1;
diff --git a/t/t4018/matlab-octave-section-2.ctx b/t/t4018/matlab-octave-section-2.ctx
deleted file mode 100644
index 5cbb77faf5..0000000000
--- a/t/t4018/matlab-octave-section-2.ctx
+++ /dev/null
@@ -1 +0,0 @@
-## RIGHT section
diff --git a/t/t4018/matlab-section b/t/t4018/matlab-section
deleted file mode 100644
index 5ea59a5de0..0000000000
--- a/t/t4018/matlab-section
+++ /dev/null
@@ -1,3 +0,0 @@
-%% RIGHT section
-% this is understood by both matlab and octave
-ChangeMe = 1;
diff --git a/t/t4018/matlab-section.ctx b/t/t4018/matlab-section.ctx
deleted file mode 100644
index e83fee6f4d..0000000000
--- a/t/t4018/matlab-section.ctx
+++ /dev/null
@@ -1 +0,0 @@
-%% RIGHT section
diff --git a/t/t4018/matlab.sh b/t/t4018/matlab.sh
new file mode 100755
index 0000000000..f62289148e
--- /dev/null
+++ b/t/t4018/matlab.sh
@@ -0,0 +1,52 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'matlab: class definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+classdef RIGHT
+EOF_HUNK
+classdef RIGHT
+    properties
+        ChangeMe
+    end
+end
+EOF_TEST
+
+test_diff_funcname 'matlab: function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function y = RIGHT()
+EOF_HUNK
+function y = RIGHT()
+x = 5;
+y = ChangeMe + x;
+end
+EOF_TEST
+
+test_diff_funcname 'matlab: octave section 1' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+%%% RIGHT section
+EOF_HUNK
+%%% RIGHT section
+# this is octave script
+ChangeMe = 1;
+EOF_TEST
+
+test_diff_funcname 'matlab: octave section 2' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+## RIGHT section
+EOF_HUNK
+## RIGHT section
+# this is octave script
+ChangeMe = 1;
+EOF_TEST
+
+test_diff_funcname 'matlab: section' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+%% RIGHT section
+EOF_HUNK
+%% RIGHT section
+% this is understood by both matlab and octave
+ChangeMe = 1;
+EOF_TEST
diff --git a/t/t4018/perl-skip-end-of-heredoc b/t/t4018/perl-skip-end-of-heredoc
deleted file mode 100644
index c22d39b256..0000000000
--- a/t/t4018/perl-skip-end-of-heredoc
+++ /dev/null
@@ -1,8 +0,0 @@
-sub RIGHTwithheredocument {
-	print <<"EOF"
-decoy here-doc
-EOF
-	# some lines of context
-	# to pad it out
-	print "ChangeMe\n";
-}
diff --git a/t/t4018/perl-skip-end-of-heredoc.ctx b/t/t4018/perl-skip-end-of-heredoc.ctx
deleted file mode 100644
index c15f4b78bd..0000000000
--- a/t/t4018/perl-skip-end-of-heredoc.ctx
+++ /dev/null
@@ -1 +0,0 @@
-sub RIGHTwithheredocument {
diff --git a/t/t4018/perl-skip-forward-decl b/t/t4018/perl-skip-forward-decl
deleted file mode 100644
index a98cb8bdad..0000000000
--- a/t/t4018/perl-skip-forward-decl
+++ /dev/null
@@ -1,10 +0,0 @@
-package RIGHT;
-
-use strict;
-use warnings;
-use parent qw(Exporter);
-our @EXPORT_OK = qw(round finalround);
-
-sub other; # forward declaration
-
-# ChangeMe
diff --git a/t/t4018/perl-skip-forward-decl.ctx b/t/t4018/perl-skip-forward-decl.ctx
deleted file mode 100644
index e0c51599ad..0000000000
--- a/t/t4018/perl-skip-forward-decl.ctx
+++ /dev/null
@@ -1 +0,0 @@
-package RIGHT;
diff --git a/t/t4018/perl-skip-sub-in-pod b/t/t4018/perl-skip-sub-in-pod
deleted file mode 100644
index e39f02462e..0000000000
--- a/t/t4018/perl-skip-sub-in-pod
+++ /dev/null
@@ -1,18 +0,0 @@
-=head1 NAME
-
-Beer - subroutine to output fragment of a drinking song
-
-=head1 SYNOPSIS_RIGHT
-
-	use Beer qw(round finalround);
-
-	sub song {
-		for (my $i = 99; $i > 0; $i--) {
-			round $i;
-		}
-		finalround;
-	}
-
-	ChangeMe;
-
-=cut
diff --git a/t/t4018/perl-skip-sub-in-pod.ctx b/t/t4018/perl-skip-sub-in-pod.ctx
deleted file mode 100644
index abddd76655..0000000000
--- a/t/t4018/perl-skip-sub-in-pod.ctx
+++ /dev/null
@@ -1 +0,0 @@
-=head1 SYNOPSIS_RIGHT
diff --git a/t/t4018/perl-sub-definition b/t/t4018/perl-sub-definition
deleted file mode 100644
index a507d1f645..0000000000
--- a/t/t4018/perl-sub-definition
+++ /dev/null
@@ -1,4 +0,0 @@
-sub RIGHT {
-	my ($n) = @_;
-	print "ChangeMe";
-}
diff --git a/t/t4018/perl-sub-definition-kr-brace b/t/t4018/perl-sub-definition-kr-brace
deleted file mode 100644
index 330b3df114..0000000000
--- a/t/t4018/perl-sub-definition-kr-brace
+++ /dev/null
@@ -1,4 +0,0 @@
-sub RIGHT
-{
-	print "ChangeMe\n";
-}
diff --git a/t/t4018/perl-sub-definition-kr-brace.ctx b/t/t4018/perl-sub-definition-kr-brace.ctx
deleted file mode 100644
index 7e5aee5cde..0000000000
--- a/t/t4018/perl-sub-definition-kr-brace.ctx
+++ /dev/null
@@ -1 +0,0 @@
-sub RIGHT
diff --git a/t/t4018/perl-sub-definition.ctx b/t/t4018/perl-sub-definition.ctx
deleted file mode 100644
index d49a63598e..0000000000
--- a/t/t4018/perl-sub-definition.ctx
+++ /dev/null
@@ -1 +0,0 @@
-sub RIGHT {
diff --git a/t/t4018/perl.sh b/t/t4018/perl.sh
new file mode 100755
index 0000000000..ac8fff7417
--- /dev/null
+++ b/t/t4018/perl.sh
@@ -0,0 +1,78 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'perl: skip end of heredoc' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHTwithheredocument {
+EOF_HUNK
+sub RIGHTwithheredocument {
+	print <<"EOF"
+decoy here-doc
+EOF
+	# some lines of context
+	# to pad it out
+	print "ChangeMe\n";
+}
+EOF_TEST
+
+test_diff_funcname 'perl: skip forward decl' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+package RIGHT;
+EOF_HUNK
+package RIGHT;
+
+use strict;
+use warnings;
+use parent qw(Exporter);
+our @EXPORT_OK = qw(round finalround);
+
+sub other; # forward declaration
+
+# ChangeMe
+EOF_TEST
+
+test_diff_funcname 'perl: skip sub in pod' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+=head1 SYNOPSIS_RIGHT
+EOF_HUNK
+=head1 NAME
+
+Beer - subroutine to output fragment of a drinking song
+
+=head1 SYNOPSIS_RIGHT
+
+	use Beer qw(round finalround);
+
+	sub song {
+		for (my $i = 99; $i > 0; $i--) {
+			round $i;
+		}
+		finalround;
+	}
+
+	ChangeMe;
+
+=cut
+EOF_TEST
+
+test_diff_funcname 'perl: sub definition' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHT {
+EOF_HUNK
+sub RIGHT {
+	my ($n) = @_;
+	print "ChangeMe";
+}
+EOF_TEST
+
+test_diff_funcname 'perl: sub definition kr brace' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+sub RIGHT
+EOF_HUNK
+sub RIGHT
+{
+	print "ChangeMe\n";
+}
+EOF_TEST
diff --git a/t/t4018/php-abstract-class b/t/t4018/php-abstract-class
deleted file mode 100644
index 5213e12494..0000000000
--- a/t/t4018/php-abstract-class
+++ /dev/null
@@ -1,4 +0,0 @@
-abstract class RIGHT
-{
-    const FOO = 'ChangeMe';
-}
diff --git a/t/t4018/php-abstract-class.ctx b/t/t4018/php-abstract-class.ctx
deleted file mode 100644
index f572d2129b..0000000000
--- a/t/t4018/php-abstract-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-abstract class RIGHT
diff --git a/t/t4018/php-abstract-method b/t/t4018/php-abstract-method
deleted file mode 100644
index ce215df75a..0000000000
--- a/t/t4018/php-abstract-method
+++ /dev/null
@@ -1,7 +0,0 @@
-abstract class Klass
-{
-    abstract public function RIGHT(): ?string
-    {
-        return 'ChangeMe';
-    }
-}
diff --git a/t/t4018/php-abstract-method.ctx b/t/t4018/php-abstract-method.ctx
deleted file mode 100644
index 14cb6df42e..0000000000
--- a/t/t4018/php-abstract-method.ctx
+++ /dev/null
@@ -1 +0,0 @@
-abstract public function RIGHT(): ?string
diff --git a/t/t4018/php-class b/t/t4018/php-class
deleted file mode 100644
index 7785b6303c..0000000000
--- a/t/t4018/php-class
+++ /dev/null
@@ -1,4 +0,0 @@
-class RIGHT
-{
-    const FOO = 'ChangeMe';
-}
diff --git a/t/t4018/php-class.ctx b/t/t4018/php-class.ctx
deleted file mode 100644
index 54bff816d6..0000000000
--- a/t/t4018/php-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT
diff --git a/t/t4018/php-final-class b/t/t4018/php-final-class
deleted file mode 100644
index 69f5710552..0000000000
--- a/t/t4018/php-final-class
+++ /dev/null
@@ -1,4 +0,0 @@
-final class RIGHT
-{
-    const FOO = 'ChangeMe';
-}
diff --git a/t/t4018/php-final-class.ctx b/t/t4018/php-final-class.ctx
deleted file mode 100644
index 4d59fb749b..0000000000
--- a/t/t4018/php-final-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-final class RIGHT
diff --git a/t/t4018/php-final-method b/t/t4018/php-final-method
deleted file mode 100644
index 537fb8ad9a..0000000000
--- a/t/t4018/php-final-method
+++ /dev/null
@@ -1,7 +0,0 @@
-class Klass
-{
-    final public function RIGHT(): string
-    {
-        return 'ChangeMe';
-    }
-}
diff --git a/t/t4018/php-final-method.ctx b/t/t4018/php-final-method.ctx
deleted file mode 100644
index b7da8f8082..0000000000
--- a/t/t4018/php-final-method.ctx
+++ /dev/null
@@ -1 +0,0 @@
-final public function RIGHT(): string
diff --git a/t/t4018/php-function b/t/t4018/php-function
deleted file mode 100644
index 35717c51c3..0000000000
--- a/t/t4018/php-function
+++ /dev/null
@@ -1,4 +0,0 @@
-function RIGHT()
-{
-    return 'ChangeMe';
-}
diff --git a/t/t4018/php-function.ctx b/t/t4018/php-function.ctx
deleted file mode 100644
index c5f3e55302..0000000000
--- a/t/t4018/php-function.ctx
+++ /dev/null
@@ -1 +0,0 @@
-function RIGHT()
diff --git a/t/t4018/php-interface b/t/t4018/php-interface
deleted file mode 100644
index 86b49ad5d9..0000000000
--- a/t/t4018/php-interface
+++ /dev/null
@@ -1,4 +0,0 @@
-interface RIGHT
-{
-    public function foo($ChangeMe);
-}
diff --git a/t/t4018/php-interface.ctx b/t/t4018/php-interface.ctx
deleted file mode 100644
index a45fa0532a..0000000000
--- a/t/t4018/php-interface.ctx
+++ /dev/null
@@ -1 +0,0 @@
-interface RIGHT
diff --git a/t/t4018/php-method b/t/t4018/php-method
deleted file mode 100644
index 03af1a6d9d..0000000000
--- a/t/t4018/php-method
+++ /dev/null
@@ -1,7 +0,0 @@
-class Klass
-{
-    public static function RIGHT()
-    {
-        return 'ChangeMe';
-    }
-}
diff --git a/t/t4018/php-method.ctx b/t/t4018/php-method.ctx
deleted file mode 100644
index eb1659ff9f..0000000000
--- a/t/t4018/php-method.ctx
+++ /dev/null
@@ -1 +0,0 @@
-public static function RIGHT()
diff --git a/t/t4018/php-trait b/t/t4018/php-trait
deleted file mode 100644
index 65b8c82a61..0000000000
--- a/t/t4018/php-trait
+++ /dev/null
@@ -1,7 +0,0 @@
-trait RIGHT
-{
-    public function foo($ChangeMe)
-    {
-        return 'foo';
-    }
-}
diff --git a/t/t4018/php-trait.ctx b/t/t4018/php-trait.ctx
deleted file mode 100644
index 57aa4c6267..0000000000
--- a/t/t4018/php-trait.ctx
+++ /dev/null
@@ -1 +0,0 @@
-trait RIGHT
diff --git a/t/t4018/php.sh b/t/t4018/php.sh
new file mode 100755
index 0000000000..e0ccb2277b
--- /dev/null
+++ b/t/t4018/php.sh
@@ -0,0 +1,106 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'php: abstract class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+abstract class RIGHT
+EOF_HUNK
+abstract class RIGHT
+{
+    const FOO = 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: abstract method' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+abstract public function RIGHT(): ?string
+EOF_HUNK
+abstract class Klass
+{
+    abstract public function RIGHT(): ?string
+    {
+        return 'ChangeMe';
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'php: class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT
+EOF_HUNK
+class RIGHT
+{
+    const FOO = 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: final class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+final class RIGHT
+EOF_HUNK
+final class RIGHT
+{
+    const FOO = 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: final method' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+final public function RIGHT(): string
+EOF_HUNK
+class Klass
+{
+    final public function RIGHT(): string
+    {
+        return 'ChangeMe';
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'php: function' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+function RIGHT()
+EOF_HUNK
+function RIGHT()
+{
+    return 'ChangeMe';
+}
+EOF_TEST
+
+test_diff_funcname 'php: interface' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+interface RIGHT
+EOF_HUNK
+interface RIGHT
+{
+    public function foo($ChangeMe);
+}
+EOF_TEST
+
+test_diff_funcname 'php: method' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static function RIGHT()
+EOF_HUNK
+class Klass
+{
+    public static function RIGHT()
+    {
+        return 'ChangeMe';
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'php: trait' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+trait RIGHT
+EOF_HUNK
+trait RIGHT
+{
+    public function foo($ChangeMe)
+    {
+        return 'foo';
+    }
+}
+EOF_TEST
diff --git a/t/t4018/python-async-def b/t/t4018/python-async-def
deleted file mode 100644
index 87640e03d2..0000000000
--- a/t/t4018/python-async-def
+++ /dev/null
@@ -1,4 +0,0 @@
-async def RIGHT(pi: int = 3.14):
-    while True:
-        break
-    return ChangeMe()
diff --git a/t/t4018/python-async-def.ctx b/t/t4018/python-async-def.ctx
deleted file mode 100644
index 468c548bbe..0000000000
--- a/t/t4018/python-async-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-async def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-class b/t/t4018/python-class
deleted file mode 100644
index ba9e741430..0000000000
--- a/t/t4018/python-class
+++ /dev/null
@@ -1,4 +0,0 @@
-class RIGHT(int, str):
-    # comment
-    # another comment
-    # ChangeMe
diff --git a/t/t4018/python-class.ctx b/t/t4018/python-class.ctx
deleted file mode 100644
index a40b755e29..0000000000
--- a/t/t4018/python-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT(int, str):
diff --git a/t/t4018/python-def b/t/t4018/python-def
deleted file mode 100644
index e50b31b0ad..0000000000
--- a/t/t4018/python-def
+++ /dev/null
@@ -1,4 +0,0 @@
-def RIGHT(pi: int = 3.14):
-    while True:
-        break
-    return ChangeMe()
diff --git a/t/t4018/python-def.ctx b/t/t4018/python-def.ctx
deleted file mode 100644
index a1a9cbad63..0000000000
--- a/t/t4018/python-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def RIGHT(pi: int = 3.14):
diff --git a/t/t4018/python-indented-async-def b/t/t4018/python-indented-async-def
deleted file mode 100644
index f5d03258af..0000000000
--- a/t/t4018/python-indented-async-def
+++ /dev/null
@@ -1,7 +0,0 @@
-class Foo:
-    async def RIGHT(self, x: int):
-        return [
-            1,
-            2,
-            ChangeMe,
-        ]
diff --git a/t/t4018/python-indented-async-def.ctx b/t/t4018/python-indented-async-def.ctx
deleted file mode 100644
index d393620a1e..0000000000
--- a/t/t4018/python-indented-async-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-async def RIGHT(self, x: int):
diff --git a/t/t4018/python-indented-class b/t/t4018/python-indented-class
deleted file mode 100644
index 19b4f35c4c..0000000000
--- a/t/t4018/python-indented-class
+++ /dev/null
@@ -1,5 +0,0 @@
-if TYPE_CHECKING:
-    class RIGHT:
-        # comment
-        # another comment
-        # ChangeMe
diff --git a/t/t4018/python-indented-class.ctx b/t/t4018/python-indented-class.ctx
deleted file mode 100644
index 0881c84dba..0000000000
--- a/t/t4018/python-indented-class.ctx
+++ /dev/null
@@ -1 +0,0 @@
-class RIGHT:
diff --git a/t/t4018/python-indented-def b/t/t4018/python-indented-def
deleted file mode 100644
index 208fbadd2b..0000000000
--- a/t/t4018/python-indented-def
+++ /dev/null
@@ -1,7 +0,0 @@
-class Foo:
-    def RIGHT(self, x: int):
-        return [
-            1,
-            2,
-            ChangeMe,
-        ]
diff --git a/t/t4018/python-indented-def.ctx b/t/t4018/python-indented-def.ctx
deleted file mode 100644
index 6e5a44b391..0000000000
--- a/t/t4018/python-indented-def.ctx
+++ /dev/null
@@ -1 +0,0 @@
-def RIGHT(self, x: int):
diff --git a/t/t4018/python.sh b/t/t4018/python.sh
new file mode 100755
index 0000000000..ecb5736d57
--- /dev/null
+++ b/t/t4018/python.sh
@@ -0,0 +1,71 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'python: async def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+async def RIGHT(pi: int = 3.14):
+EOF_HUNK
+async def RIGHT(pi: int = 3.14):
+    while True:
+        break
+    return ChangeMe()
+EOF_TEST
+
+test_diff_funcname 'python: class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT(int, str):
+EOF_HUNK
+class RIGHT(int, str):
+    # comment
+    # another comment
+    # ChangeMe
+EOF_TEST
+
+test_diff_funcname 'python: def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def RIGHT(pi: int = 3.14):
+EOF_HUNK
+def RIGHT(pi: int = 3.14):
+    while True:
+        break
+    return ChangeMe()
+EOF_TEST
+
+test_diff_funcname 'python: indented async def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+async def RIGHT(self, x: int):
+EOF_HUNK
+class Foo:
+    async def RIGHT(self, x: int):
+        return [
+            1,
+            2,
+            ChangeMe,
+        ]
+EOF_TEST
+
+test_diff_funcname 'python: indented class' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+class RIGHT:
+EOF_HUNK
+if TYPE_CHECKING:
+    class RIGHT:
+        # comment
+        # another comment
+        # ChangeMe
+EOF_TEST
+
+test_diff_funcname 'python: indented def' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+def RIGHT(self, x: int):
+EOF_HUNK
+class Foo:
+    def RIGHT(self, x: int):
+        return [
+            1,
+            2,
+            ChangeMe,
+        ]
+EOF_TEST
diff --git a/t/t4018/rust-fn b/t/t4018/rust-fn
deleted file mode 100644
index cbe02155f1..0000000000
--- a/t/t4018/rust-fn
+++ /dev/null
@@ -1,5 +0,0 @@
-pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
-    let _ = x;
-    // a comment
-    let a = ChangeMe;
-}
diff --git a/t/t4018/rust-fn.ctx b/t/t4018/rust-fn.ctx
deleted file mode 100644
index baa37cf253..0000000000
--- a/t/t4018/rust-fn.ctx
+++ /dev/null
@@ -1 +0,0 @@
-pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
diff --git a/t/t4018/rust-impl b/t/t4018/rust-impl
deleted file mode 100644
index 09df3cd93b..0000000000
--- a/t/t4018/rust-impl
+++ /dev/null
@@ -1,5 +0,0 @@
-impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
-
-    pub fn ChangeMe(&self) -> () {
-    }
-}
diff --git a/t/t4018/rust-impl.ctx b/t/t4018/rust-impl.ctx
deleted file mode 100644
index 5344c35f3f..0000000000
--- a/t/t4018/rust-impl.ctx
+++ /dev/null
@@ -1 +0,0 @@
-impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
diff --git a/t/t4018/rust-macro-rules b/t/t4018/rust-macro-rules
deleted file mode 100644
index ec610c5b62..0000000000
--- a/t/t4018/rust-macro-rules
+++ /dev/null
@@ -1,6 +0,0 @@
-macro_rules! RIGHT {
-    () => {
-        // a comment
-        let x = ChangeMe;
-    };
-}
diff --git a/t/t4018/rust-macro-rules.ctx b/t/t4018/rust-macro-rules.ctx
deleted file mode 100644
index 7520463aa0..0000000000
--- a/t/t4018/rust-macro-rules.ctx
+++ /dev/null
@@ -1 +0,0 @@
-macro_rules! RIGHT {
diff --git a/t/t4018/rust-struct b/t/t4018/rust-struct
deleted file mode 100644
index 76aff1c0d8..0000000000
--- a/t/t4018/rust-struct
+++ /dev/null
@@ -1,5 +0,0 @@
-#[derive(Debug)]
-pub(super) struct RIGHT<'a> {
-    name: &'a str,
-    age: ChangeMe,
-}
diff --git a/t/t4018/rust-struct.ctx b/t/t4018/rust-struct.ctx
deleted file mode 100644
index c1e09dc808..0000000000
--- a/t/t4018/rust-struct.ctx
+++ /dev/null
@@ -1 +0,0 @@
-pub(super) struct RIGHT<'a> {
diff --git a/t/t4018/rust-trait b/t/t4018/rust-trait
deleted file mode 100644
index ea397f09ed..0000000000
--- a/t/t4018/rust-trait
+++ /dev/null
@@ -1,5 +0,0 @@
-unsafe trait RIGHT<T> {
-    fn len(&self) -> u32;
-    fn ChangeMe(&self, n: u32) -> T;
-    fn iter<F>(&self, f: F) where F: Fn(T);
-}
diff --git a/t/t4018/rust-trait.ctx b/t/t4018/rust-trait.ctx
deleted file mode 100644
index 6af803db29..0000000000
--- a/t/t4018/rust-trait.ctx
+++ /dev/null
@@ -1 +0,0 @@
-unsafe trait RIGHT<T> {
diff --git a/t/t4018/rust.sh b/t/t4018/rust.sh
new file mode 100755
index 0000000000..ba018c6b95
--- /dev/null
+++ b/t/t4018/rust.sh
@@ -0,0 +1,60 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_diff_funcname 'rust: fn' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
+EOF_HUNK
+pub(self) fn RIGHT<T>(x: &[T]) where T: Debug {
+    let _ = x;
+    // a comment
+    let a = ChangeMe;
+}
+EOF_TEST
+
+test_diff_funcname 'rust: impl' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
+EOF_HUNK
+impl<'a, T: AsRef<[u8]>>  std::RIGHT for Git<'a> {
+
+    pub fn ChangeMe(&self) -> () {
+    }
+}
+EOF_TEST
+
+test_diff_funcname 'rust: macro rules' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+macro_rules! RIGHT {
+EOF_HUNK
+macro_rules! RIGHT {
+    () => {
+        // a comment
+        let x = ChangeMe;
+    };
+}
+EOF_TEST
+
+test_diff_funcname 'rust: struct' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+pub(super) struct RIGHT<'a> {
+EOF_HUNK
+#[derive(Debug)]
+pub(super) struct RIGHT<'a> {
+    name: &'a str,
+    age: ChangeMe,
+}
+EOF_TEST
+
+test_diff_funcname 'rust: trait' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+unsafe trait RIGHT<T> {
+EOF_HUNK
+unsafe trait RIGHT<T> {
+    fn len(&self) -> u32;
+    fn ChangeMe(&self, n: u32) -> T;
+    fn iter<F>(&self, f: F) where F: Fn(T);
+}
+EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 09/20] blame tests: don't rely on t/t4018/ directory

[Ævar Arnfjörð Bjarmason]

Refactor a test added in 9466e3809d (blame: enable funcname blaming
with userdiff driver, 2020-11-01) so that the blame tests don't rely
on stealing the contents of "t/t4018/fortran-external-function".

I'm about to refactor that directory, just moving the relevant test
file here inline is the easiest solution, and I think also the most
readable.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/annotate-tests.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/annotate-tests.sh b/t/annotate-tests.sh
index 29ce89090d..04a2c58594 100644
--- a/t/annotate-tests.sh
+++ b/t/annotate-tests.sh
@@ -482,12 +482,22 @@ test_expect_success 'blame -L ^:RE (absolute: end-of-file)' '
 test_expect_success 'setup -L :funcname with userdiff driver' '
 	echo "fortran-* diff=fortran" >.gitattributes &&
 	fortran_file=fortran-external-function &&
-	orig_file="$TEST_DIRECTORY/t4018/$fortran_file" &&
-	cp "$orig_file" . &&
+	cat >$fortran_file <<-\EOF &&
+	function RIGHT(a, b) result(c)
+
+	integer, intent(in) :: ChangeMe
+	integer, intent(in) :: b
+	integer, intent(out) :: c
+
+	c = a+b
+
+	end function RIGHT
+	EOF
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="A" GIT_AUTHOR_EMAIL="A@test.git" \
 	git commit -m "add fortran file" &&
-	sed -e "s/ChangeMe/IWasChanged/" <"$orig_file" >"$fortran_file" &&
+	sed -e "s/ChangeMe/IWasChanged/" <"$fortran_file" >"$fortran_file".tmp &&
+	mv "$fortran_file".tmp "$fortran_file" &&
 	git add "$fortran_file" &&
 	GIT_AUTHOR_NAME="B" GIT_AUTHOR_EMAIL="B@test.git" \
 	git commit -m "change fortran file"
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 10/20] userdiff tests: move custom patterns into one test file

[Ævar Arnfjörð Bjarmason]

In a preceding commit the test infrastructure got rewritten so
"t/t4018/" are now normal test files which can do things like set
config, so let's make it responsible for setting up and tearing down
the config for its tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 16 +-------
 t/t4018/custom.sh        | 84 ++++++++++++++++++++++++++++++++++++++++
 t/t4018/custom1.sh       | 27 -------------
 t/t4018/custom2.sh       | 18 ---------
 t/t4018/custom3.sh       | 27 -------------
 5 files changed, 85 insertions(+), 87 deletions(-)
 create mode 100755 t/t4018/custom.sh
 delete mode 100755 t/t4018/custom1.sh
 delete mode 100755 t/t4018/custom2.sh
 delete mode 100755 t/t4018/custom3.sh

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 3ff34c13d7..3e4c07e42b 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -12,18 +12,6 @@ test_expect_success 'setup' '
 	test_file_not_empty builtin-drivers &&
 	builtin_drivers=$(cat builtin-drivers) &&
 
-	# a non-trivial custom pattern
-	git config diff.custom1.funcname "!static
-!String
-[^ 	].*s.*" &&
-
-	# a custom pattern which matches to end of line
-	git config diff.custom2.funcname "......Beer\$" &&
-
-	# alternation in pattern
-	git config diff.custom3.funcname "Beer$" &&
-	git config diff.custom3.xfuncname "^[ 	]*((public|static).*)$" &&
-
 	# for regexp compilation tests
 	echo A >A.java &&
 	echo B >B.java
@@ -31,9 +19,7 @@ test_expect_success 'setup' '
 
 diffpatterns="
 	$builtin_drivers
-	custom1
-	custom2
-	custom3
+	custom
 "
 
 for p in $diffpatterns
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
new file mode 100755
index 0000000000..69f1f7339f
--- /dev/null
+++ b/t/t4018/custom.sh
@@ -0,0 +1,84 @@
+#!/bin/sh
+#
+# See ../t4018-diff-funcname.sh's test_diff_funcname()
+#
+
+test_expect_success 'custom: setup non-trivial custom' '
+	git config diff.custom.funcname "!static
+!String
+[^ 	].*s.*"
+'
+
+test_diff_funcname 'custom: non-trivial custom pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+int special, RIGHT;
+EOF_HUNK
+public class Beer
+{
+	int special, RIGHT;
+	public static void main(String args[])
+	{
+		String s=" ";
+		for(int x = 99; x > 0; x--)
+		{
+			System.out.print(x + " bottles of beer on the wall "
+				+ x + " bottles of beer\n" // ChangeMe
+				+ "Take one down, pass it around, " + (x - 1)
+				+ " bottles of beer on the wall.\n");
+		}
+		System.out.print("Go to the store, buy some more,\n"
+			+ "99 bottles of beer on the wall.\n");
+	}
+}
+EOF_TEST
+
+test_expect_success 'custom: setup match to end of line' '
+	git config diff.custom.funcname "......Beer\$"
+'
+
+test_diff_funcname 'custom: match to end of line' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+RIGHT_Beer
+EOF_HUNK
+public class RIGHT_Beer
+{
+	int special;
+	public static void main(String args[])
+	{
+		System.out.print("ChangeMe");
+	}
+}
+EOF_TEST
+
+test_expect_success 'custom: setup alternation in pattern' '
+	git config diff.custom.funcname "Beer$" &&
+	git config diff.custom.xfuncname "^[ 	]*((public|static).*)$"
+'
+
+test_diff_funcname 'custom: alternation in pattern' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+public static void main(String RIGHT[])
+EOF_HUNK
+public class Beer
+{
+	int special;
+	public static void main(String RIGHT[])
+	{
+		String s=" ";
+		for(int x = 99; x > 0; x--)
+		{
+			System.out.print(x + " bottles of beer on the wall "
+				+ x + " bottles of beer\n" // ChangeMe
+				+ "Take one down, pass it around, " + (x - 1)
+				+ " bottles of beer on the wall.\n");
+		}
+		System.out.print("Go to the store, buy some more,\n"
+			+ "99 bottles of beer on the wall.\n");
+	}
+}
+EOF_TEST
+
+test_expect_success 'custom: teardown' '
+	test_unconfig diff.custom.funcname &&
+	test_unconfig diff.custom.xfuncname
+'
diff --git a/t/t4018/custom1.sh b/t/t4018/custom1.sh
deleted file mode 100755
index f8bbccadb4..0000000000
--- a/t/t4018/custom1.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom1: pattern' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-int special, RIGHT;
-EOF_HUNK
-public class Beer
-{
-	int special, RIGHT;
-	public static void main(String args[])
-	{
-		String s=" ";
-		for(int x = 99; x > 0; x--)
-		{
-			System.out.print(x + " bottles of beer on the wall "
-				+ x + " bottles of beer\n" // ChangeMe
-				+ "Take one down, pass it around, " + (x - 1)
-				+ " bottles of beer on the wall.\n");
-		}
-		System.out.print("Go to the store, buy some more,\n"
-			+ "99 bottles of beer on the wall.\n");
-	}
-}
-EOF_TEST
diff --git a/t/t4018/custom2.sh b/t/t4018/custom2.sh
deleted file mode 100755
index c68421f788..0000000000
--- a/t/t4018/custom2.sh
+++ /dev/null
@@ -1,18 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom2: match to end of line' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-RIGHT_Beer
-EOF_HUNK
-public class RIGHT_Beer
-{
-	int special;
-	public static void main(String args[])
-	{
-		System.out.print("ChangeMe");
-	}
-}
-EOF_TEST
diff --git a/t/t4018/custom3.sh b/t/t4018/custom3.sh
deleted file mode 100755
index 07c5c134ff..0000000000
--- a/t/t4018/custom3.sh
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# See ../t4018-diff-funcname.sh's test_diff_funcname()
-#
-
-test_diff_funcname 'custom3: alternation in pattern' \
-	8<<\EOF_HUNK 9<<\EOF_TEST
-public static void main(String RIGHT[])
-EOF_HUNK
-public class Beer
-{
-	int special;
-	public static void main(String RIGHT[])
-	{
-		String s=" ";
-		for(int x = 99; x > 0; x--)
-		{
-			System.out.print(x + " bottles of beer on the wall "
-				+ x + " bottles of beer\n" // ChangeMe
-				+ "Take one down, pass it around, " + (x - 1)
-				+ " bottles of beer on the wall.\n");
-		}
-		System.out.print("Go to the store, buy some more,\n"
-			+ "99 bottles of beer on the wall.\n");
-	}
-}
-EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 11/20] userdiff tests: remove hack for "RIGHT" token

[Ævar Arnfjörð Bjarmason]

Now that the "RIGHT" token isn't how we select the desired hunk header
line in the test anymore we can revert a hack added in
f1b75fbaf1 (t4018: convert custom pattern test to the new
infrastructure, 2014-03-21) and go back to the regular expression we
were testing before that change.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/custom.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 69f1f7339f..20abb38451 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -33,14 +33,14 @@ public class Beer
 EOF_TEST
 
 test_expect_success 'custom: setup match to end of line' '
-	git config diff.custom.funcname "......Beer\$"
+	git config diff.custom.funcname "Beer\$"
 '
 
 test_diff_funcname 'custom: match to end of line' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
-RIGHT_Beer
+Beer
 EOF_HUNK
-public class RIGHT_Beer
+public class Beer
 {
 	int special;
 	public static void main(String args[])
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 12/20] userdiff: match "package" in diff=golang

[Ævar Arnfjörð Bjarmason]

Improve the "golang" built-in pattern to match "package" lines, as
they weren't matched before changing e.g. the imports would commonly
result in an empty hunk header, now we'll instead show the package
name.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/golang.sh | 10 ++++++++++
 userdiff.c        |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/t/t4018/golang.sh b/t/t4018/golang.sh
index bf22f58c12..cdf9d6f8aa 100755
--- a/t/t4018/golang.sh
+++ b/t/t4018/golang.sh
@@ -3,6 +3,16 @@
 # See ../t4018-diff-funcname.sh's test_diff_funcname()
 #
 
+test_diff_funcname 'golang: package' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+package main
+EOF_HUNK
+package main
+
+import "fmt"
+// ChangeMe
+EOF_TEST
+
 test_diff_funcname 'golang: complex function' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
 func (t *Test) RIGHT(a Type) (Type, error) {
diff --git a/userdiff.c b/userdiff.c
index 92b5a97e12..d99b488700 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -124,6 +124,8 @@ IPATTERN("fortran",
 IPATTERN("fountain", "^((\\.[^.]|(int|ext|est|int\\.?/ext|i/e)[. ]).*)$",
 	 "[^ \t-]+"),
 PATTERNS("golang",
+	 /* Packages */
+	 "^[ \t]*(package[ \t]*(.*))\n"
 	 /* Functions */
 	 "^[ \t]*(func[ \t]*.*(\\{[ \t]*)?)\n"
 	 /* Structs and interfaces */
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 13/20] userdiff tests + docs: document & test "diff.<driver>.x?funcname"

[Ævar Arnfjörð Bjarmason]

Add the missing documentation for "diff.<driver>.funcname" and test
for how it and "diff.<driver>.xfuncname" interact.

Between the introduction of the "diff.<driver>.xfuncname" form in
45d9414fa5 (diff.*.xfuncname which uses "extended" regex's for hunk
header selection, 2008-09-18) and when this documentation was written
in 90b94c26f7 (Documentation: Add diff.<driver>.* to config,
2011-04-07) we forgot to document the existence of
"diff.<driver>.funcname".

Let's make a mention of it here, we could also partially revert the
former commit and discuss the more verbose form in gitattributes(5),
but let's stop short of that. It makes sense to guide users towards
ERE over BRE whenever possible.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/config/diff.txt | 12 +++++++++++
 t/t4018/custom.sh             | 40 +++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt
index 2d3331f55c..5fce8021de 100644
--- a/Documentation/config/diff.txt
+++ b/Documentation/config/diff.txt
@@ -153,10 +153,22 @@ diff.<driver>.command::
 	The custom diff driver command.  See linkgit:gitattributes[5]
 	for details.
 
+diff.<driver>.funcname::
 diff.<driver>.xfuncname::
 	The regular expression that the diff driver should use to
 	recognize the hunk header.  A built-in pattern may also be used.
 	See linkgit:gitattributes[5] for details.
++
+When provided as `diff.<driver>.funcname` the regular expression is
+interpreted as a basic regular expression, with
+`diff.<driver>.xfuncname` it's interpreted as an extended regular
+expression.
++
+
+The `*.funcname` and `*.xfuncname` variables behave as if though they
+were one configuration variable for the purposes of what value
+eventually gets used. Setting `*.funcname` will override an earlier
+`*.xfuncname` and vice-versa.
 
 diff.<driver>.binary::
 	Set this option to true to make the diff driver treat files as
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 20abb38451..b68d96a8af 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -78,6 +78,46 @@ public class Beer
 }
 EOF_TEST
 
+test_expect_success 'custom; setup config precedence' '
+	git config diff.custom.funcname "foo" &&
+	git config diff.custom.xfuncname "bar"
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+bar
+EOF_HUNK
+foo
+bar
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: teardown' '
+	test_unconfig diff.custom.funcname &&
+	test_unconfig diff.custom.xfuncname
+'
+
+test_expect_success 'custom; setup config precedence' '
+	git config diff.custom.xfuncname "bar" &&
+	git config diff.custom.funcname "foo"
+
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+foo
+EOF_HUNK
+foo
+bar
+
+ChangeMe
+
+baz
+EOF_TEST
+
 test_expect_success 'custom: teardown' '
 	test_unconfig diff.custom.funcname &&
 	test_unconfig diff.custom.xfuncname
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH 13/20] userdiff tests + docs: document & test "diff.<driver>.x?funcname"

[Eric Sunshine]

On Sun, Feb 14, 2021 at 7:56 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> Add the missing documentation for "diff.<driver>.funcname" and test
> for how it and "diff.<driver>.xfuncname" interact.
> [...]
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
> diff --git a/Documentation/config/diff.txt b/Documentation/config/diff.txt
> @@ -153,10 +153,22 @@ diff.<driver>.command::
> +diff.<driver>.funcname::
>  diff.<driver>.xfuncname::
>         The regular expression that the diff driver should use to
>         recognize the hunk header.  A built-in pattern may also be used.
>         See linkgit:gitattributes[5] for details.
> ++
> +When provided as `diff.<driver>.funcname` the regular expression is
> +interpreted as a basic regular expression, with

This would be easier to understand: s/, with/. With/

> +`diff.<driver>.xfuncname` it's interpreted as an extended regular
> +expression.
> ++
> +
> +The `*.funcname` and `*.xfuncname` variables behave as if though they
> +were one configuration variable for the purposes of what value
> +eventually gets used. Setting `*.funcname` will override an earlier
> +`*.xfuncname` and vice-versa.

Extra blank line before this paragraph?

[PATCH 14/20] gitattributes doc: reword discussion of built-in userdiff patterns

[Ævar Arnfjörð Bjarmason]

Reword the discussion of the built-in userdiff patterns to make it
more natural to precede it with a discussion about the semantics of
pattern matching, instead of assuming that it follows right after the
"diff.tex.xfuncname" example which now immediately precedes it. This
will make a follow-up commit smaller.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/gitattributes.txt | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index e84e104f93..90992e2136 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -794,11 +794,17 @@ backslashes; the pattern above picks a line that begins with a
 backslash, and zero or more occurrences of `sub` followed by
 `section` followed by open brace, to the end of line.
 
-There are a few built-in patterns to make this easier, and `tex`
-is one of them, so you do not have to write the above in your
-configuration file (you still need to enable this with the
-attribute mechanism, via `.gitattributes`).  The following built in
-patterns are available:
+There are built-in patterns shipped as part of git itself. A more
+advanced version of the `tex` pattern discussed above is one of them.
+
+For built-in patterns you do not need the "diff.tex.xfuncname"
+discussed above in your configuration file, but if present it'll
+override the built-in pattern.
+
+You still need to enable built-in patterns with the the attribute
+mechanism, via `.gitattributes`).
+
+The following built in patterns are available:
 
 - `ada` suitable for source code in the Ada language.
 
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH 14/20] gitattributes doc: reword discussion of built-in userdiff patterns

[Eric Sunshine]

On Sun, Feb 14, 2021 at 7:56 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> Reword the discussion of the built-in userdiff patterns to make it
> more natural to precede it with a discussion about the semantics of
> pattern matching, instead of assuming that it follows right after the
> "diff.tex.xfuncname" example which now immediately precedes it. This
> will make a follow-up commit smaller.
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
> @@ -794,11 +794,17 @@ backslashes; the pattern above picks a line that begins with a
> -There are a few built-in patterns to make this easier, and `tex`
> -is one of them, so you do not have to write the above in your
> -configuration file (you still need to enable this with the
> -attribute mechanism, via `.gitattributes`).  The following built in
> -patterns are available:
> +There are built-in patterns shipped as part of git itself. A more
> +advanced version of the `tex` pattern discussed above is one of them.
> +
> +For built-in patterns you do not need the "diff.tex.xfuncname"
> +discussed above in your configuration file, but if present it'll
> +override the built-in pattern.

The literal "diff.tex.xfuncname" now feels disconnected from what is
being said, especially as it is preceded by a reference to generic
"built-in patterns". Perhaps it would make sense to generalize it a
bit to `diff.<lang>.xfuncname`. For instance:

    For built-in patterns, you do not need `diff.<lang>.xfuncname`
    in your configuration file as discussed above, but if present,
    it will override a built-in pattern.

> +You still need to enable built-in patterns with the the attribute
> +mechanism, via `.gitattributes`).

s/the the/the/

Although you're simply re-using the existing text, it also now feels
disconnected. Perhaps:

    Nevertheless, you need to enable built-in patterns via
    `.gitattributes` for the pattern to take effect.

[PATCH 15/20] gitattributes doc: document multi-line userdiff patterns

[Ævar Arnfjörð Bjarmason]

Document the multi-line userdiff patterns and how their matching and
the negation syntax works.

These patterns have been supported since f258475a6e (Per-path
attribute based hunk header selection., 2007-07-06), and have had
their current semantics ever since 3d8dccd74a (diff: fix "multiple
regexp" semantics to find hunk header comment, 2008-09-20).

But we had no documentation for them, let's fix that, and also add
tests showing how some of the things being discussed here work.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/gitattributes.txt | 17 ++++++++++++++++
 t/t4018/custom.sh               | 35 +++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index 90992e2136..225c17b90d 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -794,6 +794,23 @@ backslashes; the pattern above picks a line that begins with a
 backslash, and zero or more occurrences of `sub` followed by
 `section` followed by open brace, to the end of line.
 
+Multiple patterns can be supplied by seperating them with
+newlines. They will be matched one at a time and are compiled as
+separate patterns, and thus the first capture in each such pattern is
+`$1`, see further discussion of captures below.
+
+Patterns that begin with "!" are negated (to match a literal "!" at
+the start of a line use e.g. "[!]"). A matching negated pattern will
+cause the matching line to be skipped. Use it to blacklist otherwise
+matching non-negated patterns. The last pattern must not be negated,
+we'll error out if that's the case.
+
+If the pattern contains a `$1` capture it will be used instead of the
+entire matching line (`$0`) to display the hunk header. This can be
+used e.g. to strip whitespace from the beginning of the line, or to
+only display the function name as part of a longer function
+definition.
+
 There are built-in patterns shipped as part of git itself. A more
 advanced version of the `tex` pattern discussed above is one of them.
 
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index b68d96a8af..cccf468c3a 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -122,3 +122,38 @@ test_expect_success 'custom: teardown' '
 	test_unconfig diff.custom.funcname &&
 	test_unconfig diff.custom.xfuncname
 '
+
+test_expect_success 'custom: negation syntax, ! is magic' '
+	git config diff.custom.xfuncname "!negation
+line"
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+line
+EOF_HUNK
+line
+!negation
+
+ChangeMe
+
+baz
+EOF_TEST
+
+test_expect_success 'custom: negation syntax, use [!] to override ! magic' '
+	git config diff.custom.xfuncname "[!]negation
+line"
+'
+
+test_diff_funcname 'custom: config precedence' \
+	8<<\EOF_HUNK 9<<\EOF_TEST
+!negation
+EOF_HUNK
+line
+!negation
+
+ChangeMe
+
+baz
+EOF_TEST
+
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH 15/20] gitattributes doc: document multi-line userdiff patterns

[Chris Torek]

This is extremely trivial, just something to consider if you are
already doing a reroll:

On Sun, Feb 14, 2021 at 4:57 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
> index 90992e2136..225c17b90d 100644
> --- a/Documentation/gitattributes.txt
> +++ b/Documentation/gitattributes.txt
> @@ -794,6 +794,23 @@ backslashes; the pattern above picks a line that begins with a
>  backslash, and zero or more occurrences of `sub` followed by
>  `section` followed by open brace, to the end of line.
>
> +Multiple patterns can be supplied by seperating them with
> +newlines. They will be matched one at a time and are compiled as
> +separate patterns, and thus the first capture in each such pattern is
> +`$1`, see further discussion of captures below.

This is a comma splice (https://en.wikipedia.org/wiki/Comma_splice).
Use a period or semicolon to fix it.

(overall comment, haven't quite finished scanning the series yet but it
looks good!)

Chris

Re: [PATCH 15/20] gitattributes doc: document multi-line userdiff patterns

[Eric Sunshine]

On Sun, Feb 14, 2021 at 7:56 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> Document the multi-line userdiff patterns and how their matching and
> the negation syntax works.
> [...]
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
> diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
> @@ -794,6 +794,23 @@ backslashes; the pattern above picks a line that begins with a
> +Multiple patterns can be supplied by seperating them with

s/seperating/separating/

> +newlines. They will be matched one at a time and are compiled as
> +separate patterns, and thus the first capture in each such pattern is
> +`$1`, see further discussion of captures below.

I found the wording "separating them with newlines" ambiguous. I
couldn't figure out if that meant that there must be a blank line
between patterns. Would it be more accurate to say merely that the
patterns must be listed one per line?

> +Patterns that begin with "!" are negated (to match a literal "!" at
> +the start of a line use e.g. "[!]"). A matching negated pattern will
> +cause the matching line to be skipped. Use it to blacklist otherwise
> +matching non-negated patterns. The last pattern must not be negated,
> +we'll error out if that's the case.

The parenthesized comment makes it difficult to follow the discussion.
Moving the comment to the end of the paragraph would make it easier to
grok:

    Patterns that begin with "!" are negated. A matching...
    ...error out if that's the case. To match a literal "!" at
    the start of a line, use "[!]".

I think, also, you want s/matching line/matched line/.

Chris's comma-splice comment also seems applicable for the last
sentence in this paragraph.

[PATCH 16/20] userdiff tests: remove "funcname" from custom3 test

[Ævar Arnfjörð Bjarmason]

We can only have one "funcname" or "xfuncname", any later definition
overrides the earlier one, so this configuration wasn't doing
anything.

When this test was originally added in 3632cfc248 (Use compatibility
regex library for OSX/Darwin, 2008-09-07) we had no such definition of
two patters for this test. Back then this was setting the
"diff.java.funcname" configuration variable.

The stage for that second pattern being set got set later. In
45d9414fa5 (diff.*.xfuncname which uses "extended" regex's for hunk
header selection, 2008-09-18) the pattern got converted from
"funcname" to "xfuncname".

Soon after in b19d288b4d (t4018-diff-funcname: demonstrate end of line
funcname matching flaw, 2008-10-15) another test immediately preceding
this one got added, using "diff.java.funcname" for its configuration.

Then f792a0b88e (t4018 (funcname patterns): make configuration easier
to track, 2011-05-21) came along and codified this whole thing when
converting the two tests from "git config" to "test_config".

Since this was never the intent of the test let's just remove this,
the rationale in f792a0b88e for having some test for the clobbering
behavior makes sense, but I'll do that in another follow-up test, not
as a hard to read side-effect of this one.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018/custom.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index cccf468c3a..81a68aa332 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -51,7 +51,6 @@ public class Beer
 EOF_TEST
 
 test_expect_success 'custom: setup alternation in pattern' '
-	git config diff.custom.funcname "Beer$" &&
 	git config diff.custom.xfuncname "^[ 	]*((public|static).*)$"
 '
 
@@ -133,6 +132,7 @@ test_diff_funcname 'custom: config precedence' \
 line
 EOF_HUNK
 line
+
 !negation
 
 ChangeMe
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 17/20] userdiff tests: factor out test_diff_funcname() logic

[Ævar Arnfjörð Bjarmason]

Factor out logic in test_diff_funcname() into two helper functions,
these will be useful in a follow-up commit where we'll do this munging
in more than one place.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 3e4c07e42b..7a830ec57f 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -47,6 +47,17 @@ test_expect_success 'last regexp must not be negated' '
 	test_i18ngrep ": Last expression must not be negated:" msg
 '
 
+do_change_me () {
+	file=$1
+	sed -e "s/ChangeMe/IWasChanged/" <"$file" >tmp &&
+	mv tmp "$file"
+}
+
+last_diff_context_line () {
+	file=$1
+	sed -n -e "s/^.*@@\( \|$\)//p" <$file
+}
+
 test_diff_funcname () {
 	desc=$1
 	cat <&8 >arg.header &&
@@ -57,13 +68,12 @@ test_diff_funcname () {
 		cp arg.test "$what" &&
 		cp arg.header expected &&
 		git add "$what" &&
-		sed -e "s/ChangeMe/IWasChanged/" <"$what" >tmp &&
-		mv tmp "$what"
+		do_change_me "$what"
 	' &&
 
 	test_expect_success "$desc" '
 		git diff -U1 "$what" >diff &&
-		sed -n -e "s/^.*@@\( \|$\)//p" <diff >actual &&
+		last_diff_context_line diff >actual &&
 		test_cmp expected actual
 	'
 }
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 18/20] userdiff tests: test hunk headers on accumulated files

[Ævar Arnfjörð Bjarmason]

The existing tests in "t/t4018/" are unrealistic in that they're all
setting up small few-line isolated test cases with one thing we could
match as a hunk header, right above the one change in the file.

Expand those tests by accumulating changes within the same file type
in the "test_diff_funcname" function. So e.g. for "bash" we'll end up
a "bash.acc" file with 15 s/ChangeMe/IWasChanged/ changes.

This stress tests whether the hunk header selection will "jump across"
to an earlier change because the match for that is greedier.

As it turns out we had one false positive in "t/t4018/cpp.sh" and
"t4018/matlab.sh" because of how the tests were structured, we must
always give the "ChangeMe" line at least one line of separation from
the header, since it was at the end of those tests we'd select the
"wrong" header. Let's adjust the spacing to compensate.

So in the end we found nothing of interest here, regardless, I think
it is useful to continue to test in this mode. It's likely to aid in
finding bugs in combinations of our positive and negative matching as
we add more built-in patterns.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 19 +++++++++++++++++++
 t/t4018/cpp.sh           |  1 +
 t/t4018/matlab.sh        |  3 +++
 3 files changed, 23 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 7a830ec57f..0d75d93c69 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -71,10 +71,24 @@ test_diff_funcname () {
 		do_change_me "$what"
 	' &&
 
+	test_expect_success "setup: $desc (accumulated)" '
+		cat arg.test >>arg.tests &&
+		cp arg.tests "$what".acc &&
+		git add "$what".acc &&
+		do_change_me "$what".acc
+	' &&
+
 	test_expect_success "$desc" '
 		git diff -U1 "$what" >diff &&
 		last_diff_context_line diff >actual &&
 		test_cmp expected actual
+	' &&
+
+	test_expect_success "$desc (accumulated)" '
+		git diff -U1 "$what".acc >diff &&
+		last_diff_context_line diff >actual.lines &&
+		tail -n 1 actual.lines >actual &&
+		test_cmp expected actual
 	'
 }
 
@@ -92,6 +106,11 @@ do
 		echo "$what" >arg.what
 	' &&
 
+	test_expect_success "setup: hunk header for $what (accumulated)" '
+		>arg.tests &&
+		echo "$what.acc diff=$what" >>.gitattributes
+	' &&
+
 	. "$test"
 done
 
diff --git a/t/t4018/cpp.sh b/t/t4018/cpp.sh
index 185d40d5ef..e0ab749316 100755
--- a/t/t4018/cpp.sh
+++ b/t/t4018/cpp.sh
@@ -206,6 +206,7 @@ void wrong()
 struct RIGHT_iterator_tag {};
 
 int ChangeMe;
+
 EOF_TEST
 
 test_diff_funcname 'cpp: template function definition' \
diff --git a/t/t4018/matlab.sh b/t/t4018/matlab.sh
index f62289148e..fba410e6f5 100755
--- a/t/t4018/matlab.sh
+++ b/t/t4018/matlab.sh
@@ -31,6 +31,7 @@ EOF_HUNK
 %%% RIGHT section
 # this is octave script
 ChangeMe = 1;
+
 EOF_TEST
 
 test_diff_funcname 'matlab: octave section 2' \
@@ -40,6 +41,7 @@ EOF_HUNK
 ## RIGHT section
 # this is octave script
 ChangeMe = 1;
+
 EOF_TEST
 
 test_diff_funcname 'matlab: section' \
@@ -49,4 +51,5 @@ EOF_HUNK
 %% RIGHT section
 % this is understood by both matlab and octave
 ChangeMe = 1;
+
 EOF_TEST
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 19/20] userdiff tests: test hunk header selection with -U0

[Ævar Arnfjörð Bjarmason]

The userdiff tests have used a custom -U1 context since
f12c66b9bb (userdiff/perl: anchor "sub" and "package" patterns on the
left, 2011-05-21). Changing it to -U0 doesn't change the results for
any of the tests, except one.

Let's test for this case explicitly. I.e. that we go "beyond" the
selected context to find our hunk header. In many cases the desired
hunk header is part of the diff itself under -U1.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 13 +++++++++++++
 t/t4018/custom.sh        |  1 +
 2 files changed, 14 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 0d75d93c69..94026b8296 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -63,6 +63,7 @@ test_diff_funcname () {
 	cat <&8 >arg.header &&
 	cat <&9 >arg.test &&
 	what=$(cat arg.what) &&
+	arg_diff_U0=$2 &&
 
 	test_expect_success "setup: $desc" '
 		cp arg.test "$what" &&
@@ -84,6 +85,18 @@ test_diff_funcname () {
 		test_cmp expected actual
 	' &&
 
+	test_expect_success "$desc -U0" '
+		git diff -U0 "$what" >diff &&
+		last_diff_context_line diff >actual &&
+		if test -n "$arg_diff_U0"
+		then
+			echo "$arg_diff_U0" >new-expected &&
+			test_cmp new-expected actual
+		else
+			test_cmp expected actual
+		fi
+	' &&
+
 	test_expect_success "$desc (accumulated)" '
 		git diff -U1 "$what".acc >diff &&
 		last_diff_context_line diff >actual.lines &&
diff --git a/t/t4018/custom.sh b/t/t4018/custom.sh
index 81a68aa332..605e2d33ae 100755
--- a/t/t4018/custom.sh
+++ b/t/t4018/custom.sh
@@ -10,6 +10,7 @@ test_expect_success 'custom: setup non-trivial custom' '
 '
 
 test_diff_funcname 'custom: non-trivial custom pattern' \
+	'System.out.print(x + " bottles of beer on the wall "' \
 	8<<\EOF_HUNK 9<<\EOF_TEST
 int special, RIGHT;
 EOF_HUNK
-- 
2.30.0.284.gd98b1dd5eaa7

[PATCH 20/20] userdiff tests: assert empty hunk header context on -U<large>

[Ævar Arnfjörð Bjarmason]

Assert the existing behavior that under -U<large> we'll show no hunk
header context, where <large> takes us past the potential hunk header
we'd have extracted. I'm just picking a number over nine thousand as a
really large number we're unlikely to exceed in these tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t4018-diff-funcname.sh | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/t/t4018-diff-funcname.sh b/t/t4018-diff-funcname.sh
index 94026b8296..bd81974dab 100755
--- a/t/t4018-diff-funcname.sh
+++ b/t/t4018-diff-funcname.sh
@@ -103,6 +103,14 @@ test_diff_funcname () {
 		tail -n 1 actual.lines >actual &&
 		test_cmp expected actual
 	'
+
+	test_expect_success "$desc -U9001 (accumulated)" '
+		git diff -U9001 "$what".acc >diff &&
+		last_diff_context_line diff >actual.lines &&
+		tail -n 1 actual.lines >actual &&
+		echo >blank &&
+		test_cmp blank actual
+	'
 }
 
 for what in $diffpatterns
-- 
2.30.0.284.gd98b1dd5eaa7

Re: [PATCH] userdiff: add support for Emacs Lisp

[Protesilaos Stavrou]

On 2021-02-14, 19:25 +0100, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:

> On Sun, Feb 14 2021, Johannes Sixt wrote:
>
>> Am 14.02.21 um 02:41 schrieb Ævar Arnfjörð Bjarmason:
>>> Just a cursory "git log -p -- lisp" in emacs.git with your patch shows
>>> e.g. lisp/thingatpt.el where forms in a "defun" aren't indented (before
>>> it selects the "defun", after with yours it's a "put" in the middle of
>>> the function).
>>
>> Note that negative matches can be specified. We use the feature in the
>> cpp case to exclude public:/protected:/private: and label: that happen 
>> to be not indented. Perhaps that can be useful here?
>>
>> Oh, and BTW, what the patterns treat as "function" must not match what
>> the language treats as function. The purpose of the hunk header is to 
>> spot a place in the source file easily. So, it does not hurt if
>> eval-and-compile forms are captured (as was mentioned in the linked 
>> thread) if desired.
>
> Right, so having lots of test-case is helpful, e.g. for elisp maybe you
> have a top-level defun, maybe not, maybe the top-level is a "(progn" so
> you'd like a second-level more meaningful context, or maybe not...
>
> Obviously these userdiff patterns aren't a general parser and will
> always be hit-and-miss, it's just useful to at least eyeball them
> against in-the-wild test data to check their sanity & add some tests.

Our intent with this patch is to rely on a heuristic that works for most
cases.  There will always be some case that is not covered.  This point
was made explicit in the relevant emacs-devel thread.

> My cursory glance at the emacs.git version v.s. what's being submitted
> here is that this one does a worse job in *very common* cases.
>
>>> Yours also changes it from e.g.:
>>>      @@ -61,7 +61,7 @@ forward-thing
>>> to:
>>>      @@ -61,7 +61,7 @@ (defun forward-thing (thing &optional n)
>>> Is this really desired in elisp?

I think this is a matter of perspective and indeed a valid reason for
doing this in emacs.git first.

For my part, I find the verbose style more informative, especially when
it captures the previous form instead of the one directly around the
diffed lines.

Perhaps one could add to that Emacs' ability to use the same name for
different things, e.g. a function and a variable, where verbosity of
this sort can help with disambiguation.  Granted, we should not really
on such a heading style to aid us in that task, though it may be
something to consider.

>> It's common practice to extract the entire line (sans indentation if
>> applicable), not just the function name. Why would somebody want the 
>> latter? It doesn't carry as much information as could be possible.
>
> Because I'm familiar with the codebase I'm editing and I just need to
> know that the diff I'm viewing is on the "main" function, not that it's
> "main()", "int main(int argv, char **argv)", "int main(const int argv,
> const char *argv[])", or to either have a " {" or not at the end
> depending on the project's coding style.

Oftentimes we produce/read patches for projects that we are not
necessarily acquainted with.  This includes emacs.git and the multitude
of Elisp packages in that milieu.  An extra element of contextuality can
do us good.

It is true that familiarity with the code base will always benefit from
succinct headings, though I feel that whenever we are in doubt we should
err on the side of caution: which means that we must not introduce such
an assumption to the workings of this piece of functionality.

> I know our own userdiff builtin patterns don't do this, but it would be
> very useful to retrofit this capability / maybe make it a configurable
> feature, i.e. have them capture the meaningful part of the line, and you
> could either print it all, or just that part.

It would be nice to have an option that toggles verbosity.  Though I
guess this lies outside the scope of the patch in question.

In conclusion, I think we should decide on the next step forward: if you
think this should be applied to emacs.git before making its way to git
itself, then we can move the discussion there.

Thanks for your time and efforts!
Protesilaos or "Prot"

-- 
Protesilaos Stavrou
protesilaos.com

Re: [PATCH] userdiff: add support for Emacs Lisp

[Ævar Arnfjörð Bjarmason]

On Tue, Feb 16 2021, Protesilaos Stavrou wrote:

> On 2021-02-14, 19:25 +0100, Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote:
>
>> On Sun, Feb 14 2021, Johannes Sixt wrote:
>>
>>> Am 14.02.21 um 02:41 schrieb Ævar Arnfjörð Bjarmason:
>>>> Just a cursory "git log -p -- lisp" in emacs.git with your patch shows
>>>> e.g. lisp/thingatpt.el where forms in a "defun" aren't indented (before
>>>> it selects the "defun", after with yours it's a "put" in the middle of
>>>> the function).
>>>
>>> Note that negative matches can be specified. We use the feature in the
>>> cpp case to exclude public:/protected:/private: and label: that happen 
>>> to be not indented. Perhaps that can be useful here?
>>>
>>> Oh, and BTW, what the patterns treat as "function" must not match what
>>> the language treats as function. The purpose of the hunk header is to 
>>> spot a place in the source file easily. So, it does not hurt if
>>> eval-and-compile forms are captured (as was mentioned in the linked 
>>> thread) if desired.
>>
>> Right, so having lots of test-case is helpful, e.g. for elisp maybe you
>> have a top-level defun, maybe not, maybe the top-level is a "(progn" so
>> you'd like a second-level more meaningful context, or maybe not...
>>
>> Obviously these userdiff patterns aren't a general parser and will
>> always be hit-and-miss, it's just useful to at least eyeball them
>> against in-the-wild test data to check their sanity & add some tests.
>
> Our intent with this patch is to rely on a heuristic that works for most
> cases.  There will always be some case that is not covered.  This point
> was made explicit in the relevant emacs-devel thread.

Yes these heuristics will always be bad in some cases. I'm just
encouraging you to find some of the more common cases, make test-cases
out of them, to say "this is what I want".

The most common difference I've seen is that by finding comments it
means for changes like this:
    
    (def foo ()
         "..."
         (some-code)
    ;; comments inside functions are often not indented
       (blah))
    
    (def bar ()
         "..."
         CHANGE_ME)

Before we'd find "foo" as the context, now we find ";; comments inside
functions are often not indented".

There's other cases where that comment rule does the right
thing. E.g. finding the description of the file at the top, but we could
also capture that with:

    ^(;;; [^.]+\.el ---.*)'

Then you'd get things like:

    ;;; button.el --- clickable buttons

Without overmatching any and all comments. Or maybe it isn't
overmatching and is better in the common case. I don't know.

Another example is now we'll match "(progn", sometimes that's "right",
sometimes "wrong". Perhaps even in cases where that's "right" the common
"(progn" pattern doesn't provide much/any extra information, and we'd be
better to skip past it with a negative rule to the next "(def" ?

I'm typing this in mu4e, but I wouldn't call myself deeply familiar with
Elisp. My reading of the linked thread / blog posts it links to is that
some other people came up with more narrow matching rules, presumably
because they found some of these cases.

So all I'm suggesting is maybe pro-actively find those cases & loop
those people in.

>> My cursory glance at the emacs.git version v.s. what's being submitted
>> here is that this one does a worse job in *very common* cases.
>>
>>>> Yours also changes it from e.g.:
>>>>      @@ -61,7 +61,7 @@ forward-thing
>>>> to:
>>>>      @@ -61,7 +61,7 @@ (defun forward-thing (thing &optional n)
>>>> Is this really desired in elisp?
>
> I think this is a matter of perspective and indeed a valid reason for
> doing this in emacs.git first.
>
> For my part, I find the verbose style more informative, especially when
> it captures the previous form instead of the one directly around the
> diffed lines.
>
> Perhaps one could add to that Emacs' ability to use the same name for
> different things, e.g. a function and a variable, where verbosity of
> this sort can help with disambiguation.  Granted, we should not really
> on such a heading style to aid us in that task, though it may be
> something to consider.

I'm not saying it needs to be done in emacs.git first, just maybe
coordinate with people who wrote/use that rule.

I think matching the whole line makes sense (sans leading whitespace),
we do it for the rest of the git.git rules.

>>> It's common practice to extract the entire line (sans indentation if
>>> applicable), not just the function name. Why would somebody want the 
>>> latter? It doesn't carry as much information as could be possible.
>>
>> Because I'm familiar with the codebase I'm editing and I just need to
>> know that the diff I'm viewing is on the "main" function, not that it's
>> "main()", "int main(int argv, char **argv)", "int main(const int argv,
>> const char *argv[])", or to either have a " {" or not at the end
>> depending on the project's coding style.
>
> Oftentimes we produce/read patches for projects that we are not
> necessarily acquainted with.  This includes emacs.git and the multitude
> of Elisp packages in that milieu.  An extra element of contextuality can
> do us good.
>
> It is true that familiarity with the code base will always benefit from
> succinct headings, though I feel that whenever we are in doubt we should
> err on the side of caution: which means that we must not introduce such
> an assumption to the workings of this piece of functionality.
>
>> I know our own userdiff builtin patterns don't do this, but it would be
>> very useful to retrofit this capability / maybe make it a configurable
>> feature, i.e. have them capture the meaningful part of the line, and you
>> could either print it all, or just that part.
>
> It would be nice to have an option that toggles verbosity.  Though I
> guess this lies outside the scope of the patch in question.

Yes I agree that this is completely outside the scope of adding elisp
support to userdiff.

I was just replying generally to Johaness on the topic of why someone
would want a pattern to match $1 over $0, as most of our git.git
patterns do (or if it's $1, it's the whole line anyway, sans leading
whitespace).

> In conclusion, I think we should decide on the next step forward: if you
> think this should be applied to emacs.git before making its way to git
> itself, then we can move the discussion there.

I think just applying a version to this to git.git is fine, and it
should not wait for whatever 20-some patches test mechanism rewrite I
started. I can rebase on top of this.

B.t.w. emacs.git also has a rule for *.texi, perhaps you'd be interested
in hacking that up too? :)