Re: [v2.35.0 regression] some PCRE hangs under UTF-8 locale (was: [PATCH 1/2] grep/pcre2: use PCRE2_UTF even with ASCII patterns)

["René Scharfe" <l.s.r@web.de>]

Am 29.01.22 um 18:25 schrieb SZEDER Gábor:
> On Sat, Dec 18, 2021 at 08:50:02PM +0100, René Scharfe wrote:
>> compile_pcre2_pattern() currently uses the option PCRE2_UTF only for
>> patterns with non-ASCII characters.  Patterns with ASCII wildcards can
>> match non-ASCII strings, though.  Without that option PCRE2 mishandles
>> UTF-8 input, though -- it matches parts of multi-byte characters.  Fix
>> that by using PCRE2_UTF even for ASCII-only patterns.
>>
>> This is a remake of the reverted ae39ba431a (grep/pcre2: fix an edge
>> case concerning ascii patterns and UTF-8 data, 2021-10-15).  The change
>> to the condition and the test are simplified and more targeted.
>>
>> Original-patch-by: Hamza Mahfooz <someguy@effective-light.com>
>> Signed-off-by: René Scharfe <l.s.r@web.de>
>> ---
>>  grep.c                          | 2 +-
>>  t/t7812-grep-icase-non-ascii.sh | 6 ++++++
>>  2 files changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/grep.c b/grep.c
>> index fe847a0111..5badb6d851 100644
>> --- a/grep.c
>> +++ b/grep.c
>> @@ -382,7 +382,7 @@ static void compile_pcre2_pattern(struct grep_pat *p, const struct grep_opt *opt
>>  		}
>>  		options |= PCRE2_CASELESS;
>>  	}
>> -	if (!opt->ignore_locale && is_utf8_locale() && has_non_ascii(p->pattern) &&
>> +	if (!opt->ignore_locale && is_utf8_locale() &&
>>  	    !(!opt->ignore_case && (p->fixed || p->is_fixed)))
>>  		options |= (PCRE2_UTF | PCRE2_MATCH_INVALID_UTF);
>>
>
> I tried to use 'git grep -P' for the first time ever, and it hung
> right away, spinning all CPUs at 100%.  I could narrow it down, both
> the complexity of the pattern and the size of the input, see the test
> below, and it bisects to this patch.
>
>
>   ---   >8   ---
>
> #!/bin/sh
>
> test_description='test'
>
> . ./test-lib.sh
>
> test_expect_success PCRE 'test' '
> 	# LC_ALL=C works
> 	LC_ALL=en_US.UTF-8 &&
> 	cat >ascii <<-\EOF &&
> 	foo
> 	 bar
> 	 baz
> 	EOF
> 	cat >utf8 <<-\EOF &&
> 	foo
> 	 bar
> 	 báz
> 	EOF
> 	git add ascii utf8 &&
>
> 	# These all work as expected:
> 	git grep --threads=1 -P " " ascii &&
> 	git grep --threads=1 -P "^ " ascii &&
> 	git grep --threads=1 -P "\s" ascii &&
> 	git grep --threads=1 -P "^\s" ascii &&
> 	git grep --threads=1 -P " " utf8 &&
> 	git grep --threads=1 -P "^ " utf8 &&
> 	git grep --threads=1 -P "\s" utf8 &&
>
> 	# This hangs (but it does work with basic and extended regexp):
> 	git grep --threads=1 -P "^\s" utf8
> '
>
> test_done

I get the following result and no hang with PCRE2 10.39:

   utf8: bar
   utf8: báz

e0c6029 (Fix inifinite loop when a single byte newline is searched in
JIT., 2020-05-29) [1] sounds like it might have fixed it.  It's part of
version 10.36.

Do you still get the error when you disable JIT, i.e. when you use the
pattern "(*NO_JIT)^\s" instead?

René


[1] https://github.com/PhilipHazel/pcre2/commit/e0c6029a62db9c2161941ecdf459205382d4d379