Hi Ævar,

On Thu, 27 Jun 2019, Ævar Arnfjörð Bjarmason wrote:

>
> On Thu, Jun 27 2019, Johannes Schindelin wrote:
>
> > On Wed, 26 Jun 2019, Johannes Schindelin wrote:
> >
> >> On Wed, 26 Jun 2019, Ævar Arnfjörð Bjarmason wrote:
> >>
> >> > This speeds things up a lot, but as shown in the patches & tests
> >> > changed modifies the behavior where we have \0 in *patterns* (only
> >> > possible with 'grep -f <file>').
> >>
> >> I agree that it is not worth a lot to care about NULs in search patterns.
> >>
> >> So I am in favor of the goal of this patch series.
> >
> > There seems to be a Windows-specific test failure:
> > https://dev.azure.com/gitgitgadget/git/_build/results?buildId=11535&view=ms.vss-test-web.build-test-results-tab&runId=28232&resultId=101315&paneView=debug
> >
> > The output is this:
> >
> > -- snip --
> > not ok 5 - log --grep does not find non-reencoded values (latin1)
> >
> > expecting success:
> > 	git log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e >actual
> > &&
> > 	test_must_be_empty actual
> >
> > ++ git log --encoding=ISO-8859-1 --format=%s --grep=é
> > fatal: pcre2_match failed with error code -8: UTF-8 error: byte 2 top bits
> > not 0x80
> > -- snap --
> >
> > Any quick ideas? (I _could_ imagine that it is yet another case of passing
> > non-UTF-8-encoded stuff via command-line vs via file, which does not work
> > on Windows.)
>
> This is an existing issue that my patches just happen to uncover. I'm
> working on a v2 which'll fix it.

I just found yet another problem. When using a libpcre2 _without_ JIT
support, I get this:

-- snip --
$ sh t4210-log-i18n.sh -i -V -x || tail -20
test-results/t4210-log-i18n.out
ok 1 - create commits in different encodings
ok 2 - log --grep searches in log output encoding (utf8)
ok 3 # skip log --grep searches in log output encoding (latin1) (missing !MINGW)
ok 4 # skip log --grep does not find non-reencoded values (utf8) (missing !MINGW)
not ok 5 - log --grep does not find non-reencoded values (latin1)
#
#               git log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e
#               >actual &&
#               test_must_be_empty actual
#
ok 3 # skip log --grep searches in log output encoding (latin1) (missing !MINGW)

skipping test: log --grep does not find non-reencoded values (utf8)
        git log --encoding=utf8 --format=%s --grep=$latin1_e >actual &&
        test_must_be_empty actual

ok 4 # skip log --grep does not find non-reencoded values (utf8) (missing !MINGW)

expecting success:
        git log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e >actual &&
        test_must_be_empty actual

++ git log --encoding=ISO-8859-1 --format=%s --grep=é
fatal: pcre2_match failed with error code -8: UTF-8 error: byte 2 top bits not 0x80
error: last command exited with $?=128
not ok 5 - log --grep does not find non-reencoded values (latin1)
#
#               git log --encoding=ISO-8859-1 --format=%s --grep=$utf8_e
#               >actual &&
#               test_must_be_empty actual
#
-- snap --

This is actually a correct error, as we specifically feed non-UTF-8 text
to PCRE2, but we do turn on the PCRE2_UTF flag.

Funnily enough, this error only occurs when `pcre2_jit_on == 0`, i.e. when
we hit the code path that calls `pcre2_match()`. When the alternative code
path is used, `pcre2_jit_match()` is called, and it does _not_ print that
error.

Whatever the bug in libpcre2 that causes the JIT code path to fail on the
Unicode validation, it points to the problem in this code in
`compile_pcre2_pattern()`:

-- snip --
        if (is_utf8_locale() && has_non_ascii(p->pattern))
                options |= PCRE2_UTF;
-- snap --

It only asks whether there is a non-ASCII character in pattern, but we
never bother to see whether the haystack is also encoded in UTF-8. In this
case, it is not...

Ciao,
Dscho

P.S.: Yes, yes, I know, we should run PCRE2 with JIT, and I am trying to
figure out why it is not enabled on Windows when I specifically asked
`./configure` to enable it... Investigating now.

-