From: Paul Eggert <eggert@cs.ucla.edu>
To: Junio C Hamano <gitster@pobox.com>
Cc: "Carlo Arenas" <carenas@gmail.com>,
demerphq@gmail.com, 60690@debbugs.gnu.org,
"mega lith01" <megalith01@gmail.com>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
git@vger.kernel.org,
"Tukusej’s Sirs" <tukusejssirs@protonmail.com>,
pcre-dev@exim.org, Philip.Hazel@gmail.com
Subject: Re: bug#60690: -P '\d' in GNU and git grep
Date: Wed, 5 Apr 2023 11:32:38 -0700 [thread overview]
Message-ID: <6d86214a-1b80-eb88-1efb-36e61fd3203e@cs.ucla.edu> (raw)
In-Reply-To: <xmqqttxvzbo8.fsf@gitster.g>
On 2023-04-04 12:31, Junio C Hamano wrote:
> My personal inclination is to let Perl folks decide
> and follow them (even though I am skeptical about the wisdom of
> letting '\d' match anything other than [0-9])
I looked into what pcre2grep does. It has always done only 8-bit
processing unless you use the -u or --utf option, so plain "pcre2grep
'\d'" matches only ASCII digits.
Although this causes pcre2grep to mishandle Unicode characters:
$ echo 'Ævar' | pcre2grep '[Ssß]'
Ævar
it mimics Perl 5.36:
$ echo 'Ævar' | perl -ne 'print $_ if /[Ssß]/'
Ævar
so this seems to be what Perl users expect, despite its infelicities.
For better Unicode handling one can use pcre2grep's -u or --utf option,
which causes pcre2grep to behave more like GNU grep -P and git grep -P:
"echo 'Ævar' | pcre2grep -u '[Ssß]'" outputs nothing, which I think is
what most people would expect (unless they're Perl users :-).
Neither git grep -P nor the current release of pcre2grep -u have \d
matching non-ASCII digits, because they do not use PCRE2_UCP. However,
in a February 8 commit[1], Philip Hazel changed pcre2grep to use
PCRE2_UCP, so this will mean 10.43 pcre2grep -u will behave like 3.9 GNU
grep -P did (though 3.10 has changed this).
That February commit also added a --no-ucp option, to disable PCRE2_UCP.
So as I understand it, if you're in a UTF-8 locale:
* 10.43 pcre2grep -u will behave like 3.9 GNU grep -P.
* 10.43 pcre2grep -u --no-ucp will behave like git grep -P.
* Current GNU grep -P is different from everybody else.
This incompatibility is not good.
Here are two ways forward to fix this incompatibility (there are other
possibilities of course):
(A) GNU grep adds a --no-ucp option that acts like 10.43 pcre2grep
--no-ucp, and git grep -P follows suit. That is, both GNU and git grep
act like 10.43 pcre2grep -u, in that they enable PCRE2_UTF, and also
enable PCRE2_UCP unless --no-ucp is given. This would cause \d to match
non-ASCII digits unless --no-ucp is given.
(B) GNU grep -P and git grep -P mimic pcre2grep in both -u and --no-ucp.
That is, they would both do 8-bit-only by default, and use PCRE2_UTF
only when -u or --utf is given, and use PCRE2_UCP only when --no-ucp is
absent. This would cause \d to match non-ASCII digits only when -u is
given but --no-ucp is not.
Under either (A) or (B), future pcre2grep -u, GNU grep -P, and git grep
-P would be consistent.
I mildly prefer (B) but (A) would also work. (One advantage of (B) is
that it should be faster....)
[1]:
https://github.com/PCRE2Project/pcre2/commit/8385df8c97b6f8069a48e600c7e4e94cc3e3ebd9ht
next prev parent reply other threads:[~2023-04-05 18:33 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-08 6:23 [PATCH] grep: correctly identify utf-8 characters with \{b,w} in -P Carlo Marcelo Arenas Belón
2023-01-08 6:39 ` Junio C Hamano
2023-01-08 15:52 ` [PATCH v2] " Carlo Marcelo Arenas Belón
2023-01-09 11:35 ` Ævar Arnfjörð Bjarmason
2023-01-09 18:40 ` bug#60690: [PATCH v2] grep: correctly identify utf-8 characters with \{b, w} " Paul Eggert
2023-01-09 19:51 ` Ævar Arnfjörð Bjarmason
2023-01-09 23:12 ` Paul Eggert
2023-01-10 4:49 ` [PATCH v2] grep: correctly identify utf-8 characters with \{b,w} " Carlo Arenas
2023-01-16 20:48 ` Junio C Hamano
2023-04-03 21:38 ` -P '\d' in GNU and git grep Paul Eggert
2023-04-04 3:30 ` bug#60690: " Jim Meyering
2023-04-04 6:46 ` Paul Eggert
2023-04-04 15:31 ` Jim Meyering
2023-04-04 6:56 ` Carlo Arenas
2023-04-04 18:25 ` bug#60690: " Paul Eggert
2023-04-04 19:31 ` Junio C Hamano
2023-04-05 18:32 ` Paul Eggert [this message]
2023-04-05 19:04 ` Paul Eggert
2023-04-05 19:37 ` Junio C Hamano
2023-04-05 19:40 ` Jim Meyering
2023-04-05 20:03 ` Paul Eggert
2023-04-05 21:20 ` Carlo Arenas
2023-04-06 15:45 ` demerphq
2023-04-07 16:48 ` Paul Eggert
2023-04-06 13:39 ` demerphq
2023-04-07 19:00 ` Paul Eggert
2023-04-08 5:01 ` Carlo Arenas
2023-04-08 22:45 ` Paul Eggert
2023-01-17 10:51 ` [PATCH v3] grep: correctly identify utf-8 characters with \{b,w} in -P Carlo Marcelo Arenas Belón
2023-01-17 12:38 ` Ævar Arnfjörð Bjarmason
2023-01-17 15:19 ` Junio C Hamano
2023-01-18 7:35 ` Carlo Arenas
2023-01-18 11:49 ` Ævar Arnfjörð Bjarmason
2023-01-18 16:20 ` Junio C Hamano
2023-01-18 23:06 ` Ævar Arnfjörð Bjarmason
2023-01-18 23:24 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6d86214a-1b80-eb88-1efb-36e61fd3203e@cs.ucla.edu \
--to=eggert@cs.ucla.edu \
--cc=60690@debbugs.gnu.org \
--cc=Philip.Hazel@gmail.com \
--cc=avarab@gmail.com \
--cc=carenas@gmail.com \
--cc=demerphq@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=megalith01@gmail.com \
--cc=pcre-dev@exim.org \
--cc=tukusejssirs@protonmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).