[PATCH] gitweb: Measure offsets against UTF-8 flagged string

[PATCH] gitweb: Measure offsets against UTF-8 flagged string

[Shin Kojima]

Offset positions should not be counted by byte length, but by actual
character length.

>    5183 	# We need to untabify lines before split()'ing them;
>    5184 	# otherwise offsets would be invalid.

Horizontal tab is not the only case we need to consider.  Please excuse
me for using your name here, but the following URL can not find "match"
occurances while using `git-instaweb` on the git repository.

    http://127.0.0.1:1234/?p=.git&a=search&h=HEAD&st=grep&s=Nar%C4%99bski

Signed-off-by: Shin Kojima <shin@kojima.org>
---
 gitweb/gitweb.perl | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 2594a4bad..a5a9093a1 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -1697,7 +1697,7 @@ sub unquote {
 
 # escape tabs (convert tabs to spaces)
 sub untabify {
-	my $line = shift;
+	my $line = to_utf8(shift);
 
 	while ((my $pos = index($line, "\t")) != -1) {
 		if (my $count = (8 - ($pos % 8))) {
-- 
2.17.0

Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string

[Junio C Hamano]

Shin Kojima <shin@kojima.org> writes:

> Offset positions should not be counted by byte length, but by actual
> character length.
> ...
>  # escape tabs (convert tabs to spaces)
>  sub untabify {
> -	my $line = shift;
> +	my $line = to_utf8(shift);
>  
>  	while ((my $pos = index($line, "\t")) != -1) {
>  		if (my $count = (8 - ($pos % 8))) {

Some codepaths in the resulting codeflow look even hackier than they
already are.  For example, format_rem_add_lines_pair() calls
untabify() and then feeds its result to esc_html().  The first thing
done in esc_html() is to call to_utf8().  I know that to_utf8()
cheats and leaves the incoming string as-is if it is already UTF-8,
so this may be a safe conversion, but ideally we should be able to
say "function X takes non-UTF8 and works on it", "function Y takes
UTF8 and works on it", and "function Z takes non-UTF8 and gives UTF8
data back" for each functions clearly, not "function W can take
either UTF8 or any other garbage and tries to return UTF8".

Also, does it even "fix" the problem to use to_utf8() here in the
first place?  Untabify is about aligning the character after a HT to
multiple of 8 display position, so we'd want measure display width,
which is not the same as either byte count or char count.

Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string

[Shin Kojima]

> ideally we should be able to say "function X takes non-UTF8 and
> works on it", "function Y takes UTF8 and works on it", and "function
> Z takes non-UTF8 and gives UTF8 data back" for each functions
> clearly, not "function W can take either UTF8 or any other garbage
> and tries to return UTF8".

Yes, I totally agree with that.


> Some codepaths in the resulting codeflow look even harder than they
> already are.  For example, format_rem_add_lines_pair() calls
> untabify() and then feeds its result to esc_html().

Honestly, I discovered this problem exactly from
format_rem_add_lines_pair().  In my environment($fallback_encoding
= 'cp932'), some commitdiff shows garbled text only inside color-words
portions.

I added a reproduce process at the end of this message.

After my investigation, I thought format_rem_add_lines_pair() tries to
use split()/index()/substr()/etc against raw blob contents and that
produces funny characters.  These builtin functions should be used to
decoded string.

untabify() looks proper place for me to decode blob contents
beforehand, as it definitely is not to be used for binary contests
like images and compressed snapshot files.

I'm sure using to_utf8() in untabify() fixes this problem.  In fact,
there is also another similar problem in blame function that assumes
blob contents as if utf8 encoded:

    binmode $fd, ':utf8';

I personally consider text blob contents should be decoded as soon as
possible and esc_html() should not contain to_utf8(), but the
codeflow is slightly vast and I couldn't eliminate the possibility of
calls from somewhere else that does not care character encodings.

So yes, I would appreciate hearing your thoughts.


> Also, does it even "fix" the problem to use to_utf8() here in the
> first place?  Untabify is about aligning the character after a HT to
> multiple of 8 display position, so we'd want measure display width,
> which is not the same as either byte count or char count.

Following is a reproduce process:

    $ git --version
        git version 2.17.0

    $ mkdir test
    $ cd test
    $ git init
    $ echo 'モバイル' | iconv -f UTF-8 -t Shift_JIS > dummy
    $ git add .
    $ git commit -m 'init'
    $ echo 'インスタント' | iconv -f UTF-8 -t Shift_JIS > dummy
    $ git commit -am 'change'
    $ git instaweb
    $ echo 'our $fallback_encoding = "cp932";' >> .git/gitweb/gitweb_config.perl
    $ w3m -dump 'http://127.0.0.1:1234/?p=.git;a=commitdiff'

What I got:

    gitprojects / .git / commitdiff
    [commit   ] ? search: [                    ] [ ]re
    summary | shortlog | log | commit | commitdiff | tree
    raw | patch | inline | side by side (parent: 79e26fe)
    change master

    author    Shin Kojima <shin@kojima.org>
              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)
    committer Shin Kojima <shin@kojima.org>
              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)

    dummy  patch | blob | history


    diff --git a/dummy b/dummy
    index ac37f38..31fb96a 100644 (file)
    --- a/dummy
    +++ b/dummy
    @@ -1 +1 @@
    -ｃoイル
    +Cンスタント
    Unnamed repository; edit this file 'description' to name the repository.
    RSS Atom

What I expected:

    gitprojects / .git / commitdiff
    [commit   ] ? search: [                    ] [ ]re
    summary | shortlog | log | commit | commitdiff | tree
    raw | patch | inline | side by side (parent: 79e26fe)
    change master

    author    Shin Kojima <shin@kojima.org>
              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)
    committer Shin Kojima <shin@kojima.org>
              Wed, 2 May 2018 10:55:01 +0000 (19:55 +0900)

    dummy  patch | blob | history


    diff --git a/dummy b/dummy
    index ac37f38..31fb96a 100644 (file)
    --- a/dummy
    +++ b/dummy
    @@ -1 +1 @@
    -モバイル
    +インスタント
    Unnamed repository; edit this file 'description' to name the repository.
    RSS Atom


-- 
Shin Kojima

Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string

[Jakub Narebski]

Junio C Hamano <gitster@pobox.com> writes:
> Shin Kojima <shin@kojima.org> writes:
>
>> Offset positions should not be counted by byte length, but by actual
>> character length.
>> ...
>>  # escape tabs (convert tabs to spaces)
>>  sub untabify {
>> -	my $line = shift;
>> +	my $line = to_utf8(shift);
>>  
>>  	while ((my $pos = index($line, "\t")) != -1) {
>>  		if (my $count = (8 - ($pos % 8))) {
>
> Some codepaths in the resulting codeflow look even hackier than they
> already are.  For example, format_rem_add_lines_pair() calls
> untabify() and then feeds its result to esc_html().  The first thing
> done in esc_html() is to call to_utf8().  I know that to_utf8()
> cheats and leaves the incoming string as-is if it is already UTF-8,
> so this may be a safe conversion, but ideally we should be able to
> say "function X takes non-UTF8 and works on it", "function Y takes
> UTF8 and works on it", and "function Z takes non-UTF8 and gives UTF8
> data back" for each functions clearly, not "function W can take
> either UTF8 or any other garbage and tries to return UTF8".

The problem with handling encoding in sane way, that is encode it on
input (to UTF-8), and decode on output (to plain text or HTML) is the
$fallback_encoding.

Gitweb assumes that everything uses UTF-8 encoding.  If the source is
not in UTF-8, but for example uses latin-1 encoding, then there we could
stumble upon byte sequences which are not valid UTF-8.  If that happens,
then gitweb tries to convert it to UTF-8 using $fallback_encoding.  If
$fallback_encoding is single-byte encoding, like latin-1, where any byte
sequence is valid, then that's all.  If there is an error during
conversion to UTF-8, then Unicode REPLACEMENT CHARACTER, code point
U+FFFD, is used.

But there are places where gitweb outputs plain text; the intention is
to use source data as is - to have it as one would have in the console.
Some input paths are common for plain text and HTML output; because of
that problem the data is not converted to UTF-8 on input.


The to_utf8() function tries to be clever, and do not convert alredy
converted data.

> Also, does it even "fix" the problem to use to_utf8() here in the
> first place?  Untabify is about aligning the character after a HT to
> multiple of 8 display position, so we'd want measure display width,
> which is not the same as either byte count or char count.

I think the problem is not with aligning, otherwise we would simply get
bad aling, and not visible corruption.  The ACTUAL PROBLEM is most
probably because of concatenating strings marked as UTF-8 and strings
not marked as UTF-8.  Strange things happen then in Perl, unfortunetaly.


One solution would be to force conversion to UTF-8 on input via "open"
pragma (e.g. "use open ':encoding(UTF-8)';").  But there is no
UTF-8-with_fallback encoding available - we would have to write one, and
install it as module (or fake it via Perl trickery).  This mechanism is
almost the same to what we currently use in gitwbe.

Another would be to use the trick that Perl 6 uses when encountering
byte sequence that is invalid UTF-8 - encode it using private plane in
Unicode using UTF-8, thus achieving lossless conversion / decoding.  But
this also as far as I know is not available from CPAN, so we would have
to implement it ourself.

Best,
-- 
Jakub Narębski

Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string

[Shin Kojima]

> One solution would be to force conversion to UTF-8 on input via "open"
> pragma (e.g. "use open ':encoding(UTF-8)';").  But there is no
> UTF-8-with_fallback encoding available - we would have to write one, and
> install it as module (or fake it via Perl trickery).  This mechanism is
> almost the same to what we currently use in gitwbe.

Yes, I tried using `Encode::Guess` with "open" pragma, but no luck.
https://perldoc.perl.org/Encode/Guess.html

I'm also afraid of "open" pragma does not work properly while using
git_blame_common().  Let's say someone using non-ASCII characters in
his/her name, committing non-UTF8 encoded characters.  git-blame will
combine them in the same line.  Following is an example:

$ git blame dummy | xxd
00000000: 3461 6464 3565 6331 2028 e585 90e5 b3b6  4add5ec1 (......
00000010: 20e6 96b0 2032 3031 382d 3035 2d30 3320   ... 2018-05-03
00000020: 3232 3a34 383a 3432 202b 3039 3030 2031  22:48:42 +0900 1
00000030: 2920 8367 8389 8343 0a                   ) .g...C.

    * e585 90e5 b3b6 20e6 96b0 : my name, encoded with UTF-8
    * 8367 8389 8343           : "トライ" encoded with Shift_JIS

It means I need to split each lines of git-blame output at the very
beginning, then convert the first-half as UTF-8 and the second-half as
Shift_JIS.

Sincerely,

-- 
Shin Kojima

Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> I think the problem is not with aligning, otherwise we would simply get
> bad aling, and not visible corruption.  The ACTUAL PROBLEM is most
> probably because of concatenating strings marked as UTF-8 and strings
> not marked as UTF-8.  Strange things happen then in Perl, unfortunetaly.

Sounds quite right---the patch needs to be retitled, if the bug is
not about "measuring offset", which I think is what fooled me when I
sent my response.