Re: [PATCH] gitweb: Measure offsets against UTF-8 flagged string

[Junio C Hamano <gitster@pobox.com>]

Shin Kojima <shin@kojima.org> writes:

> Offset positions should not be counted by byte length, but by actual
> character length.
> ...
>  # escape tabs (convert tabs to spaces)
>  sub untabify {
> -	my $line = shift;
> +	my $line = to_utf8(shift);
>  
>  	while ((my $pos = index($line, "\t")) != -1) {
>  		if (my $count = (8 - ($pos % 8))) {

Some codepaths in the resulting codeflow look even hackier than they
already are.  For example, format_rem_add_lines_pair() calls
untabify() and then feeds its result to esc_html().  The first thing
done in esc_html() is to call to_utf8().  I know that to_utf8()
cheats and leaves the incoming string as-is if it is already UTF-8,
so this may be a safe conversion, but ideally we should be able to
say "function X takes non-UTF8 and works on it", "function Y takes
UTF8 and works on it", and "function Z takes non-UTF8 and gives UTF8
data back" for each functions clearly, not "function W can take
either UTF8 or any other garbage and tries to return UTF8".

Also, does it even "fix" the problem to use to_utf8() here in the
first place?  Untabify is about aligning the character after a HT to
multiple of 8 display position, so we'd want measure display width,
which is not the same as either byte count or char count.