[RFC PATCH] gitweb: improve title shortening heuristics

[RFC PATCH] gitweb: improve title shortening heuristics

[Julien Rouhaud]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

Hi,

First of all, this is my first time on this ML so apologies in advance if I
missed anything in the patch submission guidelines.

We got some report recently that the commit short title on the postgres gitweb
instance was sometimes being mangled (1).  After a bit of digging, it appears
to be due to some long time heuristics to remove some uninteresting parts of a
commit message (see 198066916a8 from August 2005).  In our case, it removed any
occurrence of "master." in the commit message even if the message contains
"postmaster.c" rather than a cname (or something that looks like it), leading
to the commit message:

Remove postmaster.c's reset_shared() wrapper function.

being displayed as:

Remove postc's reset_shared() wrapper function.

It's probably some corner case for which there's barely any complaint, so it
doesn't look worthwhile to spend too much effort on it.  It also seems
impossible to make the current approach entirely bullet proof, but if we simply
make sure that the prefix is preceded by at least one whitespace and isn't
followed by another one we could avoid almost all of the incorrect matches (and
all of them as far as postgres is concerned).  Would that be an acceptable
compromise?  If yes, I'm attaching a patch that does that (and also adds git://
and https:// to the list of trimmed protocols while at it).

Otherwise, would it be acceptable to disable the whole block (the "remove
leading stuff of merges to make the interesting part visible") with some new
configuration option?

Cheers,
Julien.

[1] https://www.postgresql.org/message-id/flat/4025723.1658013974%40sss.pgh.pa.us

[-- Attachment #2: v1-0001-gitweb-improve-title_short-shortening-heuristics.patch --]
[-- Type: text/plain, Size: 1423 bytes --]

From ed46dcd2796b9af6ba3f73d46a3141a88964ed11 Mon Sep 17 00:00:00 2001
From: Julien Rouhaud <julien.rouhaud@free.fr>
Date: Sun, 24 Jul 2022 13:17:19 +0800
Subject: [PATCH v1] gitweb: improve title_short shortening heuristics

In order to shorten the title, some common domain prefixes can be detected and
removed.  However, the current regex matches those prefix anywhere in the
title which makes it likely to remove it where it's not intended.

To make that case less likely, make sure that the prefix is preceded by at
least one whitespace and isn't followed by another whitespace.

While at it, also add  git:// and https:// to the list of detected and trimmed
protocols.

Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
---
 gitweb/gitweb.perl | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 1835487ab2..18dd0b93fb 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3565,10 +3565,10 @@ sub parse_commit_text {
 				$title =~ s/^Automatic //;
 				$title =~ s/^merge (of|with) /Merge ... /i;
 				if (length($title) > 50) {
-					$title =~ s/(http|rsync):\/\///;
+					$title =~ s/(git|http|https|rsync):\/\///;
 				}
 				if (length($title) > 50) {
-					$title =~ s/(master|www|rsync)\.//;
+					$title =~ s/\s+(master|www|rsync)\.([^\s])/ \2/;
 				}
 				if (length($title) > 50) {
 					$title =~ s/kernel.org:?//;
-- 
2.37.0

Re: [RFC PATCH] gitweb: improve title shortening heuristics

[Junio C Hamano]

Julien Rouhaud <rjuju123@gmail.com> writes:

> Otherwise, would it be acceptable to disable the whole block (the "remove
> leading stuff of merges to make the interesting part visible") with some new
> configuration option?

I personally find that "shortening" logic way too specific to the
kernel project hosted at kernel.org and would be inappropriate to
use it anywhere else.

	if (length($title) > 50) {
		$title =~ s/^Automatic //;
		$title =~ s/^merge (of|with) /Merge ... /i;
		if (length($title) > 50) {
			$title =~ s/(http|rsync):\/\///;
		}
		if (length($title) > 50) {
			$title =~ s/(master|www|rsync)\.//;
		}
		if (length($title) > 50) {
			$title =~ s/kernel.org:?//;
		}
		if (length($title) > 50) {
			$title =~ s/\/pub\/scm//;
		}
	}
	$co{'title_short'} = chop_str($title, 50, 5);

Of course, http:// and rsync:// are way outdated (https://, ssh://
and git:// are probably reasonable).  Equally outdated is to merge
branches from master.kernel.org, www.kernel.org, or rsync.kernel.org
(many merges are recorded as pulling from git://git.kernel.org/ or
https://git.kernel.org/ these days).

The above code is all from 198066916a8 (Kay Sievers 2005-08-07), so
it is very much understandable that its shortening heuristics no
longer match reality.

Even worse, I somehow thought that kernel.org no longer uses gitweb
but uses something else.  So I suspect that nobody sheds tears if we
remove the whole block unconditionally.  In fact, it would make the
world a better place.

Re: [RFC PATCH] gitweb: improve title shortening heuristics

[Julien Rouhaud]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

On Sun, Jul 24, 2022 at 06:30:44PM -0700, Junio C Hamano wrote:
> Julien Rouhaud <rjuju123@gmail.com> writes:
>
> > Otherwise, would it be acceptable to disable the whole block (the "remove
> > leading stuff of merges to make the interesting part visible") with some new
> > configuration option?
>
> I personally find that "shortening" logic way too specific to the
> kernel project hosted at kernel.org and would be inappropriate to
> use it anywhere else.
>
> 	if (length($title) > 50) {
> 		$title =~ s/^Automatic //;
> 		$title =~ s/^merge (of|with) /Merge ... /i;
> 		if (length($title) > 50) {
> 			$title =~ s/(http|rsync):\/\///;
> 		}
> 		if (length($title) > 50) {
> 			$title =~ s/(master|www|rsync)\.//;
> 		}
> 		if (length($title) > 50) {
> 			$title =~ s/kernel.org:?//;
> 		}
> 		if (length($title) > 50) {
> 			$title =~ s/\/pub\/scm//;
> 		}
> 	}
> 	$co{'title_short'} = chop_str($title, 50, 5);

That's probably true, although some parts (like the protocol) seemed general
enough to me to be worth considering for general use case, and some of the
really specific parts seems so specific that they shouldn't really matter if
used outside kernel.org.
>
> Of course, http:// and rsync:// are way outdated (https://, ssh://
> and git:// are probably reasonable).  Equally outdated is to merge
> branches from master.kernel.org, www.kernel.org, or rsync.kernel.org
> (many merges are recorded as pulling from git://git.kernel.org/ or
> https://git.kernel.org/ these days).
>
> Even worse, I somehow thought that kernel.org no longer uses gitweb
> but uses something else.  So I suspect that nobody sheds tears if we
> remove the whole block unconditionally.  In fact, it would make the
> world a better place.

Well, I'm obviously totally fine with getting rid of it.  FWIW, even the
protocol part wouldn't change anything for the instance hosted by postgres.

I'm attaching an updated v2 patch that removes all that logic and just keeps
the title_short chopped at 50 chars.

[-- Attachment #2: v2-0001-gitweb-Remove-title_short-shortening-heuristics.patch --]
[-- Type: text/plain, Size: 1426 bytes --]

From ea4f31c0a2276f042322a11e258e55223863cd35 Mon Sep 17 00:00:00 2001
From: Julien Rouhaud <julien.rouhaud@free.fr>
Date: Sun, 24 Jul 2022 13:17:19 +0800
Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics

Those heuristics are way outdated and too general to be useful outside of
kernel.org.  Since kernel.org doesn't use gitweb anymore and at least one
project complained about incorrect behavior, entirely remove them.

Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
---
 gitweb/gitweb.perl | 17 -----------------
 1 file changed, 17 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 1835487ab2..e66eb3d9ba 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3560,23 +3560,6 @@ sub parse_commit_text {
 		$title =~ s/^    //;
 		if ($title ne "") {
 			$co{'title'} = chop_str($title, 80, 5);
-			# remove leading stuff of merges to make the interesting part visible
-			if (length($title) > 50) {
-				$title =~ s/^Automatic //;
-				$title =~ s/^merge (of|with) /Merge ... /i;
-				if (length($title) > 50) {
-					$title =~ s/(http|rsync):\/\///;
-				}
-				if (length($title) > 50) {
-					$title =~ s/(master|www|rsync)\.//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/kernel.org:?//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/\/pub\/scm//;
-				}
-			}
 			$co{'title_short'} = chop_str($title, 50, 5);
 			last;
 		}
-- 
2.37.0

Re: [RFC PATCH] gitweb: improve title shortening heuristics

[Ævar Arnfjörð Bjarmason]

On Mon, Jul 25 2022, Julien Rouhaud wrote:

> On Sun, Jul 24, 2022 at 06:30:44PM -0700, Junio C Hamano wrote:
>> Julien Rouhaud <rjuju123@gmail.com> writes:
>>
>> > Otherwise, would it be acceptable to disable the whole block (the "remove
>> > leading stuff of merges to make the interesting part visible") with some new
>> > configuration option?
>>
>> I personally find that "shortening" logic way too specific to the
>> kernel project hosted at kernel.org and would be inappropriate to
>> use it anywhere else.
>>
>> 	if (length($title) > 50) {
>> 		$title =~ s/^Automatic //;
>> 		$title =~ s/^merge (of|with) /Merge ... /i;
>> 		if (length($title) > 50) {
>> 			$title =~ s/(http|rsync):\/\///;
>> 		}
>> 		if (length($title) > 50) {
>> 			$title =~ s/(master|www|rsync)\.//;
>> 		}
>> 		if (length($title) > 50) {
>> 			$title =~ s/kernel.org:?//;
>> 		}
>> 		if (length($title) > 50) {
>> 			$title =~ s/\/pub\/scm//;
>> 		}
>> 	}
>> 	$co{'title_short'} = chop_str($title, 50, 5);
>
> That's probably true, although some parts (like the protocol) seemed general
> enough to me to be worth considering for general use case, and some of the
> really specific parts seems so specific that they shouldn't really matter if
> used outside kernel.org.
>>
>> Of course, http:// and rsync:// are way outdated (https://, ssh://
>> and git:// are probably reasonable).  Equally outdated is to merge
>> branches from master.kernel.org, www.kernel.org, or rsync.kernel.org
>> (many merges are recorded as pulling from git://git.kernel.org/ or
>> https://git.kernel.org/ these days).
>>
>> Even worse, I somehow thought that kernel.org no longer uses gitweb
>> but uses something else.  So I suspect that nobody sheds tears if we
>> remove the whole block unconditionally.  In fact, it would make the
>> world a better place.
>
> Well, I'm obviously totally fine with getting rid of it.  FWIW, even the
> protocol part wouldn't change anything for the instance hosted by postgres.
>
> I'm attaching an updated v2 patch that removes all that logic and just keeps
> the title_short chopped at 50 chars.

Please "inline" your patches, see "Sending your patches" in
Documentation/SubmittingPatches (I.e. send it with git-send-email, or
similar). I see this as:

> [2. text/plain; v2-0001-gitweb-Remove-title_short-shortening-heuristics.patch]...

In my MUA. But manually extracting it and quoting it:

> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 1835487ab2..e66eb3d9ba 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -3560,23 +3560,6 @@ sub parse_commit_text {
>  		$title =~ s/^    //;
>  		if ($title ne "") {
>  			$co{'title'} = chop_str($title, 80, 5);
> -			# remove leading stuff of merges to make the interesting part visible
> -			if (length($title) > 50) {
> -				$title =~ s/^Automatic //;
> -				$title =~ s/^merge (of|with) /Merge ... /i;
> -				if (length($title) > 50) {
> -					$title =~ s/(http|rsync):\/\///;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/(master|www|rsync)\.//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/kernel.org:?//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/\/pub\/scm//;
> -				}
> -			}
>  			$co{'title_short'} = chop_str($title, 50, 5);
>  			last;

This looks good to me, The one thing I'd add is that we're just running:

	 git rev-list --parents --header  --max-count=1 HEAD

And parsing that, but if we're truncating things perhaps we should just
run "git log" or "git show" with the "%<(<N>[,trunc|ltrunc|mtrunc])"
syntax or similar.

That's obviously a follow-up, but if anyone's interested in deleting
even more code here...

Re: [RFC PATCH] gitweb: improve title shortening heuristics

[Julien Rouhaud]

Hi,

On Mon, Jul 25, 2022 at 07:54:34AM +0200, Ævar Arnfjörð Bjarmason wrote:
>
> Please "inline" your patches, see "Sending your patches" in
> Documentation/SubmittingPatches (I.e. send it with git-send-email, or
> similar). I see this as:
>
> > [2. text/plain; v2-0001-gitweb-Remove-title_short-shortening-heuristics.patch]...

Ah I'm sorry about that.  For the archives sake (and to make sure that I
correctly configured my MUA), I'm reattaching the v2 inline.

> This looks good to me, The one thing I'd add is that we're just running:
>
> 	 git rev-list --parents --header  --max-count=1 HEAD
>
> And parsing that, but if we're truncating things perhaps we should just
> run "git log" or "git show" with the "%<(<N>[,trunc|ltrunc|mtrunc])"
> syntax or similar.
>
> That's obviously a follow-up, but if anyone's interested in deleting
> even more code here...

Note that I didn't try to do anything about that.  I have the feeling that
gitweb isn't wildly used nowadays, especially for bigger projects, so it didn't
seems worth spending too much efforts.

From ea4f31c0a2276f042322a11e258e55223863cd35 Mon Sep 17 00:00:00 2001
From: Julien Rouhaud <julien.rouhaud@free.fr>
Date: Sun, 24 Jul 2022 13:17:19 +0800
Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics

Those heuristics are way outdated and too general to be useful outside of
kernel.org.  Since kernel.org doesn't use gitweb anymore and at least one
project complained about incorrect behavior, entirely remove them.

Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
---

Notes:
    Patch re-submitted inline rather than in attachment, sorry about the
    previous attachments.

 gitweb/gitweb.perl | 17 -----------------
 1 file changed, 17 deletions(-)

diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
index 1835487ab2..e66eb3d9ba 100755
--- a/gitweb/gitweb.perl
+++ b/gitweb/gitweb.perl
@@ -3560,23 +3560,6 @@ sub parse_commit_text {
 		$title =~ s/^    //;
 		if ($title ne "") {
 			$co{'title'} = chop_str($title, 80, 5);
-			# remove leading stuff of merges to make the interesting part visible
-			if (length($title) > 50) {
-				$title =~ s/^Automatic //;
-				$title =~ s/^merge (of|with) /Merge ... /i;
-				if (length($title) > 50) {
-					$title =~ s/(http|rsync):\/\///;
-				}
-				if (length($title) > 50) {
-					$title =~ s/(master|www|rsync)\.//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/kernel.org:?//;
-				}
-				if (length($title) > 50) {
-					$title =~ s/\/pub\/scm//;
-				}
-			}
 			$co{'title_short'} = chop_str($title, 50, 5);
 			last;
 		}
--
2.37.0

Re: [RFC PATCH] gitweb: improve title shortening heuristics

[Junio C Hamano]

Julien Rouhaud <rjuju123@gmail.com> writes:

> Hi,
>
> On Mon, Jul 25, 2022 at 07:54:34AM +0200, Ævar Arnfjörð Bjarmason wrote:
>>
>> Please "inline" your patches, see "Sending your patches" in
>> Documentation/SubmittingPatches (I.e. send it with git-send-email, or
>> similar). I see this as:
>>
>> > [2. text/plain; v2-0001-gitweb-Remove-title_short-shortening-heuristics.patch]...
>
> Ah I'm sorry about that.  For the archives sake (and to make sure that I
> correctly configured my MUA), I'm reattaching the v2 inline.

Thanks.  It is not for the archives' sake; I do not take (or notice
in the first place) patches in attachments, simply because rarely
anybody reviews attachments.

>> This looks good to me, The one thing I'd add is that we're just running:
>>
>> 	 git rev-list --parents --header  --max-count=1 HEAD
>>
>> And parsing that, but if we're truncating things perhaps we should just
>> run "git log" or "git show" with the "%<(<N>[,trunc|ltrunc|mtrunc])"
>> syntax or similar.
>>
>> That's obviously a follow-up, but if anyone's interested in deleting
>> even more code here...
>
> Note that I didn't try to do anything about that.  I have the feeling that
> gitweb isn't wildly used nowadays, especially for bigger projects, so it didn't
> seems worth spending too much efforts.
>
> From ea4f31c0a2276f042322a11e258e55223863cd35 Mon Sep 17 00:00:00 2001

When you are sending the patch in for real, after making a non-patch
discussion (all of the above), you can write a "scissors" line

----- >8 -----

right here, immediately before your "From:" and "Subject:".  What it
does is to tell "git am" on the receiving end to ignore everything
it saw up to that point and pretend the message starts there.

You do not want to include "From <object name>" line that is used as
an internal separator in the mailbox format.  You do not want to
include the "Date:" in-body header, either.  "From:" and "Subject:"
are used often because they are frequently different from the e-mail
(e.g. in the message I am responding to, the sender is rjuju123@gmail,
not julien.rouhaud@free.fr, and we want the latter on the From: line
because that is what appears on the Signed-off-by: line).

> From: Julien Rouhaud <julien.rouhaud@free.fr>
> Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics
>
> Those heuristics are way outdated and too general to be useful outside of
> kernel.org.  Since kernel.org doesn't use gitweb anymore and at least one
> project complained about incorrect behavior, entirely remove them.
>
> Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
> ---
>
> Notes:
>     Patch re-submitted inline rather than in attachment, sorry about the
>     previous attachments.

Thanks.  Will queue.


>  gitweb/gitweb.perl | 17 -----------------
>  1 file changed, 17 deletions(-)
>
> diff --git a/gitweb/gitweb.perl b/gitweb/gitweb.perl
> index 1835487ab2..e66eb3d9ba 100755
> --- a/gitweb/gitweb.perl
> +++ b/gitweb/gitweb.perl
> @@ -3560,23 +3560,6 @@ sub parse_commit_text {
>  		$title =~ s/^    //;
>  		if ($title ne "") {
>  			$co{'title'} = chop_str($title, 80, 5);
> -			# remove leading stuff of merges to make the interesting part visible
> -			if (length($title) > 50) {
> -				$title =~ s/^Automatic //;
> -				$title =~ s/^merge (of|with) /Merge ... /i;
> -				if (length($title) > 50) {
> -					$title =~ s/(http|rsync):\/\///;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/(master|www|rsync)\.//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/kernel.org:?//;
> -				}
> -				if (length($title) > 50) {
> -					$title =~ s/\/pub\/scm//;
> -				}
> -			}
>  			$co{'title_short'} = chop_str($title, 50, 5);
>  			last;
>  		}
> --
> 2.37.0

Re: [RFC PATCH] gitweb: improve title shortening heuristics

[Julien Rouhaud]

Hi,

On Tue, Jul 26, 2022 at 11:31:32PM -0700, Junio C Hamano wrote:
>
> When you are sending the patch in for real, after making a non-patch
> discussion (all of the above), you can write a "scissors" line
>
> ----- >8 -----
>
> right here, immediately before your "From:" and "Subject:".  What it
> does is to tell "git am" on the receiving end to ignore everything
> it saw up to that point and pretend the message starts there.
>
> You do not want to include "From <object name>" line that is used as
> an internal separator in the mailbox format.  You do not want to
> include the "Date:" in-body header, either.  "From:" and "Subject:"
> are used often because they are frequently different from the e-mail
> (e.g. in the message I am responding to, the sender is rjuju123@gmail,
> not julien.rouhaud@free.fr, and we want the latter on the From: line
> because that is what appears on the Signed-off-by: line).

Ok, got it.

> > From: Julien Rouhaud <julien.rouhaud@free.fr>
> > Subject: [PATCH v2] gitweb: Remove title_short shortening heuristics
> >
> > Those heuristics are way outdated and too general to be useful outside of
> > kernel.org.  Since kernel.org doesn't use gitweb anymore and at least one
> > project complained about incorrect behavior, entirely remove them.
> >
> > Signed-off-by: Julien Rouhaud <julien.rouhaud@free.fr>
> > ---
> >
> > Notes:
> >     Patch re-submitted inline rather than in attachment, sorry about the
> >     previous attachments.
>
> Thanks.  Will queue.

Thanks!