empty /manifest.js.gz response as of 520be116

empty /manifest.js.gz response as of 520be116

[Kyle Meyer]

I recently upgraded a server from 08b649735 to 5860b498a and noticed
that grok-pull didn't bring in any updates.  It looks like what's going
on is that the top-level /manifest.js.gz endpoint is now coming up
empty.

A minimal example on 5860b498a:

  $ git clone --mirror https://try.public-inbox.org/test test
  $ export PI_CONFIG="$PWD/config"
  $ public-inbox-init -V1 test test/ \
     http://localhost:8080/test test@public-inbox.org
  $ public-inbox-index -v test
  $ public-inbox-httpd &

  $ curl -fsS http://localhost:8080/manifest.js.gz | zcat | jq
  {}

On 08b649735, the output is

  {
    "/test": {
      "reference": null,
      ...
    }
  }

Bisecting with the script below points to 520be116 (www_listing: start
updating for pagination + search, 2021-06-23).

--8<---------------cut here---------------start------------->8---
#!/bin/sh

perl Makefile.PL && make && sudo make install || exit 125

# Local clone of https://try.public-inbox.org/test
pi_test=/tmp/pi-test

cd "$(mktemp -d "${TMPDIR:-/tmp}"/pi-XXXXXXX)"

export PI_CONFIG="$PWD/config"

git clone --mirror "$pi_test" test
public-inbox-init -V1 test test/ \
                  http://localhost:8080/test test@public-inbox.org
public-inbox-index -v test

trap 'trap - TERM && pkill -P $$' INT TERM EXIT
public-inbox-httpd &

sleep 1

curl -fsS http://localhost:8080/manifest.js.gz | zcat | jq -e '.["/test"]'
--8<---------------cut here---------------end--------------->8---

[PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest

[Eric Wong]

ManifestJsGz->response was not invoking the new "url_filter"
method properly.  Furthermore, fix url_filter for returning 404
responses.

Reported-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87fsx3128a.fsf@kyleam.com/
Fixes: 520be116e8a686cb ("www_listing: start updating for pagination + search")
---
 lib/PublicInbox/ManifestJsGz.pm |  4 +--
 lib/PublicInbox/WwwListing.pm   |  2 +-
 t/www_listing.t                 | 53 ++++++++++++++++++++-------------
 3 files changed, 35 insertions(+), 24 deletions(-)

diff --git a/lib/PublicInbox/ManifestJsGz.pm b/lib/PublicInbox/ManifestJsGz.pm
index 9dc10791..7fee78dd 100644
--- a/lib/PublicInbox/ManifestJsGz.pm
+++ b/lib/PublicInbox/ManifestJsGz.pm
@@ -13,11 +13,11 @@ use HTTP::Date qw(time2str);
 
 my $json = PublicInbox::Config::json();
 
-sub url_regexp {
+sub url_filter {
 	my ($ctx) = @_;
 	# grokmirror uses relative paths, so it's domain-dependent
 	# SUPER calls PublicInbox::WwwListing::url_filter
-	($ctx->url_filter('publicInbox.grokManifest', 'match=domain'))[0];
+	$ctx->SUPER::url_filter('publicInbox.grokManifest', 'match=domain');
 }
 
 sub inject_entry ($$$;$) {
diff --git a/lib/PublicInbox/WwwListing.pm b/lib/PublicInbox/WwwListing.pm
index 76c76663..98a69986 100644
--- a/lib/PublicInbox/WwwListing.pm
+++ b/lib/PublicInbox/WwwListing.pm
@@ -55,7 +55,7 @@ again:
 	} elsif ($v eq 'all') {
 		(qr/./, undef);
 	} elsif ($v eq '404') {
-		undef;
+		(undef, undef);
 	} else {
 		warn <<EOF;
 `$v' is not a valid value for `$key'
diff --git a/t/www_listing.t b/t/www_listing.t
index 6a2892de..6b3b408f 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -1,14 +1,15 @@
+#!perl -w
 # Copyright (C) 2019-2021 all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 # manifest.js.gz generation and grok-pull integration test
 use strict;
-use warnings;
+use v5.10.1;
 use Test::More;
 use PublicInbox::Spawn qw(which);
 use PublicInbox::TestCommon;
 use PublicInbox::Import;
-require_mods(qw(json URI::Escape Plack::Builder Digest::SHA
-		IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
+use IO::Uncompress::Gunzip qw(gunzip);
+require_mods(qw(json URI::Escape Plack::Builder Digest::SHA HTTP::Tiny));
 require PublicInbox::WwwListing;
 require PublicInbox::ManifestJsGz;
 use PublicInbox::Config;
@@ -32,21 +33,23 @@ like($bare->manifest_entry->{fingerprint}, qr/\A[a-f0-9]{40}\z/,
 	'got fingerprint with non-empty repo');
 
 sub tiny_test {
-	my ($json, $host, $port) = @_;
-	my $tmp;
+	my ($json, $host, $port, $html) = @_;
+	my ($tmp, $res);
 	my $http = HTTP::Tiny->new;
-	my $res = $http->get("http://$host:$port/");
-	is($res->{status}, 200, 'got HTML listing');
-	like($res->{content}, qr!</html>!si, 'listing looks like HTML');
-
-	$res = $http->get("http://$host:$port/", {'Accept-Encoding'=>'gzip'});
-	is($res->{status}, 200, 'got gzipped HTML listing');
-	IO::Uncompress::Gunzip::gunzip(\(delete $res->{content}) => \$tmp);
-	like($tmp, qr!</html>!si, 'unzipped listing looks like HTML');
-
+	if ($html) {
+		$res = $http->get("http://$host:$port/");
+		is($res->{status}, 200, 'got HTML listing');
+		like($res->{content}, qr!</html>!si, 'listing looks like HTML');
+
+		$res = $http->get("http://$host:$port/",
+				{'Accept-Encoding'=>'gzip'});
+		is($res->{status}, 200, 'got gzipped HTML listing');
+		gunzip(\(delete $res->{content}) => \$tmp);
+		like($tmp, qr!</html>!si, 'unzipped listing looks like HTML');
+	}
 	$res = $http->get("http://$host:$port/manifest.js.gz");
 	is($res->{status}, 200, 'got manifest');
-	IO::Uncompress::Gunzip::gunzip(\(delete $res->{content}) => \$tmp);
+	gunzip(\(delete $res->{content}) => \$tmp);
 	unlike($tmp, qr/"modified":\s*"/, 'modified is an integer');
 	my $manifest = $json->decode($tmp);
 	ok(my $clone = $manifest->{'/alt'}, '/alt in manifest');
@@ -95,10 +98,9 @@ SKIP: {
 		"lorelei \xc4\x80"), 0,
 		'set gitweb user');
 	ok(unlink("$bare->{git_dir}/description"), 'removed bare/description');
-	open $fh, '>', $cfgfile or die;
-	print $fh <<"" or die;
-[publicinbox]
-	wwwlisting = all
+	open $fh, '>', $cfgfile or xbail "open $cfgfile: $!";
+	$fh->autoflush(1);
+	print $fh <<"" or xbail "print $!";
 [publicinbox "bare"]
 	inboxdir = $bare->{git_dir}
 	url = http://$host/bare
@@ -112,13 +114,22 @@ SKIP: {
 	url = http://$host/v2
 	address = v2\@example.com
 
-	close $fh or die;
 	my $env = { PI_CONFIG => $cfgfile };
 	my $cmd = [ '-httpd', '-W0', "--stdout=$out", "--stderr=$err" ];
 	$td = start_script($cmd, $env, { 3 => $sock });
-	$sock = undef;
 
+	# default publicinboxGrokManifest match=domain default
 	tiny_test($json, $host, $port);
+	undef $td;
+
+	print $fh <<"" or xbail "print $!";
+[publicinbox]
+	wwwlisting = all
+
+	close $fh or xbail "close $!";
+	$td = start_script($cmd, $env, { 3 => $sock });
+	tiny_test($json, $host, $port, 1);
+	undef $sock;
 
 	my $grok_pull = which('grok-pull') or
 		skip('grok-pull not available', 12);

Re: empty /manifest.js.gz response as of 520be116

[Konstantin Ryabitsev]

On Sun, Jun 27, 2021 at 04:28:37PM -0400, Kyle Meyer wrote:
> I recently upgraded a server from 08b649735 to 5860b498a and noticed
> that grok-pull didn't bring in any updates.  It looks like what's going
> on is that the top-level /manifest.js.gz endpoint is now coming up
> empty.

BTW, I just added another hook feature to grokmirror that may be useful to
mirrors. This one is called when all new repos have finished cloning and will
hopefully help avoid situations where something like 3.git is done cloning
before any other epoch. It also helps keep things tidy and separate clone
events (that require public-inbox-init) from update events (that just need
public-inbox-index).

https://github.com/mricon/grokmirror/blob/master/grokmirror.conf#L142

It's not released yet, but hopefully 2.0.9 will be out after I do some more
testing.

-K

Re: [PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest

[Kyle Meyer]

Eric Wong writes:

> Fixes: 520be116e8a686cb ("www_listing: start updating for pagination + search")

Confirmed.  Thank you for the quick fix.

Re: empty /manifest.js.gz response as of 520be116

[Kyle Meyer]

Konstantin Ryabitsev writes:

> BTW, I just added another hook feature to grokmirror that may be useful to
> mirrors. [...]

Nice, thanks for the pointer.