user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* empty /manifest.js.gz response as of 520be116
@ 2021-06-27 20:28 Kyle Meyer
  2021-06-28 21:59 ` [PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest Eric Wong
  2021-06-29 13:02 ` empty /manifest.js.gz response as of 520be116 Konstantin Ryabitsev
  0 siblings, 2 replies; 5+ messages in thread
From: Kyle Meyer @ 2021-06-27 20:28 UTC (permalink / raw)
  To: meta

I recently upgraded a server from 08b649735 to 5860b498a and noticed
that grok-pull didn't bring in any updates.  It looks like what's going
on is that the top-level /manifest.js.gz endpoint is now coming up
empty.

A minimal example on 5860b498a:

  $ git clone --mirror https://try.public-inbox.org/test test
  $ export PI_CONFIG="$PWD/config"
  $ public-inbox-init -V1 test test/ \
     http://localhost:8080/test test@public-inbox.org
  $ public-inbox-index -v test
  $ public-inbox-httpd &

  $ curl -fsS http://localhost:8080/manifest.js.gz | zcat | jq
  {}

On 08b649735, the output is

  {
    "/test": {
      "reference": null,
      ...
    }
  }

Bisecting with the script below points to 520be116 (www_listing: start
updating for pagination + search, 2021-06-23).

--8<---------------cut here---------------start------------->8---
#!/bin/sh

perl Makefile.PL && make && sudo make install || exit 125

# Local clone of https://try.public-inbox.org/test
pi_test=/tmp/pi-test

cd "$(mktemp -d "${TMPDIR:-/tmp}"/pi-XXXXXXX)"

export PI_CONFIG="$PWD/config"

git clone --mirror "$pi_test" test
public-inbox-init -V1 test test/ \
                  http://localhost:8080/test test@public-inbox.org
public-inbox-index -v test

trap 'trap - TERM && pkill -P $$' INT TERM EXIT
public-inbox-httpd &

sleep 1

curl -fsS http://localhost:8080/manifest.js.gz | zcat | jq -e '.["/test"]'
--8<---------------cut here---------------end--------------->8---

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest
  2021-06-27 20:28 empty /manifest.js.gz response as of 520be116 Kyle Meyer
@ 2021-06-28 21:59 ` Eric Wong
  2021-06-30  2:37   ` Kyle Meyer
  2021-06-29 13:02 ` empty /manifest.js.gz response as of 520be116 Konstantin Ryabitsev
  1 sibling, 1 reply; 5+ messages in thread
From: Eric Wong @ 2021-06-28 21:59 UTC (permalink / raw)
  To: meta; +Cc: Kyle Meyer

ManifestJsGz->response was not invoking the new "url_filter"
method properly.  Furthermore, fix url_filter for returning 404
responses.

Reported-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87fsx3128a.fsf@kyleam.com/
Fixes: 520be116e8a686cb ("www_listing: start updating for pagination + search")
---
 lib/PublicInbox/ManifestJsGz.pm |  4 +--
 lib/PublicInbox/WwwListing.pm   |  2 +-
 t/www_listing.t                 | 53 ++++++++++++++++++++-------------
 3 files changed, 35 insertions(+), 24 deletions(-)

diff --git a/lib/PublicInbox/ManifestJsGz.pm b/lib/PublicInbox/ManifestJsGz.pm
index 9dc10791..7fee78dd 100644
--- a/lib/PublicInbox/ManifestJsGz.pm
+++ b/lib/PublicInbox/ManifestJsGz.pm
@@ -13,11 +13,11 @@ use HTTP::Date qw(time2str);
 
 my $json = PublicInbox::Config::json();
 
-sub url_regexp {
+sub url_filter {
 	my ($ctx) = @_;
 	# grokmirror uses relative paths, so it's domain-dependent
 	# SUPER calls PublicInbox::WwwListing::url_filter
-	($ctx->url_filter('publicInbox.grokManifest', 'match=domain'))[0];
+	$ctx->SUPER::url_filter('publicInbox.grokManifest', 'match=domain');
 }
 
 sub inject_entry ($$$;$) {
diff --git a/lib/PublicInbox/WwwListing.pm b/lib/PublicInbox/WwwListing.pm
index 76c76663..98a69986 100644
--- a/lib/PublicInbox/WwwListing.pm
+++ b/lib/PublicInbox/WwwListing.pm
@@ -55,7 +55,7 @@ again:
 	} elsif ($v eq 'all') {
 		(qr/./, undef);
 	} elsif ($v eq '404') {
-		undef;
+		(undef, undef);
 	} else {
 		warn <<EOF;
 `$v' is not a valid value for `$key'
diff --git a/t/www_listing.t b/t/www_listing.t
index 6a2892de..6b3b408f 100644
--- a/t/www_listing.t
+++ b/t/www_listing.t
@@ -1,14 +1,15 @@
+#!perl -w
 # Copyright (C) 2019-2021 all contributors <meta@public-inbox.org>
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 # manifest.js.gz generation and grok-pull integration test
 use strict;
-use warnings;
+use v5.10.1;
 use Test::More;
 use PublicInbox::Spawn qw(which);
 use PublicInbox::TestCommon;
 use PublicInbox::Import;
-require_mods(qw(json URI::Escape Plack::Builder Digest::SHA
-		IO::Compress::Gzip IO::Uncompress::Gunzip HTTP::Tiny));
+use IO::Uncompress::Gunzip qw(gunzip);
+require_mods(qw(json URI::Escape Plack::Builder Digest::SHA HTTP::Tiny));
 require PublicInbox::WwwListing;
 require PublicInbox::ManifestJsGz;
 use PublicInbox::Config;
@@ -32,21 +33,23 @@ like($bare->manifest_entry->{fingerprint}, qr/\A[a-f0-9]{40}\z/,
 	'got fingerprint with non-empty repo');
 
 sub tiny_test {
-	my ($json, $host, $port) = @_;
-	my $tmp;
+	my ($json, $host, $port, $html) = @_;
+	my ($tmp, $res);
 	my $http = HTTP::Tiny->new;
-	my $res = $http->get("http://$host:$port/");
-	is($res->{status}, 200, 'got HTML listing');
-	like($res->{content}, qr!</html>!si, 'listing looks like HTML');
-
-	$res = $http->get("http://$host:$port/", {'Accept-Encoding'=>'gzip'});
-	is($res->{status}, 200, 'got gzipped HTML listing');
-	IO::Uncompress::Gunzip::gunzip(\(delete $res->{content}) => \$tmp);
-	like($tmp, qr!</html>!si, 'unzipped listing looks like HTML');
-
+	if ($html) {
+		$res = $http->get("http://$host:$port/");
+		is($res->{status}, 200, 'got HTML listing');
+		like($res->{content}, qr!</html>!si, 'listing looks like HTML');
+
+		$res = $http->get("http://$host:$port/",
+				{'Accept-Encoding'=>'gzip'});
+		is($res->{status}, 200, 'got gzipped HTML listing');
+		gunzip(\(delete $res->{content}) => \$tmp);
+		like($tmp, qr!</html>!si, 'unzipped listing looks like HTML');
+	}
 	$res = $http->get("http://$host:$port/manifest.js.gz");
 	is($res->{status}, 200, 'got manifest');
-	IO::Uncompress::Gunzip::gunzip(\(delete $res->{content}) => \$tmp);
+	gunzip(\(delete $res->{content}) => \$tmp);
 	unlike($tmp, qr/"modified":\s*"/, 'modified is an integer');
 	my $manifest = $json->decode($tmp);
 	ok(my $clone = $manifest->{'/alt'}, '/alt in manifest');
@@ -95,10 +98,9 @@ SKIP: {
 		"lorelei \xc4\x80"), 0,
 		'set gitweb user');
 	ok(unlink("$bare->{git_dir}/description"), 'removed bare/description');
-	open $fh, '>', $cfgfile or die;
-	print $fh <<"" or die;
-[publicinbox]
-	wwwlisting = all
+	open $fh, '>', $cfgfile or xbail "open $cfgfile: $!";
+	$fh->autoflush(1);
+	print $fh <<"" or xbail "print $!";
 [publicinbox "bare"]
 	inboxdir = $bare->{git_dir}
 	url = http://$host/bare
@@ -112,13 +114,22 @@ SKIP: {
 	url = http://$host/v2
 	address = v2\@example.com
 
-	close $fh or die;
 	my $env = { PI_CONFIG => $cfgfile };
 	my $cmd = [ '-httpd', '-W0', "--stdout=$out", "--stderr=$err" ];
 	$td = start_script($cmd, $env, { 3 => $sock });
-	$sock = undef;
 
+	# default publicinboxGrokManifest match=domain default
 	tiny_test($json, $host, $port);
+	undef $td;
+
+	print $fh <<"" or xbail "print $!";
+[publicinbox]
+	wwwlisting = all
+
+	close $fh or xbail "close $!";
+	$td = start_script($cmd, $env, { 3 => $sock });
+	tiny_test($json, $host, $port, 1);
+	undef $sock;
 
 	my $grok_pull = which('grok-pull') or
 		skip('grok-pull not available', 12);

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: empty /manifest.js.gz response as of 520be116
  2021-06-27 20:28 empty /manifest.js.gz response as of 520be116 Kyle Meyer
  2021-06-28 21:59 ` [PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest Eric Wong
@ 2021-06-29 13:02 ` Konstantin Ryabitsev
  2021-06-30  2:38   ` Kyle Meyer
  1 sibling, 1 reply; 5+ messages in thread
From: Konstantin Ryabitsev @ 2021-06-29 13:02 UTC (permalink / raw)
  To: Kyle Meyer; +Cc: meta

On Sun, Jun 27, 2021 at 04:28:37PM -0400, Kyle Meyer wrote:
> I recently upgraded a server from 08b649735 to 5860b498a and noticed
> that grok-pull didn't bring in any updates.  It looks like what's going
> on is that the top-level /manifest.js.gz endpoint is now coming up
> empty.

BTW, I just added another hook feature to grokmirror that may be useful to
mirrors. This one is called when all new repos have finished cloning and will
hopefully help avoid situations where something like 3.git is done cloning
before any other epoch. It also helps keep things tidy and separate clone
events (that require public-inbox-init) from update events (that just need
public-inbox-index).

https://github.com/mricon/grokmirror/blob/master/grokmirror.conf#L142

It's not released yet, but hopefully 2.0.9 will be out after I do some more
testing.

-K

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest
  2021-06-28 21:59 ` [PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest Eric Wong
@ 2021-06-30  2:37   ` Kyle Meyer
  0 siblings, 0 replies; 5+ messages in thread
From: Kyle Meyer @ 2021-06-30  2:37 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong writes:

> Fixes: 520be116e8a686cb ("www_listing: start updating for pagination + search")

Confirmed.  Thank you for the quick fix.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: empty /manifest.js.gz response as of 520be116
  2021-06-29 13:02 ` empty /manifest.js.gz response as of 520be116 Konstantin Ryabitsev
@ 2021-06-30  2:38   ` Kyle Meyer
  0 siblings, 0 replies; 5+ messages in thread
From: Kyle Meyer @ 2021-06-30  2:38 UTC (permalink / raw)
  To: Konstantin Ryabitsev; +Cc: meta

Konstantin Ryabitsev writes:

> BTW, I just added another hook feature to grokmirror that may be useful to
> mirrors. [...]

Nice, thanks for the pointer.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-06-30  2:38 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-27 20:28 empty /manifest.js.gz response as of 520be116 Kyle Meyer
2021-06-28 21:59 ` [PATCH] www: fix manifest.js.gz for default publicInbox.grokManifest Eric Wong
2021-06-30  2:37   ` Kyle Meyer
2021-06-29 13:02 ` empty /manifest.js.gz response as of 520be116 Konstantin Ryabitsev
2021-06-30  2:38   ` Kyle Meyer

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.mail.public-inbox.meta
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.mail.public-inbox.meta
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git