user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
thread overview below | download mbox.gz: |
* Re: Bug related to (maybe?) / in Message-Id
  2023-02-16 21:36  0% ` Eric Wong
@ 2023-02-17  8:52  0%   ` Uwe Kleine-König
  0 siblings, 0 replies; 6+ results
From: Uwe Kleine-König @ 2023-02-17  8:52 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

[-- Attachment #1: Type: text/plain, Size: 6047 bytes --]

Hello Eric,

first of all: Thanks for your quick answer.

On Thu, Feb 16, 2023 at 09:36:28PM +0000, Eric Wong wrote:
> Uwe Kleine-König <u.kleine-koenig@pengutronix.de> wrote:
> > Hello,
> > 
> > The mail by Alexander Dahl that is (currently) the first hit on
> > https://lore.ptxdist.org/ptxdist/?q=ptxd_make_world_compile_commands_filter
> > results in a 404 when I follow the link.
> > 
> > The original mail has
> > 
> > 	Message-ID: <Y+07h0l/zJJAgs9s@falbala.internal.home.lespocky.de>
> > 
> > and the corresponding link is:
> > 
> > 	https://lore.ptxdist.org/ptxdist/Y+07h0l%2FzJJAgs9s@falbala.internal.home.lespocky.de/
> > 
> > I noticed this on public-inbox 1.8.0-1~bpo11+1 from Debian, upgrading to
> > 1.9.0-1~bpo11+1 didn't help.
> > 
> > Other mails with / in Message-Id are not accessible either, I tested
> > with:
> > 
> > 	YyHu/412LT8uQTy1@lenoch
> > 	Y0/5xdFZO3u0952+@lenoch
> 
> The TODO file has this:
> 
> 	* use REQUEST_URI properly for CGI / mod_perl2 compatibility
> 	  with Message-IDs which include '%' (done?)
> 
> So I guess it's not done...  To deal with '/' in the Message-ID,
> $env->{REQUEST_URI} really needs to be the raw, undecoded URI
> specified in the PSGI specs[1].
> 
> I'm not sure how to go about it Apache+CGI or mod_perl2..
> 
> Fwiw, the recommended configuration is:
> (nginx|haproxy) -> varnish -> public-inbox-{httpd,netd}
> 
> Maybe Apache2 mpm_event reverse proxy can work in lieu of
> (nginx|haproxy), but /T/, /t/, /t.mbox.gz requests are a bit
> faster on -httpd/-netd since 1.6+ on SMP machines.
> 
> > I also wonder why these mails yield the webserver's 404 page and not the
> > one provided by the public-inbox cgi?!
> 
> This may be the small size public-inbox's 404 page.  I don't
> know Apache configs well, but I know nginx did something
> similar.

> > Is this a problem in public-inbox, or is the apache configuration
> > somehow borked? Any hints welcome.
> 
> Do you have access to that server and can show us the configs?
> REQUEST_URI really needs to be raw in accordance to PSGI specs.
> 
> This can dump the request $env to stderr and show us
> REQUEST_URI, PATH_INFO, SCRIPT_NAME, and anything else
> which may enlighten us:
> 
> diff --git a/lib/PublicInbox/WWW.pm b/lib/PublicInbox/WWW.pm
> index 9ffcb879..f67fe8e6 100644
> --- a/lib/PublicInbox/WWW.pm
> +++ b/lib/PublicInbox/WWW.pm
> @@ -52,7 +52,8 @@ sub call {
>  		# none of the keys we care about will need escaping
>  		($k // '', uri_unescape($v // ''))
>  	} split(/[&;]+/, $env->{QUERY_STRING});
> -
> +	use Data::Dumper; $Data::Dumper::Useqq = 1;
> +	warn Dumper($env);
>  	my $path_info = path_info_raw($env);
>  	my $method = $env->{REQUEST_METHOD};

I added that patch and for the reported request this didn't trigger,
which I assume means that public-inbox isn't called at all.
 
Playing around with slashes got my admin and me on the right trail:
https://httpd.apache.org/docs/current/mod/core.html#allowencodedslashes

We set that to "On" and now it (mostly) works. Maybe it's worth adding
this hint to the documentation even though apache isn't the most
recommended setup? Maybe other servers have a similar security setting?

I wrote "mostly" because

	https://lore.ptxdist.org/ptxdist/Y+07h0l%2FzJJAgs9s@falbala.internal.home.lespocky.de/
	https://lore.ptxdist.org/ptxdist/Y+07h0l%2FzJJAgs9s@falbala.internal.home.lespocky.de
	https://lore.ptxdist.org/ptxdist/Y+07h0l/zJJAgs9s@falbala.internal.home.lespocky.de/

work as expected;

	https://lore.ptxdist.org/ptxdist/Y+07h0l/zJJAgs9s@falbala.internal.home.lespocky.de

however does not, that yields a short "Not Found".

With the patch applied the logged stuff for these URLs is mostly
identical. REMOTE_PORT differs which is expected. Otherwise only
PATH_INFO, PATH_TRANSLATED and REQUEST_URI differ. They are
respectively:

	"PATH_INFO" => "/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de/",
	"PATH_TRANSLATED" => "/usr/lib/cgi-bin/public-inbox.cgi/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de/",
	"REQUEST_URI" => "/ptxdist/Y+07h0l%2FzJJAgs9s\@falbala.internal.home.lespocky.de/",

	"PATH_INFO" => "/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de",
	"PATH_TRANSLATED" => "/usr/lib/cgi-bin/public-inbox.cgi/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de",
	"REQUEST_URI" => "/ptxdist/Y+07h0l%2FzJJAgs9s\@falbala.internal.home.lespocky.de",

	"PATH_INFO" => "/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de/",
	"PATH_TRANSLATED" => "/usr/lib/cgi-bin/public-inbox.cgi/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de/",
	"REQUEST_URI" => "/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de/",

	"PATH_INFO" => "/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de",
	"PATH_TRANSLATED" => "/usr/lib/cgi-bin/public-inbox.cgi/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de",
	"REQUEST_URI" => "/ptxdist/Y+07h0l/zJJAgs9s\@falbala.internal.home.lespocky.de",

which I think is all as expected. In all cases we have

	"SCRIPT_NAME" => "",

. Not sure making the last URL work is easily possible (and worth the
effort)? If a Message-Id ends in "/T" or similar the result will always
be ambigous?

One thing I just noticed is:

$ curl https://lore.ptxdist.org/ptxdist/Y+07h0l/zJJAgs9s@falbala.internal.home.lespocky.de/T
Redirecting to https://lore.ptxdist.org/ptxdist/Y+07h0l/zJJAgs9s@falbala.internal.home.lespocky.de/T

which makes Firefox say: "The page isn’t redirecting properly". It works
fine with the / replaced by %2F:

$ curl https://lore.ptxdist.org/ptxdist/Y+07h0l%2fzJJAgs9s@falbala.internal.home.lespocky.de/T
Redirecting to https://lore.ptxdist.org/ptxdist/Y+07h0l%2fzJJAgs9s@falbala.internal.home.lespocky.de/T/#u

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | https://www.pengutronix.de/ |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[relevance 0%]

* Re: Bug related to (maybe?) / in Message-Id
  2023-02-16 21:05  4% Bug related to (maybe?) / in Message-Id Uwe Kleine-König
@ 2023-02-16 21:36  0% ` Eric Wong
  2023-02-17  8:52  0%   ` Uwe Kleine-König
  0 siblings, 1 reply; 6+ results
From: Eric Wong @ 2023-02-16 21:36 UTC (permalink / raw)
  To: Uwe Kleine-König; +Cc: meta

Uwe Kleine-König <u.kleine-koenig@pengutronix.de> wrote:
> Hello,
> 
> The mail by Alexander Dahl that is (currently) the first hit on
> https://lore.ptxdist.org/ptxdist/?q=ptxd_make_world_compile_commands_filter
> results in a 404 when I follow the link.
> 
> The original mail has
> 
> 	Message-ID: <Y+07h0l/zJJAgs9s@falbala.internal.home.lespocky.de>
> 
> and the corresponding link is:
> 
> 	https://lore.ptxdist.org/ptxdist/Y+07h0l%2FzJJAgs9s@falbala.internal.home.lespocky.de/
> 
> I noticed this on public-inbox 1.8.0-1~bpo11+1 from Debian, upgrading to
> 1.9.0-1~bpo11+1 didn't help.
> 
> Other mails with / in Message-Id are not accessible either, I tested
> with:
> 
> 	YyHu/412LT8uQTy1@lenoch
> 	Y0/5xdFZO3u0952+@lenoch

The TODO file has this:

	* use REQUEST_URI properly for CGI / mod_perl2 compatibility
	  with Message-IDs which include '%' (done?)

So I guess it's not done...  To deal with '/' in the Message-ID,
$env->{REQUEST_URI} really needs to be the raw, undecoded URI
specified in the PSGI specs[1].

I'm not sure how to go about it Apache+CGI or mod_perl2..

Fwiw, the recommended configuration is:
(nginx|haproxy) -> varnish -> public-inbox-{httpd,netd}

Maybe Apache2 mpm_event reverse proxy can work in lieu of
(nginx|haproxy), but /T/, /t/, /t.mbox.gz requests are a bit
faster on -httpd/-netd since 1.6+ on SMP machines.

> I also wonder why these mails yield the webserver's 404 page and not the
> one provided by the public-inbox cgi?!

This may be the small size public-inbox's 404 page.  I don't
know Apache configs well, but I know nginx did something
similar.

> Is this a problem in public-inbox, or is the apache configuration
> somehow borked? Any hints welcome.

Do you have access to that server and can show us the configs?
REQUEST_URI really needs to be raw in accordance to PSGI specs.

This can dump the request $env to stderr and show us
REQUEST_URI, PATH_INFO, SCRIPT_NAME, and anything else
which may enlighten us:

diff --git a/lib/PublicInbox/WWW.pm b/lib/PublicInbox/WWW.pm
index 9ffcb879..f67fe8e6 100644
--- a/lib/PublicInbox/WWW.pm
+++ b/lib/PublicInbox/WWW.pm
@@ -52,7 +52,8 @@ sub call {
 		# none of the keys we care about will need escaping
 		($k // '', uri_unescape($v // ''))
 	} split(/[&;]+/, $env->{QUERY_STRING});
-
+	use Data::Dumper; $Data::Dumper::Useqq = 1;
+	warn Dumper($env);
 	my $path_info = path_info_raw($env);
 	my $method = $env->{REQUEST_METHOD};
 

[1] PSGI specs: git clone https://github.com/plack/psgi-specs

^ permalink raw reply related	[relevance 0%]

* Bug related to (maybe?) / in Message-Id
@ 2023-02-16 21:05  4% Uwe Kleine-König
  2023-02-16 21:36  0% ` Eric Wong
  0 siblings, 1 reply; 6+ results
From: Uwe Kleine-König @ 2023-02-16 21:05 UTC (permalink / raw)
  To: meta

[-- Attachment #1: Type: text/plain, Size: 1053 bytes --]

Hello,

The mail by Alexander Dahl that is (currently) the first hit on
https://lore.ptxdist.org/ptxdist/?q=ptxd_make_world_compile_commands_filter
results in a 404 when I follow the link.

The original mail has

	Message-ID: <Y+07h0l/zJJAgs9s@falbala.internal.home.lespocky.de>

and the corresponding link is:

	https://lore.ptxdist.org/ptxdist/Y+07h0l%2FzJJAgs9s@falbala.internal.home.lespocky.de/

I noticed this on public-inbox 1.8.0-1~bpo11+1 from Debian, upgrading to
1.9.0-1~bpo11+1 didn't help.

Other mails with / in Message-Id are not accessible either, I tested
with:

	YyHu/412LT8uQTy1@lenoch
	Y0/5xdFZO3u0952+@lenoch

I also wonder why these mails yield the webserver's 404 page and not the
one provided by the public-inbox cgi?!

Is this a problem in public-inbox, or is the apache configuration
somehow borked? Any hints welcome.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | https://www.pengutronix.de/ |

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[relevance 4%]

* Re: [ANNOUNCE] public-inbox 1.8.0
  2022-04-23  8:22 14% [ANNOUNCE] public-inbox 1.8.0 Eric Wong
@ 2022-04-23 20:23  5% ` Eric Wong
  0 siblings, 0 replies; 6+ results
From: Eric Wong @ 2022-04-23 20:23 UTC (permalink / raw)
  To: meta

Eric Wong <e@80x24.org> wrote:
> A minor release focused on bugfixes and minor improvements.
> Upgrades should happen transparently, but downgrading back to
> 1.7.0 will likely cause problems for lei users (and only lei
> users).

Upgrading notes: lei users need to issue an "lei daemon-kill"
to restart the daemon after upgrading.

^ permalink raw reply	[relevance 5%]

* [ANNOUNCE] public-inbox 1.8.0
@ 2022-04-23  8:22 14% Eric Wong
  2022-04-23 20:23  5% ` Eric Wong
  0 siblings, 1 reply; 6+ results
From: Eric Wong @ 2022-04-23  8:22 UTC (permalink / raw)
  To: meta

A minor release focused on bugfixes and minor improvements.
Upgrades should happen transparently, but downgrading back to
1.7.0 will likely cause problems for lei users (and only lei
users).

lei users may experience duplicate messages in Maildirs if attempting to
downgrade from 1.8.0 to 1.7.x.  public-inbox-* tools are unaffected and
may downgrade freely.

Bugfixes:

  Numerous test fixes thanks to NixOS developers.

  Long-running daemons are more robust in case of corrupt blobs
  or crashes of git-cat-file processes

  PublicInbox::WWW: all CR are removed before LF, fixing display of
  CR-CR-LF messages.

  Solver supports SHA-256 code repositories (inbox and lei store support
  is still pending).

Internal updates:

  Reduced dependencies on Inline::C for Linux users; Linux users may
  now use lei with neither Inline::C nor Socket::MsgHdr installed.

New features:

  The --dangerous flag is now supported in public-inbox-index and
  public-inbox-extindex to use the Xapian::DB_DANGEROUS flag for initial
  indexes.  This may reduce SSD/HDD wear at the expense of disallowing
  concurrency and data integrity in case of an unexpected shutdown.

Please report bugs via plain-text mail to: meta@public-inbox.org

See archives at https://public-inbox.org/meta/ for all history.
See https://public-inbox.org/TODO for what the future holds.

^ permalink raw reply	[relevance 14%]

* [PATCH] doc: add WIP release notes for 1.8
@ 2022-03-11  5:21  6% Eric Wong
  0 siblings, 0 replies; 6+ results
From: Eric Wong @ 2022-03-11  5:21 UTC (permalink / raw)
  To: meta

1.8 will be a minor release, soon (I initially expected to
release it in December, but was side-tracked).  Major features
will be for 1.9.
---
 Documentation/RelNotes/v1.8.0.wip | 36 +++++++++++++++++++++++++++++++
 MANIFEST                          |  1 +
 2 files changed, 37 insertions(+)
 create mode 100644 Documentation/RelNotes/v1.8.0.wip

diff --git a/Documentation/RelNotes/v1.8.0.wip b/Documentation/RelNotes/v1.8.0.wip
new file mode 100644
index 00000000..b662e186
--- /dev/null
+++ b/Documentation/RelNotes/v1.8.0.wip
@@ -0,0 +1,36 @@
+To: meta@public-inbox.org
+Subject: [WIP] public-inbox 1.8.0
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Disposition: inline
+
+A minor release focused on bugfixes and minor improvements.
+
+lei users may experience duplicate messages in Maildirs if attempting to
+downgrade from 1.8.0 to 1.7.x.  public-inbox-* tools are unaffected and
+may downgrade freely.
+
+Bugfixes:
+
+  Numerous test fixes thanks to NixOS developers.
+
+  Long-running daemons are more robust in case of corrupt blobs
+  or crashes of git-cat-file processes
+
+  PublicInbox::WWW: all CR are removed before LF, fixing display of
+  CR-CR-LF messages.
+
+Internal updates:
+
+  Reduced dependencies on Inline::C for Linux users.
+
+New features:
+
+  The --dangerous flag is now supported in public-inbox-index and
+  public-inbox-extindex to use the Xapian::DB_DANGEROUS flag for initial
+  indexes.  This may reduce SSD/HDD wear at the expense of
+
+Please report bugs via plain-text mail to: meta@public-inbox.org
+
+See archives at https://public-inbox.org/meta/ for all history.
+See https://public-inbox.org/TODO for what the future holds.
diff --git a/MANIFEST b/MANIFEST
index ca840210..92c9f891 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -12,6 +12,7 @@ Documentation/RelNotes/v1.5.0.eml
 Documentation/RelNotes/v1.6.0.eml
 Documentation/RelNotes/v1.6.1.eml
 Documentation/RelNotes/v1.7.0.eml
+Documentation/RelNotes/v1.8.0.wip
 Documentation/clients.txt
 Documentation/common.perl
 Documentation/dc-dlvr-spam-flow.txt

^ permalink raw reply related	[relevance 6%]

Results 1-6 of 6 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2022-03-11  5:21  6% [PATCH] doc: add WIP release notes for 1.8 Eric Wong
2022-04-23  8:22 14% [ANNOUNCE] public-inbox 1.8.0 Eric Wong
2022-04-23 20:23  5% ` Eric Wong
2023-02-16 21:05  4% Bug related to (maybe?) / in Message-Id Uwe Kleine-König
2023-02-16 21:36  0% ` Eric Wong
2023-02-17  8:52  0%   ` Uwe Kleine-König

Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).