* [PATCH 00/11] www: export SQLite altid dumps
@ 2020-03-21 2:03 6% Eric Wong
2020-03-21 2:03 7% ` [PATCH 05/11] wwwstream: oneshot sets content-length Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2020-03-21 2:03 UTC (permalink / raw)
To: meta
To improve reproducibility in mirrors, altid dumps can be
exported via "POST /$INBOX_URL/$prefix.sql.gz". $prefix is
something like "gmane" (though the search prefix is "gmane:"
with a colon).
Eric Wong (11):
qspawn: reinstate filter support, add gzip filter
gzipfilter: lazy allocate the deflate context
wwwstream: introduce oneshot API to avoid ->getline
extmsg: use WwwResponse::oneshot
wwwstream: oneshot sets content-length
mbox: need_gzip uses WwwStream::oneshot
qspawn: handle ENOENT (and other errors on exec)
search: clobber -user_pfx on query parser initialization
wwwtext: show thread endpoints info w/ indexlevel=basic
altid: warn about non-word prefixes
www: add endpoint to retrieve altid dumps
MANIFEST | 4 ++
lib/PublicInbox/AltId.pm | 3 +-
lib/PublicInbox/ExtMsg.pm | 4 +-
lib/PublicInbox/GetlineBody.pm | 21 ++++----
lib/PublicInbox/GzipFilter.pm | 59 +++++++++++++++++++++
lib/PublicInbox/Mbox.pm | 16 +++---
lib/PublicInbox/Qspawn.pm | 66 ++++++++++++++----------
lib/PublicInbox/Search.pm | 4 +-
lib/PublicInbox/ViewVCS.pm | 8 +--
lib/PublicInbox/WWW.pm | 14 ++++-
lib/PublicInbox/WwwAltId.pm | 94 ++++++++++++++++++++++++++++++++++
lib/PublicInbox/WwwStream.pm | 29 +++++++++--
lib/PublicInbox/WwwText.pm | 10 +++-
t/gzip_filter.t | 37 +++++++++++++
t/httpd-corner.psgi | 16 ++++++
t/httpd-corner.t | 48 +++++++++++++++++
t/www_altid.t | 83 ++++++++++++++++++++++++++++++
17 files changed, 452 insertions(+), 64 deletions(-)
create mode 100644 lib/PublicInbox/GzipFilter.pm
create mode 100644 lib/PublicInbox/WwwAltId.pm
create mode 100644 t/gzip_filter.t
create mode 100644 t/www_altid.t
^ permalink raw reply [relevance 6%]
* [PATCH 05/11] wwwstream: oneshot sets content-length
2020-03-21 2:03 6% [PATCH 00/11] www: export SQLite altid dumps Eric Wong
@ 2020-03-21 2:03 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2020-03-21 2:03 UTC (permalink / raw)
To: meta
PublicInbox::HTTP will chunk, otherwise, and that's
extra overhead which isn't needed.
---
lib/PublicInbox/WwwStream.pm | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
diff --git a/lib/PublicInbox/WwwStream.pm b/lib/PublicInbox/WwwStream.pm
index fceef745..985e0262 100644
--- a/lib/PublicInbox/WwwStream.pm
+++ b/lib/PublicInbox/WwwStream.pm
@@ -9,6 +9,7 @@
package PublicInbox::WwwStream;
use strict;
use warnings;
+use bytes (); # length
use PublicInbox::Hval qw(ascii_html prurl);
our $TOR_URL = 'https://www.torproject.org/';
our $CODE_URL = 'https://public-inbox.org/public-inbox.git';
@@ -170,14 +171,18 @@ sub getline {
}
sub oneshot {
- my ($ctx, $code, $strref) = @_;
+ my ($ctx, $code, $sref) = @_;
my $self = bless {
ctx => $ctx,
base_url => base_url($ctx),
}, __PACKAGE__;
- [ $code, [ 'Content-Type', 'text/html; charset=UTF-8' ], [
- _html_top($self), $strref ? $$strref : (), _html_end($self)
- ] ]
+ my @x = (_html_top($self), $sref ? $$sref : (), _html_end($self));
+ my $len = 0;
+ $len += bytes::length($_) for @x;
+ [ $code, [
+ 'Content-Type' => 'text/html; charset=UTF-8',
+ 'Content-Length' => $len
+ ], \@x ];
}
1;
^ permalink raw reply related [relevance 7%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-03-21 2:03 6% [PATCH 00/11] www: export SQLite altid dumps Eric Wong
2020-03-21 2:03 7% ` [PATCH 05/11] wwwstream: oneshot sets content-length Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).