From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 31DA41F619 for ; Sat, 21 Mar 2020 02:03:54 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/11] www: export SQLite altid dumps Date: Sat, 21 Mar 2020 02:03:43 +0000 Message-Id: <20200321020354.9056-1-e@yhbt.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: To improve reproducibility in mirrors, altid dumps can be exported via "POST /$INBOX_URL/$prefix.sql.gz". $prefix is something like "gmane" (though the search prefix is "gmane:" with a colon). Eric Wong (11): qspawn: reinstate filter support, add gzip filter gzipfilter: lazy allocate the deflate context wwwstream: introduce oneshot API to avoid ->getline extmsg: use WwwResponse::oneshot wwwstream: oneshot sets content-length mbox: need_gzip uses WwwStream::oneshot qspawn: handle ENOENT (and other errors on exec) search: clobber -user_pfx on query parser initialization wwwtext: show thread endpoints info w/ indexlevel=basic altid: warn about non-word prefixes www: add endpoint to retrieve altid dumps MANIFEST | 4 ++ lib/PublicInbox/AltId.pm | 3 +- lib/PublicInbox/ExtMsg.pm | 4 +- lib/PublicInbox/GetlineBody.pm | 21 ++++---- lib/PublicInbox/GzipFilter.pm | 59 +++++++++++++++++++++ lib/PublicInbox/Mbox.pm | 16 +++--- lib/PublicInbox/Qspawn.pm | 66 ++++++++++++++---------- lib/PublicInbox/Search.pm | 4 +- lib/PublicInbox/ViewVCS.pm | 8 +-- lib/PublicInbox/WWW.pm | 14 ++++- lib/PublicInbox/WwwAltId.pm | 94 ++++++++++++++++++++++++++++++++++ lib/PublicInbox/WwwStream.pm | 29 +++++++++-- lib/PublicInbox/WwwText.pm | 10 +++- t/gzip_filter.t | 37 +++++++++++++ t/httpd-corner.psgi | 16 ++++++ t/httpd-corner.t | 48 +++++++++++++++++ t/www_altid.t | 83 ++++++++++++++++++++++++++++++ 17 files changed, 452 insertions(+), 64 deletions(-) create mode 100644 lib/PublicInbox/GzipFilter.pm create mode 100644 lib/PublicInbox/WwwAltId.pm create mode 100644 t/gzip_filter.t create mode 100644 t/www_altid.t