user/dev discussion of public-inbox itself
 help / color / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 00/11] www: export SQLite altid dumps
Date: Sat, 21 Mar 2020 02:03:43 +0000
Message-ID: <20200321020354.9056-1-e@yhbt.net> (raw)

To improve reproducibility in mirrors, altid dumps can be
exported via "POST /$INBOX_URL/$prefix.sql.gz".  $prefix is
something like "gmane" (though the search prefix is "gmane:"
with a colon).

Eric Wong (11):
  qspawn: reinstate filter support, add gzip filter
  gzipfilter: lazy allocate the deflate context
  wwwstream: introduce oneshot API to avoid ->getline
  extmsg: use WwwResponse::oneshot
  wwwstream: oneshot sets content-length
  mbox: need_gzip uses WwwStream::oneshot
  qspawn: handle ENOENT (and other errors on exec)
  search: clobber -user_pfx on query parser initialization
  wwwtext: show thread endpoints info w/ indexlevel=basic
  altid: warn about non-word prefixes
  www: add endpoint to retrieve altid dumps

 MANIFEST                       |  4 ++
 lib/PublicInbox/AltId.pm       |  3 +-
 lib/PublicInbox/ExtMsg.pm      |  4 +-
 lib/PublicInbox/GetlineBody.pm | 21 ++++----
 lib/PublicInbox/GzipFilter.pm  | 59 +++++++++++++++++++++
 lib/PublicInbox/Mbox.pm        | 16 +++---
 lib/PublicInbox/Qspawn.pm      | 66 ++++++++++++++----------
 lib/PublicInbox/Search.pm      |  4 +-
 lib/PublicInbox/ViewVCS.pm     |  8 +--
 lib/PublicInbox/WWW.pm         | 14 ++++-
 lib/PublicInbox/WwwAltId.pm    | 94 ++++++++++++++++++++++++++++++++++
 lib/PublicInbox/WwwStream.pm   | 29 +++++++++--
 lib/PublicInbox/WwwText.pm     | 10 +++-
 t/gzip_filter.t                | 37 +++++++++++++
 t/httpd-corner.psgi            | 16 ++++++
 t/httpd-corner.t               | 48 +++++++++++++++++
 t/www_altid.t                  | 83 ++++++++++++++++++++++++++++++
 17 files changed, 452 insertions(+), 64 deletions(-)
 create mode 100644 lib/PublicInbox/GzipFilter.pm
 create mode 100644 lib/PublicInbox/WwwAltId.pm
 create mode 100644 t/gzip_filter.t
 create mode 100644 t/www_altid.t

             reply index

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-21  2:03 Eric Wong [this message]
2020-03-21  2:03 ` [PATCH 01/11] qspawn: reinstate filter support, add gzip filter Eric Wong
2020-03-21  2:03 ` [PATCH 02/11] gzipfilter: lazy allocate the deflate context Eric Wong
2020-03-21  2:03 ` [PATCH 03/11] wwwstream: introduce oneshot API to avoid ->getline Eric Wong
2020-03-21  2:03 ` [PATCH 04/11] extmsg: use WwwResponse::oneshot Eric Wong
2020-03-21  2:03 ` [PATCH 05/11] wwwstream: oneshot sets content-length Eric Wong
2020-03-21  2:03 ` [PATCH 06/11] mbox: need_gzip uses WwwStream::oneshot Eric Wong
2020-03-21  2:03 ` [PATCH 07/11] qspawn: handle ENOENT (and other errors on exec) Eric Wong
2020-03-21  2:03 ` [PATCH 08/11] search: clobber -user_pfx on query parser initialization Eric Wong
2020-03-21  2:03 ` [PATCH 09/11] wwwtext: show thread endpoint w/ indexlevel=basic Eric Wong
2020-03-21  2:03 ` [PATCH 10/11] altid: warn about non-word prefixes Eric Wong
2020-03-21  2:03 ` [PATCH 11/11] www: add endpoint to retrieve altid dumps Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200321020354.9056-1-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git