user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 00/43] www: async git cat-file w/ -httpd
Date: Sun,  5 Jul 2020 23:27:16 +0000	[thread overview]
Message-ID: <20200705232759.3161-1-e@yhbt.net> (raw)

This allows -httpd to make better use of time it spends waiting
on git-cat-file to respond.  It allows us to deal with
high-latency HDD storage without a client monopolizing the event
loop.  Even on a mid-range consumer-grade SSD, this seems to
give a 10+% speed improvement for HTTP responses requiring many
blobs, including all /T/, /t/, and /t.mbox.gz endpoints.

This only benefits indexed inboxes (both v1 and v2); I'm not
sure if anybody still uses unindexed v1 inboxes nowadays.

A new xt/httpd-async-stream.t maintainer test ensures checksums
for responses before and after this series match exactly as
before.

This builds off a branch I started several months ago (but never
published here) to integrate gzip responses into our codebase
and remove our optional dependency on Plack::Middleware::Deflater.

We already gzip a bunch of things independent of
Plack::Middleware::Deflater: manifest.js.gz, altid SQLite3 dumps
and all the *.mbox.gz endpoints; so being able to use gzip on
all of our responses without an extra dependency seemed logical.

Being able to consistently use our GzipFilter API to perform
buffering via ->zmore made it significantly easier to reason
about small response chunks for ghost messages interspersed with
large ones when streaming /$INBOX/$MSGID/t/ endpoints.

I'm not yet maximizing use of ->zmore for all buffering of HTTP
responses, yet; measurements need to happen, first.  That may
happen in the 1.7 time frame.  In particular, we would need to
ensure the Perl method dispatch and DSO overhead to Zlib.so and
libz.so of making many ->zmore calls doesn't cause performance
regressions compared to the current `.=' use and calling
->zmore/->translate fewer times.

Eric Wong (43):
  gzipfilter: minor cleanups
  wwwstream: oneshot: perform gzip without middleware
  www*stream: gzip ->getline responses
  wwwtext: gzip text/plain responses, as well
  wwwtext: switch to html_oneshot
  www: need: use WwwStream::html_oneshot
  wwwlisting: use GzipFilter for HTML
  gzipfilter: replace Compress::Raw::Deflate usages
  {gzip,noop}filter: ->zmore returns undef, always
  mbox: remove html_oneshot import
  wwwstatic: support gzipped response directory listings
  qspawn: learn to gzip streaming responses
  stop auto-loading Plack::Middleware::Deflater
  mboxgz: do asynchronous git blob retrievals
  mboxgz: reduce object hash depth
  mbox: async blob retrieval for "single message" raw mboxrd
  wwwatomstream: simplify feed_update callers
  wwwatomstream: use PublicInbox::Inbox->modified for feed_updated
  wwwatomstream: reuse $ctx as $self
  xt/httpd-async-stream: allow more options
  wwwatomstream: support asynchronous blob retrievals
  wwwstream: reduce object graph depth
  wwwstream: reduce blob retrieval paths for ->getline
  www: start making gzipfilter the parent response class
  remove unused/redundant zlib-related imports
  wwwstream: use parent.pm and no warnings
  wwwstream: subclass off GzipFilter
  view: wire up /$INBOX/$MESSAGE_ID/ permalink to async
  view: /$INBOX/$MSGID/t/ reads blobs asynchronously
  view: update /$INBOX/$MSGID/T/ to be async
  feed: generate_i: eliminate pointless loop
  feed: /$INBOX/new.html retrieves blobs asynchronously
  ssearchview: /$INBOX/?q=$QUERY&x=t uses async blobs
  view: eml_entry: reduce parameters
  view: /$INBOX/$MSGID/t/: avoid extra hash lookup in eml case
  wwwstream: eliminate ::response, use html_oneshot
  www: update internal docs
  view: simplify eml_entry callers further
  wwwtext: simplify gzf_maybe use
  wwwattach: support async blob retrievals
  gzipfilter: drop HTTP connection on bugs or data corruption
  daemon: warn on missing blobs
  gzipfilter: check http->{forward} for client disconnects

 Documentation/mknews.perl            |  20 +--
 Documentation/public-inbox-httpd.pod |   1 -
 Documentation/technical/ds.txt       |   4 +-
 INSTALL                              |   5 -
 MANIFEST                             |   2 +
 ci/deps.perl                         |   1 -
 examples/cgit.psgi                   |   8 -
 examples/newswww.psgi                |   8 -
 examples/public-inbox.psgi           |   9 --
 examples/unsubscribe.psgi            |   1 -
 lib/PublicInbox/CompressNoop.pm      |  22 +++
 lib/PublicInbox/Feed.pm              |  22 ++-
 lib/PublicInbox/GetlineBody.pm       |   4 +-
 lib/PublicInbox/GzipFilter.pm        | 168 +++++++++++++++++---
 lib/PublicInbox/HTTP.pm              |   7 +
 lib/PublicInbox/HTTPD.pm             |   5 +-
 lib/PublicInbox/IMAP.pm              |   1 +
 lib/PublicInbox/Mbox.pm              | 137 +++++++++--------
 lib/PublicInbox/MboxGz.pm            |  81 ++++------
 lib/PublicInbox/NNTP.pm              |   1 +
 lib/PublicInbox/Qspawn.pm            |   6 +-
 lib/PublicInbox/SearchView.pm        |  40 ++---
 lib/PublicInbox/View.pm              | 219 ++++++++++++++-------------
 lib/PublicInbox/WWW.pm               |   9 +-
 lib/PublicInbox/WwwAtomStream.pm     |  66 ++++----
 lib/PublicInbox/WwwAttach.pm         |  63 ++++++--
 lib/PublicInbox/WwwListing.pm        |  24 +--
 lib/PublicInbox/WwwStatic.pm         |  14 +-
 lib/PublicInbox/WwwStream.pm         | 110 ++++++++------
 lib/PublicInbox/WwwText.pm           |  26 ++--
 script/public-inbox-httpd            |   9 --
 script/public-inbox.cgi              |   7 -
 t/httpd-corner.psgi                  |   7 +
 t/httpd-corner.t                     |   9 +-
 t/plack.t                            |   4 +
 t/psgi_attach.t                      | 162 +++++++++++---------
 t/psgi_text.t                        |  33 +++-
 t/psgi_v2.t                          |  80 ++++++++--
 t/www_listing.t                      |   8 +-
 t/www_static.t                       |  11 +-
 xt/httpd-async-stream.t              | 104 +++++++++++++
 41 files changed, 964 insertions(+), 554 deletions(-)
 create mode 100644 lib/PublicInbox/CompressNoop.pm
 create mode 100644 xt/httpd-async-stream.t

             reply	other threads:[~2020-07-05 23:27 UTC|newest]

Thread overview: 44+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-07-05 23:27 Eric Wong [this message]
2020-07-05 23:27 ` [PATCH 01/43] gzipfilter: minor cleanups Eric Wong
2020-07-05 23:27 ` [PATCH 02/43] wwwstream: oneshot: perform gzip without middleware Eric Wong
2020-07-05 23:27 ` [PATCH 03/43] www*stream: gzip ->getline responses Eric Wong
2020-07-05 23:27 ` [PATCH 04/43] wwwtext: gzip text/plain responses, as well Eric Wong
2020-07-05 23:27 ` [PATCH 05/43] wwwtext: switch to html_oneshot Eric Wong
2020-07-05 23:27 ` [PATCH 06/43] www: need: use WwwStream::html_oneshot Eric Wong
2020-07-05 23:27 ` [PATCH 07/43] wwwlisting: use GzipFilter for HTML Eric Wong
2020-07-05 23:27 ` [PATCH 08/43] gzipfilter: replace Compress::Raw::Deflate usages Eric Wong
2020-07-05 23:27 ` [PATCH 09/43] {gzip,noop}filter: ->zmore returns undef, always Eric Wong
2020-07-05 23:27 ` [PATCH 10/43] mbox: remove html_oneshot import Eric Wong
2020-07-05 23:27 ` [PATCH 11/43] wwwstatic: support gzipped directory listings Eric Wong
2020-07-05 23:27 ` [PATCH 12/43] qspawn: learn to gzip streaming responses Eric Wong
2020-07-05 23:27 ` [PATCH 13/43] stop auto-loading Plack::Middleware::Deflater Eric Wong
2020-07-05 23:27 ` [PATCH 14/43] mboxgz: do asynchronous git blob retrievals Eric Wong
2020-07-05 23:27 ` [PATCH 15/43] mboxgz: reduce hash depth Eric Wong
2020-07-05 23:27 ` [PATCH 16/43] mbox: async blob fetch for "single message" raw mboxrd Eric Wong
2020-07-05 23:27 ` [PATCH 17/43] wwwatomstream: simplify feed_update callers Eric Wong
2020-07-05 23:27 ` [PATCH 18/43] wwwatomstream: use PublicInbox::Inbox->modified for feed_updated Eric Wong
2020-07-05 23:27 ` [PATCH 19/43] wwwatomstream: reuse $ctx as $self Eric Wong
2020-07-05 23:27 ` [PATCH 20/43] xt/httpd-async-stream: allow more options Eric Wong
2020-07-05 23:27 ` [PATCH 21/43] wwwatomstream: support async blob fetch Eric Wong
2020-07-05 23:27 ` [PATCH 22/43] wwwstream: reduce object graph depth Eric Wong
2020-07-05 23:27 ` [PATCH 23/43] wwwstream: reduce blob fetch paths for ->getline Eric Wong
2020-07-05 23:27 ` [PATCH 24/43] www: start making gzipfilter the parent response class Eric Wong
2020-07-05 23:27 ` [PATCH 25/43] remove unused/redundant zlib-related imports Eric Wong
2020-07-05 23:27 ` [PATCH 26/43] wwwstream: use parent.pm and no warnings Eric Wong
2020-07-05 23:27 ` [PATCH 27/43] wwwstream: subclass off GzipFilter Eric Wong
2020-07-05 23:27 ` [PATCH 28/43] view: make /$INBOX/$MSGID/ permalink async Eric Wong
2020-07-05 23:27 ` [PATCH 29/43] view: /$INBOX/$MSGID/t/ reads blobs asynchronously Eric Wong
2020-07-05 23:27 ` [PATCH 30/43] view: update /$INBOX/$MSGID/T/ to be async Eric Wong
2020-07-05 23:27 ` [PATCH 31/43] feed: generate_i: eliminate pointless loop Eric Wong
2020-07-05 23:27 ` [PATCH 32/43] feed: /$INBOX/new.html fetches blobs asynchronously Eric Wong
2020-07-05 23:27 ` [PATCH 33/43] ssearchview: /$INBOX/?q=$QUERY&x=t uses async blobs Eric Wong
2020-07-05 23:27 ` [PATCH 34/43] view: eml_entry: reduce parameters Eric Wong
2020-07-05 23:27 ` [PATCH 35/43] view: /$INBOX/$MSGID/t/: avoid extra hash lookup in eml case Eric Wong
2020-07-05 23:27 ` [PATCH 36/43] wwwstream: eliminate ::response, use html_oneshot Eric Wong
2020-07-05 23:27 ` [PATCH 37/43] www: update internal docs Eric Wong
2020-07-05 23:27 ` [PATCH 38/43] view: simplify eml_entry callers further Eric Wong
2020-07-05 23:27 ` [PATCH 39/43] wwwtext: simplify gzf_maybe use Eric Wong
2020-07-05 23:27 ` [PATCH 40/43] wwwattach: support async blob retrievals Eric Wong
2020-07-05 23:27 ` [PATCH 41/43] gzipfilter: drop HTTP connection on bugs or data corruption Eric Wong
2020-07-05 23:27 ` [PATCH 42/43] daemon: warn on missing blobs Eric Wong
2020-07-05 23:27 ` [PATCH 43/43] gzipfilter: check http->{forward} for client disconnects Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200705232759.3161-1-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    --subject='Re: [PATCH 00/43] www: async git cat-file w/ -httpd' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.mail.public-inbox.meta
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.mail.public-inbox.meta
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git