[PATCH 0/7] optimize V2

["Eric Wong (Contractor, The Linux Foundation)" <e@80x24.org>]

The switch to the SQLite overview DB introduced a regression in
the use of OFFSET.  While Xapian largely unaffected by the
offset for pagination, SQLite basically gets the full result and
cuts it at the offset, so using large offsets gets slower as
time goes on.

There are some good side-effects of this, too:

* Using last-hop-Received timestamps for HTML pagination ought
  to improve cacheability on the client-side and in search engines.

* Switching to using NNTP article numbers gives us race avoidance
  for NEWNEWS so duplicates won't show up while iterating
  through result batches (not that I've seen any logs this
  command being used in the wild...)

I don't believe there are any more regressions from switching
to the Xapian skeleton to a SQLite over(view) DB.  Homepage
performance seems consistent whether it's 2.8 million messages
or 2-3x that.

Now, NNTP performance should be improved across the board and
commands like XOVER/XHDR are over 20x faster than they were
in v1 with Xapian.

There's also a few cleanups and code simplifications
which should make future work easier.

Eric Wong (Contractor, The Linux Foundation) (7):
  t/thread-all.t: modernize test to support modern inboxes
  rename+rewrite test using Benchmark module
  nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster
  view: avoid offset during pagination
  mbox: remove remaining OFFSET usage in SQLite
  msgmap: replace id_batch with ids_after
  nntp: simplify the long_response API

 MANIFEST                  |   3 +-
 lib/PublicInbox/Feed.pm   |  25 +--------
 lib/PublicInbox/Inbox.pm  |   4 +-
 lib/PublicInbox/Mbox.pm   |  30 ++++++++---
 lib/PublicInbox/Msgmap.pm |  29 +++++++----
 lib/PublicInbox/NNTP.pm   | 114 ++++++++++++++++++----------------------
 lib/PublicInbox/Over.pm   |  50 ++++++++++++------
 lib/PublicInbox/Search.pm |   8 +--
 lib/PublicInbox/View.pm   |  90 +++++++++++++++++++++++---------
 t/nntpd.t                 |   1 +
 t/perf-nntpd.t            | 130 ++++++++++++++++++++++++++++++++++++++++++++++
 t/perf-threading.t        |  32 ++++++++++++
 t/psgi_v2.t               |  22 +++++++-
 t/thread-all.t            |  38 --------------
 t/v2writable.t            |  17 ++++++
 15 files changed, 403 insertions(+), 190 deletions(-)
 create mode 100644 t/perf-nntpd.t
 create mode 100644 t/perf-threading.t
 delete mode 100644 t/thread-all.t

-- 
EW