From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 7715C1F424 for ; Tue, 3 Apr 2018 11:09:12 +0000 (UTC) From: "Eric Wong (Contractor, The Linux Foundation)" To: meta@public-inbox.org Subject: [PATCH 0/7] optimize V2 Date: Tue, 3 Apr 2018 11:09:05 +0000 Message-Id: <20180403110912.24231-1-e@80x24.org> List-Id: The switch to the SQLite overview DB introduced a regression in the use of OFFSET. While Xapian largely unaffected by the offset for pagination, SQLite basically gets the full result and cuts it at the offset, so using large offsets gets slower as time goes on. There are some good side-effects of this, too: * Using last-hop-Received timestamps for HTML pagination ought to improve cacheability on the client-side and in search engines. * Switching to using NNTP article numbers gives us race avoidance for NEWNEWS so duplicates won't show up while iterating through result batches (not that I've seen any logs this command being used in the wild...) I don't believe there are any more regressions from switching to the Xapian skeleton to a SQLite over(view) DB. Homepage performance seems consistent whether it's 2.8 million messages or 2-3x that. Now, NNTP performance should be improved across the board and commands like XOVER/XHDR are over 20x faster than they were in v1 with Xapian. There's also a few cleanups and code simplifications which should make future work easier. Eric Wong (Contractor, The Linux Foundation) (7): t/thread-all.t: modernize test to support modern inboxes rename+rewrite test using Benchmark module nntp: make XOVER, XHDR, OVER, HDR and NEWNEWS faster view: avoid offset during pagination mbox: remove remaining OFFSET usage in SQLite msgmap: replace id_batch with ids_after nntp: simplify the long_response API MANIFEST | 3 +- lib/PublicInbox/Feed.pm | 25 +-------- lib/PublicInbox/Inbox.pm | 4 +- lib/PublicInbox/Mbox.pm | 30 ++++++++--- lib/PublicInbox/Msgmap.pm | 29 +++++++---- lib/PublicInbox/NNTP.pm | 114 ++++++++++++++++++---------------------- lib/PublicInbox/Over.pm | 50 ++++++++++++------ lib/PublicInbox/Search.pm | 8 +-- lib/PublicInbox/View.pm | 90 +++++++++++++++++++++++--------- t/nntpd.t | 1 + t/perf-nntpd.t | 130 ++++++++++++++++++++++++++++++++++++++++++++++ t/perf-threading.t | 32 ++++++++++++ t/psgi_v2.t | 22 +++++++- t/thread-all.t | 38 -------------- t/v2writable.t | 17 ++++++ 15 files changed, 403 insertions(+), 190 deletions(-) create mode 100644 t/perf-nntpd.t create mode 100644 t/perf-threading.t delete mode 100644 t/thread-all.t -- EW