From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.0 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id EACB91F404 for ; Mon, 2 Apr 2018 00:04:56 +0000 (UTC) From: "Eric Wong (Contractor, The Linux Foundation)" To: meta@public-inbox.org Subject: [PATCH 0/5] v2: drop Xapian skeleton for SQLite overview DB Date: Mon, 2 Apr 2018 00:04:51 +0000 Message-Id: <20180402000456.13446-1-e@80x24.org> List-Id: 400ms to load the top-level /$INBOX/ for /git/ is pretty bad on my systems as-is. 800ms for /lkml/ was not unacceptable (though probably 400ms on decent servers). It's now around 60ms across the board with the new overview DB. This also affects OVER/XOVER/NEWNEWS NNTP commands, as well as Atom feeds. Eric Wong (Contractor, The Linux Foundation) (5): replace Xapian skeleton with SQLite overview DB v2writable: simplify barrier vs checkpoints t/over: test empty Subject: line matching www: rework query responses to avoid COUNT in SQLite over: speedup get_thread by avoiding JOIN MANIFEST | 5 +- lib/PublicInbox/Feed.pm | 11 +- lib/PublicInbox/Inbox.pm | 15 +- lib/PublicInbox/Mbox.pm | 12 +- lib/PublicInbox/Msgmap.pm | 1 + lib/PublicInbox/NNTP.pm | 29 +- lib/PublicInbox/Over.pm | 134 ++++++++ lib/PublicInbox/OverIdx.pm | 370 +++++++++++++++++++++ .../{SearchIdxSkeleton.pm => OverIdxFork.pm} | 131 +++++--- lib/PublicInbox/Search.pm | 124 ++----- lib/PublicInbox/SearchIdx.pm | 214 +++--------- lib/PublicInbox/SearchIdxPart.pm | 16 +- lib/PublicInbox/SearchMsg.pm | 26 +- lib/PublicInbox/V2Writable.pm | 114 +++---- lib/PublicInbox/View.pm | 17 +- script/public-inbox-compact | 32 +- t/altid.t | 9 +- t/convert-compact.t | 6 +- t/over.t | 63 ++++ t/psgi_search.t | 6 +- t/search-thr-index.t | 9 +- t/search.t | 92 ++--- t/v2-add-remove-add.t | 6 +- t/v2mda.t | 4 +- t/v2writable.t | 2 +- t/watch_maildir_v2.t | 26 +- 26 files changed, 907 insertions(+), 567 deletions(-) create mode 100644 lib/PublicInbox/Over.pm create mode 100644 lib/PublicInbox/OverIdx.pm rename lib/PublicInbox/{SearchIdxSkeleton.pm => OverIdxFork.pm} (54%) create mode 100644 t/over.t -- EW