From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 565E41F4B4 for ; Sun, 10 Jan 2021 12:15:19 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/22] lei query overview views Date: Sun, 10 Jan 2021 12:14:57 +0000 Message-Id: <20210110121519.17044-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: Usage summary: lei add-external /path/to/v1-or-v2-inbox lei add-external /path/to/another-inbox-or-ext-index # URLs aren't supported, yet :< lei q SEARCH TERMS GO HERE... # pager should open with JSON output For faster startup time than what Inline::C can give: apt-get install libsocket-msghdr-perl # Socket::Msghdr Having neither Inline::C nor Socket::Msghdr means parallel queries won't work. I went back-and-forth on a bunch of things but ultimately gave up trying to support IO::FDPass since it got too fragile and difficult to test with the work-queue distribution. The pager runs from the client process (if using Socket::MsgHdr or Inline::C), now. It took at fair amount of work from my slow brain to get pager shutdown to be instantaneous, though queries which haven't output anything aren't easily interruptible... The wq_* IPC stuff will be reused in the normal read-only WWW/IMAP search at some point, too. Eric Wong (22): lei query + pagination sorta working lei q: deduplicate smsg ds: block signals when reaping ipc: add support for asynchronous callbacks cmd_ipc: send FDs with buffer payload ipc: avoid excessive evals ipc: work queue support via SOCK_SEQPACKET ipc: eliminate ipc_worker_stop method ipc: wq: support dynamic worker count change ipc: drop -ipc_parent_pid field ipc: DESTROY and wq_workers methods lei: rename $w to $wpager for warning message lei: fix oneshot TTY detection by passing STD*{GLOB} lei: query: ensure pager exit is instantaneous ipc: start supporting sending/receiving more than 3 FDs ipc: fix IO::FDPass use with a worker limit of 1 ipc: drop unused fields, default sighandlers for wq lei: get rid of client {pid} field lei: fork + FD cleanup lei: run pager in client script lei_xsearch: transfer 4 FDs internally, drop IO::FDPass lei: query: restore JSON output overview MANIFEST | 4 + lib/PublicInbox/CmdIPC4.pm | 36 ++++ lib/PublicInbox/DS.pm | 16 +- lib/PublicInbox/Daemon.pm | 10 +- lib/PublicInbox/ExtSearchIdx.pm | 4 +- lib/PublicInbox/IPC.pm | 280 ++++++++++++++++++++++++++++---- lib/PublicInbox/LEI.pm | 180 +++++++++++++------- lib/PublicInbox/LeiDedupe.pm | 29 +++- lib/PublicInbox/LeiExternal.pm | 33 ++-- lib/PublicInbox/LeiOverview.pm | 188 +++++++++++++++++++++ lib/PublicInbox/LeiQuery.pm | 92 +++++++++++ lib/PublicInbox/LeiStore.pm | 2 +- lib/PublicInbox/LeiToMail.pm | 2 + lib/PublicInbox/LeiXSearch.pm | 118 +++++++++++++- lib/PublicInbox/Search.pm | 10 +- lib/PublicInbox/SearchView.pm | 10 +- lib/PublicInbox/Sigfd.pm | 12 +- lib/PublicInbox/Spawn.pm | 85 ++++++---- lib/PublicInbox/Watch.pm | 8 +- script/lei | 76 +++++---- script/public-inbox-watch | 4 +- t/cmd_ipc.t | 82 ++++++++++ t/ipc.t | 115 ++++++++++++- t/lei.t | 31 +++- t/lei_dedupe.t | 14 ++ t/lei_xsearch.t | 5 + t/spawn.t | 33 +--- 27 files changed, 1233 insertions(+), 246 deletions(-) create mode 100644 lib/PublicInbox/CmdIPC4.pm create mode 100644 lib/PublicInbox/LeiOverview.pm create mode 100644 lib/PublicInbox/LeiQuery.pm create mode 100644 t/cmd_ipc.t