From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id F15EC1F8C4; Sat, 6 Feb 2021 00:01:06 +0000 (UTC) Date: Sat, 6 Feb 2021 00:01:06 +0000 From: Eric Wong To: James Bottomley Cc: Konstantin Ryabitsev , meta@public-inbox.org Subject: Re: Thoughts on search-based imap mailboxes Message-ID: <20210206000106.GA2096@dcvr> References: <20201002143444.itpfbhg75wqy3eva@chatter.i7.local> <20201002200830.GA17908@dcvr> <20201003145006.wm5ehgwkhilmst73@chatter.i7.local> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: List-Id: James Bottomley wrote: > On Sat, 2020-10-03 at 10:50 -0400, Konstantin Ryabitsev wrote: > > On Fri, Oct 02, 2020 at 08:08:30PM +0000, Eric Wong wrote: > [...] > > > A client-side tool is likely required anyways, I'm thinking > > > having saved search functionality in a local tool writing to > > > Maildir/mbox might be the best way forward as w.r.t. scalability > > > and offline access (and reading + saving flags from the > > > Maildir/mbox). > > Colour me annoyed by the inability to mark email messages as Seen too. Yup, working on it with "lei" :> ("local email interface") > > > External/detached index I'm working on for global search will > > > also be appropriate for client-side tooling, I think, as it's > > > being designed with non-public-inbox data sources in mind, too. > > > > Instead of a "client-side" can it just be an intermediary tool that > > can run either on the client, or on any other piece of > > infrastructure? The main reason I'm interested in that is because > > several people I know use very thin clients for their work > > (chromebooks) and I would like to offer them this service as a perk > > of kernel.org instead of forcing them to set it up and maintain on > > their own. The add-external + "lei q --remote" example below covers that, I think (it doesn't cache+memoize yet, but it will). > Actually, I'd like a "server" side solution to this as well because I > want the message "seen" status to be updated on all my devices when I > read a message on any of them. Could this be implemented as something > like a pass-through imap server which has a login infrastructure for > storing the flag metadata but sends most other queries to the backend? > If such a beast existed I'd be happy to run it locally on my cloud > system, and I'm sure Konstantin could run it for all those of use with > kernel.org addresses. Right now the plan is to have a way to import+export keywords(*) or entire messages to/from existing (IMAP|Maildir|mbox) mail sources. I don't think there needs to be new servers; existing read-write IMAP servers should be usable as-is. # import any unimported messages from a personal mailbox # into ~/.local/share/lei/store: lei import imaps://user@example.com/INBOX # (IMAP doesn't work yet, but mboxes do, Maildir next) # add LKML as a (remote) external source lei add-external https://lore.kernel.org/lkml/ # search for everything imported and in remote external sources # Anything from remote sources will be cached/memoized locally # by default. lei q --remote -o ~/tmp/results-Maildir SEARCH_TERMS... # "-o imaps://..." is on horizon # open results-Maildir (or mbox or IMAP folder) $YOUR_FAVOURITE_MUA ~/tmp/results-Maildir # lei will continue monitoring ~/tmp/results-Maildir for # keyword changes in background (not implemented) # Nothing below is implemented, yet :< # send keywords changes from background monitoring to IMAP # source (this won't put new messages in the IMAP source) lei export --kw-only imaps://user@example.com/INBOX # if your MUA interacted from an IMAP server directly, # just fetch keywords: lei import --kw-only imaps://user@example.com/INBOX # via ssh, assuming $HOST has lei installed, too: lei export --kw-only ssh://user@$HOST/ "add-external" and "q" are somewhat usable at the moment but will get fleshed out. "q" has yet to gain "-o imaps://..." support. "lei q" will eventually support saved searches + results metadata caching. My goal is to have "lei q" return usable results in <100ms if rerunning a previous query (and if the results folder didn't get clobbered by the user). I've barely started working on "lei import" (only mboxes so far), and haven't started on export, yet... WARNING: "q" matches mairix(1) behavior by clobbering the output folder before writing results (-a/--augment disables it just as it does in mairix(1). As a safety feature, I will probably have "q" check and import any edited or uncached messages into its internal store. "lei import" needs to be fleshed out, first. The first release will not support removing any imported/cached messages at all. (*) IMAP uses "flags", JMAP calls them "keywords". I'm trying to favor "keywords" since flags may be limited to the few flags supported by mbox/Maildir formats. I've used "flags" for a long time and habits are hard-to-break.