user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <>
To: James Bottomley <>
Cc: Konstantin Ryabitsev <>,
Subject: Re: Thoughts on search-based imap mailboxes
Date: Sat, 6 Feb 2021 00:01:06 +0000	[thread overview]
Message-ID: <20210206000106.GA2096@dcvr> (raw)
In-Reply-To: <>

James Bottomley <> wrote:
> On Sat, 2020-10-03 at 10:50 -0400, Konstantin Ryabitsev wrote:
> > On Fri, Oct 02, 2020 at 08:08:30PM +0000, Eric Wong wrote:
> [...]
> > > A client-side tool is likely required anyways, I'm thinking
> > > having saved search functionality in a local tool writing to
> > > Maildir/mbox might be the best way forward as w.r.t. scalability
> > > and offline access (and reading + saving flags from the
> > > Maildir/mbox).
> Colour me annoyed by the inability to mark email messages as Seen too.

Yup, working on it with "lei" :>  ("local email interface")

> > > External/detached index I'm working on for global search will
> > > also be appropriate for client-side tooling, I think, as it's
> > > being designed with non-public-inbox data sources in mind, too.
> > 
> > Instead of a "client-side" can it just be an intermediary tool that
> > can run either on the client, or on any other piece of
> > infrastructure? The main reason I'm interested in that is because
> > several people I know use very thin clients for their work
> > (chromebooks) and I would like to offer them this service as a perk
> > of instead of forcing them to set  it up and maintain on
> > their own.

The add-external + "lei q --remote" example below covers that,
I think (it doesn't cache+memoize yet, but it will).

> Actually, I'd like a "server" side solution to this as well because I
> want the message "seen" status to be updated on all my devices when I
> read a message on any of them.  Could this be implemented as something
> like a pass-through imap server which has a login infrastructure for
> storing the flag metadata but sends most other queries to the backend? 
> If such a beast existed I'd be happy to run it locally on my cloud
> system, and I'm sure Konstantin could run it for all those of use with
> addresses.

Right now the plan is to have a way to import+export keywords(*)
or entire messages to/from existing (IMAP|Maildir|mbox) mail

I don't think there needs to be new servers; existing read-write
IMAP servers should be usable as-is.

	# import any unimported messages from a personal mailbox
	# into ~/.local/share/lei/store:
	lei import imaps://
	# (IMAP doesn't work yet, but mboxes do, Maildir next)

	# add LKML as a (remote) external source
	lei add-external

	# search for everything imported and in remote external sources
	# Anything from remote sources will be cached/memoized locally
	# by default.
	lei q --remote -o ~/tmp/results-Maildir SEARCH_TERMS...
	# "-o imaps://..." is on horizon

	# open results-Maildir (or mbox or IMAP folder)
	$YOUR_FAVOURITE_MUA ~/tmp/results-Maildir

	# lei will continue monitoring ~/tmp/results-Maildir for
	# keyword changes in background (not implemented)

	# Nothing below is implemented, yet :<

	# send keywords changes from background monitoring to IMAP
	# source (this won't put new messages in the IMAP source)
	lei export --kw-only imaps://

	# if your MUA interacted from an IMAP server directly,
	# just fetch keywords:
	lei import --kw-only imaps://

	# via ssh, assuming $HOST has lei installed, too:
	lei export --kw-only ssh://user@$HOST/

"add-external" and "q" are somewhat usable at the moment but
will get fleshed out.  "q" has yet to gain "-o imaps://..."

"lei q" will eventually support saved searches + results metadata
caching.  My goal is to have "lei q" return usable results in
<100ms if rerunning a previous query (and if the results folder
didn't get clobbered by the user).

I've barely started working on "lei import" (only mboxes so
far), and haven't started on export, yet...


	"q" matches mairix(1) behavior by clobbering the output
	folder before writing results (-a/--augment disables it
	just as it does in mairix(1).

	As a safety feature, I will probably have "q" check and
	import any edited or uncached messages into its internal
	store.  "lei import" needs to be fleshed out, first.

	The first release will not support removing any
	imported/cached messages at all.

(*) IMAP uses "flags", JMAP calls them "keywords".  I'm trying
    to favor "keywords" since flags may be limited to the few
    flags supported by mbox/Maildir formats.  I've used "flags"
    for a long time and habits are hard-to-break.

      reply	other threads:[~2021-02-06  0:01 UTC|newest]

Thread overview: 6+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-02 14:34 Thoughts on search-based imap mailboxes Konstantin Ryabitsev
2020-10-02 20:08 ` Eric Wong
2020-10-03 14:50   ` Konstantin Ryabitsev
2020-10-03 19:40     ` Eric Wong
2021-02-05 22:37     ` James Bottomley
2021-02-06  0:01       ` Eric Wong [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210206000106.GA2096@dcvr \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).