user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: Re: lei: per-message keywords and externals
Date: Tue, 2 Mar 2021 09:28:02 +0000	[thread overview]
Message-ID: <20210302092802.GA19386@dcvr> (raw)
In-Reply-To: <20210226092648.GA30618@dcvr>

Eric Wong <e@80x24.org> wrote:
> Eric Wong <e@80x24.org> wrote:
> > Something I've been pondering for a bit is how to handle
> > keywords (Seen, Important, Replied, ...) for messages stored in
> > externals.
> > 
> > I want "kw:" prefix to be a usable search term, like:
> > 
> > 	lei q something interesting kw:seen
> > 	lei q something interesting NOT kw:seen
> > 
> > This is no problem for imported messages in ~/.local/share/lei/store.
> > All the keyword info is stored in line with the rest of the
> > Xapian index data.
> > 
> > But, I also don't want to be wasting users' space by duplicating
> > index data if they're already hosting inboxes for public
> > consumption.  So, it's looking like parsing out kw: ourselves
> > and do extra filtering on our end when externals are in play is
> > going to be a requirement...
> 
> Something I considered a few weeks ago, but decided against, but
> am again coming around to is indexing just the overview header
> info in lei/store.  In other words:
> 
> 	$sto->set_eml($eml->header_obj, @kw)
> 
> instead of:
> 
> 	$sto->set_eml($eml, @kw)
> 
> > Or, just don't support searching using "kw:" with externals, for
> > now; but still stash keywords somewhere when writing to
> > traditional mail stores.
> 
> Maybe it'll be another instance of LeiStore in a separate dir
> for external keywords: ~/.local/share/lei/xkw-store

I'm leaning that way.  For deduplication purposes (that is:
merging keywords from cross-posted messages), OID will be
indexed as a boolean term for repeat lookups (along with
Message-ID).  I'm not 100% sure if I want this to be SQLite or
Xapian, yet.  Leaning towards Xapian since that would give us
more flexibility w.r.t keyword searches and also let us do
filtering on common headers without too much cost.

  reply	other threads:[~2021-03-02  9:28 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-24 20:49 Eric Wong
2021-02-26  9:26 ` Eric Wong
2021-03-02  9:28   ` Eric Wong [this message]
2021-03-03 14:01 ` [RFC] lei_xkw: eXternal KeyWord index Eric Wong
2021-03-13 11:50   ` Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210302092802.GA19386@dcvr \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    --subject='Re: lei: per-message keywords and externals' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).