user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: ebiederm@xmission.com (Eric W. Biederman)
To: Johannes Berg <johannes@sipsolutions.net>
Cc: Eric Wong <e@80x24.org>,  meta@public-inbox.org
Subject: Re: IMAP server [was: Q: V2 format]
Date: Mon, 01 Oct 2018 10:51:21 +0200	[thread overview]
Message-ID: <87y3biyrqu.fsf@xmission.com> (raw)
In-Reply-To: <1538379966.3126.13.camel@sipsolutions.net> (Johannes Berg's message of "Mon, 01 Oct 2018 09:46:06 +0200")

Johannes Berg <johannes@sipsolutions.net> writes:

> On Fri, 2018-09-28 at 23:01 +0200, Eric W. Biederman wrote:
>> 
>> I have looked at gnus and there is support in there for performing
>> searches via the old gmane web interface.  Public inbox already provides
>> an attribute that tells you what the web server is.  So all it will
>> really take is someone with a little time to wire up the search
>> interface.
>
> That's ... interesting, but of course completely out-of-band. I'm not
> sure it should or could be advocated that every email client actually
> implement that :-)
>
> But if you think broader than that, you don't even necessarily need a
> web server to run p-i.

>> Beyond that if you have the archives local (and that is easy) it is
>> quite possible to just git grep through them and find things of
>> interest.
>
> That also doesn't use the index, not sure how that's any better?

So for linux-kernel.  I have 7G for the git email archive and 65G more
for the indexes.  Which makes the indexes quite expensive.  So for
personal use I am not certain an archive is a benefit.  Especially when
the email archive fits in ram and the index does not.

I have to wonder if there is a way to make the indexes an order of
magnitude smaller.

>> I should verify this but I don't think IMAP has a good version of the
>> NNTP overview database.  Which seems to make IMAP quite a bit slower for
>> reading news.  Certainly gnus+public-inbox locally is running quite a
>> bit faster than my old gnus+cyrus-imap configuration.
>
> IMAP servers typically should do header/MIME parsing, so you should be
> able to query such a thing - but not as easily as XOVER, I suppose.
>
> However, I think FETCH could be made to return the data similar to
> XOVER, though it may not be backed by a pre-created database file, and
> it depends on what the client does to show the overview in the first
> place.

>> I tried to read through the IMAP search specification to see how it
>> compares with what public-inbox makes available and I did not get
>> particularly far.   It was not easy to match up the various search
>> capabilities.  The biggest issue is that IMAP tends to not talk
>> about message-ids.  Where that is fundamentally one of the most
>> important fields to index if you are dealing with threaded mail.
>
> You can search for arbitrary headers in search by using
>
> HEADER <field-name> <string>
>
> where the string is "contains", so you can use it for both Message-Id
> and References headers.

>> So long story short while I am not opposed to a read-only IMAP
>> configuration I think NNTP has much to recommend it.  I do think we need
>> little things like SSL support for NNTP.  Just to prevent inappropriate
>> access to traffic in flight.
>
> Sure. I'm not saying NNTP is bad, just saying that the choice of clients
> is rather limited. Also, posting isn't supported over NNTP, so if I had
> it all in my email client I could read in the public-inbox archive, and
> respond via normal email.

The thing I can  confirm and I have gotten as far as  is that nntp has a
sequential  message  id,  and  IMAP  has a  sequential  message  id  and
public-inbox has  a sequential message  id (now reliably based  upon the
order of the  messages in the git  archive).  So it is  very possible to
have a read-only IMAP view.

The really noticable downside of IMAP is that it does want to keep the
status of messages you have read on the server.  That makes a read-only
archive a bit of a pain.

So I am not certain the choice of clients when you restrict IMAP to what
is an advantage.  Nor am I certain the general IMAP search functionality
maps well to what public-inbox indexes or people want to search for.

Which is me again saying while things can make I am not certain IMAP is
the best protocol for the job.

>> It won't be for a while yet but I have some scripts I need to push at
>> least to the public-inbox scripts directory that simplify the process
>> taking a single email address subscribing to email and sorting it out
>> into different public-inbox git archives.  Currently I have every
>> mailling list I am subscribed to pushed into public-inbox.
>
> :-)

I do love that public-inbox makes it very easy to archive all my content
and still be able to take it all with me when I travel.

Eric

      reply	other threads:[~2018-10-01  8:51 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-07-11 20:01 Q: V2 format Eric W. Biederman
2018-07-11 21:18 ` Konstantin Ryabitsev
2018-07-11 21:41   ` Eric W. Biederman
2018-07-12  1:47 ` Eric Wong
2018-07-12 13:58   ` Eric W. Biederman
2018-07-12 23:09     ` Eric Wong
2018-07-13 13:39       ` Eric W. Biederman
2018-07-13 20:03         ` Eric W. Biederman
2018-07-13 22:22           ` msgmap serial number regeneration [was: Q: V2 format] Eric Wong
2018-07-14 19:01             ` Eric W. Biederman
2018-07-15  3:18               ` Eric Wong
2018-07-16 15:20                 ` Eric W. Biederman
2018-07-13 22:02         ` bug: v2 deletes on incremental fetch " Eric Wong
2018-07-13 22:51           ` Eric W. Biederman
2018-07-14  0:46           ` [PATCH] v2writable: unindex deleted messages after incremental fetch Eric Wong
2018-07-13 23:07         ` IMAP server [was: Q: V2 format] Eric Wong
2018-07-13 23:12           ` Eric W. Biederman
2018-09-28 20:10           ` Johannes Berg
2018-09-28 21:01             ` Eric W. Biederman
2018-10-01  7:46               ` Johannes Berg
2018-10-01  8:51                 ` Eric W. Biederman [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87y3biyrqu.fsf@xmission.com \
    --to=ebiederm@xmission.com \
    --cc=e@80x24.org \
    --cc=johannes@sipsolutions.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).