From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS6315 166.70.0.0/16 X-Spam-Status: No, score=-3.7 required=3.0 tests=BAYES_00,RCVD_IN_DNSWL_LOW, SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 897CE1F453; Mon, 1 Oct 2018 08:51:32 +0000 (UTC) Received: from in02.mta.xmission.com ([166.70.13.52]) by out03.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1g6tvN-0001U1-7g; Mon, 01 Oct 2018 02:51:29 -0600 Received: from [105.184.227.67] (helo=x220.xmission.com) by in02.mta.xmission.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1g6tvM-0001g2-2S; Mon, 01 Oct 2018 02:51:29 -0600 From: ebiederm@xmission.com (Eric W. Biederman) To: Johannes Berg Cc: Eric Wong , meta@public-inbox.org References: <87k1q1bky6.fsf@xmission.com> <20180712014715.dn5aouayoa3uejp4@dcvr> <87k1q07dyc.fsf@xmission.com> <20180712230946.mqv3yjw4aabf7xrf@dcvr.yhbt.net> <878t6f1ch7.fsf@xmission.com> <20180713230746.GA2680@dcvr> <1538165409.14416.92.camel@sipsolutions.net> <87zhw1cp5s.fsf@xmission.com> <1538379966.3126.13.camel@sipsolutions.net> Date: Mon, 01 Oct 2018 10:51:21 +0200 In-Reply-To: <1538379966.3126.13.camel@sipsolutions.net> (Johannes Berg's message of "Mon, 01 Oct 2018 09:46:06 +0200") Message-ID: <87y3biyrqu.fsf@xmission.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1g6tvM-0001g2-2S;;;mid=<87y3biyrqu.fsf@xmission.com>;;;hst=in02.mta.xmission.com;;;ip=105.184.227.67;;;frm=ebiederm@xmission.com;;;spf=neutral X-XM-AID: U2FsdGVkX1/cdIL2Ipu4ypcFSZkdIrcc6Z298W5RYfg= X-SA-Exim-Connect-IP: 105.184.227.67 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: IMAP server [was: Q: V2 format] X-SA-Exim-Version: 4.2.1 (built Thu, 05 May 2016 13:38:54 -0600) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) List-Id: Johannes Berg writes: > On Fri, 2018-09-28 at 23:01 +0200, Eric W. Biederman wrote: >> >> I have looked at gnus and there is support in there for performing >> searches via the old gmane web interface. Public inbox already provides >> an attribute that tells you what the web server is. So all it will >> really take is someone with a little time to wire up the search >> interface. > > That's ... interesting, but of course completely out-of-band. I'm not > sure it should or could be advocated that every email client actually > implement that :-) > > But if you think broader than that, you don't even necessarily need a > web server to run p-i. >> Beyond that if you have the archives local (and that is easy) it is >> quite possible to just git grep through them and find things of >> interest. > > That also doesn't use the index, not sure how that's any better? So for linux-kernel. I have 7G for the git email archive and 65G more for the indexes. Which makes the indexes quite expensive. So for personal use I am not certain an archive is a benefit. Especially when the email archive fits in ram and the index does not. I have to wonder if there is a way to make the indexes an order of magnitude smaller. >> I should verify this but I don't think IMAP has a good version of the >> NNTP overview database. Which seems to make IMAP quite a bit slower for >> reading news. Certainly gnus+public-inbox locally is running quite a >> bit faster than my old gnus+cyrus-imap configuration. > > IMAP servers typically should do header/MIME parsing, so you should be > able to query such a thing - but not as easily as XOVER, I suppose. > > However, I think FETCH could be made to return the data similar to > XOVER, though it may not be backed by a pre-created database file, and > it depends on what the client does to show the overview in the first > place. >> I tried to read through the IMAP search specification to see how it >> compares with what public-inbox makes available and I did not get >> particularly far. It was not easy to match up the various search >> capabilities. The biggest issue is that IMAP tends to not talk >> about message-ids. Where that is fundamentally one of the most >> important fields to index if you are dealing with threaded mail. > > You can search for arbitrary headers in search by using > > HEADER > > where the string is "contains", so you can use it for both Message-Id > and References headers. >> So long story short while I am not opposed to a read-only IMAP >> configuration I think NNTP has much to recommend it. I do think we need >> little things like SSL support for NNTP. Just to prevent inappropriate >> access to traffic in flight. > > Sure. I'm not saying NNTP is bad, just saying that the choice of clients > is rather limited. Also, posting isn't supported over NNTP, so if I had > it all in my email client I could read in the public-inbox archive, and > respond via normal email. The thing I can confirm and I have gotten as far as is that nntp has a sequential message id, and IMAP has a sequential message id and public-inbox has a sequential message id (now reliably based upon the order of the messages in the git archive). So it is very possible to have a read-only IMAP view. The really noticable downside of IMAP is that it does want to keep the status of messages you have read on the server. That makes a read-only archive a bit of a pain. So I am not certain the choice of clients when you restrict IMAP to what is an advantage. Nor am I certain the general IMAP search functionality maps well to what public-inbox indexes or people want to search for. Which is me again saying while things can make I am not certain IMAP is the best protocol for the job. >> It won't be for a while yet but I have some scripts I need to push at >> least to the public-inbox scripts directory that simplify the process >> taking a single email address subscribing to email and sorting it out >> into different public-inbox git archives. Currently I have every >> mailling list I am subscribed to pushed into public-inbox. > > :-) I do love that public-inbox makes it very easy to archive all my content and still be able to take it all with me when I travel. Eric