From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_BL,RCVD_IN_MSPIKE_L5,SPF_HELO_NONE, SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out03.mta.xmission.com (out03.mta.xmission.com [166.70.13.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 1643F1F55B; Sat, 16 May 2020 20:13:04 +0000 (UTC) Received: from in02.mta.xmission.com ([166.70.13.52]) by out03.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1ja3B8-00048C-Pa; Sat, 16 May 2020 14:13:02 -0600 Received: from ip68-227-160-95.om.om.cox.net ([68.227.160.95] helo=x220.xmission.com) by in02.mta.xmission.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.87) (envelope-from ) id 1ja3B7-0001ft-CH; Sat, 16 May 2020 14:13:02 -0600 From: ebiederm@xmission.com (Eric W. Biederman) To: Eric Wong Cc: meta@public-inbox.org References: <87eeyvmx74.fsf@x220.int.ebiederm.org> <20200513193144.GA9299@dcvr> <87ftc3mrq6.fsf@x220.int.ebiederm.org> <20200513221715.GA11718@dcvr> <877dxelmsr.fsf@x220.int.ebiederm.org> <87ftc0c3r4.fsf_-_@x220.int.ebiederm.org> <87a728c3p3.fsf_-_@x220.int.ebiederm.org> <87o8qoao09.fsf@x220.int.ebiederm.org> <20200515225644.GB4131@dcvr> <87mu6888cw.fsf@x220.int.ebiederm.org> <20200516191220.GA24349@dcvr> Date: Sat, 16 May 2020 15:09:25 -0500 In-Reply-To: <20200516191220.GA24349@dcvr> (Eric Wong's message of "Sat, 16 May 2020 19:12:20 +0000") Message-ID: <877dxb8wwa.fsf@x220.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1ja3B7-0001ft-CH;;;mid=<877dxb8wwa.fsf@x220.int.ebiederm.org>;;;hst=in02.mta.xmission.com;;;ip=68.227.160.95;;;frm=ebiederm@xmission.com;;;spf=neutral X-XM-AID: U2FsdGVkX1+lU8KogKHc0KNh8o98lYrXpQxniyFUoqU= X-SA-Exim-Connect-IP: 68.227.160.95 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: Re: [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox X-SA-Exim-Version: 4.2.1 (built Thu, 05 May 2016 13:38:54 -0600) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) List-Id: Eric Wong writes: > "Eric W. Biederman" wrote: >> Eric Wong writes: >> > "Eric W. Biederman" wrote: >> >> > The email messages are placed without modification into the public >> >> > inbox repository so minimize changes of corruption or of loosing >> >> > valuable information. I use the command imap_fetch for all of my >> >> > email and not just a mailling list mirror so I don't want automation >> >> > to accidentally cause something important to be lost. >> > >> > Btw, Email::MIME usage is gone from 1.5.0 due to nasty >> > performance problems and replaced by PublicInbox::Eml. Eml >> > should be completely non-destructive unless somebody sends an >> > abusive message which exceeds the new safety limits; in which >> > case it won't OOM or burn CPU like E::M did. >> > >> > That said, {-public_inbox_raw} still works and Eml looks >> > like a drop-in replacement as far as imap_fetch is concerned. >> >> I almost did that. But I looked and saw PublicInbox::MIME still present >> and a number of other references to Email::MIME so I wasn't certain >> exactly how that was being handled. But since Email::MIME still >> worked I didn't mess with that. > > I think the Import .pod documentation is the only place aside > from some random comments and maintainer tests in xt/*, right? I am looking at 1.5.0 so you may have made a bit more progress but Import.pm still uses Email::MIME, and PublicInbox::MIME still uses Email::MIME as a base class. > I'll definitely keep the import + indexing parts compatible with > Email::MIME. > > I'm not sure how much of a Perl API I want to expose in case > this gets reimplemented/migrated to another language someday, > but maybe exposing the most heavily-used parts is OK. > > I also don't think another language is likely, at this point. >> >> > No email messages are deleted from the server instead IMAPTracker >> >> > is used to remember which messages were downloaded. >> > >> > Yup. I've integrated IMAPTracker into a local branch, already. >> > I think it could be reused for tracking NNTP fetches, too, >> > since UIDs and NNTP article numbers seem interchangeable. >> >> True. I guess it is possible to make the equivalent of my imap_fetch >> for NNTP queries as well. Does NNTP have the equivalent of IDLE where >> it is possible to get timely updates of new messages? > > I haven't seen anything in NNTP like IDLE. Maybe there's > something in the draft stages, but AFAIK nothing widely > implemented. > >> I suppose I should see if the git protocol does as well. I am trying to >> figure out what the best way to track other public inbox git repositories. > > I'll likely implement a custom HTTP endpoint and usable with > (curl -XIDLE ...); but the timeout will need to vary depending > on firewalls. > >> Currently I have a script that every N minutes does git remote update. >> Which is ok. But I think something like imap_fetch might be nicer. > > Definitely. > >> >> Bah. I sent this a little too soon. The patch needs this small >> >> incremental fix to actually work. >> > >> > No worries. >> > >> > Btw, any reason you create the SSLSocket yourself instead of >> > passing (Ssl => \@SSL_Socket_options) to IMAPClient->new? >> >> When I read the documentation it looked like that was the way to do >> things. Even now when I reread the documentation that looks like the >> way to go. Especially if I wanted to be certain the connection was >> encrypted. > > There seems more than one way to do it, but `Starttls' and `Ssl' > are just as documented from what I tell (in v3.38). > Socket/RawSocket seem useful for using an external command to > connect/launch an IMAP tunnel or server; so it'll be used to > mimic the `imap.tunnel' support of git-imap-send. Now that you point it out I can see it. Commands like starttls are a bit dangerous as they are subject to man in the middle attacks. But I think that is the difference of just tossing something together for yourself versus making something that works with everyone's setup. The one challenge I ran into was getting ssl verification to work on RHEL7. Apparently IO::Socket::SSL::default_ca() does not exist in the old version of perl that comes with RHEL7. Which is why I have the %ca and the eval. Eric