user/dev discussion of public-inbox itself
 help / color / Atom feed
From: ebiederm@xmission.com (Eric W. Biederman)
To: Eric Wong <e@yhbt.net>
Cc: meta@public-inbox.org
Subject: Re: [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox
Date: Sat, 16 May 2020 05:47:11 -0500
Message-ID: <87mu6888cw.fsf@x220.int.ebiederm.org> (raw)
In-Reply-To: <20200515225644.GB4131@dcvr> (Eric Wong's message of "Fri, 15 May 2020 22:56:44 +0000")

Eric Wong <e@yhbt.net> writes:

> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> ebiederm@xmission.com (Eric W. Biederman) writes:
>> 
>> > The command imap_fetch connects to the specified imap mailbox and
>> > fetches any unfetch messages than waits with imap idle until there are
>> > more messages to fetch.
>> >
>> > By default messages are placed in the specified public inbox mailbox.
>> > The value of List-ID is consulted and if it is present used to
>> > select an alternate public-inbox mailbox to place the messages in.
>> >
>> > The email messages are placed without modification into the public
>> > inbox repository so minimize changes of corruption or of loosing
>> > valuable information.  I use the command imap_fetch for all of my
>> > email and not just a mailling list mirror so I don't want automation
>> > to accidentally cause something important to be lost.
>
> Btw, Email::MIME usage is gone from 1.5.0 due to nasty
> performance problems and replaced by PublicInbox::Eml.  Eml
> should be completely non-destructive unless somebody sends an
> abusive message which exceeds the new safety limits; in which
> case it won't OOM or burn CPU like E::M did.
>
> That said, {-public_inbox_raw} still works and Eml looks
> like a drop-in replacement as far as imap_fetch is concerned.

I almost did that. But I looked and saw PublicInbox::MIME still present
and a number of other references to Email::MIME so I wasn't certain
exactly how that was being handled.  But since Email::MIME still
worked I didn't mess with that.

>> > No email messages are deleted from the server instead IMAPTracker
>> > is used to remember which messages were downloaded.
>
> Yup.  I've integrated IMAPTracker into a local branch, already.
> I think it could be reused for tracking NNTP fetches, too,
> since UIDs and NNTP article numbers seem interchangeable.

True.  I guess it is possible to make the equivalent of my imap_fetch
for NNTP queries as well.  Does NNTP have the equivalent of IDLE where
it is possible to get timely updates of new messages?

I suppose I should see if the git protocol does as well.  I am trying to
figure out what the best way to track other public inbox git repositories.

Currently I have a script that every N minutes does git remote update.
Which is ok.  But I think something like imap_fetch might be nicer.

>> Bah.  I sent this a little too soon.  The patch needs this small
>> incremental fix to actually work.
>
> No worries.
>
> Btw, any reason you create the SSLSocket yourself instead of
> passing (Ssl => \@SSL_Socket_options) to IMAPClient->new?

When I read the documentation it looked like that was the way to do
things.  Even now when I reread the documentation that looks like the
way to go.  Especially if I wanted to be certain the connection was
encrypted.

Eric

  reply index

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-10-29 14:40 I have figured out IMAP IDLE Eric W. Biederman
2019-10-29 22:31 ` Eric Wong
2019-10-29 23:12   ` WWW::Curl [was: I have figured out IMAP IDLE] Eric Wong
2019-11-03 16:28   ` I have figured out IMAP IDLE Eric W. Biederman
2020-05-13 19:31 ` Eric Wong
2020-05-13 21:48   ` Eric W. Biederman
2020-05-13 22:17     ` Eric Wong
2020-05-14 12:32       ` Eric W. Biederman
2020-05-14 16:15         ` Eric Wong
2020-05-15 21:00         ` [PATCH 1/2] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric W. Biederman
2020-05-15 21:02           ` [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox Eric W. Biederman
2020-05-15 21:26             ` Eric W. Biederman
2020-05-15 22:56               ` Eric Wong
2020-05-16 10:47                 ` Eric W. Biederman [this message]
2020-05-16 19:12                   ` Eric Wong
2020-05-16 20:09                     ` Eric W. Biederman
2020-05-16 22:53                       ` [PATCH] confine Email::MIME use even further Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87mu6888cw.fsf@x220.int.ebiederm.org \
    --to=ebiederm@xmission.com \
    --cc=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git