Re: I have figured out IMAP IDLE

[ebiederm@xmission.com (Eric W. Biederman)]

Eric Wong <e@80x24.org> writes:

> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> 
>> A few days ago I stumbled upon this magic decoder ring for IMAP.
>> The "Ten Commandments of How to Write an IMAP client"
>> 
>> https://www.washington.edu/imap/documentation/commndmt.txt
>> 
>> The part I was most clearly missing was that for IMAP it is better to
>> open multiple sockets (one per mail folder on the server) than it is to
>> switch between folders with the same ssl socket.
>> 
>> The core loop of my imap code now does:
>> 
>> 	for (;;) {
>> 		my $more;
>> 		do {
>> 			$more = fetch_mailbox($config, $tracker, $client, $mailbox, $validity);
>> 		} while ($more > 0);
>> 
>> 		my @idle_data;
>> 		do {
>> 			my $max_idle = 15;
>> 			$client->idle() or die("idle failed!\n");
>
> 			b) window closed
>
>> 			@idle_data = $client->idle_data($max_idle*60);
>> 			$client->done();
>
> 			a) window opens
>
>> 		} while (scalar(@idle_data) == 0);
>> 	}
>
> It seems between a) and b), there's a small window where a message
> can appear and not be noticed right away.

Hmm.

Looking at the source of IMAPClient point (b) can be moved
to after "$client->done()" by reading the output "$client->done()".

About the window for unsolicited untagged responses before the
"$client->idle()" command.  Those unsolicited responses should queue
up in the imap socket until the next read.  That next read should
happen after the "$client->idle()" command.  So in practice unsolicited
untagged responses should just look like a reply to imap idle.

So the race you have noticed does not look fundamental rather it looks
like it simply takes bug fixing to make go away.

> I think the only reliable way to do this without experiencing
> delays or an opportunity for missed messages is to have two(!)
> connections per folder:

That sounds nice.  But it is also _not_ recommended.  Quoting from
the "Ten Commandments of How to Write an IMAP Client"

    1. Thou shalt not assume that it is alright to open multiple IMAP
    sessions selected on the same mailbox simultaneously, lest thou face
    the righteous wrath of mail stores that doth not permit such access.
    Instead, thou shalt labor mightily, even unto having to use thy brain
    to thinketh the matter through, such that thy client use existing
    sessions that are already open.

> 1) fetcher - this connection only fetches, either at startup or
>    when triggered by the idler described below.
>
> 2) idler - just runs ->idle, ->idle_data and ->done
>    If ->idle times out or sees ->idle_data, it triggers
>    fetcher.  The trigger should probably have a timestamp
>    so the fetcher can merge overlapping triggers.
>
> Unfortunately, using widely-available Perl IMAP client libraries
> means two Perl processes for every folder.  But I suppose
> fetcher can be lazy-started to not waste resources.

>
> I wonder if WWW::Curl can do it all in one Perl process with an
> event loop.  curl can issue arbitrary commands for HTTP and I've
> been using it to learn some XS, so maybe it can do IDLE.
>
> WWW::Curl is also packaged for CentOS/RHEL7, so it should not be
> tough to install.
>
>> Where all I do with idle is wait until it gives me something, and
>> reissue it every 15 minutes so the connection doesn't time out.
>
> Btw, why not 29 minutes? (accounting for 30 minute timeout).

Silliness.  In the example I started from someone had a comment
about firewalls not honoring a full 30 minutes so I picked 15 minutes
arbitrarily.

>> The only filter I have found useful to add is when imap idle returns
>> nothing to just loop instead of trying to fetch mail.  I have not found
>> anything in the data idle returns that would save me work in the mailbox
>> fetching loop.
>
> Yeah, I recall that being the case when I last worked with
> IMAP in other places.
>
>> In my limited testing firing up a bunch of ssl sockets and fetching from
>> the all simultaneously appears much faster, and maybe a little more
>> stable.  Than switching between mailboxes.
>
> Thanks for the info!
>
> Fwiw, I favor using stunnel for developing/testing client-side
> mail tools with TLS.  It's easier to debug/trace a tool that's
> talking over loopback to stunnel using tools like strace or tcpdump.

I may check that out.  Certainly something that gives you the raw
protocol messages is a little bit cleaner.

I haven't had enough challenges that I have been looking for the raw
protocol messages yet.

Eric