From: ebiederm@xmission.com (Eric W. Biederman)
To: Eric Wong <e@yhbt.net>
Cc: meta@public-inbox.org
Subject: Re: [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox
Date: Sat, 16 May 2020 15:09:25 -0500 [thread overview]
Message-ID: <877dxb8wwa.fsf@x220.int.ebiederm.org> (raw)
In-Reply-To: <20200516191220.GA24349@dcvr> (Eric Wong's message of "Sat, 16 May 2020 19:12:20 +0000")
Eric Wong <e@yhbt.net> writes:
> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> Eric Wong <e@yhbt.net> writes:
>> > "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> >> > The email messages are placed without modification into the public
>> >> > inbox repository so minimize changes of corruption or of loosing
>> >> > valuable information. I use the command imap_fetch for all of my
>> >> > email and not just a mailling list mirror so I don't want automation
>> >> > to accidentally cause something important to be lost.
>> >
>> > Btw, Email::MIME usage is gone from 1.5.0 due to nasty
>> > performance problems and replaced by PublicInbox::Eml. Eml
>> > should be completely non-destructive unless somebody sends an
>> > abusive message which exceeds the new safety limits; in which
>> > case it won't OOM or burn CPU like E::M did.
>> >
>> > That said, {-public_inbox_raw} still works and Eml looks
>> > like a drop-in replacement as far as imap_fetch is concerned.
>>
>> I almost did that. But I looked and saw PublicInbox::MIME still present
>> and a number of other references to Email::MIME so I wasn't certain
>> exactly how that was being handled. But since Email::MIME still
>> worked I didn't mess with that.
>
> I think the Import .pod documentation is the only place aside
> from some random comments and maintainer tests in xt/*, right?
I am looking at 1.5.0 so you may have made a bit more progress
but Import.pm still uses Email::MIME,
and PublicInbox::MIME still uses Email::MIME as a base class.
> I'll definitely keep the import + indexing parts compatible with
> Email::MIME.
>
> I'm not sure how much of a Perl API I want to expose in case
> this gets reimplemented/migrated to another language someday,
> but maybe exposing the most heavily-used parts is OK.
>
> I also don't think another language is likely, at this point.
>> >> > No email messages are deleted from the server instead IMAPTracker
>> >> > is used to remember which messages were downloaded.
>> >
>> > Yup. I've integrated IMAPTracker into a local branch, already.
>> > I think it could be reused for tracking NNTP fetches, too,
>> > since UIDs and NNTP article numbers seem interchangeable.
>>
>> True. I guess it is possible to make the equivalent of my imap_fetch
>> for NNTP queries as well. Does NNTP have the equivalent of IDLE where
>> it is possible to get timely updates of new messages?
>
> I haven't seen anything in NNTP like IDLE. Maybe there's
> something in the draft stages, but AFAIK nothing widely
> implemented.
>
>> I suppose I should see if the git protocol does as well. I am trying to
>> figure out what the best way to track other public inbox git repositories.
>
> I'll likely implement a custom HTTP endpoint and usable with
> (curl -XIDLE ...); but the timeout will need to vary depending
> on firewalls.
>
>> Currently I have a script that every N minutes does git remote update.
>> Which is ok. But I think something like imap_fetch might be nicer.
>
> Definitely.
>
>> >> Bah. I sent this a little too soon. The patch needs this small
>> >> incremental fix to actually work.
>> >
>> > No worries.
>> >
>> > Btw, any reason you create the SSLSocket yourself instead of
>> > passing (Ssl => \@SSL_Socket_options) to IMAPClient->new?
>>
>> When I read the documentation it looked like that was the way to do
>> things. Even now when I reread the documentation that looks like the
>> way to go. Especially if I wanted to be certain the connection was
>> encrypted.
>
> There seems more than one way to do it, but `Starttls' and `Ssl'
> are just as documented from what I tell (in v3.38).
> Socket/RawSocket seem useful for using an external command to
> connect/launch an IMAP tunnel or server; so it'll be used to
> mimic the `imap.tunnel' support of git-imap-send.
Now that you point it out I can see it. Commands like starttls
are a bit dangerous as they are subject to man in the middle attacks.
But I think that is the difference of just tossing something together
for yourself versus making something that works with everyone's setup.
The one challenge I ran into was getting ssl verification to work on
RHEL7. Apparently IO::Socket::SSL::default_ca() does not exist in
the old version of perl that comes with RHEL7. Which is why I have
the %ca and the eval.
Eric
next prev parent reply other threads:[~2020-05-16 20:13 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-29 14:40 I have figured out IMAP IDLE Eric W. Biederman
2019-10-29 22:31 ` Eric Wong
2019-10-29 23:12 ` WWW::Curl [was: I have figured out IMAP IDLE] Eric Wong
2019-11-03 16:28 ` I have figured out IMAP IDLE Eric W. Biederman
2020-05-13 19:31 ` Eric Wong
2020-05-13 21:48 ` Eric W. Biederman
2020-05-13 22:17 ` Eric Wong
2020-05-14 12:32 ` Eric W. Biederman
2020-05-14 16:15 ` Eric Wong
2020-05-15 21:00 ` [PATCH 1/2] IMAPTracker: Add a helper to track our place in reading imap mailboxes Eric W. Biederman
2020-05-15 21:02 ` [PATCH 2/2] imap_fetch: Add a command to continuously fetch from an imap mailbox Eric W. Biederman
2020-05-15 21:26 ` Eric W. Biederman
2020-05-15 22:56 ` Eric Wong
2020-05-16 10:47 ` Eric W. Biederman
2020-05-16 19:12 ` Eric Wong
2020-05-16 20:09 ` Eric W. Biederman [this message]
2020-05-16 22:53 ` [PATCH] confine Email::MIME use even further Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=877dxb8wwa.fsf@x220.int.ebiederm.org \
--to=ebiederm@xmission.com \
--cc=e@yhbt.net \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).