From: Ricardo Ribalda <ribalda@chromium.org>
To: Eric Wong <e@80x24.org>
Cc: meta@public-inbox.org
Subject: Re: [PATCH v2] lei: bail out earlier on IMAP writer failures
Date: Sat, 10 Sep 2022 21:34:00 +0200 [thread overview]
Message-ID: <CANiDSCvSdYY1KkV-XDa5ZH+db0Nq6VDHBfJuh6pr7RghMo0dKQ@mail.gmail.com> (raw)
In-Reply-To: <20220910011859.M68532@dcvr>
Hi Eric
On Sat, 10 Sept 2022 at 03:19, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > The patch did not seem to have any effect :(, I never get a "IMAP
> > LastError: " message
>
> Yeah, I guess IMAP servers will just shutdown the socket w/o
> saying anything. At least I didn't get anything from dovecot...
>
> The below patch is a refinement of what I posted originally
> and should stop the process instead of attempting to continue
> and spew.
>
> > On the other hand, the -j worked! I can go up to -j ,15 without any error.
>
> Good to know.
>
> I wonder if making the default `-j ,4' for IMAP is reasonable if
> unspecified. That's the default limit for HTTP(S) hosts, and I
> seem to recall 4 being a reasonable limit for browsers.
>
> Thanks for the report and followup!
Similar output:
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660159987..)+AND+dt%3A20220908193300..
1486767 lei2mail 17 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486762 lei2mail 12 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486782 lei2mail 32 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486771 lei2mail 21 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486766 lei2mail 16 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486788 lei2mail 38 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486761 lei2mail 11 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486791 lei2mail 41 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486775 lei2mail 25 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486765 lei2mail 15 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486755 lei2mail 5 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486793 lei2mail 43 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486795 lei2mail 45 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486763 lei2mail 13 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486779 lei2mail 29 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486773 lei2mail 23 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486751 lei2mail 1 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486787 lei2mail 37 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486796 lei2mail 46 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486776 lei2mail 26 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486752 lei2mail 2 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486786 lei2mail 36 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486777 lei2mail 27 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486764 lei2mail 14 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486794 lei2mail 44 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486756 lei2mail 6 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486790 lei2mail 40 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486784 lei2mail 34 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486792 lei2mail 42 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486768 lei2mail 18 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486774 lei2mail 24 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486757 lei2mail 7 wq_worker: do_post_auth: Can't call method
"uidvalidity" on an undefined value at
/usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
1486750 lei2mail 0 wq_worker: write_mail: Can't use an undefined value
as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
line 783.
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)
ribalda@denia:/tmp/public-inbox$ lei up imaps://imap.gmail.com/lei/me -j ,15
# https://lore.kernel.org/all/ limiting to 2022-09-08 19:33 +0000 and newer
# /usr/local/google/home/ribalda/.local/share/lei/store 12/12
# /usr/bin/curl -Sf -s -d ''
https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660160023..)+AND+dt%3A20220908193307..
# https://lore.kernel.org/all/ 20/20
# 0 written to imaps://imap.gmail.com/lei/me (32 matches)
Thanks!
>
> ------8<------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: bail out earlier on IMAP writer failures
>
> Excessive IMAP connections can overload IMAP servers and cause
> clients to be disconnected without diagnostic messages.
> Use $lei->fail on these exceptions to propagate errors to the
> CLI ASAP to avoid further errors down the line.
>
> This ought to make problems more apparent for users using IMAP
> destinations.
>
> Reported-by: Ricardo Ribalda <ribalda@chromium.org>
> Link: https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
> lib/PublicInbox/LeiToMail.pm | 10 +++++++---
> lib/PublicInbox/NetReader.pm | 8 +++++++-
> 2 files changed, 14 insertions(+), 4 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..03cbde3b 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,11 @@ sub _imap_write_cb ($$) {
> my $dedupe = $lei->{dedupe};
> $dedupe->prepare_dedupe if $dedupe;
> my $append = $lei->{net}->can('imap_append');
> - my $uri = $self->{uri};
> - my $mic = $lei->{net}->mic_get($uri);
> + my $uri = $self->{uri} // die 'BUG: no {uri}';
> + my $mic = $lei->{net}->mic_get($uri) // die <<EOM;
> +E: $uri connection failed.
> +E: Consider using `--jobs ,1' to limit IMAP connections
> +EOM
> my $folder = $uri->mailbox;
> $uri->uidvalidity($mic->uidvalidity($folder));
> my $lse = $lei->{lse}; # may be undef
> @@ -749,7 +752,8 @@ sub do_post_auth {
> $au_peers->[1] = undef;
> sysread($au_peers->[0], my $barrier1, 1);
> }
> - $self->{wcb} = $self->write_cb($lei);
> + eval { $self->{wcb} = $self->write_cb($lei) };
> + $lei->fail($@) if $@;
> if ($au_peers) { # wait for peer l2m to set write_cb
> $au_peers->[3] = undef;
> sysread($au_peers->[2], my $barrier2, 1);
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
> }
> my $mic = mic_new($self, $mic_arg, $sec, $uri);
> $cached //= {}; # invalid placeholder if no cache enabled
> - $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> + if ($mic && $mic->IsConnected) {
> + $cached->{$sec} = $mic;
> + } else {
> + warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> + warn "IMAP errno: $!\n" if $!;
> + undef;
> + }
> }
>
> sub imap_each {
--
Ricardo Ribalda
next prev parent reply other threads:[~2022-09-10 19:34 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-09 10:09 imap: "Can't use an undefined value as a subroutine reference" Ricardo Ribalda
2022-09-09 17:44 ` [PATCH] lei: add diagnostics for IMAP writer failures Eric Wong
2022-09-09 18:00 ` [PATCH] doc: document --jobs for `lei q' and `lei up' Eric Wong
2022-09-09 20:35 ` [PATCH] lei: add diagnostics for IMAP writer failures Ricardo Ribalda
2022-09-10 1:18 ` [PATCH v2] lei: bail out earlier on " Eric Wong
2022-09-10 19:34 ` Ricardo Ribalda [this message]
2022-09-10 19:50 ` Eric Wong
2022-09-10 19:53 ` Ricardo Ribalda
2022-09-10 20:19 ` Eric Wong
2022-11-14 8:07 ` [PATCH] lei q|up: limit default write --jobs for IMAP(S) Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CANiDSCvSdYY1KkV-XDa5ZH+db0Nq6VDHBfJuh6pr7RghMo0dKQ@mail.gmail.com \
--to=ribalda@chromium.org \
--cc=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).