Re: [PATCH] lei: add diagnostics for IMAP writer failures

[Ricardo Ribalda <ribalda@chromium.org>]

Hi Eric

On Fri, 9 Sept 2022 at 19:45, Eric Wong <e@80x24.org> wrote:
>
> Ricardo Ribalda <ribalda@chromium.org> wrote:
> > Hi
> >
> > I am getting a lot of those messages when using lei in imap mode.
> >
> > Nonetheless the mail seems to arrive fine to its destination (on the
> > first run, some of the mail was lost, but for small batches of mails
> > it seems to work fine).
> >
> > It is a valid message or just some red-herring.
> >
> > I am using debian testing/
> >
> > Regards!
> >
> > # https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer
> > # /usr/local/google/home/ribalda/.local/share/lei/store 13/13
> > # /usr/bin/curl -Sf -s -d ''
> > https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321..
> > 1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method
> > "uidvalidity" on an undefined value at
> > /usr/share/perl5/PublicInbox/LeiToMail.pm line 313.
>
> <snip>
>
> > 1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value
> > as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm
> > line 783.
> > # https://lore.kernel.org/all/ 18/18
> > # 14 written to imaps://imap.gmail.com/lei/me (31 matches)
>
> I wonder if it's excessive parallelism for gmail's IMAP.
> I haven't tested IMAP destinations, much...
>
> Can you try the patch at the bottom?
>
> There's also another patch coming to document the `--jobs|-j' CLI
> switch for `lei up' and `lei q', but trying `-j ,1' may help you
> if it's parallelism.  Note the comma before `1', it accepts
> `-j $Q,$W' since $Q is the number of query processes and $W is
> the number of LeiToMail writers.

The patch did not seem to have any effect :(, I never  get a "IMAP
LastError: " message

On the other hand, the -j worked! I can go up to -j ,15 without any error.

>
> -------8<-------
> From: Eric Wong <e@80x24.org>
> Subject: [PATCH] lei: add diagnostics for IMAP writer failures
>
> This may help diagnose the problem with IMAP destinations
> encountered at:
> https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/
> ---
>  lib/PublicInbox/LeiToMail.pm | 4 ++--
>  lib/PublicInbox/NetReader.pm | 8 +++++++-
>  2 files changed, 9 insertions(+), 3 deletions(-)
>
> diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm
> index 2aa3977e..bc00b96a 100644
> --- a/lib/PublicInbox/LeiToMail.pm
> +++ b/lib/PublicInbox/LeiToMail.pm
> @@ -310,8 +310,8 @@ sub _imap_write_cb ($$) {
>         my $dedupe = $lei->{dedupe};
>         $dedupe->prepare_dedupe if $dedupe;
>         my $append = $lei->{net}->can('imap_append');
> -       my $uri = $self->{uri};
> -       my $mic = $lei->{net}->mic_get($uri);
> +       my $uri = $self->{uri} // die 'BUG: no {uri}';
> +       my $mic = $lei->{net}->mic_get($uri) // die 'BUG: no $mic';
>         my $folder = $uri->mailbox;
>         $uri->uidvalidity($mic->uidvalidity($folder));
>         my $lse = $lei->{lse}; # may be undef
> diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
> index c1af03a3..4de2583e 100644
> --- a/lib/PublicInbox/NetReader.pm
> +++ b/lib/PublicInbox/NetReader.pm
> @@ -685,7 +685,13 @@ sub mic_get {
>         }
>         my $mic = mic_new($self, $mic_arg, $sec, $uri);
>         $cached //= {}; # invalid placeholder if no cache enabled
> -       $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef;
> +       if ($mic && $mic->IsConnected) {
> +               $cached->{$sec} = $mic;
> +       } else {
> +               warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic;
> +               warn "IMAP errno: $!\n" if $!;
> +               undef;
> +       }
>  }
>
>  sub imap_each {



-- 
Ricardo Ribalda