From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-3.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 063921F54E for ; Fri, 9 Sep 2022 20:35:20 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="dOT9/UYi"; dkim-atps=neutral Received: by mail-ej1-x62e.google.com with SMTP id r18so6640166eja.11 for ; Fri, 09 Sep 2022 13:35:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=kufebovl6QHoM9IYVYX8BVICaQxkUtUHN7Wbk+esLzk=; b=dOT9/UYi9B4zlejvX+vLqbH0/Ocepr3/pqnGz6fqs7lMVGeMhcfKXiN/L6mHLZFrgq OjFVowqYxiyG/+In2EsTB7c9wS3sA+XjapvKj/E03RtlMff2HBiGEqUuiHS6UpRpZguh MBU8Ry2rJSJcUKV2jvCQDEdAqABEqnCOUG4gU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=kufebovl6QHoM9IYVYX8BVICaQxkUtUHN7Wbk+esLzk=; b=g8kgTHOLEH0MprAKIEUKpTNfPUeTLhzF+WsWbDqaXytoQalI0P5Kx8rHVxyxkilZAT pHX8ZdSVibQp6f+pu6EkU/BoMS68q0tvT5HY/A/7Uo1F+6QtghoDcwkNtgYTCysWLxIB ShQXMin8rMdTleT1wLfWTjDMPmpNslMYG11OLLZoPyKHQ0G5xtroVTwDVh4WbEhmeMA7 lxycTYUtwB3NbArH+v2KxIzVXlwv2/aCZQJLqrH5H5nmbvq3nKGx6e5ycAE+Xt1X5WTu Q6qITx5ZowNz6DZxAGsrvUPUwnX4aAvXCKC8zbERL1B/ehBA+Q+NJ+56g2Q8T5ANHE5P XJRg== X-Gm-Message-State: ACgBeo3lKd2jbUxqO9Ebt+vIjy5abrn3jF2hbtFNwgLm2S8NOTntTgDR qbOodloSPJDMH+h69ovkvfQuajWMK+4lel+VEJ4= X-Google-Smtp-Source: AA6agR7Fbf1wS5v4cssKOJrAZqMXrDyt3HSFRvN1eZwZESf1dpdyFXsR6fkvFIU3PpLWqZcl8WpdXw== X-Received: by 2002:a17:907:a427:b0:761:2bd0:3e72 with SMTP id sg39-20020a170907a42700b007612bd03e72mr10905086ejc.346.1662755718292; Fri, 09 Sep 2022 13:35:18 -0700 (PDT) Received: from mail-ej1-f50.google.com (mail-ej1-f50.google.com. [209.85.218.50]) by smtp.gmail.com with ESMTPSA id e20-20020a170906c01400b0073c8d4c9f38sm709269ejz.177.2022.09.09.13.35.17 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 09 Sep 2022 13:35:17 -0700 (PDT) Received: by mail-ej1-f50.google.com with SMTP id y3so6722749ejc.1 for ; Fri, 09 Sep 2022 13:35:17 -0700 (PDT) X-Received: by 2002:a17:907:2ccb:b0:76f:908:bc56 with SMTP id hg11-20020a1709072ccb00b0076f0908bc56mr11392509ejc.763.1662755717153; Fri, 09 Sep 2022 13:35:17 -0700 (PDT) MIME-Version: 1.0 References: <20220909174410.M560915@dcvr> In-Reply-To: <20220909174410.M560915@dcvr> From: Ricardo Ribalda Date: Fri, 9 Sep 2022 22:35:06 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH] lei: add diagnostics for IMAP writer failures To: Eric Wong Cc: meta@public-inbox.org Content-Type: text/plain; charset="UTF-8" List-Id: Hi Eric On Fri, 9 Sept 2022 at 19:45, Eric Wong wrote: > > Ricardo Ribalda wrote: > > Hi > > > > I am getting a lot of those messages when using lei in imap mode. > > > > Nonetheless the mail seems to arrive fine to its destination (on the > > first run, some of the mail was lost, but for small batches of mails > > it seems to work fine). > > > > It is a valid message or just some red-herring. > > > > I am using debian testing/ > > > > Regards! > > > > # https://lore.kernel.org/all/ limiting to 2022-09-07 10:03 +0000 and newer > > # /usr/local/google/home/ribalda/.local/share/lei/store 13/13 > > # /usr/bin/curl -Sf -s -d '' > > https://lore.kernel.org/all/?x=m&t=1&q=((ribalda)+AND+rt%3A1660039651..)+AND+dt%3A20220907100321.. > > 1285740 lei2mail 6 wq_worker: do_post_auth: Can't call method > > "uidvalidity" on an undefined value at > > /usr/share/perl5/PublicInbox/LeiToMail.pm line 313. > > > > > 1285735 lei2mail 1 wq_worker: write_mail: Can't use an undefined value > > as a subroutine reference at /usr/share/perl5/PublicInbox/LeiToMail.pm > > line 783. > > # https://lore.kernel.org/all/ 18/18 > > # 14 written to imaps://imap.gmail.com/lei/me (31 matches) > > I wonder if it's excessive parallelism for gmail's IMAP. > I haven't tested IMAP destinations, much... > > Can you try the patch at the bottom? > > There's also another patch coming to document the `--jobs|-j' CLI > switch for `lei up' and `lei q', but trying `-j ,1' may help you > if it's parallelism. Note the comma before `1', it accepts > `-j $Q,$W' since $Q is the number of query processes and $W is > the number of LeiToMail writers. The patch did not seem to have any effect :(, I never get a "IMAP LastError: " message On the other hand, the -j worked! I can go up to -j ,15 without any error. > > -------8<------- > From: Eric Wong > Subject: [PATCH] lei: add diagnostics for IMAP writer failures > > This may help diagnose the problem with IMAP destinations > encountered at: > https://public-inbox.org/meta/CANiDSCsDfutAUMBLPZbxdyka+_jnhv+4YNYdL9QPRoC=wNUGCQ@mail.gmail.com/ > --- > lib/PublicInbox/LeiToMail.pm | 4 ++-- > lib/PublicInbox/NetReader.pm | 8 +++++++- > 2 files changed, 9 insertions(+), 3 deletions(-) > > diff --git a/lib/PublicInbox/LeiToMail.pm b/lib/PublicInbox/LeiToMail.pm > index 2aa3977e..bc00b96a 100644 > --- a/lib/PublicInbox/LeiToMail.pm > +++ b/lib/PublicInbox/LeiToMail.pm > @@ -310,8 +310,8 @@ sub _imap_write_cb ($$) { > my $dedupe = $lei->{dedupe}; > $dedupe->prepare_dedupe if $dedupe; > my $append = $lei->{net}->can('imap_append'); > - my $uri = $self->{uri}; > - my $mic = $lei->{net}->mic_get($uri); > + my $uri = $self->{uri} // die 'BUG: no {uri}'; > + my $mic = $lei->{net}->mic_get($uri) // die 'BUG: no $mic'; > my $folder = $uri->mailbox; > $uri->uidvalidity($mic->uidvalidity($folder)); > my $lse = $lei->{lse}; # may be undef > diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm > index c1af03a3..4de2583e 100644 > --- a/lib/PublicInbox/NetReader.pm > +++ b/lib/PublicInbox/NetReader.pm > @@ -685,7 +685,13 @@ sub mic_get { > } > my $mic = mic_new($self, $mic_arg, $sec, $uri); > $cached //= {}; # invalid placeholder if no cache enabled > - $mic && $mic->IsConnected ? ($cached->{$sec} = $mic) : undef; > + if ($mic && $mic->IsConnected) { > + $cached->{$sec} = $mic; > + } else { > + warn 'IMAP LastError: ',$mic->LastError, "\n" if $mic; > + warn "IMAP errno: $!\n" if $!; > + undef; > + } > } > > sub imap_each { -- Ricardo Ribalda