From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 628041F727 for ; Wed, 29 Jun 2022 16:53:23 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.b="Kq25QlGQ"; dkim-atps=neutral Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CA47361E11 for ; Wed, 29 Jun 2022 16:53:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 991B7C34114 for ; Wed, 29 Jun 2022 16:53:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1656521602; bh=8xsnpAZcnc17AuyJUS30q/1VjGm56LelG2h5DugzxkM=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=Kq25QlGQW8jmT7Otlt081tTZhlxabFsQG1xOq6mArKfAyXKEEesU/myyrFOUNGKMd pZjoIf1HbEnjAyP7GTzPASLVZ3SsnWLtIQvaRhSJ6ZUuK5Mh+XVNtjOtdSk7yhd5w0 bQ4iiM+Hs8LOmV9E1XJUTBYrGOSiyL8YwdBWuOgqke5uSCBrZYI59mXSLG/7nYN13e t9q3OHApcCqc2tTGTJLMCbJQe+mZrOVmPzxIIujc79N00CO0NLsRj0NMCzE8olCqd9 B1tuIbGkqWutiFGh3ydsrHk58yIe65l7rdHNs/x2LNMcu6xpFSWseRdajWKZJxh8P9 O+mz8G9/pWGFA== Received: by mail-vk1-f177.google.com with SMTP id a15so2718242vkl.10 for ; Wed, 29 Jun 2022 09:53:22 -0700 (PDT) X-Gm-Message-State: AJIora+62IWyIsDGP/oPdAMhOwr5MOTNSMCJtrDqYGfKOwhAkvMnkklw kgP7fL0strd4uMo1FndiCRB7vwmGh4lbYY/ifg== X-Google-Smtp-Source: AGRyM1tthHYMgQW+TC8mk9kC6atZrFkTUk8pa1hfhHeBATthJ2aFs+6R1mtn8fNF6xMhai9gSfiOkum8dDcBMlY+Eig= X-Received: by 2002:a05:6122:91a:b0:370:1b1b:f3c6 with SMTP id j26-20020a056122091a00b003701b1bf3c6mr5812113vka.15.1656521601430; Wed, 29 Jun 2022 09:53:21 -0700 (PDT) MIME-Version: 1.0 References: <20220629163033.GA14412@dcvr> In-Reply-To: <20220629163033.GA14412@dcvr> From: Rob Herring Date: Wed, 29 Jun 2022 10:53:10 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: lei missing mails To: Eric Wong Cc: meta@public-inbox.org Content-Type: text/plain; charset="UTF-8" List-Id: On Wed, Jun 29, 2022 at 10:30 AM Eric Wong wrote: > > Rob Herring wrote: > > Hi, > > > > I'm using lei with lore where I have 2 queries which overlap. Really, > > one is a subset of the other. On those overlapping threads, I'm > > finding that sometimes new messages are written to one mailbox and not > > the other. (At least sometimes, the messages may be missing from all > > mailboxes sometimes too. I'm not certain.) Using --remote-fudge-time > > to force refetching seems to get the missing mails. I haven't found > > anything strange in timestamps of the missing mails, but otherwise am > > not sure how to debug this further. The queries are retrieving full > > threads and the missing mails are in the threads, but not direct > > matches to the queries. I realize that's not a lot of detail to go on. > > Suggestions on debugging this further? > > Is this with 1.8 or 1.7? Commit 68b53c888911 actually. So post 1.8. > I forgot to note in the release notes, but there were some > SQLite usage-related fixes which could avoid missing messages. > > You'll need "lei daemon-kill" after upgrading to 1.8 to ensure > the new code is running. It's possible I haven't done that since updating though I do vaguely recall seeing something about needing to do that. Is there any way to tell before I restart it? > What might be interesting is to use the URLs lei prints and > comparing the results w/o lei. > > I'll have to double-check if overlapping affects things, but it > shouldn't; since the dedupe logic is per-output. > > Is this exclusively with HTTPS endpoints and writing to Maildirs > (or something else?) Yes. It's querying lore and writing to a maildir. Here's one of the queries: [lei] q = (dfn:drivers OR dfn:arch OR dfn:Documentation/* OR dfn:include OR dfn:scripts) AND \ f:robh@kernel.org AND rt:6.month.ago.. [lei "q"] include = https://lore.kernel.org/all/ external = 1 local = 1 remote = 1 threads = 1 dedupe = mid output = maildir:/home/rob/Mail/my-patches > > > It might be helpful if lei could print out message-ids of messages > > written to mailboxes. > > That could get very noisy, especially as mailboxes are written > in parallel. Verbose mode already is. Maybe specifying what info you want to be verbose would help. The network side is mostly uninteresting in this case for example. Is there any tool to list new messages in a maildir? I could do that before and after. I've done the clearing the new flag in mutt between runs, but that's not really ideal. Rob