From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 6BCE51F5AE for ; Fri, 23 Apr 2021 07:28:15 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH] net_reader: restart on first UID when UIDVALIDITY changes Date: Fri, 23 Apr 2021 07:28:15 +0000 Message-Id: <20210423072815.12896-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: In other words, treat the same IMAP folder with a different UIDVALIDITY a completely different folder. If the UIDVALIDITY changes, we can start from UID=1 without falling behind or losing data. If the UIDVALIDITY gets reset to a previously known-good message, we can still resume where we left off before the first UIDVALIDITY change. This affects public-inbox-watch and "lei import" One potential downside of this is for rare altid users, but that's mainly intended for NNTP article numbers which are/were often publicized; not IMAP UIDs which are rarely publicized. The other potential downside is bandwidth waste in in the rare case UIDVALIDITY changes while IMAP folder contents remain unchanged. There's no extra storage used due to existing (v1|v2|lei/store) deduplication mechanisms. Before this change, we were matching offlineimap behavior and stopped synching an IMAP folder when its UIDVALIDITY changed. offlineimap behavior made sense for IMAP <=> Maildir synchronization since Maildirs had no sense of UIDVALIDITY and could only rely on name mapping. --- lib/PublicInbox/IMAPTracker.pm | 16 ++++++++++++---- lib/PublicInbox/NetReader.pm | 11 +++-------- 2 files changed, 15 insertions(+), 12 deletions(-) diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm index bcf7af2e..fe813582 100644 --- a/lib/PublicInbox/IMAPTracker.pm +++ b/lib/PublicInbox/IMAPTracker.pm @@ -39,12 +39,20 @@ sub dbh_new ($) { $dbh; } -sub get_last ($) { - my ($self) = @_; - my $sth = $self->{dbh}->prepare_cached(<<'', undef, 1); +sub get_last ($;$) { + my ($self, $validity) = @_; + my $sth; + if (defined $validity) { + $sth = $self->{dbh}->prepare_cached(<<'', undef, 1); +SELECT uid_validity, uid FROM imap_last WHERE url = ? AND uid_validity = ? + + $sth->execute($self->{url}, $validity); + } else { + $sth = $self->{dbh}->prepare_cached(<<'', undef, 1); SELECT uid_validity, uid FROM imap_last WHERE url = ? - $sth->execute($self->{url}); + $sth->execute($self->{url}); + } $sth->fetchrow_array; } diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm index c7b43f01..39129b34 100644 --- a/lib/PublicInbox/NetReader.pm +++ b/lib/PublicInbox/NetReader.pm @@ -397,17 +397,12 @@ sub _imap_fetch_all ($$$) { $r_uidnext //= $mic->uidnext($mbx) // return "E: $uri cannot get UIDNEXT"; my $itrk = _itrk($self, $uri); - my ($l_uidval, $l_uid) = $itrk ? $itrk->get_last : (); - $l_uidval //= $r_uidval; # first time + my $l_uid; + $l_uid = $itrk->get_last($r_uidval) if $itrk; $l_uid //= 0; - if ($l_uidval != $r_uidval) { - return "E: $uri UIDVALIDITY mismatch\n". - "E: local=$l_uidval != remote=$r_uidval"; - } my $r_uid = $r_uidnext - 1; if ($l_uid > $r_uid) { - return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n". - "E: $uri strangely, UIDVALIDLITY matches ($l_uidval)\n"; + return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n"; } return if $l_uid >= $r_uid; # nothing to do $l_uid ||= 1;