user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* [PATCH] net_reader: restart on first UID when UIDVALIDITY changes
@ 2021-04-23  7:28 Eric Wong
  2021-04-23 11:26 ` Eric Wong
  0 siblings, 1 reply; 2+ messages in thread
From: Eric Wong @ 2021-04-23  7:28 UTC (permalink / raw)
  To: meta

In other words, treat the same IMAP folder with a different
UIDVALIDITY a completely different folder.  If the UIDVALIDITY
changes, we can start from UID=1 without falling behind or
losing data.  If the UIDVALIDITY gets reset to a previously
known-good message, we can still resume where we left off
before the first UIDVALIDITY change.

This affects public-inbox-watch and "lei import"

One potential downside of this is for rare altid users, but
that's mainly intended for NNTP article numbers which are/were
often publicized; not IMAP UIDs which are rarely publicized.

The other potential downside is bandwidth waste in in the rare
case UIDVALIDITY changes while IMAP folder contents remain
unchanged.  There's no extra storage used due to existing
(v1|v2|lei/store) deduplication mechanisms.

Before this change, we were matching offlineimap behavior and
stopped synching an IMAP folder when its UIDVALIDITY changed.
offlineimap behavior made sense for IMAP <=> Maildir
synchronization since Maildirs had no sense of UIDVALIDITY and
could only rely on name mapping.
---
 lib/PublicInbox/IMAPTracker.pm | 16 ++++++++++++----
 lib/PublicInbox/NetReader.pm   | 11 +++--------
 2 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
index bcf7af2e..fe813582 100644
--- a/lib/PublicInbox/IMAPTracker.pm
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -39,12 +39,20 @@ sub dbh_new ($) {
 	$dbh;
 }
 
-sub get_last ($) {
-	my ($self) = @_;
-	my $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+sub get_last ($;$) {
+	my ($self, $validity) = @_;
+	my $sth;
+	if (defined $validity) {
+		$sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+SELECT uid_validity, uid FROM imap_last WHERE url = ? AND uid_validity = ?
+
+		$sth->execute($self->{url}, $validity);
+	} else {
+		$sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
 SELECT uid_validity, uid FROM imap_last WHERE url = ?
 
-	$sth->execute($self->{url});
+		$sth->execute($self->{url});
+	}
 	$sth->fetchrow_array;
 }
 
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index c7b43f01..39129b34 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -397,17 +397,12 @@ sub _imap_fetch_all ($$$) {
 	$r_uidnext //= $mic->uidnext($mbx) //
 		return "E: $uri cannot get UIDNEXT";
 	my $itrk = _itrk($self, $uri);
-	my ($l_uidval, $l_uid) = $itrk ? $itrk->get_last : ();
-	$l_uidval //= $r_uidval; # first time
+	my $l_uid;
+	$l_uid = $itrk->get_last($r_uidval) if $itrk;
 	$l_uid //= 0;
-	if ($l_uidval != $r_uidval) {
-		return "E: $uri UIDVALIDITY mismatch\n".
-			"E: local=$l_uidval != remote=$r_uidval";
-	}
 	my $r_uid = $r_uidnext - 1;
 	if ($l_uid > $r_uid) {
-		return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n".
-			"E: $uri strangely, UIDVALIDLITY matches ($l_uidval)\n";
+		return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n";
 	}
 	return if $l_uid >= $r_uid; # nothing to do
 	$l_uid ||= 1;

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [PATCH] net_reader: restart on first UID when UIDVALIDITY changes
  2021-04-23  7:28 [PATCH] net_reader: restart on first UID when UIDVALIDITY changes Eric Wong
@ 2021-04-23 11:26 ` Eric Wong
  0 siblings, 0 replies; 2+ messages in thread
From: Eric Wong @ 2021-04-23 11:26 UTC (permalink / raw)
  To: meta

Eric Wong <e@80x24.org> wrote:
> In other words, treat the same IMAP folder with a different
> UIDVALIDITY a completely different folder.  If the UIDVALIDITY

"as a completely different folder"

Pushed as commit 7632d8f7590daf70c65d4270e750c36552fa9389

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2021-04-23 11:26 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-23  7:28 [PATCH] net_reader: restart on first UID when UIDVALIDITY changes Eric Wong
2021-04-23 11:26 ` Eric Wong

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.mail.public-inbox.meta
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.mail.public-inbox.meta
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git