user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
* [PATCH 0/4] some lei synchronization work
@ 2021-04-29  9:46 Eric Wong
  2021-04-29  9:46 ` [PATCH 1/4] content_hash: git_sha: allow unblessed SCALAR refs Eric Wong
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Eric Wong @ 2021-04-29  9:46 UTC (permalink / raw)
  To: meta

2/4 is a fairly big change to recent-ish behavior; but ought to
help us avoid problems by avoiding mismatched sources of truth.
-watch once again halts on UIDVALIDITY changes (matching <=1.6
behavior).

Tor .onion URLs for NNTP and IMAP also aren't supported unless
the entire lei-daemon process is running under torsocks.
Net::NNTP doesn't seem to have provisions for using
user-supplied classes like IO::Socket::SOCKS, either.
Mail::IMAPClient supports ->Socket and ->RawSocket, at least...

Eric Wong (4):
  content_hash: git_sha: allow unblessed SCALAR refs
  lei import: avoid IMAPTracker, use LeiMailSync more
  lei import: support UIDVALIDITY in IMAP URL
  lei import: support shell completion of known folders

 Documentation/lei-store-format.pod     |  2 +-
 contrib/completion/lei-completion.bash |  1 +
 lib/PublicInbox/ContentHash.pm         |  6 +--
 lib/PublicInbox/IMAPTracker.pm         | 35 +++++---------
 lib/PublicInbox/LEI.pm                 |  2 +-
 lib/PublicInbox/LeiExternal.pm         | 22 +++++----
 lib/PublicInbox/LeiImport.pm           | 20 ++++----
 lib/PublicInbox/LeiInput.pm            | 20 +++++---
 lib/PublicInbox/LeiMailSync.pm         |  2 +-
 lib/PublicInbox/LeiUp.pm               |  4 +-
 lib/PublicInbox/LeiViewText.pm         |  1 -
 lib/PublicInbox/NetReader.pm           | 67 +++++++++++++++++---------
 lib/PublicInbox/URIimap.pm             |  2 +
 t/lei-import-imap.t                    |  8 ++-
 t/lei-import-nntp.t                    |  4 +-
 t/solver_git.t                         |  7 +--
 16 files changed, 112 insertions(+), 91 deletions(-)

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/4] content_hash: git_sha: allow unblessed SCALAR refs
  2021-04-29  9:46 [PATCH 0/4] some lei synchronization work Eric Wong
@ 2021-04-29  9:46 ` Eric Wong
  2021-04-29  9:46 ` [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more Eric Wong
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-04-29  9:46 UTC (permalink / raw)
  To: meta

This will be convenient to avoid the overhead of
PublicInbox::Eml for verifying synchronization in lei.
---
 lib/PublicInbox/ContentHash.pm | 6 +++---
 lib/PublicInbox/LeiViewText.pm | 1 -
 t/solver_git.t                 | 7 ++-----
 3 files changed, 5 insertions(+), 9 deletions(-)

diff --git a/lib/PublicInbox/ContentHash.pm b/lib/PublicInbox/ContentHash.pm
index 112b1ea6..cc4a54c9 100644
--- a/lib/PublicInbox/ContentHash.pm
+++ b/lib/PublicInbox/ContentHash.pm
@@ -97,9 +97,9 @@ sub content_hash ($) {
 sub git_sha ($$) {
 	my ($n, $eml) = @_;
 	my $dig = Digest::SHA->new($n);
-	my $buf = $eml->as_string;
-	$dig->add('blob '.length($buf)."\0");
-	$dig->add($buf);
+	my $bref = ref($eml) eq 'SCALAR' ? $eml : \($eml->as_string);
+	$dig->add('blob '.length($$bref)."\0");
+	$dig->add($$bref);
 	$dig;
 }
 
diff --git a/lib/PublicInbox/LeiViewText.pm b/lib/PublicInbox/LeiViewText.pm
index d0f8b7f4..340a6648 100644
--- a/lib/PublicInbox/LeiViewText.pm
+++ b/lib/PublicInbox/LeiViewText.pm
@@ -7,7 +7,6 @@ package PublicInbox::LeiViewText;
 use strict;
 use v5.10.1;
 use PublicInbox::MsgIter qw(msg_part_text);
-use PublicInbox::ContentHash qw(git_sha);
 use PublicInbox::MID qw(references);
 use PublicInbox::View;
 use PublicInbox::Hval;
diff --git a/t/solver_git.t b/t/solver_git.t
index 6875e26b..75387b2a 100644
--- a/t/solver_git.t
+++ b/t/solver_git.t
@@ -7,7 +7,6 @@ use PublicInbox::TestCommon;
 use Cwd qw(abs_path);
 require_git(2.6);
 use PublicInbox::ContentHash qw(git_sha);
-use PublicInbox::Eml;
 use PublicInbox::Spawn qw(popen_rd which);
 require_mods(qw(DBD::SQLite Search::Xapian Plack::Util));
 my $git_dir = xqx([qw(git rev-parse --git-dir)], undef, {2 => \(my $null)});
@@ -41,8 +40,7 @@ test_lei({tmpdir => $tmpdir}, sub {
 		"--mail won't run solver");
 
 	lei_ok('blob', '69df7d5', '-I', $ibx->{inboxdir});
-	is(git_sha(1, PublicInbox::Eml->new($lei_out))->hexdigest,
-		$expect, 'blob contents output');
+	is(git_sha(1, \$lei_out)->hexdigest, $expect, 'blob contents output');
 	my $prev = $lei_out;
 	lei_ok(qw(blob --no-mail 69df7d5 -I), $ibx->{inboxdir});
 	is($lei_out, $prev, '--no-mail works');
@@ -239,8 +237,7 @@ EOF
 		test_lei({tmpdir => "$tmpdir/ext"}, sub {
 			my $rurl = "$url/$name";
 			lei_ok(qw(blob --no-mail 69df7d5 -I), $rurl);
-			my $eml = PublicInbox::Eml->new($lei_out);
-			is(git_sha(1, $eml)->hexdigest, $expect,
+			is(git_sha(1, \$lei_out)->hexdigest, $expect,
 				'blob contents output');
 			ok(!lei(qw(blob -I), $rurl, $non_existent),
 					'non-existent blob fails');

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more
  2021-04-29  9:46 [PATCH 0/4] some lei synchronization work Eric Wong
  2021-04-29  9:46 ` [PATCH 1/4] content_hash: git_sha: allow unblessed SCALAR refs Eric Wong
@ 2021-04-29  9:46 ` Eric Wong
  2021-04-29  9:46 ` [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL Eric Wong
  2021-04-29  9:46 ` [PATCH 4/4] lei import: support shell completion of known folders Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-04-29  9:46 UTC (permalink / raw)
  To: meta

IMAPTracker has a UNIQUE constraint on the `url' column,
which may cause compatibility and/or rollback problems
in attempting to deal with UIDVALIDITY changes.

Having multiple sources of truth leads to confusion and bugs,
so relying on LeiMailSync exclusively ought to simplify things.

Furthermore, since LeiMailSync is only written to by LeiStore,
it is safer in that it won't mark a UID or article as imported
until git-fast-import has seen it, and the SQLite commit always
happens after "done\n" is sent to fast-import.

This mostly reverts recent commits to IMAPTracker to support
lei, those are:

1) commit 7632d8f7590daf70c65d4270e750c36552fa9389
   ("net_reader: restart on first UID when UIDVALIDITY changes")
2) commit 311a5d37ad275cd75b1e64d87827c4d13fe4bfab
   ("imap_tracker: prepare for use with lei").

This means public-inbox-watch will not change between 1.6 and
1.7: -watch stops synching a folder when UIDVALIDITY changes.
---
 Documentation/lei-store-format.pod |  2 +-
 lib/PublicInbox/IMAPTracker.pm     | 35 ++++++-----------
 lib/PublicInbox/LEI.pm             |  2 +-
 lib/PublicInbox/LeiImport.pm       | 12 ++----
 lib/PublicInbox/LeiInput.pm        | 10 ++---
 lib/PublicInbox/LeiMailSync.pm     |  2 +-
 lib/PublicInbox/NetReader.pm       | 60 ++++++++++++++++++------------
 lib/PublicInbox/URIimap.pm         |  2 +
 t/lei-import-imap.t                |  4 +-
 t/lei-import-nntp.t                |  4 +-
 10 files changed, 64 insertions(+), 69 deletions(-)

diff --git a/Documentation/lei-store-format.pod b/Documentation/lei-store-format.pod
index 3e1ddc65..71aa72cb 100644
--- a/Documentation/lei-store-format.pod
+++ b/Documentation/lei-store-format.pod
@@ -32,7 +32,7 @@ prevent them from being accidentally treated as a v2 inbox.
   ~/.local/share/lei/store
   - ipc.lock                        # lock file for internal lei IPC
   - local/$EPOCH.git                # normal bare git repositories
-  - net_last.sqlite3                # import state for IMAP & NNTP
+  - mail_sync.sqlite3               # sync state IMAP, Maildir, NNTP
 
 Additionally, the following share the same roles they do in extindex:
 
diff --git a/lib/PublicInbox/IMAPTracker.pm b/lib/PublicInbox/IMAPTracker.pm
index fe813582..5eb33cf7 100644
--- a/lib/PublicInbox/IMAPTracker.pm
+++ b/lib/PublicInbox/IMAPTracker.pm
@@ -39,20 +39,12 @@ sub dbh_new ($) {
 	$dbh;
 }
 
-sub get_last ($;$) {
-	my ($self, $validity) = @_;
-	my $sth;
-	if (defined $validity) {
-		$sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
-SELECT uid_validity, uid FROM imap_last WHERE url = ? AND uid_validity = ?
-
-		$sth->execute($self->{url}, $validity);
-	} else {
-		$sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
+sub get_last ($) {
+	my ($self) = @_;
+	my $sth = $self->{dbh}->prepare_cached(<<'', undef, 1);
 SELECT uid_validity, uid FROM imap_last WHERE url = ?
 
-		$sth->execute($self->{url});
-	}
+	$sth->execute($self->{url});
 	$sth->fetchrow_array;
 }
 
@@ -70,19 +62,16 @@ VALUES (?, ?, ?)
 }
 
 sub new {
-	my ($class, $url, $dbname) = @_;
+	my ($class, $url) = @_;
 
-	unless (defined($dbname)) {
-		# original name for compatibility with old setups:
-		$dbname = PublicInbox::Config->config_dir() . '/imap.sqlite3';
+	# original name for compatibility with old setups:
+	my $dbname = PublicInbox::Config->config_dir() . '/imap.sqlite3';
 
-		# use the new XDG-compliant name for new setups:
-		if (!-f $dbname) {
-			$dbname = ($ENV{XDG_DATA_HOME} //
-					(($ENV{HOME} // '/nonexistent').
-					 '/.local/share')) .
-				'/public-inbox/imap.sqlite3';
-		}
+	# use the new XDG-compliant name for new setups:
+	if (!-f $dbname) {
+		$dbname = ($ENV{XDG_DATA_HOME} //
+			(($ENV{HOME} // '/nonexistent').'/.local/share')) .
+			'/public-inbox/imap.sqlite3';
 	}
 	if (!-f $dbname) {
 		require File::Path;
diff --git a/lib/PublicInbox/LEI.pm b/lib/PublicInbox/LEI.pm
index 1ea7c9ca..52ce8ec2 100644
--- a/lib/PublicInbox/LEI.pm
+++ b/lib/PublicInbox/LEI.pm
@@ -211,7 +211,7 @@ our %CMD = ( # sorted in order of importance/use:
 'import' => [ 'LOCATION...|--stdin',
 	'one-time import/update from URL or filesystem',
 	qw(stdin| offset=i recursive|r exclude=s include|I=s
-	lock=s@ in-format|F=s kw! verbose|v+ incremental! sync!), @c_opt ],
+	lock=s@ in-format|F=s kw! verbose|v+ incremental! mail-sync!), @c_opt ],
 'convert' => [ 'LOCATION...|--stdin',
 	'one-time conversion from URL or filesystem to another format',
 	qw(stdin| in-format|F=s out-format|f=s output|mfolder|o=s
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 26127ece..277f4f95 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -41,18 +41,13 @@ sub input_maildir_cb { # maildir_each_eml cb
 	input_eml_cb($self, $eml, $vmd);
 }
 
-sub input_imap_cb { # imap_each
+sub input_net_cb { # imap_each / nntp_each
 	my ($url, $uid, $kw, $eml, $self) = @_;
 	my $vmd = $self->{-import_kw} ? { kw => $kw } : undef;
 	$vmd->{sync_info} = [ $url, $uid ] if $self->{-mail_sync};
 	input_eml_cb($self, $eml, $vmd);
 }
 
-sub input_nntp_cb { # nntp_each
-	my ($url, $num, $kw, $eml, $self) = @_;
-	input_eml_cb($self, $eml, $self->{-import_kw} ? { kw => $kw } : undef);
-}
-
 sub net_merge_complete { # callback used by LeiAuth
 	my ($self) = @_;
 	$self->wq_io_do('process_inputs');
@@ -69,7 +64,7 @@ sub lei_import { # the main "lei import" method
 	return $lei->fail(join("\n", @{$vmd_mod->{err}})) if $vmd_mod->{err};
 	$self->{all_vmd} = $vmd_mod if scalar keys %$vmd_mod;
 	$self->prepare_inputs($lei, \@inputs) or return;
-	$self->{-mail_sync} = $lei->{opt}->{sync} // 1;
+	$self->{-mail_sync} = $lei->{opt}->{'mail-sync'} // 1;
 
 	$lei->ale; # initialize for workers to read
 	my $j = $lei->{opt}->{jobs} // scalar(@{$self->{inputs}}) || 1;
@@ -77,8 +72,7 @@ sub lei_import { # the main "lei import" method
 		# $j = $net->net_concurrency($j); TODO
 		if ($lei->{opt}->{incremental} // 1) {
 			$net->{incremental} = 1;
-			$net->{itrk_fn} = $lei->store_path .
-						'/net_last.sqlite3';
+			$net->{-lms_ro} = $lei->_lei_store->search->lms // 0;
 		}
 	} else {
 		my $nproc = $self->detect_nproc;
diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index 785e607d..ce675f40 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -110,14 +110,12 @@ sub input_path_url {
 	my $ifmt = lc($lei->{opt}->{'in-format'} // '');
 	# TODO auto-detect?
 	if ($input =~ m!\Aimaps?://!i) {
-		$lei->{net}->imap_each($input, $self->can('input_imap_cb') //
-						$self->can('input_net_cb'),
-					$self, @args);
+		$lei->{net}->imap_each($input, $self->can('input_net_cb'),
+						$self, @args);
 		return;
 	} elsif ($input =~ m!\A(?:nntps?|s?news)://!i) {
-		$lei->{net}->nntp_each($input, $self->can('input_nntp_cb') //
-						$self->can('input_net_cb'),
-					$self, @args);
+		$lei->{net}->nntp_each($input, $self->can('input_net_cb'),
+						$self, @args);
 		return;
 	} elsif ($input =~ m!\Ahttps?://!i) {
 		handle_http_input($self, $input, @args);
diff --git a/lib/PublicInbox/LeiMailSync.pm b/lib/PublicInbox/LeiMailSync.pm
index 52f26d69..2ce189fa 100644
--- a/lib/PublicInbox/LeiMailSync.pm
+++ b/lib/PublicInbox/LeiMailSync.pm
@@ -143,7 +143,7 @@ sub each_src {
 }
 
 sub location_stats {
-	my ($self, $folder, $cb, @args) = @_;
+	my ($self, $folder) = @_;
 	my $dbh = $self->{dbh} //= dbh_new($self);
 	my $fid;
 	my $ret = {};
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 5978752f..81d25ead 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -235,7 +235,7 @@ sub imap_common_init ($;$) {
 	$self->{quiet} = 1 if $lei && $lei->{opt}->{quiet};
 	eval { require PublicInbox::IMAPClient } or
 		die "Mail::IMAPClient is required for IMAP:\n$@\n";
-	eval { require PublicInbox::IMAPTracker } or
+	($lei || eval { require PublicInbox::IMAPTracker }) or
 		die "DBD::SQLite is required for IMAP\n:$@\n";
 	require PublicInbox::URIimap;
 	my $cfg = $self->{pi_cfg} // $lei->_lei_cfg;
@@ -283,7 +283,7 @@ sub nntp_common_init ($;$) {
 	$self->{quiet} = 1 if $lei && $lei->{opt}->{quiet};
 	eval { require Net::NNTP } or
 		die "Net::NNTP is required for NNTP:\n$@\n";
-	eval { require PublicInbox::IMAPTracker } or
+	($lei || eval { require PublicInbox::IMAPTracker }) or
 		die "DBD::SQLite is required for NNTP\n:$@\n";
 	my $cfg = $self->{pi_cfg} // $lei->_lei_cfg;
 	my $nn_args = {}; # scheme://authority => Net::NNTP->new arg
@@ -373,17 +373,28 @@ sub run_commit_cb ($) {
 	$cb->(@args);
 }
 
-sub _itrk ($$) {
-	my ($self, $uri) = @_;
-	return unless $self->{incremental};
-	# itrk_fn is set by lei
-	PublicInbox::IMAPTracker->new($$uri, $self->{itrk_fn});
+sub _itrk_last ($$;$) {
+	my ($self, $uri, $r_uidval) = @_;
+	return (undef, undef, $r_uidval) unless $self->{incremental};
+	my ($itrk, $l_uid, $l_uidval);
+	if (defined(my $lms = $self->{-lms_ro})) { # LeiMailSync or 0
+		$uri->uidvalidity($r_uidval) if defined $r_uidval;
+		my $x;
+		$l_uid = ($lms && ($x = $lms->location_stats($$uri))) ?
+				$x->{'uid.max'} : undef;
+		# itrk remains undef, lei/store worker writes to
+		# mail_sync.sqlite3
+	} else {
+		$itrk = PublicInbox::IMAPTracker->new($$uri);
+		($l_uidval, $l_uid) = $itrk->get_last($$uri);
+	}
+	($itrk, $l_uid, $l_uidval //= $r_uidval);
 }
 
 sub _imap_fetch_all ($$$) {
-	my ($self, $mic, $uri) = @_;
-	my $sec = uri_section($uri);
-	my $mbx = $uri->mailbox;
+	my ($self, $mic, $orig_uri) = @_;
+	my $sec = uri_section($orig_uri);
+	my $mbx = $orig_uri->mailbox;
 	$mic->Clear(1); # trim results history
 	$mic->examine($mbx) or return "E: EXAMINE $mbx ($sec) failed: $!";
 	my ($r_uidval, $r_uidnext);
@@ -393,20 +404,22 @@ sub _imap_fetch_all ($$$) {
 		last if $r_uidval && $r_uidnext;
 	}
 	$r_uidval //= $mic->uidvalidity($mbx) //
-		return "E: $uri cannot get UIDVALIDITY";
+		return "E: $orig_uri cannot get UIDVALIDITY";
 	$r_uidnext //= $mic->uidnext($mbx) //
-		return "E: $uri cannot get UIDNEXT";
-	my $url = ref($uri)->new($$uri);
-	$url->uidvalidity($r_uidval);
-	$url = $$url;
-	my $itrk = _itrk($self, $uri);
-	my $l_uid;
-	$l_uid = $itrk->get_last($r_uidval) if $itrk;
+		return "E: $orig_uri cannot get UIDNEXT";
+	my $uri = $orig_uri->clone;
+	my ($itrk, $l_uid, $l_uidval) = _itrk_last($self, $uri, $r_uidval);
+	return <<EOF if $l_uidval != $r_uidval;
+E: $uri UIDVALIDITY mismatch
+E: local=$l_uidval != remote=$r_uidval
+EOF
+	$uri->uidvalidity($r_uidval);
 	$l_uid //= 0;
 	my $r_uid = $r_uidnext - 1;
-	if ($l_uid > $r_uid) {
-		return "E: $uri local UID exceeds remote ($l_uid > $r_uid)\n";
-	}
+	return <<EOF if $l_uid > $r_uid;
+E: $uri local UID exceeds remote ($l_uid > $r_uid)
+E: $uri strangely, UIDVALIDLITY matches ($l_uidval)
+EOF
 	return if $l_uid >= $r_uid; # nothing to do
 	$l_uid ||= 1;
 	my ($mod, $shard) = @{$self->{shard_info} // []};
@@ -458,7 +471,7 @@ sub _imap_fetch_all ($$$) {
 				# messages get deleted, so holes appear
 				my $per_uid = delete $r->{$uid} // next;
 				my $raw = delete($per_uid->{$key}) // next;
-				_imap_do_msg($self, $url, $uid, \$raw,
+				_imap_do_msg($self, $$uri, $uid, \$raw,
 						$per_uid->{FLAGS});
 				$last_uid = $uid;
 				last if $self->{quit};
@@ -547,8 +560,7 @@ sub _nntp_fetch_all ($$$) {
 	# IMAPTracker is also used for tracking NNTP, UID == article number
 	# LIST.ACTIVE can get the equivalent of UIDVALIDITY, but that's
 	# expensive.  So we assume newsgroups don't change:
-	my $itrk = _itrk($self, $uri);
-	my (undef, $l_art) = $itrk ? $itrk->get_last : ();
+	my ($itrk, $l_art) = _itrk_last($self, $uri);
 
 	# allow users to specify articles to refetch
 	# cf. https://tools.ietf.org/id/draft-gilman-news-url-01.txt
diff --git a/lib/PublicInbox/URIimap.pm b/lib/PublicInbox/URIimap.pm
index df9f5fd9..f6244137 100644
--- a/lib/PublicInbox/URIimap.pm
+++ b/lib/PublicInbox/URIimap.pm
@@ -144,4 +144,6 @@ sub scheme {
 
 sub as_string { ${$_[0]} }
 
+sub clone { ref($_[0])->new(as_string($_[0])) }
+
 1;
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index cf1fa49d..611328b4 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -45,8 +45,8 @@ test_lei({ tmpdir => $tmpdir }, sub {
 	is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
 	lei_ok([qw(tag +kw:seen), $url], undef, undef);
 
-	my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
-	ok(-s $f, 'net tracked for redundant imports');
+	my $f = "$ENV{HOME}/.local/share/lei/store/mail_sync.sqlite3";
+	ok(-s $f, 'mail_sync tracked for redundant imports');
 	lei_ok('inspect', "blob:$out->[5]->{blob}");
 	my $x = json_utf8->decode($lei_out);
 	is(ref($x->{'lei/store'}), 'ARRAY', 'lei/store in inspect');
diff --git a/t/lei-import-nntp.t b/t/lei-import-nntp.t
index d795a86a..12bb002a 100644
--- a/t/lei-import-nntp.t
+++ b/t/lei-import-nntp.t
@@ -27,7 +27,7 @@ test_lei({ tmpdir => $tmpdir }, sub {
 	for (@$out) { $r{ref($_)}++ }
 	is_deeply(\%r, { 'HASH' => scalar(@$out) }, 'all hashes');
 
-	my $f = "$ENV{HOME}/.local/share/lei/store/net_last.sqlite3";
-	ok(-s $f, 'net tracked for redundant imports');
+	my $f = "$ENV{HOME}/.local/share/lei/store/mail_sync.sqlite3";
+	ok(-s $f, 'mail_sync exists tracked for redundant imports');
 });
 done_testing;

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL
  2021-04-29  9:46 [PATCH 0/4] some lei synchronization work Eric Wong
  2021-04-29  9:46 ` [PATCH 1/4] content_hash: git_sha: allow unblessed SCALAR refs Eric Wong
  2021-04-29  9:46 ` [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more Eric Wong
@ 2021-04-29  9:46 ` Eric Wong
  2021-04-29  9:46 ` [PATCH 4/4] lei import: support shell completion of known folders Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-04-29  9:46 UTC (permalink / raw)
  To: meta

Specifying a UIDVALIDITY value allows the user to enforce
a strict match and force failure.  This necessitated changes
to NetReader to allow die() and make error reporting more
suitable for CLI usage rather than daemonized usage of -watch.
---
 lib/PublicInbox/LeiInput.pm  | 10 +++++++++-
 lib/PublicInbox/NetReader.pm |  7 +++++++
 t/lei-import-imap.t          |  4 ++++
 3 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/LeiInput.pm b/lib/PublicInbox/LeiInput.pm
index ce675f40..277ad88d 100644
--- a/lib/PublicInbox/LeiInput.pm
+++ b/lib/PublicInbox/LeiInput.pm
@@ -293,6 +293,7 @@ $input is `eml', not --in-format=$in_fmt
 		$lei->err("# --sync is not supported for: @{$sync->{no}}");
 	}
 	if ($net) {
+		$net->{-can_die} = 1;
 		if (my $err = $net->errors) {
 			return $lei->fail($err);
 		}
@@ -306,10 +307,17 @@ $input is `eml', not --in-format=$in_fmt
 
 sub process_inputs {
 	my ($self) = @_;
+	my $err;
 	for my $input (@{$self->{inputs}}) {
-		$self->input_path_url($input);
+		eval { $self->input_path_url($input) };
+		next unless $@;
+		$err = "$input: $@";
+		last;
 	}
+	# always commit first, even on error partial work is acceptable for
+	# lei <import|tag|convert>
 	my $wait = $self->{lei}->{sto}->ipc_do('done') if $self->{lei}->{sto};
+	$self->{lei}->fail($err) if $err;
 }
 
 sub input_only_atfork_child {
diff --git a/lib/PublicInbox/NetReader.pm b/lib/PublicInbox/NetReader.pm
index 81d25ead..3fc37b10 100644
--- a/lib/PublicInbox/NetReader.pm
+++ b/lib/PublicInbox/NetReader.pm
@@ -407,6 +407,11 @@ sub _imap_fetch_all ($$$) {
 		return "E: $orig_uri cannot get UIDVALIDITY";
 	$r_uidnext //= $mic->uidnext($mbx) //
 		return "E: $orig_uri cannot get UIDNEXT";
+	my $expect = $orig_uri->uidvalidity // $r_uidval;
+	return <<EOF if $expect != $r_uidval;
+E: $orig_uri UIDVALIDITY mismatch (got $r_uidval)
+EOF
+
 	my $uri = $orig_uri->clone;
 	my ($itrk, $l_uid, $l_uidval) = _itrk_last($self, $uri, $r_uidval);
 	return <<EOF if $l_uidval != $r_uidval;
@@ -520,6 +525,7 @@ sub imap_each {
 	} else {
 		$err = "E: <$uri> not connected: $!";
 	}
+	die $err if $err && $self->{-can_die};
 	warn $err if $err;
 	$mic;
 }
@@ -620,6 +626,7 @@ sub nntp_each {
 	} else {
 		$err = "E: <$uri> not connected: $!";
 	}
+	die $err if $err && $self->{-can_die};
 	warn $err if $err;
 	$nn;
 }
diff --git a/t/lei-import-imap.t b/t/lei-import-imap.t
index 611328b4..c977c68e 100644
--- a/t/lei-import-imap.t
+++ b/t/lei-import-imap.t
@@ -24,6 +24,10 @@ test_lei({ tmpdir => $tmpdir }, sub {
 	lei_ok('import', $url);
 	lei_ok 'ls-sync';
 	like($lei_out, qr!\A\Q$url\E;UIDVALIDITY=\d+\n\z!, 'ls-sync');
+	chomp(my $u = $lei_out);
+	lei_ok('import', $u, \'UIDVALIDITY match in URL');
+	$u =~ s/;UIDVALIDITY=(\d+)\s*/;UIDVALIDITY=9$1/s;
+	ok(!lei('import', $u), 'UIDVALIDITY mismatch in URL rejected');
 
 	lei_ok('inspect', $url);
 	my $inspect = json_utf8->decode($lei_out);

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 4/4] lei import: support shell completion of known folders
  2021-04-29  9:46 [PATCH 0/4] some lei synchronization work Eric Wong
                   ` (2 preceding siblings ...)
  2021-04-29  9:46 ` [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL Eric Wong
@ 2021-04-29  9:46 ` Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2021-04-29  9:46 UTC (permalink / raw)
  To: meta

This also fixes completion of "lei up" for IMAP folders.
---
 contrib/completion/lei-completion.bash |  1 +
 lib/PublicInbox/LeiExternal.pm         | 22 ++++++++++++----------
 lib/PublicInbox/LeiImport.pm           |  8 ++++++++
 lib/PublicInbox/LeiUp.pm               |  4 ++--
 4 files changed, 23 insertions(+), 12 deletions(-)

diff --git a/contrib/completion/lei-completion.bash b/contrib/completion/lei-completion.bash
index 2c28d44a..5c137e68 100644
--- a/contrib/completion/lei-completion.bash
+++ b/contrib/completion/lei-completion.bash
@@ -9,6 +9,7 @@ _lei() {
 	*':'* | *'='* | '//'*) compopt -o nospace ;;
 	*) compopt +o nospace ;; # the default
 	esac
+	wordlist="${wordlist//;/\\\\;}" # escape ';' for ';UIDVALIDITY' and such
 	COMPREPLY=($(compgen -W "$wordlist" -- "${COMP_WORDS[COMP_CWORD]}"))
 	return 0
 }
diff --git a/lib/PublicInbox/LeiExternal.pm b/lib/PublicInbox/LeiExternal.pm
index 3858085e..6fd3efef 100644
--- a/lib/PublicInbox/LeiExternal.pm
+++ b/lib/PublicInbox/LeiExternal.pm
@@ -215,7 +215,8 @@ sub lei_forget_external {
 	}
 }
 
-sub complete_url_common {
+# returns an anonymous sub which returns an array of potential results
+sub complete_url_prepare {
 	my $argv = $_[-1];
 	# Workaround bash word-splitting URLs to ['https', ':', '//' ...]
 	# Maybe there's a better way to go about this in
@@ -239,37 +240,38 @@ sub complete_url_common {
 		}
 		$re = quotemeta($re);
 	}
-	($cur, $re);
+	my $match_cb = sub {
+		# only return the part specified on the CLI
+		# don't duplicate if already 100% completed
+		$_[0] =~ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ()
+	};
+	wantarray ? ($re, $cur, $match_cb) : $match_cb;
 }
 
 # shell completion helper called by lei__complete
 sub _complete_forget_external {
 	my ($self, @argv) = @_;
 	my $cfg = $self->_lei_cfg;
-	my ($cur, $re) = complete_url_common(\@argv);
+	my ($cur, $re, $match_cb) = complete_url_prepare(\@argv);
 	# FIXME: bash completion off "http:" or "https:" when the last
 	# character is a colon doesn't work properly even if we're
 	# returning "//$HTTP_HOST/$PATH_INFO/", not sure why, could
 	# be a bash issue.
 	map {
-		my $x = substr($_, length('external.'));
-		# only return the part specified on the CLI
-		# don't duplicate if already 100% completed
-		$x =~ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ();
+		$match_cb->(substr($_, length('external.')));
 	} grep(/\Aexternal\.$re\Q$cur/, @{$cfg->{-section_order}});
 }
 
 sub _complete_add_external { # for bash, this relies on "compopt -o nospace"
 	my ($self, @argv) = @_;
 	my $cfg = $self->_lei_cfg;
-	my ($cur, $re) = complete_url_common(\@argv);
+	my $match_cb = complete_url_prepare(\@argv);
 	require URI;
 	map {
 		my $u = URI->new(substr($_, length('external.')));
 		my ($base) = ($u->path =~ m!((?:/?.*)?/)[^/]+/?\z!);
 		$u->path($base);
-		$u = $u->as_string;
-		$u =~ /\A$re(\Q$cur\E.*)/ ? ($cur eq $1 ? () : $1) : ();
+		$match_cb->($u->as_string);
 	} grep(m!\Aexternal\.https?://!, @{$cfg->{-section_order}});
 }
 
diff --git a/lib/PublicInbox/LeiImport.pm b/lib/PublicInbox/LeiImport.pm
index 277f4f95..def121ab 100644
--- a/lib/PublicInbox/LeiImport.pm
+++ b/lib/PublicInbox/LeiImport.pm
@@ -89,6 +89,14 @@ sub lei_import { # the main "lei import" method
 	$op_c->op_wait_event($ops);
 }
 
+sub _complete_import {
+	my ($lei, @argv) = @_;
+	my $sto = $lei->_lei_store or return;
+	my $lms = $sto->search->lms or return;
+	my $match_cb = $lei->complete_url_prepare(\@argv);
+	map { $match_cb->($_) } $lms->folders;
+}
+
 no warnings 'once';
 *ipc_atfork_child = \&PublicInbox::LeiInput::input_only_atfork_child;
 
diff --git a/lib/PublicInbox/LeiUp.pm b/lib/PublicInbox/LeiUp.pm
index f4ff070b..4399c4fb 100644
--- a/lib/PublicInbox/LeiUp.pm
+++ b/lib/PublicInbox/LeiUp.pm
@@ -99,8 +99,8 @@ sub lei_up {
 
 sub _complete_up {
 	my ($lei, @argv) = @_;
-	my ($cur, $re) = $lei->complete_url_common(\@argv);
-	grep(/\A$re\Q$cur/, PublicInbox::LeiSavedSearch::list($lei));
+	my $match_cb = $lei->complete_url_prepare(\@argv);
+	map { $match_cb->($_) } PublicInbox::LeiSavedSearch::list($lei);
 }
 
 1;

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-04-29  9:46 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-29  9:46 [PATCH 0/4] some lei synchronization work Eric Wong
2021-04-29  9:46 ` [PATCH 1/4] content_hash: git_sha: allow unblessed SCALAR refs Eric Wong
2021-04-29  9:46 ` [PATCH 2/4] lei import: avoid IMAPTracker, use LeiMailSync more Eric Wong
2021-04-29  9:46 ` [PATCH 3/4] lei import: support UIDVALIDITY in IMAP URL Eric Wong
2021-04-29  9:46 ` [PATCH 4/4] lei import: support shell completion of known folders Eric Wong

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ https://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://7fh6tueqddpjyxjmgtdiueylzoqt6pt7hec3pukyptlmohoowvhde4yd.onion/inbox.comp.mail.public-inbox.meta
	nntp://ie5yzdi7fg72h7s4sdcztq5evakq23rdt33mfyfcddc5u3ndnw24ogqd.onion/inbox.comp.mail.public-inbox.meta
	nntp://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git