user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 43/82] imap: require ".$UID_MIN-$UID_END" suffix
Date: Wed, 10 Jun 2020 07:04:40 +0000	[thread overview]
Message-ID: <20200610070519.18252-44-e@yhbt.net> (raw)
In-Reply-To: <20200610070519.18252-1-e@yhbt.net>

Finish up the IMAP-only portion of iterative config reloading,
which allows us to create all sub-ranges of an inbox up front.
The InboxIdler still uses ->each_inbox which will struggle with
100K inboxes.

Having messages in the top-level newsgroup name of an inbox will
still waste bandwidth for clients which want to do full syncs
once there's a rollover to a new 50K range.  So instead, make
every inbox accessible exclusively via 50K slices in the form of
"$NEWSGROUP.$UID_MIN-$UID_END".

This introduces the DummyInbox, which makes $NEWSGROUP
and every parent component a selectable, empty inbox.
This aids navigation with mutt and possibly other MUAs.

Finally, the xt/perf-imap-list maintainer test is broken, now,
so remove it.  The grep perlfunc is already proven effective,
and we'll have separate tests for mocking out ~100k inboxes.
---
 MANIFEST                      |  2 +-
 lib/PublicInbox/Config.pm     |  7 ++-
 lib/PublicInbox/DummyInbox.pm | 21 +++++++++
 lib/PublicInbox/IMAP.pm       | 64 +++++++++----------------
 lib/PublicInbox/IMAPD.pm      | 88 ++++++++++++++++-------------------
 t/imap.t                      | 33 +++++++++++--
 t/imapd.t                     | 22 +++++----
 xt/imapd-mbsync-oimap.t       |  5 +-
 xt/imapd-validate.t           |  9 ++--
 xt/perf-imap-list.t           | 38 ---------------
 10 files changed, 139 insertions(+), 150 deletions(-)
 create mode 100644 lib/PublicInbox/DummyInbox.pm
 delete mode 100644 xt/perf-imap-list.t

diff --git a/MANIFEST b/MANIFEST
index 957228250a7..a133b71f881 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -105,6 +105,7 @@ lib/PublicInbox/DS.pm
 lib/PublicInbox/DSKQXS.pm
 lib/PublicInbox/DSPoll.pm
 lib/PublicInbox/Daemon.pm
+lib/PublicInbox/DummyInbox.pm
 lib/PublicInbox/Emergency.pm
 lib/PublicInbox/Eml.pm
 lib/PublicInbox/EmlContentFoo.pm
@@ -357,7 +358,6 @@ xt/imapd-validate.t
 xt/mem-msgview.t
 xt/msgtime_cmp.t
 xt/nntpd-validate.t
-xt/perf-imap-list.t
 xt/perf-msgview.t
 xt/perf-nntpd.t
 xt/perf-threading.t
diff --git a/lib/PublicInbox/Config.pm b/lib/PublicInbox/Config.pm
index c18c9c75b4f..19535beb973 100644
--- a/lib/PublicInbox/Config.pm
+++ b/lib/PublicInbox/Config.pm
@@ -105,13 +105,16 @@ sub iterate_start {
 	$self->{-iter} = [ \$i, $cb, $arg ];
 }
 
-# for PublicInbox::DS::next_tick
+# for PublicInbox::DS::next_tick, we only call this is if
+# PublicInbox::DS is already loaded
 sub event_step {
 	my ($self) = @_;
 	my ($i, $cb, $arg) = @{$self->{-iter}};
 	my $section = $self->{-section_order}->[$$i++];
 	delete($self->{-iter}) unless defined($section);
-	$cb->($self, $section, $arg);
+	eval { $cb->($self, $section, $arg) };
+	warn "E: $@ in ${self}::event_step" if $@;
+	PublicInbox::DS::requeue($self) if defined($section);
 }
 
 sub lookup_newsgroup {
diff --git a/lib/PublicInbox/DummyInbox.pm b/lib/PublicInbox/DummyInbox.pm
new file mode 100644
index 00000000000..e38f9e5a432
--- /dev/null
+++ b/lib/PublicInbox/DummyInbox.pm
@@ -0,0 +1,21 @@
+# Copyright (C) 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+#
+# An EXAMINE-able, PublicInbox::Inbox-like object for IMAP.  Some
+# IMAP clients don't like having unselectable parent mailboxes,
+# so we have a dummy
+package PublicInbox::DummyInbox;
+use strict;
+
+sub created_at { 0 } # Msgmap::created_at
+sub mm { shift }
+sub max { undef } # Msgmap::max
+sub msg_range { [] } # Msgmap::msg_range
+
+no warnings 'once';
+*query_xover = \&msg_range;
+*over = \&mm;
+*subscribe_unlock = *unsubscribe_unlock =
+	*get_art = *description = *base_url = \&max;
+
+1;
diff --git a/lib/PublicInbox/IMAP.pm b/lib/PublicInbox/IMAP.pm
index 959dcfbe328..6f64dff9958 100644
--- a/lib/PublicInbox/IMAP.pm
+++ b/lib/PublicInbox/IMAP.pm
@@ -192,26 +192,23 @@ sub cmd_done ($$) {
 	"$idle_tag OK Idle done\r\n";
 }
 
-sub ensure_old_ranges_exist ($$$) {
-	my ($self, $ibx, $uid_min) = @_;
-	my $groups = $self->{imapd}->{groups};
-	my $mailbox = $ibx->{newsgroup};
+sub ensure_ranges_exist ($$$) {
+	my ($imapd, $ibx, $max) = @_;
+	my $mailboxes = $imapd->{mailboxes};
+	my $mb_top = $ibx->{newsgroup};
 	my @created;
-	$uid_min -= UID_BLOCK;
+	my $uid_min = UID_BLOCK * int($max/UID_BLOCK) + 1;
 	my $uid_end = $uid_min + UID_BLOCK - 1;
 	while ($uid_min > 0) {
-		my $sub_mailbox = "$mailbox.$uid_min-$uid_end";
-		last if exists $groups->{$sub_mailbox};
-		$groups->{$sub_mailbox} = $ibx;
+		my $sub_mailbox = "$mb_top.$uid_min-$uid_end";
+		last if exists $mailboxes->{$sub_mailbox};
+		$mailboxes->{$sub_mailbox} = $ibx;
 		$uid_end -= UID_BLOCK;
 		$uid_min -= UID_BLOCK;
 		push @created, $sub_mailbox;
 	}
 	return unless @created;
-	my $l = $self->{imapd}->{inboxlist};
-	grep {
-		/ \Q$mailbox\E\r\n\z/ and s/\(\\HasNoChildren/\(\\HasChildren/;
-	} @$l;
+	my $l = $imapd->{inboxlist} or return;
 	push @$l, map { qq[* LIST (\\HasNoChildren) "." $_\r\n] } @created;
 }
 
@@ -222,41 +219,23 @@ sub cmd_examine ($$$) {
 	if ($mailbox =~ /\A(.+)\.([0-9]+)-([0-9]+)\z/) {
 		# old mail: inbox.comp.foo.$uid_min-$uid_end
 		my ($mb_top, $uid_min, $uid_end) = ($1, $2 + 0, $3 + 0);
-		$ibx = $self->{imapd}->{groups}->{lc $mb_top};
-		if (!$ibx || ($uid_end % UID_BLOCK) != 0 ||
-				($uid_min + UID_BLOCK - 1) != $uid_end) {
-			return "$tag NO Mailbox doesn't exist: $mailbox\r\n";
-		}
-		$mm = $ibx->mm;
-		$max = $mm->max // 0;
 
-		# don't let users create inboxes w/ not-yet-possible range:
-		$uid_min > $max and
+		$ibx = $self->{imapd}->{mailboxes}->{lc $mailbox} or
 			return "$tag NO Mailbox doesn't exist: $mailbox\r\n";
 
-		$max = $uid_min + UID_BLOCK + 1;
+		$mm = $ibx->mm;
+		$max = $mm->max // 0;
 		$self->{uid_min} = $uid_min;
-		ensure_old_ranges_exist($self, $ibx, $uid_min);
-	} else { # current mailbox (most recent UID_BLOCK messages)
-		$ibx = $self->{imapd}->{groups}->{lc $mailbox} or
+		ensure_ranges_exist($self->{imapd}, $ibx, $max);
+		$max = $uid_end if $max > $uid_end;
+	} else { # check for dummy inboxes
+		$ibx = $self->{imapd}->{mailboxes}->{lc $mailbox} or
 			return "$tag NO Mailbox doesn't exist: $mailbox\r\n";
-
+		delete $self->{uid_min};
+		$max = 0;
 		$mm = $ibx->mm;
-		$max = $mm->max // 0;
-
-		my $uid_min = UID_BLOCK * int($max/UID_BLOCK) + 1;
-		if ($uid_min == 1) { # normal inbox with <UID_BLOCK messages
-			delete $self->{uid_min}; # implicit cmd_close
-		} else { # we have a giant inbox:
-			$self->{uid_min} = $uid_min;
-			ensure_old_ranges_exist($self, $ibx, $uid_min);
-		}
 	}
 
-	# RFC 3501 2.3.1.1 -  "A good UIDVALIDITY value to use in
-	# this case is a 32-bit representation of the creation
-	# date/time of the mailbox"
-	my $uidvalidity = $mm->created_at or return "$tag BAD UIDVALIDITY\r\n";
 	my $uidnext = $max + 1;
 
 	# XXX: do we need this? RFC 5162/7162
@@ -269,7 +248,7 @@ sub cmd_examine ($$$) {
 * OK [PERMANENTFLAGS ()] Read-only mailbox\r
 * OK [UNSEEN $max]\r
 * OK [UIDNEXT $uidnext]\r
-* OK [UIDVALIDITY $uidvalidity]\r
+* OK [UIDVALIDITY $ibx->{uidvalidity}]\r
 $tag OK [READ-ONLY] EXAMINE/SELECT done\r
 EOF
 }
@@ -561,7 +540,7 @@ sub uid_fetch_m { # long_response
 
 sub cmd_status ($$$;@) {
 	my ($self, $tag, $mailbox, @items) = @_;
-	my $ibx = $self->{imapd}->{groups}->{$mailbox} or
+	my $ibx = $self->{imapd}->{mailboxes}->{$mailbox} or
 		return "$tag NO Mailbox doesn't exist: $mailbox\r\n";
 	return "$tag BAD no items\r\n" if !scalar(@items);
 	($items[0] !~ s/\A\(//s || $items[-1] !~ s/\)\z//s) and
@@ -577,8 +556,7 @@ sub cmd_status ($$$;@) {
 		} elsif ($it eq 'UIDNEXT') {
 			push(@it, ($max //= $mm->max // 0) + 1);
 		} elsif ($it eq 'UIDVALIDITY') {
-			push(@it, $mm->created_at //
-				return("$tag BAD UIDVALIDITY\r\n"));
+			push(@it, $ibx->{uidvalidity});
 		} else {
 			return "$tag BAD invalid item\r\n";
 		}
diff --git a/lib/PublicInbox/IMAPD.pm b/lib/PublicInbox/IMAPD.pm
index a10fabffb1f..b647c940505 100644
--- a/lib/PublicInbox/IMAPD.pm
+++ b/lib/PublicInbox/IMAPD.pm
@@ -5,82 +5,75 @@
 # see script/public-inbox-imapd for how it is used
 package PublicInbox::IMAPD;
 use strict;
-use parent qw(PublicInbox::NNTPD);
+use PublicInbox::Config;
 use PublicInbox::InboxIdle;
 use PublicInbox::IMAP;
-# *UID_BLOCK = \&PublicInbox::IMAP::UID_BLOCK;
+use PublicInbox::DummyInbox;
+my $dummy = bless { uidvalidity => 0 }, 'PublicInbox::DummyInbox';
 
 sub new {
 	my ($class) = @_;
 	bless {
-		groups => {},
+		mailboxes => {},
 		err => \*STDERR,
 		out => \*STDOUT,
-		grouplist => [],
 		# accept_tls => { SSL_server => 1, ..., SSL_reuse_ctx => ... }
 		# pi_config => PublicInbox::Config
 		# idler => PublicInbox::InboxIdle
 	}, $class;
 }
 
-sub refresh_inboxlist ($) {
-	my ($self) = @_;
-	my @names = map { $_->{newsgroup} } @{delete $self->{grouplist}};
-	my %ns; # "\Noselect \HasChildren"
-
-	if (my @uc = grep(/[A-Z]/, @names)) {
-		warn "Uppercase not allowed for IMAP newsgroup(s):\n",
-			map { "\t$_\n" } @uc;
-		my %uc = map { $_ => 1 } @uc;
-		@names = grep { !$uc{$_} } @names;
-	}
-	for (@names) {
-		my $up = $_;
-		while ($up =~ s/\.[^\.]+\z//) {
-			$ns{$up} = '\\Noselect \\HasChildren';
-		}
-	}
-	@names = map {;
-		my $at = delete($ns{$_}) ? '\\HasChildren' : '\\HasNoChildren';
-		qq[* LIST ($at) "." $_\r\n]
-	} @names;
-	push(@names, map { qq[* LIST ($ns{$_}) "." $_\r\n] } keys %ns);
-	@names = sort {
-		my ($xa) = ($a =~ / (\S+)\r\n/g);
-		my ($xb) = ($b =~ / (\S+)\r\n/g);
-		length($xa) <=> length($xb);
-	} @names;
-	$self->{inboxlist} = \@names;
-}
-
 sub imapd_refresh_ibx { # pi_config->each_inbox cb
 	my ($ibx, $imapd) = @_;
 	my $ngname = $ibx->{newsgroup} or return;
 	if (ref $ngname) {
 		warn 'multiple newsgroups not supported: '.
 			join(', ', @$ngname). "\n";
+		return;
 	} elsif ($ngname =~ m![^a-z0-9/_\.\-\~\@\+\=:]! ||
 		 $ngname =~ /\.[0-9]+-[0-9]+\z/) {
-		warn "mailbox name invalid: `$ngname'\n";
+		warn "mailbox name invalid: newsgroup=`$ngname'\n";
+		return;
 	}
-
+	$ibx->over or return;
+	$ibx->{over} = undef;
 	my $mm = $ibx->mm or return;
 	$ibx->{mm} = undef;
+
+	# RFC 3501 2.3.1.1 -  "A good UIDVALIDITY value to use in
+	# this case is a 32-bit representation of the creation
+	# date/time of the mailbox"
 	defined($ibx->{uidvalidity} = $mm->created_at) or return;
-	$imapd->{tmp_groups}->{$ngname} = $ibx;
+	PublicInbox::IMAP::ensure_ranges_exist($imapd, $ibx, $mm->max // 1);
 
 	# preload to avoid fragmentation:
 	$ibx->description;
 	$ibx->base_url;
-	# my $max = $mm->max // 0;
-	# my $uid_min = UID_BLOCK * int($max/UID_BLOCK) + 1;
+
+	# ensure dummies are selectable
+	my $dummies = $imapd->{dummies};
+	do {
+		$dummies->{$ngname} = $dummy;
+	} while ($ngname =~ s/\.[^\.]+\z//);
 }
 
 sub imapd_refresh_finalize {
 	my ($imapd, $pi_config) = @_;
-	$imapd->{groups} = delete $imapd->{tmp_groups};
-	$imapd->{grouplist} = [ values %{$imapd->{groups}} ];
-	refresh_inboxlist($imapd);
+	my $mailboxes;
+	if (my $next = delete $imapd->{imapd_next}) {
+		$imapd->{mailboxes} = delete $next->{mailboxes};
+		$mailboxes = delete $next->{dummies};
+	} else {
+		$mailboxes = delete $imapd->{dummies};
+	}
+	%$mailboxes = (%$mailboxes, %{$imapd->{mailboxes}});
+	$imapd->{mailboxes} = $mailboxes;
+	$imapd->{inboxlist} = [
+		map {
+			my $no = $mailboxes->{$_} == $dummy ? '' : 'No';
+			qq[* LIST (\\Has${no}Children) "." $_\r\n]
+		} sort { length($a) <=> length($b) } keys %$mailboxes
+	];
 	$imapd->{pi_config} = $pi_config;
 	if (my $idler = $imapd->{idler}) {
 		$idler->refresh($pi_config);
@@ -92,8 +85,8 @@ sub imapd_refresh_step { # pi_config->iterate_start cb
 	if (defined($section)) {
 		return if $section !~ m!\Apublicinbox\.([^/]+)\z!;
 		my $ibx = $pi_config->lookup_name($1) or return;
-		imapd_refresh_ibx($ibx, $imapd);
-	} else { # "EOF"
+		imapd_refresh_ibx($ibx, $imapd->{imapd_next});
+	} else { # undef == "EOF"
 		imapd_refresh_finalize($imapd, $pi_config);
 	}
 }
@@ -101,11 +94,12 @@ sub imapd_refresh_step { # pi_config->iterate_start cb
 sub refresh_groups {
 	my ($self, $sig) = @_;
 	my $pi_config = PublicInbox::Config->new;
-	$self->{tmp_groups} = {};
-	if (0 && $sig) { # SIGHUP
+	if ($sig) { # SIGHUP is handled through the event loop
+		$self->{imapd_next} = { dummies => {}, mailboxes => {} };
 		$pi_config->iterate_start(\&imapd_refresh_step, $self);
 		PublicInbox::DS::requeue($pi_config); # call event_step
-	} else { # initial start
+	} else { # initial start is synchronous
+		$self->{dummies} = {};
 		$pi_config->each_inbox(\&imapd_refresh_ibx, $self);
 		imapd_refresh_finalize($self, $pi_config);
 	}
diff --git a/t/imap.t b/t/imap.t
index aa262a19632..af59ef69386 100644
--- a/t/imap.t
+++ b/t/imap.t
@@ -6,16 +6,39 @@ use strict;
 use Test::More;
 use PublicInbox::IMAP;
 use PublicInbox::IMAPD;
+use PublicInbox::TestCommon;
+require_mods(qw(DBD::SQLite));
+require_git 2.6;
 
-{ # make sure we get '%' globbing right
+my ($tmpdir, $for_destroy) = tmpdir();
+my $cfgfile = "$tmpdir/config";
+{
+	open my $fh, '>', $cfgfile or BAIL_OUT $!;
+	print $fh <<EOF or BAIL_OUT $!;
+[publicinbox "a"]
+	inboxdir = $tmpdir/a
+	newsgroup = x.y.z
+[publicinbox "b"]
+	inboxdir = $tmpdir/b
+	newsgroup = x.z.y
+[publicinbox "c"]
+	inboxdir = $tmpdir/c
+	newsgroup = IGNORE.THIS
+EOF
+	close $fh or BAIL_OUT $!;
+	local $ENV{PI_CONFIG} = $cfgfile;
+	for my $x (qw(a b c)) {
+		ok(run_script(['-init', '-Lbasic', '-V2', $x, "$tmpdir/$x",
+				"https://example.com/$x", "$x\@example.com"]),
+			"init $x");
+	}
+	my $imapd = PublicInbox::IMAPD->new;
 	my @w;
 	local $SIG{__WARN__} = sub { push @w, @_ };
-	my @n = map { { newsgroup => $_ } } (qw(x.y.z x.z.y IGNORE.THIS));
-	my $self = { imapd => { grouplist => \@n } };
-	PublicInbox::IMAPD::refresh_inboxlist($self->{imapd});
+	$imapd->refresh_groups;
+	my $self = { imapd => $imapd };
 	is(scalar(@w), 1, 'got a warning for upper-case');
 	like($w[0], qr/IGNORE\.THIS/, 'warned about upper-case');
-
 	my $res = PublicInbox::IMAP::cmd_list($self, 'tag', 'x', '%');
 	is(scalar($$res =~ tr/\n/\n/), 2, 'only one result');
 	like($$res, qr/ x\r\ntag OK/, 'saw expected');
diff --git a/t/imapd.t b/t/imapd.t
index a5324f78da0..017bfa0b27b 100644
--- a/t/imapd.t
+++ b/t/imapd.t
@@ -16,6 +16,9 @@ if ($can_compress) { # hope this gets fixed upstream, soon
 	$imap_client = 'PublicInbox::IMAPClient';
 }
 
+require_ok 'PublicInbox::IMAP';
+my $first_range = '1-'.PublicInbox::IMAP::UID_BLOCK();
+
 my $level = '-Lbasic';
 SKIP: {
 	require_mods('Search::Xapian', 1);
@@ -87,11 +90,13 @@ ok(!$mic->examine('foo') && ($e = $@), 'EXAMINE non-existent');
 like($e, qr/\bNO\b/, 'got a NO on EXAMINE for non-existent');
 ok(!$mic->select('foo') && ($e = $@), 'EXAMINE non-existent');
 like($e, qr/\bNO\b/, 'got a NO on EXAMINE for non-existent');
-ok($mic->select('inbox.i1'), 'SELECT succeeds');
-ok($mic->examine('INBOX.i1'), 'EXAMINE succeeds');
-my @raw = $mic->status('inbox.i1', qw(Messages uidnext uidvalidity));
+my $mailbox1 = "inbox.i1.$first_range";
+ok($mic->select('inbox.i1'), 'SELECT on parent succeeds');
+ok($mic->select($mailbox1), 'SELECT succeeds');
+ok($mic->examine($mailbox1), 'EXAMINE succeeds');
+my @raw = $mic->status($mailbox1, qw(Messages uidnext uidvalidity));
 is(scalar(@raw), 2, 'got status response');
-like($raw[0], qr/\A\*\x20STATUS\x20inbox\.i1\x20
+like($raw[0], qr/\A\*\x20STATUS\x20inbox\.i1\.$first_range\x20
 	\(MESSAGES\x20\d+\x20UIDNEXT\x20\d+\x20UIDVALIDITY\x20\d+\)\r\n/sx);
 like($raw[1], qr/\A\S+ OK /, 'finished status response');
 
@@ -99,7 +104,7 @@ like($raw[1], qr/\A\S+ OK /, 'finished status response');
 like($raw[0], qr/^\* LIST \(.*?\) "\." inbox/,
 	'got an inbox');
 like($raw[-1], qr/^\S+ OK /, 'response ended with OK');
-is(scalar(@raw), scalar(@V) + 2, 'default LIST response');
+is(scalar(@raw), scalar(@V) + 4, 'default LIST response');
 @raw = $mic->list('', 'inbox.i1');
 is(scalar(@raw), 2, 'limited LIST response');
 like($raw[0], qr/^\* LIST \(.*?\) "\." inbox/,
@@ -199,7 +204,7 @@ SKIP: {
 	skip 'Mail::IMAPClient too old for ->compress', 2 if !$can_compress;
 	my $c = $imap_client->new(%mic_opt);
 	ok($c && $c->compress, 'compress enabled');
-	ok($c->examine('inbox.i1'), 'EXAMINE succeeds after COMPRESS');
+	ok($c->examine($mailbox1), 'EXAMINE succeeds after COMPRESS');
 	$ret = $c->search('uid 1:*') or BAIL_OUT "SEARCH FAIL $@";
 	is_deeply($ret, [ 1 ], 'search UID 1:* works after compression');
 }
@@ -216,11 +221,12 @@ $pi_config->each_inbox(sub {
 	my $ng = $ibx->{newsgroup};
 	my $mic = $imap_client->new(%mic_opt);
 	ok($mic && $mic->login && $mic->IsAuthenticated, "authed $name");
-	my $uidnext = $mic->uidnext($ng); # we'll fetch BODYSTRUCTURE on this
+	my $mb = "$ng.$first_range";
+	my $uidnext = $mic->uidnext($mb); # we'll fetch BODYSTRUCTURE on this
 	ok($uidnext, 'got uidnext for later fetch');
 	is_deeply([$mic->has_capability('IDLE')], ['IDLE'], "IDLE capa $name");
 	ok(!$mic->idle, "IDLE fails w/o SELECT/EXAMINE $name");
-	ok($mic->examine($ng), "EXAMINE $ng succeeds");
+	ok($mic->examine($mb), "EXAMINE $ng succeeds");
 	ok(my $idle_tag = $mic->idle, "IDLE succeeds on $ng");
 
 	open(my $fh, '<', 't/data/message_embed.eml') or BAIL_OUT("open: $!");
diff --git a/xt/imapd-mbsync-oimap.t b/xt/imapd-mbsync-oimap.t
index d2237a24bef..b2cb8737f82 100644
--- a/xt/imapd-mbsync-oimap.t
+++ b/xt/imapd-mbsync-oimap.t
@@ -13,12 +13,13 @@ my $inboxdir = $ENV{GIANT_INBOX_DIR};
 plan skip_all => "bad characters in $inboxdir" if $inboxdir =~ m![^\w\.\-/]!;
 my ($tmpdir, $for_destroy) = tmpdir();
 my $cfg = "$tmpdir/cfg";
-my $mailbox = 'inbox.test';
+my $newsgroup = 'inbox.test';
+my $mailbox = "$newsgroup.1-50000";
 {
 	open my $fh, '>', $cfg or BAIL_OUT "open: $!";
 	print $fh <<EOF or BAIL_OUT "print: $!";
 [publicinbox "test"]
-	newsgroup = $mailbox
+	newsgroup = $newsgroup
 	address = oimap\@example.com
 	inboxdir = $inboxdir
 EOF
diff --git a/xt/imapd-validate.t b/xt/imapd-validate.t
index f96ec8791b9..b7b66d0583f 100644
--- a/xt/imapd-validate.t
+++ b/xt/imapd-validate.t
@@ -30,7 +30,7 @@ if ($test_tls && !-r $key || !-r $cert) {
 }
 my ($tmpdir, $for_destroy) = tmpdir();
 my %OPT = qw(User u Password p);
-my (%STARTTLS_OPT, %IMAPS_OPT, $td, $mailbox, $make_local_server);
+my (%STARTTLS_OPT, %IMAPS_OPT, $td, $newsgroup, $mailbox, $make_local_server);
 if (($ENV{IMAP_TEST_URL} // '') =~ m!\Aimap://([^/]+)/(.+)\z!) {
 	($OPT{Server}, $mailbox) = ($1, $2);
 	$OPT{Server} =~ s/:([0-9]+)\z// and $OPT{Port} = $1 + 0;
@@ -39,6 +39,7 @@ if (($ENV{IMAP_TEST_URL} // '') =~ m!\Aimap://([^/]+)/(.+)\z!) {
 } else {
 	require_mods(qw(DBD::SQLite));
 	$make_local_server->();
+	$mailbox = "$newsgroup.1-50000";
 }
 
 my %opts = (imap => \%OPT, 'imap+compress' => { %OPT, Compress => 1 });
@@ -124,15 +125,15 @@ BEGIN {
 
 $make_local_server = sub {
 	require PublicInbox::Inbox;
-	$mailbox = 'inbox.test';
-	my $ibx = { inboxdir => $inbox_dir, newsgroup => $mailbox };
+	$newsgroup = 'inbox.test';
+	my $ibx = { inboxdir => $inbox_dir, newsgroup => $newsgroup };
 	$ibx = PublicInbox::Inbox->new($ibx);
 	my $pi_config = "$tmpdir/config";
 	{
 		open my $fh, '>', $pi_config or die "open($pi_config): $!";
 		print $fh <<"" or die "print $pi_config: $!";
 [publicinbox "test"]
-	newsgroup = $mailbox
+	newsgroup = $newsgroup
 	inboxdir = $inbox_dir
 	address = test\@example.com
 
diff --git a/xt/perf-imap-list.t b/xt/perf-imap-list.t
deleted file mode 100644
index 0f00f487991..00000000000
--- a/xt/perf-imap-list.t
+++ /dev/null
@@ -1,38 +0,0 @@
-#!perl -w
-# Copyright (C) 2020 all contributors <meta@public-inbox.org>
-# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
-use Test::More;
-use_ok 'PublicInbox::IMAP';
-use_ok 'PublicInbox::IMAPD';
-use PublicInbox::DS;
-use Benchmark qw(:all);
-my @n = map { { newsgroup => "inbox.comp.foo.bar.$_" } } (0..50000);
-push @n, map { { newsgroup => "xobni.womp.foo.bar.$_" } } (0..50000);
-my $self = { imapd => { grouplist => \@n } };
-my $n = scalar @n;
-my $t = timeit(1, sub {
-	PublicInbox::IMAPD::refresh_inboxlist($self->{imapd});
-});
-diag timestr($t). "refresh $n inboxes";
-
-open my $null, '>', '/dev/null' or BAIL_OUT "open: $!";
-my $ds = { sock => $null };
-my $nr = 200;
-diag "starting benchmark...";
-my $cmd_list = \&PublicInbox::IMAP::cmd_list;
-$t = timeit(1, sub {
-	for (0..$nr) {
-		my $res = $cmd_list->($self, 'tag', '', '*');
-		PublicInbox::DS::write($ds, $res);
-	}
-});
-diag timestr($t). "list all for $n inboxes $nr times";
-$nr = 20;
-$t = timeit(1, sub {
-	for (0..$nr) {
-		my $res = $cmd_list->($self, 'tag', 'inbox.', '%');
-		PublicInbox::DS::write($ds, $res);
-	}
-});
-diag timestr($t). "list partial for $n inboxes $nr times";
-done_testing;

  parent reply	other threads:[~2020-06-10  7:06 UTC|newest]

Thread overview: 84+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-06-10  7:03 [PATCH 00/82] public-inbox-imapd: read-only IMAP server Eric Wong
2020-06-10  7:03 ` [PATCH 01/82] doc: add some IMAP standards Eric Wong
2020-06-10  7:03 ` [PATCH 02/82] nntpd: restrict allowed newsgroup names Eric Wong
2020-06-10  7:04 ` [PATCH 03/82] preliminary imap server implementation Eric Wong
2020-06-10  7:04 ` [PATCH 04/82] inboxidle: new class to detect inbox changes Eric Wong
2020-06-10  7:04 ` [PATCH 05/82] imap: support IDLE Eric Wong
2020-06-10  7:04 ` [PATCH 06/82] msgmap: split ->max into its own method Eric Wong
2020-06-10  7:04 ` [PATCH 07/82] imap: delay InboxIdle start, support refresh Eric Wong
2020-06-10  7:04 ` [PATCH 08/82] imap: implement STATUS command Eric Wong
2020-06-10  7:04 ` [PATCH 09/82] imap: use Text::ParseWords::parse_line to handle quoted words Eric Wong
2020-06-10  7:04 ` [PATCH 10/82] imap: support LIST command Eric Wong
2020-06-10  7:04 ` [PATCH 11/82] t/imapd: support FakeInotify and KQNotify Eric Wong
2020-06-10  7:04 ` [PATCH 12/82] imap: support fetch for BODYSTRUCTURE and BODY Eric Wong
2020-06-10  7:04 ` [PATCH 13/82] eml: each_part: single part $idx is 1 Eric Wong
2020-06-10  7:04 ` [PATCH 14/82] imap: allow fetch of partial of BODY[...] and headers Eric Wong
2020-06-10  7:04 ` [PATCH 15/82] imap: always include `resp-text' in responses Eric Wong
2020-06-10  7:04 ` [PATCH 16/82] imap: split out unit tests and benchmarks Eric Wong
2020-06-10  7:04 ` [PATCH 17/82] imap: fix multi-message partial header fetches Eric Wong
2020-06-10  7:04 ` [PATCH 18/82] imap: simplify partial fetch structure Eric Wong
2020-06-10  7:04 ` [PATCH 19/82] imap: support sequence number FETCH Eric Wong
2020-06-10  7:04 ` [PATCH 20/82] imap: do not include ".PEEK" in responses Eric Wong
2020-06-10  7:04 ` [PATCH 21/82] imap: support the CLOSE command Eric Wong
2020-06-10  7:04 ` [PATCH 22/82] imap: speed up HEADER.FIELDS[.NOT] range fetches Eric Wong
2020-06-10  7:04 ` [PATCH 23/82] git: async: flatten the inflight array Eric Wong
2020-06-10  7:04 ` [PATCH 24/82] git: do our own read buffering for cat-file Eric Wong
2020-06-10  7:04 ` [PATCH 25/82] imap: use git-cat-file asynchronously Eric Wong
2020-06-10  7:04 ` [PATCH 26/82] git: idle rbuf for async Eric Wong
2020-06-10  7:04 ` [PATCH 27/82] imap: support LSUB command Eric Wong
2020-06-10  7:04 ` [PATCH 28/82] imap: FETCH: support comma-delimited ranges Eric Wong
2020-06-10  7:04 ` [PATCH 29/82] add imapd compression test Eric Wong
2020-06-10  7:04 ` [PATCH 30/82] testcommon: tcp_(server|connect): BAIL_OUT on failure Eric Wong
2020-06-10  7:04 ` [PATCH 31/82] *deflate: drop invalid comment about rbuf Eric Wong
2020-06-10  7:04 ` [PATCH 32/82] imap: fix pipelining with async git Eric Wong
2020-06-10  7:04 ` [PATCH 33/82] git: cat_async: provide requested OID + "missing" on missing blobs Eric Wong
2020-06-10  7:04 ` [PATCH 34/82] git: move async_cat reference to PublicInbox::Git Eric Wong
2020-06-10  7:04 ` [PATCH 35/82] git: async: automatic retry on alternates change Eric Wong
2020-06-10  7:04 ` [PATCH 36/82] imapclient: wrapper for Mail::IMAPClient Eric Wong
2020-06-10  7:04 ` [PATCH 37/82] xt: add imapd-validate and imapd-mbsync-oimap Eric Wong
2020-06-10  7:04 ` [PATCH 38/82] imap: support out-of-bounds ranges Eric Wong
2020-06-10  7:04 ` [PATCH 39/82] xt/perf-imap-list: time refresh_inboxlist Eric Wong
2020-06-10  7:04 ` [PATCH 40/82] imap: case-insensitive mailbox name comparisons Eric Wong
2020-06-10  7:04 ` [PATCH 41/82] imap: break giant inboxes into sub-inboxes of 50K messages Eric Wong
2020-06-10  7:04 ` [PATCH 42/82] imap: start doing iterative config reloading Eric Wong
2020-06-10  7:04 ` Eric Wong [this message]
2020-06-10  7:04 ` [PATCH 44/82] imapd: ensure LIST is sorted alphabetically, for now Eric Wong
2020-06-10  7:04 ` [PATCH 45/82] imap: omit $UID_END from mailbox name, use index Eric Wong
2020-06-10  7:04 ` [PATCH 46/82] t/config.t: always compare against git bool behavior Eric Wong
2020-06-10  7:04 ` [PATCH 47/82] xt/*: show some tunable parameters Eric Wong
2020-06-10  7:04 ` [PATCH 48/82] imap: STATUS and LIST are case-insensitive, too Eric Wong
2020-06-10  7:04 ` [PATCH 49/82] imap: EXAMINE/STATUS: return correct counts Eric Wong
2020-06-10  7:04 ` [PATCH 50/82] imap: avoid uninitialized warnings on incomplete commands Eric Wong
2020-06-10  7:04 ` [PATCH 51/82] imap: start parsing out queries for SQLite and Xapian Eric Wong
2020-06-10  7:04 ` [PATCH 52/82] imap: SEARCH: clamp results to the 50K UID range Eric Wong
2020-06-10  7:04 ` [PATCH 53/82] imap: allow UID range search on timestamps Eric Wong
2020-06-10  7:04 ` [PATCH 54/82] over: get_art: use dbh->prepare_cached Eric Wong
2020-06-10  7:04 ` [PATCH 55/82] search: index byte size of a message for IMAP search Eric Wong
2020-06-10  7:04 ` [PATCH 56/82] search: index UID for IMAP search, too Eric Wong
2020-06-10  7:04 ` [PATCH 57/82] imap: remove dummies from sequence number FETCH Eric Wong
2020-06-10  7:04 ` [PATCH 58/82] imap: compile UID FETCH to opcodes Eric Wong
2020-06-10  7:04 ` [PATCH 59/82] imap: UID FETCH: optimize for smsg-only case Eric Wong
2020-06-10  7:04 ` [PATCH 60/82] imap: UID FETCH: optimize (UID FLAGS) harder Eric Wong
2020-06-10  7:04 ` [PATCH 61/82] imap: IDLE: avoid extraneous wakeups, keep-alive Eric Wong
2020-06-10  7:04 ` [PATCH 62/82] imap: 30 minute auto-logout timer Eric Wong
2020-06-10  7:05 ` [PATCH 63/82] imap: split ->logged_in attribute into a separate class Eric Wong
2020-06-10  7:05 ` [PATCH 64/82] searchidx: v1 (re)-index uses git asynchronously Eric Wong
2020-06-10  7:05 ` [PATCH 65/82] index: account for CRLF conversion when storing bytes Eric Wong
2020-06-10  7:05 ` [PATCH 66/82] imap: rely on smsg->{bytes} for RFC822.SIZE Eric Wong
2020-06-10  7:05 ` [PATCH 67/82] imap: UID FETCH requires at least one data item Eric Wong
2020-06-10  7:05 ` [PATCH 68/82] imap: LIST shows "INBOX" in all caps Eric Wong
2020-06-10  7:05 ` [PATCH 69/82] imap: support 8000 octet lines Eric Wong
2020-06-10  7:05 ` [PATCH 70/82] imap: reinstate some message sequence number support Eric Wong
2020-06-10  7:05 ` [PATCH 71/82] imap: cleanup ->{uid_base} usage Eric Wong
2020-06-10  7:05 ` [PATCH 72/82] imap: FETCH: more granular CRLF conversion Eric Wong
2020-06-10  7:05 ` [PATCH 73/82] imap: further speed up HEADER.FIELDS FETCH requests Eric Wong
2020-06-10  7:05 ` [PATCH 74/82] imap: FETCH: try to make fake MSNs sequentially Eric Wong
2020-06-10  7:05 ` [PATCH 75/82] imap: STATUS/EXAMINE: rely on SQLite overview Eric Wong
2020-06-10  7:05 ` [PATCH 76/82] imap: UID SEARCH: support multiple ranges Eric Wong
2020-06-10  7:05 ` [PATCH 77/82] imap: wire up Xapian, MSN SEARCH and multi sequence-sets Eric Wong
2020-06-10  7:05 ` [PATCH 78/82] imap: misc cleanups and notes Eric Wong
2020-06-10  7:05 ` [PATCH 79/82] imapd: don't bother sorting LIST output Eric Wong
2020-06-10  7:05 ` [PATCH 80/82] imap: remove non-UID SEARCH for now Eric Wong
2020-06-10  7:05 ` [PATCH 81/82] over: uid_range: remove LIMIT Eric Wong
2020-06-10  7:05 ` [PATCH 82/82] imap: FETCH: proper MSN => UID mapping for requests Eric Wong
2020-06-12 23:49 ` [PATCH 83/82] imap: introduce memory-efficient uo2m mapping Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200610070519.18252-44-e@yhbt.net \
    --to=e@yhbt.net \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/public-inbox.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).