user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
* [PATCH 0/2] imapd: async prefetch + test fix
@ 2020-07-26  6:57 Eric Wong
  2020-07-26  6:57 ` [PATCH 1/2] imap: introduce and use Git->async_prefetch Eric Wong
  2020-07-26  6:57 ` [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity Eric Wong
  0 siblings, 2 replies; 3+ messages in thread
From: Eric Wong @ 2020-07-26  6:57 UTC (permalink / raw)
  To: meta

prefetch has been on the table since the earliest days of
-imapd, but the interaction with periodic cleanup made things
unreliable.  Now it won't start a git process if it's in cleanup
so things should be all good.  Uncached FETCH with mutt is
noticeably faster.

Now, the HTTP bits will be a little trickier, I think...

Eric Wong (2):
  imap: introduce and use Git->async_prefetch
  xt/imapd-mbsync-oimapd: fix noop due to case sensitivity

 lib/PublicInbox/Git.pm  | 16 ++++++++++++++++
 lib/PublicInbox/IMAP.pm |  7 ++++++-
 xt/imapd-mbsync-oimap.t |  2 +-
 3 files changed, 23 insertions(+), 2 deletions(-)

^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH 1/2] imap: introduce and use Git->async_prefetch
  2020-07-26  6:57 [PATCH 0/2] imapd: async prefetch + test fix Eric Wong
@ 2020-07-26  6:57 ` Eric Wong
  2020-07-26  6:57 ` [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity Eric Wong
  1 sibling, 0 replies; 3+ messages in thread
From: Eric Wong @ 2020-07-26  6:57 UTC (permalink / raw)
  To: meta

We can keep the git process more active by sending another
request to it while fetch_run_ops() is running.  This
parallelization speeds up mutt's initial FETCH for headers by
around ~35%(!).
---
 lib/PublicInbox/Git.pm  | 16 ++++++++++++++++
 lib/PublicInbox/IMAP.pm |  7 ++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index ffc464eb3..7b2ada243 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -410,6 +410,22 @@ sub cat_async ($$$;$) {
 	push(@$inflight, $oid, $cb, $arg);
 }
 
+# this is safe to call inside $cb, but not guaranteed to enqueue
+# returns true if successful, undef if not.
+sub async_prefetch {
+	my ($self, $oid, $cb, $arg) = @_;
+	if (defined($self->{async_cat}) && (my $inflight = $self->{inflight})) {
+		# we could use MAX_INFLIGHT here w/o the halving,
+		# but lets not allow one client to monopolize a git process
+		if (scalar(@$inflight) < int(MAX_INFLIGHT/2)) {
+			print { $self->{out} } $oid, "\n" or
+						fail($self, "write error: $!");
+			return push(@$inflight, $oid, $cb, $arg);
+		}
+	}
+	undef;
+}
+
 sub extract_cmt_time {
 	my ($bref, undef, undef, undef, $modified) = @_;
 
diff --git a/lib/PublicInbox/IMAP.pm b/lib/PublicInbox/IMAP.pm
index 8ab4b1e7d..3d66f9306 100644
--- a/lib/PublicInbox/IMAP.pm
+++ b/lib/PublicInbox/IMAP.pm
@@ -626,8 +626,13 @@ sub fetch_blob_cb { # called by git->cat_async via git_async_cat
 	} else {
 		$smsg->{blob} eq $oid or die "BUG: $smsg->{blob} != $oid";
 	}
+	my $pre;
+	if (!$self->{wbuf} && (my $nxt = $msgs->[0])) {
+		$pre = $self->{ibx}->git->async_prefetch($nxt->{blob},
+						\&fetch_blob_cb, $fetch_arg);
+	}
 	fetch_run_ops($self, $smsg, $bref, $ops, $partial);
-	requeue_once($self);
+	$pre ? $self->zflush : requeue_once($self);
 }
 
 sub emit_rfc822 {

^ permalink raw reply	[flat|nested] 3+ messages in thread

* [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity
  2020-07-26  6:57 [PATCH 0/2] imapd: async prefetch + test fix Eric Wong
  2020-07-26  6:57 ` [PATCH 1/2] imap: introduce and use Git->async_prefetch Eric Wong
@ 2020-07-26  6:57 ` Eric Wong
  1 sibling, 0 replies; 3+ messages in thread
From: Eric Wong @ 2020-07-26  6:57 UTC (permalink / raw)
  To: meta

mbsync was not retrieving anything since it was looking for
"inbox" when we need to return "INBOX" as a special case
for IMAP.

Fixes: 8af34015e9aa94e5 (imap: LIST shows "INBOX" in all caps)
---
 xt/imapd-mbsync-oimap.t | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xt/imapd-mbsync-oimap.t b/xt/imapd-mbsync-oimap.t
index edf111fde..f8641d06d 100644
--- a/xt/imapd-mbsync-oimap.t
+++ b/xt/imapd-mbsync-oimap.t
@@ -99,7 +99,7 @@ UseNamespace no
 # DisableExtension COMPRESS=DEFLATE
 
 Channel "test"
-Master ":remote:inbox"
+Master ":remote:INBOX"
 Slave ":local:test"
 Expunge None
 Sync PullNew

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2020-07-26  6:57 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-26  6:57 [PATCH 0/2] imapd: async prefetch + test fix Eric Wong
2020-07-26  6:57 ` [PATCH 1/2] imap: introduce and use Git->async_prefetch Eric Wong
2020-07-26  6:57 ` [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity Eric Wong

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ http://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git