* [PATCH 0/2] imapd: async prefetch + test fix @ 2020-07-26 6:57 Eric Wong 2020-07-26 6:57 ` [PATCH 1/2] imap: introduce and use Git->async_prefetch Eric Wong 2020-07-26 6:57 ` [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity Eric Wong 0 siblings, 2 replies; 3+ messages in thread From: Eric Wong @ 2020-07-26 6:57 UTC (permalink / raw) To: meta prefetch has been on the table since the earliest days of -imapd, but the interaction with periodic cleanup made things unreliable. Now it won't start a git process if it's in cleanup so things should be all good. Uncached FETCH with mutt is noticeably faster. Now, the HTTP bits will be a little trickier, I think... Eric Wong (2): imap: introduce and use Git->async_prefetch xt/imapd-mbsync-oimapd: fix noop due to case sensitivity lib/PublicInbox/Git.pm | 16 ++++++++++++++++ lib/PublicInbox/IMAP.pm | 7 ++++++- xt/imapd-mbsync-oimap.t | 2 +- 3 files changed, 23 insertions(+), 2 deletions(-) ^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH 1/2] imap: introduce and use Git->async_prefetch 2020-07-26 6:57 [PATCH 0/2] imapd: async prefetch + test fix Eric Wong @ 2020-07-26 6:57 ` Eric Wong 2020-07-26 6:57 ` [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity Eric Wong 1 sibling, 0 replies; 3+ messages in thread From: Eric Wong @ 2020-07-26 6:57 UTC (permalink / raw) To: meta We can keep the git process more active by sending another request to it while fetch_run_ops() is running. This parallelization speeds up mutt's initial FETCH for headers by around ~35%(!). --- lib/PublicInbox/Git.pm | 16 ++++++++++++++++ lib/PublicInbox/IMAP.pm | 7 ++++++- 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm index ffc464eb3..7b2ada243 100644 --- a/lib/PublicInbox/Git.pm +++ b/lib/PublicInbox/Git.pm @@ -410,6 +410,22 @@ sub cat_async ($$$;$) { push(@$inflight, $oid, $cb, $arg); } +# this is safe to call inside $cb, but not guaranteed to enqueue +# returns true if successful, undef if not. +sub async_prefetch { + my ($self, $oid, $cb, $arg) = @_; + if (defined($self->{async_cat}) && (my $inflight = $self->{inflight})) { + # we could use MAX_INFLIGHT here w/o the halving, + # but lets not allow one client to monopolize a git process + if (scalar(@$inflight) < int(MAX_INFLIGHT/2)) { + print { $self->{out} } $oid, "\n" or + fail($self, "write error: $!"); + return push(@$inflight, $oid, $cb, $arg); + } + } + undef; +} + sub extract_cmt_time { my ($bref, undef, undef, undef, $modified) = @_; diff --git a/lib/PublicInbox/IMAP.pm b/lib/PublicInbox/IMAP.pm index 8ab4b1e7d..3d66f9306 100644 --- a/lib/PublicInbox/IMAP.pm +++ b/lib/PublicInbox/IMAP.pm @@ -626,8 +626,13 @@ sub fetch_blob_cb { # called by git->cat_async via git_async_cat } else { $smsg->{blob} eq $oid or die "BUG: $smsg->{blob} != $oid"; } + my $pre; + if (!$self->{wbuf} && (my $nxt = $msgs->[0])) { + $pre = $self->{ibx}->git->async_prefetch($nxt->{blob}, + \&fetch_blob_cb, $fetch_arg); + } fetch_run_ops($self, $smsg, $bref, $ops, $partial); - requeue_once($self); + $pre ? $self->zflush : requeue_once($self); } sub emit_rfc822 { ^ permalink raw reply related [flat|nested] 3+ messages in thread
* [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity 2020-07-26 6:57 [PATCH 0/2] imapd: async prefetch + test fix Eric Wong 2020-07-26 6:57 ` [PATCH 1/2] imap: introduce and use Git->async_prefetch Eric Wong @ 2020-07-26 6:57 ` Eric Wong 1 sibling, 0 replies; 3+ messages in thread From: Eric Wong @ 2020-07-26 6:57 UTC (permalink / raw) To: meta mbsync was not retrieving anything since it was looking for "inbox" when we need to return "INBOX" as a special case for IMAP. Fixes: 8af34015e9aa94e5 (imap: LIST shows "INBOX" in all caps) --- xt/imapd-mbsync-oimap.t | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xt/imapd-mbsync-oimap.t b/xt/imapd-mbsync-oimap.t index edf111fde..f8641d06d 100644 --- a/xt/imapd-mbsync-oimap.t +++ b/xt/imapd-mbsync-oimap.t @@ -99,7 +99,7 @@ UseNamespace no # DisableExtension COMPRESS=DEFLATE Channel "test" -Master ":remote:inbox" +Master ":remote:INBOX" Slave ":local:test" Expunge None Sync PullNew ^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2020-07-26 6:57 UTC | newest] Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-07-26 6:57 [PATCH 0/2] imapd: async prefetch + test fix Eric Wong 2020-07-26 6:57 ` [PATCH 1/2] imap: introduce and use Git->async_prefetch Eric Wong 2020-07-26 6:57 ` [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity Eric Wong
Code repositories for project(s) associated with this public inbox https://80x24.org/public-inbox.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).