From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 1/2] imap: introduce and use Git->async_prefetch
Date: Sun, 26 Jul 2020 06:57:30 +0000 [thread overview]
Message-ID: <20200726065731.8100-2-e@yhbt.net> (raw)
In-Reply-To: <20200726065731.8100-1-e@yhbt.net>
We can keep the git process more active by sending another
request to it while fetch_run_ops() is running. This
parallelization speeds up mutt's initial FETCH for headers by
around ~35%(!).
---
lib/PublicInbox/Git.pm | 16 ++++++++++++++++
lib/PublicInbox/IMAP.pm | 7 ++++++-
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm
index ffc464eb3..7b2ada243 100644
--- a/lib/PublicInbox/Git.pm
+++ b/lib/PublicInbox/Git.pm
@@ -410,6 +410,22 @@ sub cat_async ($$$;$) {
push(@$inflight, $oid, $cb, $arg);
}
+# this is safe to call inside $cb, but not guaranteed to enqueue
+# returns true if successful, undef if not.
+sub async_prefetch {
+ my ($self, $oid, $cb, $arg) = @_;
+ if (defined($self->{async_cat}) && (my $inflight = $self->{inflight})) {
+ # we could use MAX_INFLIGHT here w/o the halving,
+ # but lets not allow one client to monopolize a git process
+ if (scalar(@$inflight) < int(MAX_INFLIGHT/2)) {
+ print { $self->{out} } $oid, "\n" or
+ fail($self, "write error: $!");
+ return push(@$inflight, $oid, $cb, $arg);
+ }
+ }
+ undef;
+}
+
sub extract_cmt_time {
my ($bref, undef, undef, undef, $modified) = @_;
diff --git a/lib/PublicInbox/IMAP.pm b/lib/PublicInbox/IMAP.pm
index 8ab4b1e7d..3d66f9306 100644
--- a/lib/PublicInbox/IMAP.pm
+++ b/lib/PublicInbox/IMAP.pm
@@ -626,8 +626,13 @@ sub fetch_blob_cb { # called by git->cat_async via git_async_cat
} else {
$smsg->{blob} eq $oid or die "BUG: $smsg->{blob} != $oid";
}
+ my $pre;
+ if (!$self->{wbuf} && (my $nxt = $msgs->[0])) {
+ $pre = $self->{ibx}->git->async_prefetch($nxt->{blob},
+ \&fetch_blob_cb, $fetch_arg);
+ }
fetch_run_ops($self, $smsg, $bref, $ops, $partial);
- requeue_once($self);
+ $pre ? $self->zflush : requeue_once($self);
}
sub emit_rfc822 {
next prev parent reply other threads:[~2020-07-26 6:57 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-26 6:57 [PATCH 0/2] imapd: async prefetch + test fix Eric Wong
2020-07-26 6:57 ` Eric Wong [this message]
2020-07-26 6:57 ` [PATCH 2/2] xt/imapd-mbsync-oimapd: fix noop due to case sensitivity Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200726065731.8100-2-e@yhbt.net \
--to=e@yhbt.net \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).