From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 8D06F1F5B1 for ; Sun, 26 Jul 2020 06:57:31 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 1/2] imap: introduce and use Git->async_prefetch Date: Sun, 26 Jul 2020 06:57:30 +0000 Message-Id: <20200726065731.8100-2-e@yhbt.net> In-Reply-To: <20200726065731.8100-1-e@yhbt.net> References: <20200726065731.8100-1-e@yhbt.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: We can keep the git process more active by sending another request to it while fetch_run_ops() is running. This parallelization speeds up mutt's initial FETCH for headers by around ~35%(!). --- lib/PublicInbox/Git.pm | 16 ++++++++++++++++ lib/PublicInbox/IMAP.pm | 7 ++++++- 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/lib/PublicInbox/Git.pm b/lib/PublicInbox/Git.pm index ffc464eb3..7b2ada243 100644 --- a/lib/PublicInbox/Git.pm +++ b/lib/PublicInbox/Git.pm @@ -410,6 +410,22 @@ sub cat_async ($$$;$) { push(@$inflight, $oid, $cb, $arg); } +# this is safe to call inside $cb, but not guaranteed to enqueue +# returns true if successful, undef if not. +sub async_prefetch { + my ($self, $oid, $cb, $arg) = @_; + if (defined($self->{async_cat}) && (my $inflight = $self->{inflight})) { + # we could use MAX_INFLIGHT here w/o the halving, + # but lets not allow one client to monopolize a git process + if (scalar(@$inflight) < int(MAX_INFLIGHT/2)) { + print { $self->{out} } $oid, "\n" or + fail($self, "write error: $!"); + return push(@$inflight, $oid, $cb, $arg); + } + } + undef; +} + sub extract_cmt_time { my ($bref, undef, undef, undef, $modified) = @_; diff --git a/lib/PublicInbox/IMAP.pm b/lib/PublicInbox/IMAP.pm index 8ab4b1e7d..3d66f9306 100644 --- a/lib/PublicInbox/IMAP.pm +++ b/lib/PublicInbox/IMAP.pm @@ -626,8 +626,13 @@ sub fetch_blob_cb { # called by git->cat_async via git_async_cat } else { $smsg->{blob} eq $oid or die "BUG: $smsg->{blob} != $oid"; } + my $pre; + if (!$self->{wbuf} && (my $nxt = $msgs->[0])) { + $pre = $self->{ibx}->git->async_prefetch($nxt->{blob}, + \&fetch_blob_cb, $fetch_arg); + } fetch_run_ops($self, $smsg, $bref, $ops, $partial); - requeue_once($self); + $pre ? $self->zflush : requeue_once($self); } sub emit_rfc822 {