From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-3.3 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.6 Received: from hurricane.the-brannons.com (hurricane.the-brannons.com [IPv6:2602:ff06:725:1:20::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 0185E1F47D for ; Thu, 5 Jan 2023 01:08:52 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=the-brannons.com header.i=@the-brannons.com header.a=rsa-sha256 header.s=hurricane header.b=aBFYH1aY; dkim-atps=neutral DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=hurricane; bh=aIpPP2wqyF EgX/Zsp4F6MWXJjwkTdTV9NVL3NjZtTzg=; h=in-reply-to:date:references: subject:cc:to:from; d=the-brannons.com; b=aBFYH1aYdMXjxaP8sSimFqUr4w4c S2KianN9K8DCuQXCl3335kZ3nycQTedmFeQcENKwils2N2IyrLXSx3RSBirQFJCAOrFQzp atKoiBlAUYWTWXp+6jOeuUiV4zXKePzekNosFcsbamY2qmFemEGLc9hN6O5RmOF3mQR+OA dJY= Received: by hurricane.the-brannons.com (OpenSMTPD) with ESMTPSA id d8b617cb (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) auth=yes user=chris; Wed, 4 Jan 2023 17:08:49 -0800 (PST) From: Chris Brannon To: Eric Wong Cc: meta@public-inbox.org Subject: Re: [PATCH] git: fix asynchronous batching for deep pipelines References: <875ye5m1wo.fsf@the-brannons.com> <20221221122102.M600156@dcvr> <871qosna30.fsf@the-brannons.com> <20221221194855.GA5179@dcvr> <87mt7glc23.fsf@the-brannons.com> <20221221211114.M412849@dcvr> <87edssl7u0.fsf@the-brannons.com> <20221221232210.M865188@dcvr> <877cykl387.fsf@the-brannons.com> <20230104034934.M147841@dcvr> Autocrypt: addr=chris@the-brannons.com; prefer-encrypt=mutual; keydata= mDMEX37pCBYJKwYBBAHaRw8BAQdASkFrVm/MHGNpj7Oa+nq4b7eBg5LX5qnvS6ojH04ZE9+0LENo cmlzdG9waGVyIEJyYW5ub24gPGNocmlzQHRoZS1icmFubm9ucy5jb20+iJAEExYIADgWIQTFfzWT HzF7+QWPg09oJ1i8YNj4egUCX37pCAIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRBoJ1i8 YNj4esVsAQCJXDxhj5gZjcrNMmSH6Z+X5RDdbvsTRvS5wwQXM3g4kAEAjOrdx/fvZDiKL8dylDhU YUVVSkwWeOD2UtmYjm/QtwG4OARffukIEgorBgEEAZdVAQUBAQdAOMaQorDR8QUdM34/Q9mUyBvS 9WFK0Mb76dHyNkERx2kDAQgHiHgEGBYIACAWIQTFfzWTHzF7+QWPg09oJ1i8YNj4egUCX37pCAIb DAAKCRBoJ1i8YNj4eo/8AQC4AX6NMv65KQQU7s95T/1Y4E+Z7KTbvhKJeSjdDfj88AEA/qzPnDvl dLmqqNTqsfZCK81teOxSoNhRlhWcQ8QQ/wY= Date: Wed, 04 Jan 2023 17:08:48 -0800 In-Reply-To: <20230104034934.M147841@dcvr> (Eric Wong's message of "Wed, 4 Jan 2023 03:49:34 +0000") Message-ID: <87tu15dbz3.fsf@the-brannons.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain List-Id: Eric Wong writes: > > --------8<------- > Subject: [PATCH] git: fix asynchronous batching for deep pipelines > > ...By using non-blocking pipe writes. This avoids problems for > musl (and other libc) where getdelim(3) used by `git cat-file --batch*' > uses a smaller input buffer than glibc or FreeBSD libc. > > My key mistake was our check against MAX_INFLIGHT is only useful > for the initial batch of requests. It is not useful for > subsequent requests since git will drain the pipe at > unpredictable rates due to libc differences. > > To fix this problem, I initially tried to drain the read pipe > as long as readable data was pending. However, reading git > output without giving git more work would also limit parallelism > opportunities since we don't want git to sit idle, either. This > change ensures we keep both pipes reasonably full to reduce > stalls and maximize parallelism between git and public-inbox. > > While the limit set a few weeks ago in commit > 56e6e587745c (git: cap MAX_INFLIGHT value to POSIX minimum, 2022-12-21) > remains in place, any higher or lower limit will work. It may > be worth it to use an even lower limit to improve interactivity > w.r.t. Ctrl-C interrupts. > > I've tested the pre-56e6e587745c and even higher values on an > Alpine VM in the GCC Farm > > Reported-by: Chris Brannon > Tested-by: Chris Brannon > Link: https://public-inbox.org/meta/87edssl7u0.fsf@the-brannons.com/T/ *SNIP* Thanks! The conversion seems to have been successful, but it logged very many of the following: E: Resource temporarily unavailable at /usr/share/perl5/vendor_perl/PublicInbox/Git.pm line 320. I'm sorry, I didn't count how many. It seemed like multiple screenfuls at least. -- Chris