From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.8 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by dcvr.yhbt.net (Postfix) with ESMTP id 35ED81F54E for ; Tue, 6 Sep 2022 22:59:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229795AbiIFW6Y (ORCPT ); Tue, 6 Sep 2022 18:58:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229593AbiIFW6W (ORCPT ); Tue, 6 Sep 2022 18:58:22 -0400 Received: from cloud.peff.net (cloud.peff.net [104.130.231.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34A7F4D267 for ; Tue, 6 Sep 2022 15:58:21 -0700 (PDT) Received: (qmail 19784 invoked by uid 109); 6 Sep 2022 22:58:20 -0000 Received: from Unknown (HELO peff.net) (10.0.1.2) by cloud.peff.net (qpsmtpd/0.94) with ESMTP; Tue, 06 Sep 2022 22:58:20 +0000 Authentication-Results: cloud.peff.net; auth=none Received: (qmail 6612 invoked by uid 111); 6 Sep 2022 22:58:21 -0000 Received: from coredump.intra.peff.net (HELO sigill.intra.peff.net) (10.0.0.2) by peff.net (qpsmtpd/0.94) with (TLS_AES_256_GCM_SHA384 encrypted) ESMTPS; Tue, 06 Sep 2022 18:58:21 -0400 Authentication-Results: peff.net; auth=none Date: Tue, 6 Sep 2022 18:58:19 -0400 From: Jeff King To: =?utf-8?B?56iL5rSL?= Cc: Derrick Stolee , "git@vger.kernel.org" , =?utf-8?B?5L2V5rWp?= , Xin7 Ma =?utf-8?B?6ams6ZGr?= , =?utf-8?B?55+z5aWJ5YW1?= , =?utf-8?B?5Yeh5Yab6L6J?= , =?utf-8?B?546L5rGJ5Z+6?= Subject: [PATCH 0/3] speeding up on-demand fetch for blobs in partial clone Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org On Tue, Sep 06, 2022 at 02:38:41PM -0400, Jeff King wrote: > On Mon, Sep 05, 2022 at 11:17:21AM +0000, 程洋 wrote: > > > Sorry, I told you the wrong branch. It should be "android-t-preview-1" > > git clone --filter=blob:none --no-local -b android-t-preview-1 grade-plugin > > > > Can you try this one? > > Yes, I see more slow-down there. There are many more blobs there, but I > don't think it's really the number of them, but their sizes. > > The problem is that both upload-pack and pack-objects are keen to call > parse_object() on their inputs. For commits, etc, that is usually > sensible; we have to parse the object to see what it points to. But for > blobs, the only thing we do is inflate a ton of bytes in order to check > the sha1. That's not really productive here; if there is a bit > corruption, the client will notice it on the receiving side. So here's a cleaned-up series which makes this a lot faster. The special sauce is in patch 2, along with timings. The first one is just preparing, and the final one is a small cleanup it enables. I based these directly on the current tip of master. There will be a small conflict in the test script when merging with the "rev-list --verify-objects" series I just sent in: https://lore.kernel.org/git/Yxe0k++LA%2FUfFLF%2F@coredump.intra.peff.net/ The resolution is to just keep the tests added by both sides. [1/3]: parse_object(): allow skipping hash check [2/3]: upload-pack: skip parse-object re-hashing of "want" objects [3/3]: parse_object(): check commit-graph when skip_hash set object.c | 21 ++++++++++++++++++--- object.h | 6 ++++++ revision.c | 14 +++----------- t/t1450-fsck.sh | 20 ++++++++++++++++++++ upload-pack.c | 8 ++------ 5 files changed, 49 insertions(+), 20 deletions(-) -Peff