From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by dcvr.yhbt.net (Postfix) with ESMTP id 10B6E1F66E for ; Mon, 24 Aug 2020 21:19:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726995AbgHXVTl (ORCPT ); Mon, 24 Aug 2020 17:19:41 -0400 Received: from pb-smtp21.pobox.com ([173.228.157.53]:63102 "EHLO pb-smtp21.pobox.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726090AbgHXVTl (ORCPT ); Mon, 24 Aug 2020 17:19:41 -0400 Received: from pb-smtp21.pobox.com (unknown [127.0.0.1]) by pb-smtp21.pobox.com (Postfix) with ESMTP id C0746DAE19; Mon, 24 Aug 2020 17:19:39 -0400 (EDT) (envelope-from junio@pobox.com) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=jB8oPgUksGDtA1Q8PXT7WmmyxEk=; b=O2fjqA iBYxM1JENtWi6auudKJPdeQO92S60uQ/hnOYdb7oTlpJRdoaR1kHZgUk4fa868Ab JDLGlJn6k5hH1OY9GFcOd2u9aap+AAImDDetDTBOhKiVji+ao+255jYfrwwscCN5 4FTlb+5GXmjcxn+YiGSRB0NblPwtpAEmUbAfE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=R619iYy01MxUdGpHG/zzq/1JrALO39rR kMZ5bG4CX8SA4TYYDO42Xgnwys/xd28Pb00b8DjQ5/SvfK/l1vAPG6UYkMjOgppI QJfDzxEDPTlrcizvCMQveLcUsxUHhV4JT0s4svu2s8qRTsUoHiIBOd90dlKAn4Xo /DB6A3nrKo8= Received: from pb-smtp21.sea.icgroup.com (unknown [127.0.0.1]) by pb-smtp21.pobox.com (Postfix) with ESMTP id B96CCDAE18; Mon, 24 Aug 2020 17:19:39 -0400 (EDT) (envelope-from junio@pobox.com) Received: from pobox.com (unknown [34.75.7.245]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pb-smtp21.pobox.com (Postfix) with ESMTPSA id 0DD9EDAE16; Mon, 24 Aug 2020 17:19:36 -0400 (EDT) (envelope-from junio@pobox.com) From: Junio C Hamano To: Jonathan Tan Cc: git@vger.kernel.org, peff@peff.net, steadmon@google.com Subject: Re: [PATCH 7/7] index-pack: make quantum of work smaller References: <3e69f41f19535fa2c04fae6adc78bcc4f052d523.1598296530.git.jonathantanmy@google.com> Date: Mon, 24 Aug 2020 14:19:35 -0700 In-Reply-To: <3e69f41f19535fa2c04fae6adc78bcc4f052d523.1598296530.git.jonathantanmy@google.com> (Jonathan Tan's message of "Mon, 24 Aug 2020 12:16:38 -0700") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: 837823D2-E64F-11EA-B8F3-843F439F7C89-77302942!pb-smtp21.pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Jonathan Tan writes: > Currently, when index-pack resolves deltas, it does not split up delta > trees into threads: each delta base root (an object that is not a > REF_DELTA or OFS_DELTA) can go into its own thread, but all deltas on > that root (direct or indirect) are processed in the same thread. > > This is a problem when a repository contains a large text file (thus, > delta-able) that is modified many times - delta resolution time during > fetching is dominated by processing the deltas corresponding to that > text file. We favor wide and shallow delta trees for both reduced storage footprint and lower runtime overhead, so optimizing the index-pack for that use case makes a lot of sense.