From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-3.8 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by dcvr.yhbt.net (Postfix) with ESMTP id 31C0A1F506 for ; Thu, 22 Sep 2022 20:13:49 +0000 (UTC) Authentication-Results: dcvr.yhbt.net; dkim=pass (1024-bit key; unprotected) header.d=pobox.com header.i=@pobox.com header.b="RHT3iXq1"; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231143AbiIVUNm (ORCPT ); Thu, 22 Sep 2022 16:13:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232766AbiIVUNl (ORCPT ); Thu, 22 Sep 2022 16:13:41 -0400 Received: from pb-smtp2.pobox.com (pb-smtp2.pobox.com [64.147.108.71]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BEBFF10E5DB for ; Thu, 22 Sep 2022 13:13:38 -0700 (PDT) Received: from pb-smtp2.pobox.com (unknown [127.0.0.1]) by pb-smtp2.pobox.com (Postfix) with ESMTP id 2AD1A133A07; Thu, 22 Sep 2022 16:13:36 -0400 (EDT) (envelope-from junio@pobox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type:content-transfer-encoding; s=sasl; bh=BIRLx6w22jEw /EmBBHLQ/kvjZUIVz02h/jiG2VyWa2o=; b=RHT3iXq1mE6ga3+vffVcv9k5/87n cLiI9k7aeFpEKMtL25rKFQzZqgLp6Ch7GMFskBLWA+tTbGP6A8INEX/le/UdKjhr glpvI2qHXd8w6b2cyW9XwLQWsB9wKkFZHoPMdDqzl9P67wh0e43DzfpdHjOIVyEJ pp22ZYe+RNwqHvw= Received: from pb-smtp2.nyi.icgroup.com (unknown [127.0.0.1]) by pb-smtp2.pobox.com (Postfix) with ESMTP id 1E987133A06; Thu, 22 Sep 2022 16:13:36 -0400 (EDT) (envelope-from junio@pobox.com) Received: from pobox.com (unknown [34.83.5.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pb-smtp2.pobox.com (Postfix) with ESMTPSA id 85A9C133A05; Thu, 22 Sep 2022 16:13:35 -0400 (EDT) (envelope-from junio@pobox.com) From: Junio C Hamano To: "Scheffenegger, Richard" Cc: "git@vger.kernel.org" Subject: Re: git --archive References: Date: Thu, 22 Sep 2022 13:13:34 -0700 In-Reply-To: (Richard Scheffenegger's message of "Thu, 22 Sep 2022 08:57:10 +0000") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 X-Pobox-Relay-ID: 09C23090-3AB3-11ED-8840-307A8E0A682E-77302942!pb-smtp2.pobox.com Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org "Scheffenegger, Richard" writes: > Unless I=E2=80=99m mistaken, the procedure to create a zip archive read= s like a recursive collection of all relevant objects, and then writing t= hem out sequentially, in a single thread.=20 > > Is this assessment correct? > > I was wondering if a highly concurrent fetching phase could be optional= ly added=E2=80=A6 The details matter here, I think. Enumerating and slurping down the contents to be archived out of the repository/object store to the core can indeed be made parallel, but the end result product being a zip archive or a tarball, which is fairly a serialized output format, there is only so much you can hold in core, and it is not clear what your plan is to do this without filling all the memory.