From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS31976 209.132.180.0/23 X-Spam-Status: No, score=-4.0 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by dcvr.yhbt.net (Postfix) with ESMTP id 1972F1F619 for ; Thu, 12 Mar 2020 20:43:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727079AbgCLUnG (ORCPT ); Thu, 12 Mar 2020 16:43:06 -0400 Received: from pb-smtp21.pobox.com ([173.228.157.53]:60186 "EHLO pb-smtp21.pobox.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725268AbgCLUnF (ORCPT ); Thu, 12 Mar 2020 16:43:05 -0400 Received: from pb-smtp21.pobox.com (unknown [127.0.0.1]) by pb-smtp21.pobox.com (Postfix) with ESMTP id 793BACE55C; Thu, 12 Mar 2020 16:43:03 -0400 (EDT) (envelope-from junio@pobox.com) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=GuDW6YeqqYLx0W7WpJT1xOlOYn8=; b=PMOFmS CNf55RLSK5ACg4U7jdRFzdkIYlN5W3n5iKluSCzFLY9frA52z+LW8fAXLBceDMXq pb6xwvO/542TA3+XpWKS4ipeWPbP6NruB06W/2CyqBQdfkINp5vAkaIOBxO1fpwD 1jVc/J5Nmp95ljo3Z5yGMo5RZMjfwXTom29as= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=huLq0Mm2h0N4RJr39cMHDTbFzO62uaZt u2N9pq4qciKeEOmNRsBeQ3qjno2VQ8+0Q4r9GM2dVYCPngN5eaPapB/BGJ+t1cec bLEPFwBX7AzVByCuve1O4j+lz40Ry/bfd7e1ZYzq7CpuH+dcRuX0/8hkpIVTFEf/ 38sRdsPtITY= Received: from pb-smtp21.sea.icgroup.com (unknown [127.0.0.1]) by pb-smtp21.pobox.com (Postfix) with ESMTP id 2F60ECE55A; Thu, 12 Mar 2020 16:43:03 -0400 (EDT) (envelope-from junio@pobox.com) Received: from pobox.com (unknown [34.74.119.39]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pb-smtp21.pobox.com (Postfix) with ESMTPSA id 6AB2CCE556; Thu, 12 Mar 2020 16:43:00 -0400 (EDT) (envelope-from junio@pobox.com) From: Junio C Hamano To: "Derrick Stolee via GitGitGadget" Cc: git@vger.kernel.org, jonathantanmy@google.com, me@ttaylorr.com, Derrick Stolee Subject: Re: [PATCH] connected.c: reprepare packs for corner cases References: Date: Thu, 12 Mar 2020 13:42:58 -0700 In-Reply-To: (Derrick Stolee via GitGitGadget's message of "Thu, 12 Mar 2020 15:36:43 +0000") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: 1006416A-64A2-11EA-8FDB-8D86F504CC47-77302942!pb-smtp21.pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org "Derrick Stolee via GitGitGadget" writes: > From: Derrick Stolee > > While updating the microsoft/git fork on top of v2.26.0-rc0 and > consuming that build into Scalar, I noticed a corner case bug around > partial clone. > > The "scalar clone" command can create a Git repository with the > proper config for using partial clone with the "blob:none" filter. > Instead of calling "git clone", it runs "git init" then sets a few > more config values before running "git fetch". > > In our builds on v2.26.0-rc0, we noticed that our "git fetch" > command was failing with > > error: https://github.com/microsoft/scalar did not send all necessary objects > > This does not happen if you copy the config file from a repository > created by "git clone --filter=blob:none ", but it does happen > when adding the config option "core.logAllRefUpdates = true". > > By debugging, I was able to see that the loop inside > check_connnected() that checks if all refs are contained in > promisor packs actually did not have any packfiles in the packed_git > list. > I'm not sure what corner-case issues caused this config option to > prevent the reprepare_packed_git() from being called at the proper > spot during the fetch operation. Even worse, I have failed to create > a test case to prevent a regression. > > Placing a reprepare_packed_git() call inside chck_connected() before > looping through the packed_git list seems like the safest way to > avoid this issue in the future. Hmmm. I am not sure if I am convinced that check_connected() is the best place to do this. Do we know the place that adds a new pack to the repository, yet forgets to add it to the packed-git list, that caused the failure you were observing? Doing this change, without describing the answer to the question in the log message, makes it smell rather like a random hack than a designed solution to me. If lazy fetching of objects happen in multiple fetches before a single check_connected() sweeps them to check for connectivity, then perhaps the lazy fetching codepath needs to remember the fact that it added a new pack that is still not known to the packed-git list (or just add it immediately, without having to scan at all), and check_connected() would need to rescan only when there is at least one such new pack? That way, you do not have to penalize normal callers of check_connected() that do not use lazy fetches at all, right? Thanks.