From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS3215 2.6.0.0/16 X-Spam-Status: No, score=-0.1 required=3.0 tests=AWL,BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE, URIBL_CSS,URIBL_CSS_A shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by dcvr.yhbt.net (Postfix) with ESMTP id 8D5BA1F8C4 for ; Wed, 16 Mar 2022 16:14:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350070AbiCPQPu (ORCPT ); Wed, 16 Mar 2022 12:15:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350101AbiCPQPs (ORCPT ); Wed, 16 Mar 2022 12:15:48 -0400 Received: from pb-smtp2.pobox.com (pb-smtp2.pobox.com [64.147.108.71]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3F4AB7CF for ; Wed, 16 Mar 2022 09:14:32 -0700 (PDT) Received: from pb-smtp2.pobox.com (unknown [127.0.0.1]) by pb-smtp2.pobox.com (Postfix) with ESMTP id 0071E11DCCF; Wed, 16 Mar 2022 12:14:29 -0400 (EDT) (envelope-from junio@pobox.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:message-id:mime-version:content-type; s=sasl; bh=ANOontaN2zIMnw59aakyVBCENoRkGY6Z/Ag0OkCyfPw=; b=N+1O +v+Qnll/+hcCIqzWT8VOD5NmqNIbE4uj95yvnu8aeyS7r4tlBoYnomi/74etV1oo M1FYo3qnRIrjgnDPCnh5weibDV7qviZoMzza4+lg0W2arHcsbsQQHZhlx2SKAwb0 jfAaFwoEdwClHFJBzBLuVnOMIhyqVvZv7iLwv2I= Received: from pb-smtp2.nyi.icgroup.com (unknown [127.0.0.1]) by pb-smtp2.pobox.com (Postfix) with ESMTP id EBFFD11DCCE; Wed, 16 Mar 2022 12:14:28 -0400 (EDT) (envelope-from junio@pobox.com) Received: from pobox.com (unknown [34.82.80.254]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by pb-smtp2.pobox.com (Postfix) with ESMTPSA id 6340811DCCD; Wed, 16 Mar 2022 12:14:28 -0400 (EDT) (envelope-from junio@pobox.com) From: Junio C Hamano To: Neeraj Singh Cc: Neeraj Singh via GitGitGadget , Git List , Johannes Schindelin , =?utf-8?B?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Patrick Steinhardt , "Neeraj K. Singh" Subject: Re: [PATCH 1/7] bulk-checkin: rename 'state' variable and separate 'plugged' boolean References: Date: Wed, 16 Mar 2022 09:14:27 -0700 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: 27B5F9F6-A544-11EC-8B2D-CB998F0A682E-77302942!pb-smtp2.pobox.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Neeraj Singh writes: > I think my new usage is congruent with the existing API, which seems > to be about combining multiple add operations into a large transaction, > where we can do some cleanup operations once we're finished. In the > preexisting code, the transaction is about adding a bunch of large objects > to a single pack file (while leaving small objects loose), and then completing > the packfile when the adds are finished. OK, so it was part me, and part a suboptimal presentation, I guess ;-) Let me rephrase the idea to see if I got it right this time. The bulk-checkin API has two interesting entry points, "plug" that signals that we are about to repeat possibly many operations to add new objects to the object store, and "unplug" that signals that we are done such adding. They are meant to serve as a hint for the object layer to optimize its operation. So far the only way the hint was used was that the logic that sends an overly large object into a packfile (instead of storing it loose, which leaves it subject to expensive repacking later) can shove more than one such objects in the same packfile. This series invents another use of the "plug"-"unplug" hint. By knowing that many loose object files are created and when the series of object creation ended, we can avoid having to fsync each and every one of them on certain filesystems and achieve the same robustness. The new "batch" option to core.fsyncmethod triggers this mechanism. Did I get it right, more-or-less? Thanks.