From: Junio C Hamano <gitster@pobox.com>
To: Jeff Hostetler <git@jeffhostetler.com>
Cc: "Neeraj K. Singh via GitGitGadget" <gitgitgadget@gmail.com>,
git@vger.kernel.org, "Neeraj K. Singh" <neerajsi@microsoft.com>,
Neeraj Singh <neerajsi@ntdev.microsoft.com>
Subject: Re: [PATCH] read-cache: make the index write buffer size 128K
Date: Fri, 19 Feb 2021 19:28:00 -0800 [thread overview]
Message-ID: <xmqqv9ana05b.fsf@gitster.g> (raw)
In-Reply-To: <f52df30b-4ab0-fd6f-17f8-70daed81df39@jeffhostetler.com> (Jeff Hostetler's message of "Fri, 19 Feb 2021 14:12:42 -0500")
Jeff Hostetler <git@jeffhostetler.com> writes:
> On 2/17/21 9:48 PM, Neeraj K. Singh via GitGitGadget wrote:
>> From: Neeraj Singh <neerajsi@ntdev.microsoft.com>
>> Writing an index 8K at a time invokes the OS filesystem and caching
>> code
>> very frequently, introducing noticeable overhead while writing large
>> indexes. When experimenting with different write buffer sizes on Windows
>> writing the Windows OS repo index (260MB), most of the benefit came by
>> bumping the index write buffer size to 64K. I picked 128K to ensure that
>> we're past the knee of the curve.
>> With this change, the time under do_write_index for an index with 3M
>> files goes from ~1.02s to ~0.72s.
>
> [...]
>
>> -#define WRITE_BUFFER_SIZE 8192
>> +#define WRITE_BUFFER_SIZE (128 * 1024)
>> static unsigned char write_buffer[WRITE_BUFFER_SIZE];
>> static unsigned long write_buffer_len;
>
> [...]
>
> Very nice.
I wonder if we gain more by going say 4M buffer size or even larger?
Is this something we can make the system auto-tune itself? This is
not about reading but writing, so we already have enough information
to estimate how much we would need to write out.
Thanks.
next prev parent reply other threads:[~2021-02-20 3:29 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-02-18 2:48 [PATCH] read-cache: make the index write buffer size 128K Neeraj K. Singh via GitGitGadget
2021-02-19 19:12 ` Jeff Hostetler
2021-02-20 3:28 ` Junio C Hamano [this message]
2021-02-20 7:56 ` Neeraj Singh
2021-02-21 12:51 ` Junio C Hamano
2021-02-24 20:56 ` Neeraj Singh
2021-02-25 5:41 ` Junio C Hamano
2021-02-25 6:58 ` Chris Torek
2021-02-25 7:16 ` Junio C Hamano
2021-02-25 7:36 ` Neeraj Singh
2021-02-25 7:57 ` Chris Torek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqv9ana05b.fsf@gitster.g \
--to=gitster@pobox.com \
--cc=git@jeffhostetler.com \
--cc=git@vger.kernel.org \
--cc=gitgitgadget@gmail.com \
--cc=neerajsi@microsoft.com \
--cc=neerajsi@ntdev.microsoft.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).