Re: [PATCH 2/5] add: allow operating on a sparse-only index

[Derrick Stolee <stolee@gmail.com>]

On 7/23/2021 1:45 PM, Elijah Newren wrote:
> On Wed, Jul 21, 2021 at 2:07 PM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
...
>> Test                                  HEAD~1           HEAD
>> ------------------------------------------------------------------------------
>> 2000.6: git add -A (full-index-v3)    0.35(0.30+0.05)  0.37(0.29+0.06) +5.7%
>> 2000.7: git add -A (full-index-v4)    0.31(0.26+0.06)  0.33(0.27+0.06) +6.5%
>> 2000.8: git add -A (sparse-index-v3)  0.57(0.53+0.07)  0.05(0.04+0.08) -91.2%
>> 2000.9: git add -A (sparse-index-v4)  0.58(0.55+0.06)  0.05(0.05+0.06) -91.4%
>>
>> While the 91% improvement seems impressive, it's important to recognize
>> that previously we had significant overhead for expanding the
>> sparse-index. Comparing to the full index case, 'git add -A' goes from
>> 0.37s to 0.05s, which is "only" an 86% improvement.
> 
> Hehe.  Yep, it's so "disappointing" to "only" have the code be 7x faster.  :-)
> 
> Out of curiosity, IIRC any operation involving the index took ~10s on
> some of the Microsoft repos.  What does the speedup look like over
> there for these changes to git-add?

The latest numbers I have for a repo with ~2 million tracked files is that
 index reads take about half a second (because of the threaded reads) and
writes take at least one second. There was a lot of work by Ben Peart, Jeff
Hostetler, and Kevin Willford to reduce this cost as much as possible a few
years ago. VFS for Git is still limited by this bottleneck, but Scalar's
use of sparse-checkout enables the use of the sparse index.

We have an experimental release [1] out to users right now, and I will
report to the mailing list about how that went after we get sufficient
adoption that the data can be significant. When focusing on individual
users I can find things like one user seeing "git commit" going from 4.3s
to 0.35s and "git add" going from 6.1s to 0.13s. (The "git add" time
might also be conflated with a change from the FS Monitor hook to the
builtin FS Monitor.)

[1] https://github.com/microsoft/git/releases/tag/v2.32.0.vfs.0.102.exp

Thanks,
-Stolee