git commit results in many lstat()s

git commit results in many lstat()s

[Gumbel, Matthew K]

Hello,

My high level problem is to speed up git commit on a large repository stored on NFS filesystem. I see via strace that it is slow because it makes a large number (~50,000) of lstat() calls in serial. Every call is a round-trip to the NFS server.

I do not understand why git commit must call lstat() on every file in the repository, even when I specify the name of the file I want to commit on the command line. Can somebody explain why it must call lstat on every file?

My command-line looks like this: git commit -uno -o -m asdf file-to-commit.txt

Secondly, are there any optimizations I can make to avoid this behavior?

Thanks,
Matt

Re: git commit results in many lstat()s

[Junio C Hamano]

"Gumbel, Matthew K" <matthew.k.gumbel@intel.com> writes:

> I do not understand why git commit must call lstat() on every file
> in the repository, even when I specify the name of the file I want
> to commit on the command line.

Assuming the "COPYING" and "README.md" files are already tracked:

    $ >COPYING
    $ >README.md
    $ git commit COPYING

would open an editor, in which you would see list of files under
"Changes to be committed", "Changes not staged for commit", etc.
Among the second class you would see README.md listed.

To figure out what paths are "changed", without having to open all
files and compare their contents with what is recorded in the commit
you are building on top of, we do lstat(2) to see if the timestamp
(and other information in the inode) of the files are the same since
you checked them out of HEAD.

    $ git commit --no-status COPYING

would reduce the number of lstat(2) somewhat, because the codepath
is told that it does not have to make the list to be shown in the
editor.  So would

    $ git commit -m "empty COPYING" COPYING

These two only halve the number of lstat(2), by taking advantage of
the fact that the list of "modified files" does not have to be
built.  There probably are other things that can be optimized.

RE: git commit results in many lstat()s

[Gumbel, Matthew K]

"Junio C Hamano" <jch2355@gmail.com> writes:
> There probably are other things that can be optimized.

Yes, I think that when the user passes --only flag to git-commit, then git does not
need to call refresh_cache() in prepare_index() in builtin/commit.c.

I may experiment with that. Do you see any downside or negative side-effects?

Matt

Re: git commit results in many lstat()s

[Junio C Hamano]

"Gumbel, Matthew K" <matthew.k.gumbel@intel.com> writes:

> "Junio C Hamano" <jch2355@gmail.com> writes:
>> There probably are other things that can be optimized.
>
> Yes, I think that when the user passes --only flag to git-commit, then git does not
> need to call refresh_cache() in prepare_index() in builtin/commit.c.
>
> I may experiment with that. Do you see any downside or negative side-effects?

There may be other fallouts, but one that immediately comes to mind
is that it may break pre-commit hook.

When we get "--only", we prepare an temporary index to create the
commit out of, and give it to the pre-commit hook.  The hook expects
that the cached stat information is up-to-date, iow, it does not
have to do 'update-index --refresh' before using plumbing commands
like "diff-index" to do its own inspection of the working tree.

RE: git commit results in many lstat()s

[Gumbel, Matthew K]

"Junio C Hamano <mailto:jch2355@gmail.com> writes:

"Gumbel, Matthew K" <matthew.k.gumbel@intel.com> writes:

>> Yes, I think that when the user passes --only flag to git-commit, then git does not
>> need to call refresh_cache() in prepare_index() in builtin/commit.c.
>>
>> I may experiment with that. Do you see any downside or negative side-effects?

> There may be other fallouts, but one that immediately comes to mind
> is that it may break pre-commit hook.

If pre-commit hook exists, we can fall-back to original behavior and call
refresh_cache(). Many repos will not have pre-commit hook and can 
benefit from the speedup.

I'm testing such a change locally. Git test suite seems to be running for quite
a while. Do you know any way to run it in parallel or otherwise speed it
up?

Thanks,
Matt

Re: git commit results in many lstat()s

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Thu, Feb 02, 2017 at 12:14:30AM +0000, Gumbel, Matthew K wrote:
> I'm testing such a change locally. Git test suite seems to be running for quite
> a while. Do you know any way to run it in parallel or otherwise speed it
> up?

I usually do something like the following:

  make -j3 all && (cd t && GIT_PROVE_OPTS=-j3 make prove)

This, of course, requires that you have Perl's prove installed, which
has been part of core Perl since 5.10.1.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]