"git commit" of single file takes 5 minutes, mounted fileystem/diskimage with 50G GIT repo + 900G of builds articles

"git commit" of single file takes 5 minutes, mounted fileystem/diskimage with 50G GIT repo + 900G of builds articles

[Saravanan Shanmugham (sarvi)]

Based on a previous thread “First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles”

The context is that 
  1. git wokspace was clone(50G)
  2. some 30 platorms build(900G)
  3. This tree was then copied into a new diskimage/filesystem + git update-index --refresh" was done to update the index to the new filesystem, then frozen.
  4. New workspaces created by cloning this frozen diskimage(< 30 seconds)
  5. This diskimage was mounted on a new machine
  6. A file was modified and "git add/commit" was done

I have done “git update-index –refresh”, in the mounted filesystem, as above
This has improved “git status/diff” timing from 40+ minutes to 1.5 minutes for the first time and <5 seconds for subsequent calls.

But "git commit -m "dummy commit" of a 1 line change in 1 file takes about 5-6 minutes, everytime in this workspace.
Tracing shows a whole bunch. The entire 5-6 minutes worth of the following sort of trace logs.
3:13:50.320930 trace git-lfs: filepathfilter: rejecting "x/y/z.o.command" via []
13:13:50.320940 trace git-lfs: filepathfilter: accepting " x/y/z.o.command "
13:13:50.320862 trace git-lfs: filepathfilter: rejecting "a/b/c/d.o.command" via []
13:13:50.320972 trace git-lfs: filepathfilter: accepting " a/b/c/d..o.command"

Does anyone have any insights on what could be causing this?

On the other hand, if I had 
 
Thanks,
Sarvi
Occam’s Razor Rules

Re: "git commit" of single file takes 5 minutes, mounted fileystem/diskimage with 50G GIT repo + 900G of builds articles

[Saravanan Shanmugham (sarvi)]

Any suggestion or advice how to optimize this in GIT ?

 
Thanks,
Sarvi
Occam’s Razor Rules

On 8/26/19, 1:43 PM, "Saravanan Shanmugham (sarvi)" <sarvi@cisco.com> wrote:

    
    Based on a previous thread “First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles”
    
    The context is that 
      1. git wokspace was clone(50G)
      2. some 30 platorms build(900G)
      3. This tree was then copied into a new diskimage/filesystem + git update-index --refresh" was done to update the index to the new filesystem, then frozen.
      4. New workspaces created by cloning this frozen diskimage(< 30 seconds)
      5. This diskimage was mounted on a new machine
      6. A file was modified and "git add/commit" was done
    
    I have done “git update-index –refresh”, in the mounted filesystem, as above
    This has improved “git status/diff” timing from 40+ minutes to 1.5 minutes for the first time and <5 seconds for subsequent calls.
    
    But "git commit -m "dummy commit" of a 1 line change in 1 file takes about 5-6 minutes, everytime in this workspace.
    Tracing shows a whole bunch. The entire 5-6 minutes worth of the following sort of trace logs.
    3:13:50.320930 trace git-lfs: filepathfilter: rejecting "x/y/z.o.command" via []
    13:13:50.320940 trace git-lfs: filepathfilter: accepting " x/y/z.o.command "
    13:13:50.320862 trace git-lfs: filepathfilter: rejecting "a/b/c/d.o.command" via []
    13:13:50.320972 trace git-lfs: filepathfilter: accepting " a/b/c/d..o.command"
    
    Does anyone have any insights on what could be causing this?
         
    Thanks,
    Sarvi
    Occam’s Razor Rules
    
    

Re: "git commit" of single file takes 5 minutes, mounted fileystem/diskimage with 50G GIT repo + 900G of builds articles

[Taylor Blau]

Hi Saravanan,

On Mon, Aug 26, 2019 at 08:43:34PM +0000, Saravanan Shanmugham (sarvi) wrote:
>
> Based on a previous thread “First Git status takes 40+ minutes, when mounting fileystem/diskimage with 50G GIT repo + 900G of builds articles”
>
> The context is that
>   1. git wokspace was clone(50G)
>   2. some 30 platorms build(900G)
>   3. This tree was then copied into a new diskimage/filesystem + git update-index --refresh" was done to update the index to the new filesystem, then frozen.
>   4. New workspaces created by cloning this frozen diskimage(< 30 seconds)
>   5. This diskimage was mounted on a new machine
>   6. A file was modified and "git add/commit" was done
>
> I have done “git update-index –refresh”, in the mounted filesystem, as above
> This has improved “git status/diff” timing from 40+ minutes to 1.5 minutes for the first time and <5 seconds for subsequent calls.
>
> But "git commit -m "dummy commit" of a 1 line change in 1 file takes about 5-6 minutes, everytime in this workspace.
> Tracing shows a whole bunch. The entire 5-6 minutes worth of the following sort of trace logs.
> 3:13:50.320930 trace git-lfs: filepathfilter: rejecting "x/y/z.o.command" via []
> 13:13:50.320940 trace git-lfs: filepathfilter: accepting " x/y/z.o.command "
> 13:13:50.320862 trace git-lfs: filepathfilter: rejecting "a/b/c/d.o.command" via []
> 13:13:50.320972 trace git-lfs: filepathfilter: accepting " a/b/c/d..o.command"
>
> Does anyone have any insights on what could be causing this?

These are messages from the Git LFS tool, and this list does not provide
support for Git LFS, since it is a separate project not affiliated with
Git.

I would recommend that you create an issue on Git LFS's tracker at:

  https://github.com/git-lfs/git-lfs/issues/new

and mention that you are seeing lots of 'filepathfilter' logs. My
speculation is that this could be a slowdown in the 'filepathfilter'
code paths, or an entire-tree scan where 'filepathfilter' is invoked
every so often. I'd mention both of those as possibilities (or send a
link to this email on public-inbox), and see if anything comes up.

> On the other hand, if I had
>
> Thanks,
> Sarvi
> Occam’s Razor Rules

Thanks,
Taylor