Suboptimal performance of git add -A with untracked cache enabled

Suboptimal performance of git add -A with untracked cache enabled

[Utsav Shah]

I'm running git 2.27 on MacOS catalina on a repo with a size of
roughly ~300k files. Also, I'm using fsmonitor and untracked cache.
Operations like git status are fast as expected and complete under 1s.
But git add -A often takes 10 seconds to complete. The trace output
shows most of the time spent in "read directory" and "diff-files". I'm
mostly interested in solving the slowness in "read directory".

It seems that the untracked cache is skipped by git add -A since the
dir_flags are different between git status and git add
(https://github.com/git/git/blob/master/dir.c#L2671), which leads to
the slowness. In the specific case of add -A, the DIR_COLLECT_IGNORED
flag is set, (https://github.com/git/git/blob/master/builtin/add.c#L543),
which is one reason why the untracked cache is not used. Another is
that git status sets DIR_SHOW_OTHER_DIRECTORIES |
DIR_HIDE_EMPTY_DIRECTORIES
(https://github.com/git/git/blob/master/wt-status.c#L709).

I'm wondering if we can leverage knowledge from fsmonitor to relax
some constraints on using the untracked cache, or skip directory
traversal. In theory, if fsmonitor is aware that there haven't been
any changes to files (tracked or not), it seems that we can skip
opening and iterating over directories.

If anyone has guidance if this is something that can be fixed and has
a recommendation for an approach, I'd love to hear it. I tried working
on a patch, but it seems that the untracked cache is correctly being
skipped for correctness reasons, and I'm not sure if relaxing its
constraints is the right approach.

Thanks