Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled

Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled

[Josh Smeaton]

git write-tree and then git diff-index reports thousands of files as `deleted`
when core.fsmonitor is true, even though the files are still present on the
file system.

What did you do before the bug happened? (Steps to reproduce your issue)

```
git config core.fsmonitor true
touch new-file.txt
git add new-file.txt
git diff-index --ignore-submodules --exit-code --no-color
--no-ext-diff $(git write-tree) --
```

What did you expect to happen? (Expected behavior)

I expected that no files would be reported as having changes.

What happened instead? (Actual behavior)

Thousands of unrelated files were reported as `deleted`.

```
git diff-index --ignore-submodules --exit-code --no-color
--no-ext-diff $(git write-tree) --
:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
0000000000000000000000000000000000000000 D src/tests/__init__.py
:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
0000000000000000000000000000000000000000 D src/tests/site/__init__.py
:100644 000000 d8edaad92120781998e811277f966551b12fc823
0000000000000000000000000000000000000000 D
src/tests/site/test_models.py

 ... thousands of other files
```

What's different between what you expected and what actually happened?

Thousands of unrelated files were reported as having been deleted.


Anything else you want to add:

Myself and a few co-workers all ran into this issue in the last few days. It
presented itself when pre-commit hooks were run (specifically with the
pre-commit
tool from github pre-commit).

When committing changes, the pre-commit tool runs the equivalent of:

`git diff-index --ignore-submodules --exit-code --no-color
--no-ext-diff $(git write-tree) --`

It then stashes and unstashes those changes, causing the files to be
removed from
the working tree.


```
$ git config core.fsmonitor true
$ git checkout -b test-branch
Switched to a new branch 'test-branch'

$ echo "test 3" >> newfile.txt
$ git add .
$ git status
On branch test-branch
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
new file:   newfile.txt

$ git commit -a -m "will it fail?"
[WARNING] Unstaged files detected.
[INFO] Stashing unstaged files to
/Users/josh/.cache/pre-commit/patch1677888420-10129.
trim trailing whitespace.................................................Passed
check python ast.....................................(no files to check)Skipped
check for case conflicts.................................................Passed
[INFO] Restored changes from
/Users/josh/.cache/pre-commit/patch1677888420-10129.
[test-branch 7c899582c19] will it fail?
 1 file changed, 1 insertion(+)
 create mode 100644 newfile.txt

$ git status
On branch test-branch
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
deleted:    src/tests/__init__.py
deleted:    src/tests/site/__init__.py
deleted:    src/tests/site/test_models.py
        ... lots more files ...

no changes added to commit (use "git add" and/or "git commit -a")

$ git status | wc -l
   15298

$ git restore .

$ git status

On branch test-branch
nothing to commit, working tree clean
```

It can be reproduced, in our repo, without using pre-commit at all:

```

git diff-index --ignore-submodules --exit-code --no-color
--no-ext-diff $(git write-tree) --
:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
0000000000000000000000000000000000000000 D src/tests/__init__.py
:100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
0000000000000000000000000000000000000000 D src/tests/site/__init__.py
:100644 000000 d8edaad92120781998e811277f966551b12fc823
0000000000000000000000000000000000000000 D
src/tests/site/test_models.py

 ... thousands of other files
```

Using `update-index` with the `--no-fsmonitor` flag prior to diff-index does the
correct thing:

```
$ git update-index --really-refresh --no-fsmonitor
warning: core.fsmonitor is set; remove it if you really want to
disable fsmonitor

$ git diff-index --ignore-submodules --exit-code --no-color
--no-ext-diff $(git write-tree) --
> no output
```

The problem persists when running `update-index` without disabling fsmonitor.

I have reported this issue to the pre-commit project, but it looks like an issue
with fsmonitor + write-tree.

See: pre-commit issues on github #2795



[System Info]
git version:
git version 2.39.2 (others reproduced with 2.39.1)
cpu: arm64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37
PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64
compiler info: clang: 14.0.0 (clang-1400.0.29.202)
libc info: no libc information available
$SHELL (typically, interactive shell): /bin/zsh


[Enabled Hooks]
pre-commit

Re: Bug: diff-index reports unrelated files as deleted when fsmonitor is enabled

[Johannes Schindelin]

Hi Josh,

On Sat, 4 Mar 2023, Josh Smeaton wrote:

> git write-tree and then git diff-index reports thousands of files as
> `deleted` when core.fsmonitor is true, even though the files are still
> present on the file system.

I suspect that the issue is a missing `git update-index --refresh`. While
`git diff` refreshes the index automatically, `git diff-index` does not
(when you use low-level commands like `diff-index` you need to do such
things explicitly).

If that does not fix your problem, I encourage you to distill the issue
into a simple MCVE (https://stackoverflow.com/help/mcve), preferably in
the form of a patch, say, to t7527 (see
https://github.com/git/git/blob/v2.40.0/t/t7527-builtin-fsmonitor.sh).

Ciao,
Johannes

>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
> ```
> git config core.fsmonitor true
> touch new-file.txt
> git add new-file.txt
> git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> ```
>
> What did you expect to happen? (Expected behavior)
>
> I expected that no files would be reported as having changes.
>
> What happened instead? (Actual behavior)
>
> Thousands of unrelated files were reported as `deleted`.
>
> ```
> git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/__init__.py
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/site/__init__.py
> :100644 000000 d8edaad92120781998e811277f966551b12fc823
> 0000000000000000000000000000000000000000 D
> src/tests/site/test_models.py
>
>  ... thousands of other files
> ```
>
> What's different between what you expected and what actually happened?
>
> Thousands of unrelated files were reported as having been deleted.
>
>
> Anything else you want to add:
>
> Myself and a few co-workers all ran into this issue in the last few days. It
> presented itself when pre-commit hooks were run (specifically with the
> pre-commit
> tool from github pre-commit).
>
> When committing changes, the pre-commit tool runs the equivalent of:
>
> `git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --`
>
> It then stashes and unstashes those changes, causing the files to be
> removed from
> the working tree.
>
>
> ```
> $ git config core.fsmonitor true
> $ git checkout -b test-branch
> Switched to a new branch 'test-branch'
>
> $ echo "test 3" >> newfile.txt
> $ git add .
> $ git status
> On branch test-branch
> Changes to be committed:
>   (use "git restore --staged <file>..." to unstage)
> new file:   newfile.txt
>
> $ git commit -a -m "will it fail?"
> [WARNING] Unstaged files detected.
> [INFO] Stashing unstaged files to
> /Users/josh/.cache/pre-commit/patch1677888420-10129.
> trim trailing whitespace.................................................Passed
> check python ast.....................................(no files to check)Skipped
> check for case conflicts.................................................Passed
> [INFO] Restored changes from
> /Users/josh/.cache/pre-commit/patch1677888420-10129.
> [test-branch 7c899582c19] will it fail?
>  1 file changed, 1 insertion(+)
>  create mode 100644 newfile.txt
>
> $ git status
> On branch test-branch
> Changes not staged for commit:
>   (use "git add/rm <file>..." to update what will be committed)
>   (use "git restore <file>..." to discard changes in working directory)
> deleted:    src/tests/__init__.py
> deleted:    src/tests/site/__init__.py
> deleted:    src/tests/site/test_models.py
>         ... lots more files ...
>
> no changes added to commit (use "git add" and/or "git commit -a")
>
> $ git status | wc -l
>    15298
>
> $ git restore .
>
> $ git status
>
> On branch test-branch
> nothing to commit, working tree clean
> ```
>
> It can be reproduced, in our repo, without using pre-commit at all:
>
> ```
>
> git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/__init__.py
> :100644 000000 e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
> 0000000000000000000000000000000000000000 D src/tests/site/__init__.py
> :100644 000000 d8edaad92120781998e811277f966551b12fc823
> 0000000000000000000000000000000000000000 D
> src/tests/site/test_models.py
>
>  ... thousands of other files
> ```
>
> Using `update-index` with the `--no-fsmonitor` flag prior to diff-index does the
> correct thing:
>
> ```
> $ git update-index --really-refresh --no-fsmonitor
> warning: core.fsmonitor is set; remove it if you really want to
> disable fsmonitor
>
> $ git diff-index --ignore-submodules --exit-code --no-color
> --no-ext-diff $(git write-tree) --
> > no output
> ```
>
> The problem persists when running `update-index` without disabling fsmonitor.
>
> I have reported this issue to the pre-commit project, but it looks like an issue
> with fsmonitor + write-tree.
>
> See: pre-commit issues on github #2795
>
>
>
> [System Info]
> git version:
> git version 2.39.2 (others reproduced with 2.39.1)
> cpu: arm64
> no commit associated with this build
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> feature: fsmonitor--daemon
> uname: Darwin 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:38:37
> PST 2023; root:xnu-8792.81.3~2/RELEASE_ARM64_T6000 arm64
> compiler info: clang: 14.0.0 (clang-1400.0.29.202)
> libc info: no libc information available
> $SHELL (typically, interactive shell): /bin/zsh
>
>
> [Enabled Hooks]
> pre-commit
>