Git on macOS shows committed files as untracked

Git on macOS shows committed files as untracked

[roeder.git]

In Git on macOS (git version 2.13.2 | brew install git) the status command will show folders as untracked even though they are committed and checked out from the repository. Does not reproduce on Windows and Ubuntu.

Repro steps:

1. Download https://www.dropbox.com/s/0q5pbpqpckwzj7b/gitstatusrepro.zip?dl=0
2. unzip gitstatusrepro.zip && cd gitstatusrepro
3. git reset --hard

HEAD is now at 95fcd7e add folder with unicode name

4. git status

On branch master
Untracked files:
  (use "git add <file>..." to include in what will be committed)

	"d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/"

nothing added to commit but untracked files present (use "git add" to track)


----------
This message was sent from a MailNull anti-spam account.  You can get
your free account and take control over your email by visiting the
following URL.

   http://mailnull.com/

Re: Git on macOS shows committed files as untracked

[Jeff King]

On Wed, Jul 12, 2017 at 06:21:28PM -0400, roeder.git@mailnull.com wrote:

> In Git on macOS (git version 2.13.2 | brew install git) the status
> command will show folders as untracked even though they are committed
> and checked out from the repository. Does not reproduce on Windows and
> Ubuntu.
> [...]
> 
> 	"d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/"

Probably the issue has to do with Unicode normalization, and you have
files in your repository that can't be represented on your filesystem.
For example, the first two code-points above are "d" followed by U+0307,
"COMBINING DOT ABOVE". That pair can also be represented as U+1E0B,
"LATIN SMALL LETTER D WITH DOT ABOVE".

I don't recall which form HFS+ normalizes to, but basically what happens
is that Git opens the file with some name, and the filesystem quietly
rewrites that under the hood to a different, normalized name. Then when
Git walks the directory later to ask which files are present, it sees
this other filename that it has no clue about.

Generally the solution is to commit the normalized name. There's some
logic inside Git to "precompose" names to the right normalization, but I
think that only affects new files you add. Existing committed files with
the wrong normalization run into this issue.

-Peff

Re: Git on macOS shows committed files as untracked

[Torsten Bögershausen]

On 13/07/17 01:15, Jeff King wrote:
> On Wed, Jul 12, 2017 at 06:21:28PM -0400, roeder.git@mailnull.com wrote:
> 
>> In Git on macOS (git version 2.13.2 | brew install git) the status
>> command will show folders as untracked even though they are committed
>> and checked out from the repository. Does not reproduce on Windows and
>> Ubuntu.
>> [...]
>>
>> 	"d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/"
> 
> Probably the issue has to do with Unicode normalization, and you have
> files in your repository that can't be represented on your filesystem.
> For example, the first two code-points above are "d" followed by U+0307,
> "COMBINING DOT ABOVE". That pair can also be represented as U+1E0B,
> "LATIN SMALL LETTER D WITH DOT ABOVE".
> 
> I don't recall which form HFS+ normalizes to, but basically what happens
> is that Git opens the file with some name, and the filesystem quietly
> rewrites that under the hood to a different, normalized name. Then when
> Git walks the directory later to ask which files are present, it sees
> this other filename that it has no clue about.
> 
> Generally the solution is to commit the normalized name. There's some
> logic inside Git to "precompose" names to the right normalization, but I
> think that only affects new files you add. Existing committed files with
> the wrong normalization run into this issue.
> 
> -Peff
> 
Thanks for the fast analyzes -
in short:
what does
git -c core.precomposeunicode=true status
say ?


The easiest thing may be to set
git config --global core.precomposeunicode true

Re: Git on macOS shows committed files as untracked

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

> Thanks for the fast analyzes -
> in short:
> what does
> git -c core.precomposeunicode=true status
> say ?
>
> The easiest thing may be to set
> git config --global core.precomposeunicode true

Good suggestion.

I learned a new thing today.  I somehow thought that precompose
trick was only about argv[] when a program starts up and did not
apply to paths readdir(3) finds through dir.c, e.g.

    $ git add .

But apparently there is replacement readdir() used in compat/ for
MacOSX so the paths from the system are also covered by the
configuration.

Re: Git on macOS shows committed files as untracked

[Lutz Roeder]

Using precomposeunicode still reproduces the issue:

Repro steps:

1. Download https://www.dropbox.com/s/0q5pbpqpckwzj7b/gitstatusrepro.zip?dl=0
2. unzip gitstatusrepro.zip && cd gitstatusrepro
3. git reset --hard
4. git -c core.precomposeunicode=true status

On branch master
Untracked files:
  (use "git add <file>..." to include in what will be committed)

"d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/"

nothing added to commit but untracked files present (use "git add" to track)

> From: Torsten Bögershausen <tboegi@web.de>
>
> Thanks for the fast analyzes -
> in short:
> what does
> git -c core.precomposeunicode=true status
> say ?
>
>
> The easiest thing may be to set
> git config --global core.precomposeunicode true

Re: Git on macOS shows committed files as untracked

[Torsten Bögershausen]

On 14/07/17 06:49, Lutz Roeder wrote:
> Using precomposeunicode still reproduces the issue:
> 
> Repro steps:
> 
> 1. Download https://www.dropbox.com/s/0q5pbpqpckwzj7b/gitstatusrepro.zip?dl=0
> 2. unzip gitstatusrepro.zip && cd gitstatusrepro
> 3. git reset --hard
> 4. git -c core.precomposeunicode=true status
> 
> On branch master
> Untracked files:
>    (use "git add <file>..." to include in what will be committed)

A (short) investigation shows that this seems to be invalid unicode,
at least from a MacOSX point of view ?

Unzipping your repo shows this:
  git status -u
   deleted: 
"\341\270\213\341\274\262\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\323\262/test.txt" 


===============
If I run this:
  xx=$(printf 
"d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/")

echo   $xx | iconv -f UTF-8-MAC -t UTF-16 | xxd

iconv: (stdin):1:5: cannot convert
0000000: feff 1e0b 1f32 2553 03b5 fbaf            .....2%S....
===============================
So I don't know if we can do something in Git to improve your repo.
How did  you end up with such a directory name ?
And would it be possible to rename it ?

Re: Git on macOS shows committed files as untracked

[Elliot Chandler]

For what it's worth, the file looks normal in Gentoo GNU/Linux (name
appears "ḋἲ╓εﮯ𒑏○╓Ӳ" and it seems to work like any other directory).


On Fri, Jul 14, 2017 at 4:41 AM, Torsten Bögershausen <tboegi@web.de> wrote:
>
>
> On 14/07/17 06:49, Lutz Roeder wrote:
>>
>> Using precomposeunicode still reproduces the issue:
>>
>> Repro steps:
>>
>> 1. Download
>> https://www.dropbox.com/s/0q5pbpqpckwzj7b/gitstatusrepro.zip?dl=0
>> 2. unzip gitstatusrepro.zip && cd gitstatusrepro
>> 3. git reset --hard
>> 4. git -c core.precomposeunicode=true status
>>
>> On branch master
>> Untracked files:
>>    (use "git add <file>..." to include in what will be committed)
>
>
> A (short) investigation shows that this seems to be invalid unicode,
> at least from a MacOSX point of view ?
>
> Unzipping your repo shows this:
>  git status -u
>   deleted:
> "\341\270\213\341\274\262\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\323\262/test.txt"
>
> ===============
> If I run this:
>  xx=$(printf
> "d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/")
>
> echo   $xx | iconv -f UTF-8-MAC -t UTF-16 | xxd
>
> iconv: (stdin):1:5: cannot convert
> 0000000: feff 1e0b 1f32 2553 03b5 fbaf            .....2%S....
> ===============================
> So I don't know if we can do something in Git to improve your repo.
> How did  you end up with such a directory name ?
> And would it be possible to rename it ?
>
>

Re: Git on macOS shows committed files as untracked

[Torsten Bögershausen]

(Please no top-posting)
On 14/07/17 11:45, Elliot Chandler wrote:
> For what it's worth, the file looks normal in Gentoo GNU/Linux (name appears 
> "ḋἲ╓εﮯ𒑏○╓Ӳ" and it seems to work like any other directory).
> 
Thanks for testing -
Normal and Normal ;-)
For me the 6th code point does look strange
The "box" with 04142F:
Code point 4142F is unassigned and is outside any currently defined block range.

So this is not valid Unicode, so we have a problem here under MacOS -
not much Git can do about it.



> On Fri, Jul 14, 2017 at 4:41 AM, Torsten Bögershausen <tboegi@web.de 
> <mailto:tboegi@web.de>> wrote:
> 
> 
> 
>     On 14/07/17 06:49, Lutz Roeder wrote:
> 
>         Using precomposeunicode still reproduces the issue:
> 
>         Repro steps:
> 
>         1. Download
>         https://www.dropbox.com/s/0q5pbpqpckwzj7b/gitstatusrepro.zip?dl=0
>         <https://www.dropbox.com/s/0q5pbpqpckwzj7b/gitstatusrepro.zip?dl=0>
>         2. unzip gitstatusrepro.zip && cd gitstatusrepro
>         3. git reset --hard
>         4. git -c core.precomposeunicode=true status
> 
>         On branch master
>         Untracked files:
>             (use "git add <file>..." to include in what will be committed)
> 
> 
>     A (short) investigation shows that this seems to be invalid unicode,
>     at least from a MacOSX point of view ?
> 
>     Unzipping your repo shows this:
>       git status -u
>        deleted:
>     "\341\270\213\341\274\262\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\323\262/test.txt"
> 
> 
>     ===============
>     If I run this:
>       xx=$(printf
>     "d\314\207\316\271\314\223\314\200\342\225\223\316\265\357\256\257\360\222\221\217\342\227\213\342\225\223\320\243\314\213/")
> 
>     echo   $xx | iconv -f UTF-8-MAC -t UTF-16 | xxd
> 
>     iconv: (stdin):1:5: cannot convert
>     0000000: feff 1e0b 1f32 2553 03b5 fbaf            .....2%S....
>     ===============================
>     So I don't know if we can do something in Git to improve your repo.
>     How did  you end up with such a directory name ?
>     And would it be possible to rename it ?
> 
> 
> 

Re: Git on macOS shows committed files as untracked

[Jeff King]

On Fri, Jul 14, 2017 at 02:35:52PM +0200, Torsten Bögershausen wrote:

> (Please no top-posting)
> On 14/07/17 11:45, Elliot Chandler wrote:
> > For what it's worth, the file looks normal in Gentoo GNU/Linux (name
> > appears "ḋἲ╓εﮯ𒑏○╓Ӳ" and it seems to work like any other directory).
> > 
> Thanks for testing -
> Normal and Normal ;-)
> For me the 6th code point does look strange
> The "box" with 04142F:
> Code point 4142F is unassigned and is outside any currently defined block range.

Are you sure? I get u+1244f. I don't have a glyph for it in my font, but
at least it's a real unicode codepoint (cuneiform 1).

> So this is not valid Unicode, so we have a problem here under MacOS -
> not much Git can do about it.

I do still suspect this is the root of the problem. There's something
about the string that HFS+ doesn't want to store, and there's nothing we
can do to fix that.

-Peff

Re: Git on macOS shows committed files as untracked

[Andreas Schwab]

On Jul 14 2017, Torsten Bögershausen <tboegi@web.de> wrote:

> (Please no top-posting)
> On 14/07/17 11:45, Elliot Chandler wrote:
>> For what it's worth, the file looks normal in Gentoo GNU/Linux (name
>> appears "ḋἲ╓εﮯ𒑏○╓Ӳ" and it seems to work like any other directory).
>>
> Thanks for testing -
> Normal and Normal ;-)
> For me the 6th code point does look strange
> The "box" with 04142F:

This is actually 1244f (CUNEIFORM NUMERIC SIGN ONE BAN2).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."