[BUG?] Fresh clone of jquery.git shows modifications?

[BUG?] Fresh clone of jquery.git shows modifications?

[Will Palmer]

I am on Linux, and core.autocrlf is not set. .gitattributes in the
JQuery repository contains:
* crlf=input

Bisect reveals:
fd6cce9e89ab5ac1125a3b5f5611048ad22379e7  v1.7.0-3-gfd6cce9
Add per-repository eol normalization

is the first git commit which shows this problem.

I am confused as to why I am seeing what I am seeing, though. I was
under the impression that "crlf=input" meant: "convert crlf to lf on
input", which I would take to mean that it would never have any effect
whatsoever on "git status"'s output.

If I clone using a version of git before v1.7.0-3-gfd6cce9, then "git
status" from a newer git, then everything also appears to work normally
(though I haven't dug much into this aspect)

Is there a git option for "just give me what's in the repository, don't
ever perform any conversions, one way or the other, just act sane" ?

I thought I had finally understood all this autocrlf nonsense, but once
again I see this as being only an "act stupid mode". I have no idea
what's going on here, and I just want to be able to rely on the
following:
 - git clone someproj.git && cd someproj && git status;
   should NEVER report changes
 - git reset --hard HEAD && git status;
   should NEVER report changes

and, why I ran into this situation this time around:
 - git clone someproj.git && cd someproj && git checkout sometag;
   should ALWAYS work

Can anyone explain why these aren't valid assertions? If they're not,
then what is the proper way to tell git "give me a real pristine copy"?

Below is a log of the problem in-action.

[shruggar@shruggar jquery]$ git --version
git version 1.7.2.1.224.g2f41ea
[shruggar@shruggar jquery]$ uname -a
Linux shruggar 2.6.32-24-generic #39-Ubuntu SMP Wed Jul 28 05:14:15 UTC
2010 x86_64 GNU/Linux
[shruggar@shruggar extra]$ git clone git://github.com/jquery/jquery.git 
Cloning into jquery...
remote: Counting objects: 12750, done.
remote: Compressing objects: 100% (3921/3921), done.
remote: Total 12750 (delta 8690), reused 12276 (delta 8312)
Receiving objects: 100% (12750/12750), 8.95 MiB | 775 KiB/s, done.
Resolving deltas: 100% (8690/8690), done.
[shruggar@shruggar extra]$ cd jquery/
[shruggar@shruggar jquery]$ git st
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working
directory)
#
#	modified:   build/google-compiler-20091218.jar
#	modified:   build/js.jar
#	modified:   speed/jquery-basis.js
#	modified:   test/data/text.php
#	modified:   test/delegatetest.html
#
no changes added to commit (use "git add" and/or "git commit -a")
[shruggar@shruggar jquery]$ git reset --hard HEAD
HEAD is now at 6f031c1 Replace usage of the removed global reset()
method with QUnit.reset().
[shruggar@shruggar jquery]$ git st
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working
directory)
#
#	modified:   build/google-compiler-20091218.jar
#	modified:   build/js.jar
#	modified:   speed/jquery-basis.js
#	modified:   test/data/text.php
#	modified:   test/delegatetest.html
#
no changes added to commit (use "git add" and/or "git commit -a")
[shruggar@shruggar jquery]$ GIT_EDITOR=cat git config --edit
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = git://github.com/jquery/jquery.git
[branch "master"]
	remote = origin
	merge = refs/heads/master
[shruggar@shruggar jquery]$

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Eyvind Bernhardsen]

On 18. aug. 2010, at 14.40, Will Palmer wrote:

> I am on Linux, and core.autocrlf is not set. .gitattributes in the
> JQuery repository contains:
> * crlf=input

That's a strange setting; in older versions of git, it means something like "if core.autocrlf is enabled, only convert CRLF on input"; that is, it effectively turns core.autocrlf=true into core.autocrlf=input.  I'm not sure why anybody would want that.

In any case, it doesn't appear to have prevented CRLFs from entering the repository, which is what's causing the behaviour you see.

> Bisect reveals:
> fd6cce9e89ab5ac1125a3b5f5611048ad22379e7  v1.7.0-3-gfd6cce9
> Add per-repository eol normalization
> 
> is the first git commit which shows this problem.

That's because from that commit, "crlf=input" turns on autocrlf implicitly (it is equivalent to "eol=lf").  Previously, the crlf attribute would only have an effect with core.autocrlf turned on.

> I am confused as to why I am seeing what I am seeing, though. I was
> under the impression that "crlf=input" meant: "convert crlf to lf on
> input", which I would take to mean that it would never have any effect
> whatsoever on "git status"'s output.

Files containing CRLFs in your repository keep the CRLFs in the working tree.  If you check them in now git will convert CRLF to LF, so they are marked as modified.
 
> If I clone using a version of git before v1.7.0-3-gfd6cce9, then "git
> status" from a newer git, then everything also appears to work normally
> (though I haven't dug much into this aspect)

If you turn on core.autocrlf=input or core.autocrlf=true you should get the same behaviour with older gits as well.  Note that msysgit has had autocrlf enabled by default for years, so no positive action is required to do this.

> Is there a git option for "just give me what's in the repository, don't
> ever perform any conversions, one way or the other, just act sane" ?

Not with that .gitattributes :)  If you don't want conversion, don't set the crlf, eol or text attributes.

> I thought I had finally understood all this autocrlf nonsense, but once
> again I see this as being only an "act stupid mode". I have no idea
> what's going on here, and I just want to be able to rely on the
> following:
> - git clone someproj.git && cd someproj && git status;
>   should NEVER report changes
> - git reset --hard HEAD && git status;
>   should NEVER report changes
> 
> and, why I ran into this situation this time around:
> - git clone someproj.git && cd someproj && git checkout sometag;
>   should ALWAYS work
> 
> Can anyone explain why these aren't valid assertions? If they're not,
> then what is the proper way to tell git "give me a real pristine copy"?

Git _is_ giving you a real pristine copy, it's just informing you that your repository is not consistent with the attributes you have set.

> Below is a log of the problem in-action.

[...]

> #	modified:   build/google-compiler-20091218.jar
> #	modified:   build/js.jar

Heh.  I was about to ask if there was anything funny about these presumably binary files, but then I realized that .gitattributes is forcing crlf=input for _all_ files.  I suggest removing the crlf setting from .gitattributes.

- Eyvind

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Will Palmer]

On Wed, 2010-08-18 at 15:18 +0200, Eyvind Bernhardsen wrote:
> On 18. aug. 2010, at 14.40, Will Palmer wrote:
> 
> > I am on Linux, and core.autocrlf is not set. .gitattributes in the
> > JQuery repository contains:
> > * crlf=input
> 
> That's a strange setting; in older versions of git, it means something like "if core.autocrlf is enabled, only convert CRLF on input"; that is, it effectively turns core.autocrlf=true into core.autocrlf=input.  I'm not sure why anybody would want that.
> 
> In any case, it doesn't appear to have prevented CRLFs from entering the repository, which is what's causing the behaviour you see.

I don't know when the .gitattributes file was added, surely git should
account for varied settings over time, especially when the meanings of
those settings have changed over time? Something like "only apply when
there are other changes to this file", or only when the newline-status
of the file itself has otherwise changed, for that matter.

> 
> > Bisect reveals:
> > fd6cce9e89ab5ac1125a3b5f5611048ad22379e7  v1.7.0-3-gfd6cce9
> > Add per-repository eol normalization
> > 
> > is the first git commit which shows this problem.
> 
> That's because from that commit, "crlf=input" turns on autocrlf implicitly (it is equivalent to "eol=lf").  Previously, the crlf attribute would only have an effect with core.autocrlf turned on.
> 
> > I am confused as to why I am seeing what I am seeing, though. I was
> > under the impression that "crlf=input" meant: "convert crlf to lf on
> > input", which I would take to mean that it would never have any effect
> > whatsoever on "git status"'s output.
> 
> Files containing CRLFs in your repository keep the CRLFs in the working tree.  If you check them in now git will convert CRLF to LF, so they are marked as modified.

So how do I tell git that I don't want those files to be modified? How
do I tell git "discard all local changes, even those which some random
file has asked you to make"?

>  
> > If I clone using a version of git before v1.7.0-3-gfd6cce9, then "git
> > status" from a newer git, then everything also appears to work normally
> > (though I haven't dug much into this aspect)
> 
> If you turn on core.autocrlf=input or core.autocrlf=true you should get the same behaviour with older gits as well.  Note that msysgit has had autocrlf enabled by default for years, so no positive action is required to do this.

This was one of the first things I tried, but with no luck. Perhaps
there is some extra step involved to tell git "update the working copy
using the new setting"? I have tried both git checkout --force HEAD, and
git reset --hard HEAD, but neither seem to have an effect.

> 
> > Is there a git option for "just give me what's in the repository, don't
> > ever perform any conversions, one way or the other, just act sane" ?
> 
> Not with that .gitattributes :)  If you don't want conversion, don't set the crlf, eol or text attributes.

How about an "ignore .gitattributes" option? Or a "use
alternative .gitattributes" option? This is not my repository. I don't
have control over what's in it, and while I expect I'll be maintaining a
local patchset, I don't really want to maintain patches of the "my git
is more-broken than the upstream's git" variety.

> 
> > I thought I had finally understood all this autocrlf nonsense, but once
> > again I see this as being only an "act stupid mode". I have no idea
> > what's going on here, and I just want to be able to rely on the
> > following:
> > - git clone someproj.git && cd someproj && git status;
> >   should NEVER report changes
> > - git reset --hard HEAD && git status;
> >   should NEVER report changes
> > 
> > and, why I ran into this situation this time around:
> > - git clone someproj.git && cd someproj && git checkout sometag;
> >   should ALWAYS work
> > 
> > Can anyone explain why these aren't valid assertions? If they're not,
> > then what is the proper way to tell git "give me a real pristine copy"?
> 
> Git _is_ giving you a real pristine copy, it's just informing you that your repository is not consistent with the attributes you have set.

"pristine" to me means "no changes". If there is an inconsistency which
git doesn't like, git should complain about that, refuse to checkout,
declare "broken repository! Oh no!" and refuse all operations, etc. It
git can determine that it is unable to act in a sane manner, I don't
want git to just go ahead making wild guesses.

> 
> > Below is a log of the problem in-action.
> 
> [...]
> 
> > #	modified:   build/google-compiler-20091218.jar
> > #	modified:   build/js.jar
> 
> Heh.  I was about to ask if there was anything funny about these presumably binary files, but then I realized that .gitattributes is forcing crlf=input for _all_ files.  I suggest removing the crlf setting from .gitattributes.

Just to re-state: This is not "my repository". That is, I don't consider
it to be. So I would prefer only solutions which involve editing
something unversioned, such as .git/config, rather than making random
patches which feel very much like work-arounds to a git problem.

If it were "my repository", and there were some problem like this, then
I would probably use filter-branch to remove the error from history.

> 
> - Eyvind

- Will

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Thomas Berg]

Hi,

On Wed, 2010-08-18 at 15:18 +0200, Eyvind Bernhardsen wrote:
> Git _is_ giving you a real pristine copy, it's just informing you that your repository is not consistent with the attributes you have set.

Marking the files as modified is sometimes a very user-unfriendly way
of informing the user, at least if just want to work with something
unrelated, and not actually fix those issues. Maybe there is a better
way?

On Wed, Aug 18, 2010 at 4:14 PM, Will Palmer <wmpalmer@gmail.com> wrote:
> "pristine" to me means "no changes". If there is an inconsistency which
> git doesn't like, git should complain about that, refuse to checkout,
> declare "broken repository! Oh no!" and refuse all operations, etc. It
> git can determine that it is unable to act in a sane manner, I don't
> want git to just go ahead making wild guesses.
>
[snip]
> Just to re-state: This is not "my repository". That is, I don't consider
> it to be. So I would prefer only solutions which involve editing
> something unversioned, such as .git/config, rather than making random
> patches which feel very much like work-arounds to a git problem.

Such a solution exists, you can create a file .git/info/attributes,
which will override the faulty .gitattributes file in the repository
[1]. Then you could make git treat all files as binary files, making
no conversions. Using this unversioned file also enables you to jump
back in history, rebase etc. without problems.

Hope this helps,
Thomas

[1] http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Will Palmer]

On Wed, 2010-08-18 at 17:50 +0200, Thomas Berg wrote:
> Hi,
> [snip]
> > Just to re-state: This is not "my repository". That is, I don't consider
> > it to be. So I would prefer only solutions which involve editing
> > something unversioned, such as .git/config, rather than making random
> > patches which feel very much like work-arounds to a git problem.
> 
> Such a solution exists, you can create a file .git/info/attributes,
> which will override the faulty .gitattributes file in the repository
> [1]. Then you could make git treat all files as binary files, making
> no conversions. Using this unversioned file also enables you to jump
> back in history, rebase etc. without problems.
> 

.git/info/attributes
* -crlf

seems to be what I was looking for! Thank you.

Though I very much think the way git currently acts in this situation is
quite inexcusably broken.

> Hope this helps,
> Thomas
> 
> [1] http://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Eyvind Bernhardsen]

On 18. aug. 2010, at 18.37, Will Palmer wrote:

[...]

> .git/info/attributes
> * -crlf
> 
> seems to be what I was looking for! Thank you.

Excellent!  Thanks for testing, and thanks to Thomas for the suggestion.

> Though I very much think the way git currently acts in this situation is
> quite inexcusably broken.

It's certainly not ideal, but the old behaviour was broken in an arguably more insidious way in that only people with autocrlf enabled saw the problem.  At least this way repository owners will see it themselves as they upgrade to newer versions of git, and (hopefully) fix it at the source.

In the meantime, I'm glad that .git/info/attributes let you work around the problem.

- Eyvind

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Eyvind Bernhardsen]

On 18. aug. 2010, at 17.50, Thomas Berg wrote:

> Hi,
> 
> On Wed, 2010-08-18 at 15:18 +0200, Eyvind Bernhardsen wrote:
>> Git _is_ giving you a real pristine copy, it's just informing you that your repository is not consistent with the attributes you have set.
> 
> Marking the files as modified is sometimes a very user-unfriendly way
> of informing the user, at least if just want to work with something
> unrelated, and not actually fix those issues. Maybe there is a better
> way?

There may be, but I'm not sure it would be worth the effort.  The reason I started working on CRLF normalization in the first place was because I got tired of seeing spuriously modified files everywhere when using core.autocrlf, so I know how annoying it is.  The difference now is that only people using "mis-attributed" repositories are annoyed, instead of anyone who uses core.autocrlf, and as I wrote elsewhere, I don't expect there to be many repositories where this is a problem.

To be clear: triggering the problem not only requires that a crlf attribute is set to something other than "false", but also that the corresponding file actually contains CRs.

- Eyvind

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Thomas Berg]

On Wed, Aug 18, 2010 at 9:19 PM, Eyvind Bernhardsen
<eyvind.bernhardsen@gmail.com> wrote:
> On 18. aug. 2010, at 17.50, Thomas Berg wrote:
>> Maybe there is a better way?
>
> There may be, but I'm not sure it would be worth the effort.  The reason I started working on CRLF normalization in the first place was because I got tired of seeing spuriously modified files everywhere when using core.autocrlf, so I know how annoying it is.  The difference now is that only people using "mis-attributed" repositories are annoyed, instead of anyone who uses core.autocrlf, and as I wrote elsewhere, I don't expect there to be many repositories where this is a problem.

I guess you're right, probably not worth the effort (although I think
that the spuriously modified files are just as "bad" in this case).
Thanks a lot for your work on this, it scratched all my CRLF itches at
least.

I'm maintaining my own personal patches on top of quite a few git-svn
mirrors and git-hg mirrors.  Before git 1.7.2, rebasing these on
Windows was a pain. I even maintained my own .gitattributes files for
many of them, but still had problems. Now all is solved!

- Thomas

Re: [BUG?] Fresh clone of jquery.git shows modifications?

[Eyvind Bernhardsen]

On 18. aug. 2010, at 16.14, Will Palmer wrote:

[...]

> I don't know when the .gitattributes file was added, surely git should
> account for varied settings over time, especially when the meanings of
> those settings have changed over time? Something like "only apply when
> there are other changes to this file", or only when the newline-status
> of the file itself has otherwise changed, for that matter.

"core.autocrlf" now does something like this: it only converts a file if it contains no CRs in the repository.  The downside to this behaviour is that line endings are no longer normalised, but since normalisation never worked very well as a configuration setting, that's a worthwhile tradeoff.

The attributes are used to enable normalisation unconditionally, and this _should_ be okay since they are versioned along with the contents of the repository so those contents _should_ reflect the attributes' settings.  Unfortunately, as you discovered, this isn't necessarily true in older repositories.  Fortunately, as Thomas Rast noted elsewhere, .git/info/attributes can work around the problem.

[...]

>> If you turn on core.autocrlf=input or core.autocrlf=true you should get the same behaviour with older gits as well.  Note that msysgit has had autocrlf enabled by default for years, so no positive action is required to do this.
> 
> This was one of the first things I tried, but with no luck. Perhaps
> there is some extra step involved to tell git "update the working copy
> using the new setting"? I have tried both git checkout --force HEAD, and
> git reset --hard HEAD, but neither seem to have an effect.

"rm .git/index && git reset" usually does the trick for me.  If the index is intact, git will know that you haven't touched any files and won't bother to check them.

[...]

>> Git _is_ giving you a real pristine copy, it's just informing you that your repository is not consistent with the attributes you have set.
> 
> "pristine" to me means "no changes". If there is an inconsistency which
> git doesn't like, git should complain about that, refuse to checkout,
> declare "broken repository! Oh no!" and refuse all operations, etc. It
> git can determine that it is unable to act in a sane manner, I don't
> want git to just go ahead making wild guesses.

I don't see how refusing to check out would be helpful.  How would you recover from that?  I think git does the least it can in this situation: no files are modified in any way, but the user is notified that some of them will be modified if they are "git add"ed.  If they are then committed, the problem is solved and future checkouts _will_ be "pristine".

In your case, simply committing the normalised files isn't an option since some of them are binaries and will be corrupted, so modifying .gitattributes is the way to go there.

[...]

> If it were "my repository", and there were some problem like this, then
> I would probably use filter-branch to remove the error from history.

Hopefully, the owner of the repository will at least remove the error from future versions, if not actually filter it out of existence.  But it _is_ an error, it's just that prior to git 1.7.2 only people with autocrlf set could see it.

I did consider this as a potential problem when implementing the new crlf conversion behaviour, but reasoned that very few repositories would be affected in this way: there was no reason to set the crlf attribute unless you worked with people who had core.autocrlf set, and if you did, they would notice the spurious "modifications".

I'm sorry you hit a repository with an incorrectly set crlf, and I hope there aren't many more out there.

- Eyvind