Ability to ignore EOL changes for certain projects

Ability to ignore EOL changes for certain projects

[Scott Richmond]

The Problem Domain
In certain dev environments (Unity3D projects) there is (AFAIK) an
unsolvable problem where some files are often modified with line
endings that aren't the native system or not the committed line
endings for that file. Secondarily, in this case line endings don't
matter - Nothing in the dev environment "cares" which kind of line
ending is used.

The Problem
Git always cares about EOL. Git has options to transparently modify
EOLs when files are checked in or out. However it is not possible to
tell Git to ignore EOLs in other commands:
Git status shows the file modified.
Merging/Pulling has to care because it can't merge with a modified
working tree. Which means the user has to care - They have to either
stash the EOL changes or wipe them out. Sometimes, if the user has a
particular app running, it may automatically reload that file and
recreate the modified EOL changes, causing an endless loop. This
problem is often made unclear to the user how to solve, especially if
they aren't domain experts.

To be clear, in this particular dev environment, I can say with
confidence that this particular issue is a major and common pain point
for users. It is made worse as many users in this environment aren't
programmers by trade and aren't domain experts in version control. I
also believe this environment is becoming a non-trivial portion of the
Git userbase and it would be worthwhile looking into resolving.

Solution Request
It would be fantastic if we could tell Git to stop caring about EOL
changes on a per-repo basis, with the effective output being that git
status shouldn't show changes for files with differing EOLs.

I'm experienced with Git, though I am not expert enough to consider
creating such a change myself - It is unclear to me just how
complicated a change may be. However maybe I could look into it if it
was made clear that this improvement is possible and has no serious
side effects.

Regards,

Scott Richmond.
  Director, Producer, Programmer
  Brightrock Games
  T: 07480795661

Re: Ability to ignore EOL changes for certain projects

[Torsten Bögershausen]

On Wed, Dec 18, 2019 at 11:10:27AM +0000, Scott Richmond wrote:
> The Problem Domain
> In certain dev environments (Unity3D projects) there is (AFAIK) an
> unsolvable problem where some files are often modified with line
> endings that aren't the native system or not the committed line
> endings for that file. Secondarily, in this case line endings don't
> matter - Nothing in the dev environment "cares" which kind of line
> ending is used.
>
> The Problem
> Git always cares about EOL. Git has options to transparently modify
> EOLs when files are checked in or out. However it is not possible to
> tell Git to ignore EOLs in other commands:
> Git status shows the file modified.
> Merging/Pulling has to care because it can't merge with a modified
> working tree. Which means the user has to care - They have to either
> stash the EOL changes or wipe them out. Sometimes, if the user has a
> particular app running, it may automatically reload that file and
> recreate the modified EOL changes, causing an endless loop. This
> problem is often made unclear to the user how to solve, especially if
> they aren't domain experts.
>
> To be clear, in this particular dev environment, I can say with
> confidence that this particular issue is a major and common pain point
> for users. It is made worse as many users in this environment aren't
> programmers by trade and aren't domain experts in version control. I
> also believe this environment is becoming a non-trivial portion of the
> Git userbase and it would be worthwhile looking into resolving.
>
> Solution Request
> It would be fantastic if we could tell Git to stop caring about EOL
> changes on a per-repo basis, with the effective output being that git
> status shouldn't show changes for files with differing EOLs.
>
> I'm experienced with Git, though I am not expert enough to consider
> creating such a change myself - It is unclear to me just how
> complicated a change may be. However maybe I could look into it if it
> was made clear that this improvement is possible and has no serious
> side effects.

Hej Scott,
I think that you problem can be solved.
For each repository, you can tell Git to ignore the line endings,
CRLF vs LF.

If you start with a fresh repo,
you can do something like:

echo "* text=auto" >.gitattributes
git add .gitattributes
git commit -m "Add .gitattributes"

For existing repos, we need to take another step:

echo "* text=auto" >.gitattributes
git add .gitattributes
git add  --renormlize .
git commit -m "Add .gitattributes"

More information can be found e.g. here:
https://git-scm.com/docs/git-add

Once you done that, you can merge branches
into the master branch with help of the renormalize

git merge -X renormalze branch-name

See even here:
https://git-scm.com/docs/git-merge


This is just a (too) short introduction, I hope that it
helps and that you find the time to dig somewhat deeper into
the documentation.

Other developers have that problem as well, you are not alone.

If you have a public repo, I could help with one example.


>
> Regards,
>
> Scott Richmond.
>   Director, Producer, Programmer
>   Brightrock Games
>   T: 07480795661

Re: Ability to ignore EOL changes for certain projects

[Scott Richmond]

Hey Torsten,

Thanks for the reply!

Correct me if am wrong, but those steps don't tell git to "ignore"
line endings. That just causes git to check all text files in and out
with a specific EOL type (Either automatically chosen, or not). If an
app in the dev env changes a files' EOL to something else, git will
notice the change locally.

Regards,

Scott Richmond.
  Director, Producer, Programmer
  Brightrock Games
  T: 07480795661

On Wed, Dec 18, 2019 at 7:27 PM Torsten Bögershausen <tboegi@web.de> wrote:
>
> On Wed, Dec 18, 2019 at 11:10:27AM +0000, Scott Richmond wrote:
> > The Problem Domain
> > In certain dev environments (Unity3D projects) there is (AFAIK) an
> > unsolvable problem where some files are often modified with line
> > endings that aren't the native system or not the committed line
> > endings for that file. Secondarily, in this case line endings don't
> > matter - Nothing in the dev environment "cares" which kind of line
> > ending is used.
> >
> > The Problem
> > Git always cares about EOL. Git has options to transparently modify
> > EOLs when files are checked in or out. However it is not possible to
> > tell Git to ignore EOLs in other commands:
> > Git status shows the file modified.
> > Merging/Pulling has to care because it can't merge with a modified
> > working tree. Which means the user has to care - They have to either
> > stash the EOL changes or wipe them out. Sometimes, if the user has a
> > particular app running, it may automatically reload that file and
> > recreate the modified EOL changes, causing an endless loop. This
> > problem is often made unclear to the user how to solve, especially if
> > they aren't domain experts.
> >
> > To be clear, in this particular dev environment, I can say with
> > confidence that this particular issue is a major and common pain point
> > for users. It is made worse as many users in this environment aren't
> > programmers by trade and aren't domain experts in version control. I
> > also believe this environment is becoming a non-trivial portion of the
> > Git userbase and it would be worthwhile looking into resolving.
> >
> > Solution Request
> > It would be fantastic if we could tell Git to stop caring about EOL
> > changes on a per-repo basis, with the effective output being that git
> > status shouldn't show changes for files with differing EOLs.
> >
> > I'm experienced with Git, though I am not expert enough to consider
> > creating such a change myself - It is unclear to me just how
> > complicated a change may be. However maybe I could look into it if it
> > was made clear that this improvement is possible and has no serious
> > side effects.
>
> Hej Scott,
> I think that you problem can be solved.
> For each repository, you can tell Git to ignore the line endings,
> CRLF vs LF.
>
> If you start with a fresh repo,
> you can do something like:
>
> echo "* text=auto" >.gitattributes
> git add .gitattributes
> git commit -m "Add .gitattributes"
>
> For existing repos, we need to take another step:
>
> echo "* text=auto" >.gitattributes
> git add .gitattributes
> git add  --renormlize .
> git commit -m "Add .gitattributes"
>
> More information can be found e.g. here:
> https://git-scm.com/docs/git-add
>
> Once you done that, you can merge branches
> into the master branch with help of the renormalize
>
> git merge -X renormalze branch-name
>
> See even here:
> https://git-scm.com/docs/git-merge
>
>
> This is just a (too) short introduction, I hope that it
> helps and that you find the time to dig somewhat deeper into
> the documentation.
>
> Other developers have that problem as well, you are not alone.
>
> If you have a public repo, I could help with one example.
>
>
> >
> > Regards,
> >
> > Scott Richmond.
> >   Director, Producer, Programmer
> >   Brightrock Games
> >   T: 07480795661

Re: Ability to ignore EOL changes for certain projects

[Philip Oakley]

On 18/12/2019 21:33, Scott Richmond wrote:
> Hey Torsten,
>
> Thanks for the reply!
>
> Correct me if am wrong, but those steps don't tell git to "ignore"
> line endings. That just causes git to check all text files in and out
> with a specific EOL type (Either automatically chosen, or not). If an
> app in the dev env changes a files' EOL to something else, git will
> notice the change locally.

A broader question is "how does the dev environment handle lone `CR`
characters? are they also considered as EOLs?"

If I recall previous discussions, part of the issue is determining what
the alternate abuse cases are, such as lone CRs, or alternate character
sets to the 'pure' utf-8.

You will still have the difficulty of how `identicality` is determined,
which currently uses the sha1/oid value. In essence you need a way of
ensuring that all checkins are always of one defined EOL, but then need
git to have a broader allowance of changed EOL values (including no lone
CRs that are not EOL, etc).

Does the environment (on any of the OS's) change the files in the
background, such that the file time stamps will indicate a modification?

Philip

> Regards,
>
> Scott Richmond.
>   Director, Producer, Programmer
>   Brightrock Games
>   T: 07480795661
>
> On Wed, Dec 18, 2019 at 7:27 PM Torsten Bögershausen <tboegi@web.de> wrote:
>> On Wed, Dec 18, 2019 at 11:10:27AM +0000, Scott Richmond wrote:
>>> The Problem Domain
>>> In certain dev environments (Unity3D projects) there is (AFAIK) an
>>> unsolvable problem where some files are often modified with line
>>> endings that aren't the native system or not the committed line
>>> endings for that file. Secondarily, in this case line endings don't
>>> matter - Nothing in the dev environment "cares" which kind of line
>>> ending is used.
>>>
>>> The Problem
>>> Git always cares about EOL. Git has options to transparently modify
>>> EOLs when files are checked in or out. However it is not possible to
>>> tell Git to ignore EOLs in other commands:
>>> Git status shows the file modified.
>>> Merging/Pulling has to care because it can't merge with a modified
>>> working tree. Which means the user has to care - They have to either
>>> stash the EOL changes or wipe them out. Sometimes, if the user has a
>>> particular app running, it may automatically reload that file and
>>> recreate the modified EOL changes, causing an endless loop. This
>>> problem is often made unclear to the user how to solve, especially if
>>> they aren't domain experts.
>>>
>>> To be clear, in this particular dev environment, I can say with
>>> confidence that this particular issue is a major and common pain point
>>> for users. It is made worse as many users in this environment aren't
>>> programmers by trade and aren't domain experts in version control. I
>>> also believe this environment is becoming a non-trivial portion of the
>>> Git userbase and it would be worthwhile looking into resolving.
>>>
>>> Solution Request
>>> It would be fantastic if we could tell Git to stop caring about EOL
>>> changes on a per-repo basis, with the effective output being that git
>>> status shouldn't show changes for files with differing EOLs.
>>>
>>> I'm experienced with Git, though I am not expert enough to consider
>>> creating such a change myself - It is unclear to me just how
>>> complicated a change may be. However maybe I could look into it if it
>>> was made clear that this improvement is possible and has no serious
>>> side effects.
>> Hej Scott,
>> I think that you problem can be solved.
>> For each repository, you can tell Git to ignore the line endings,
>> CRLF vs LF.
>>
>> If you start with a fresh repo,
>> you can do something like:
>>
>> echo "* text=auto" >.gitattributes
>> git add .gitattributes
>> git commit -m "Add .gitattributes"
>>
>> For existing repos, we need to take another step:
>>
>> echo "* text=auto" >.gitattributes
>> git add .gitattributes
>> git add  --renormlize .
>> git commit -m "Add .gitattributes"
>>
>> More information can be found e.g. here:
>> https://git-scm.com/docs/git-add
>>
>> Once you done that, you can merge branches
>> into the master branch with help of the renormalize
>>
>> git merge -X renormalze branch-name
>>
>> See even here:
>> https://git-scm.com/docs/git-merge
>>
>>
>> This is just a (too) short introduction, I hope that it
>> helps and that you find the time to dig somewhat deeper into
>> the documentation.
>>
>> Other developers have that problem as well, you are not alone.
>>
>> If you have a public repo, I could help with one example.
>>
>>
>>> Regards,
>>>
>>> Scott Richmond.
>>>   Director, Producer, Programmer
>>>   Brightrock Games
>>>   T: 07480795661

Re: Ability to ignore EOL changes for certain projects

[Torsten Bögershausen]

Scott, please avoid top-posting in this list, see my answers at the end.

> On Wed, Dec 18, 2019 at 7:27 PM Torsten Bögershausen <tboegi@web.de> wrote:
> >
> > On Wed, Dec 18, 2019 at 11:10:27AM +0000, Scott Richmond wrote:
> > > The Problem Domain
> > > In certain dev environments (Unity3D projects) there is (AFAIK) an
> > > unsolvable problem where some files are often modified with line
> > > endings that aren't the native system or not the committed line
> > > endings for that file. Secondarily, in this case line endings don't
> > > matter - Nothing in the dev environment "cares" which kind of line
> > > ending is used.
> > >
> > > The Problem
> > > Git always cares about EOL. Git has options to transparently modify
> > > EOLs when files are checked in or out. However it is not possible to
> > > tell Git to ignore EOLs in other commands:
> > > Git status shows the file modified.
> > > Merging/Pulling has to care because it can't merge with a modified
> > > working tree. Which means the user has to care - They have to either
> > > stash the EOL changes or wipe them out. Sometimes, if the user has a
> > > particular app running, it may automatically reload that file and
> > > recreate the modified EOL changes, causing an endless loop. This
> > > problem is often made unclear to the user how to solve, especially if
> > > they aren't domain experts.
> > >
> > > To be clear, in this particular dev environment, I can say with
> > > confidence that this particular issue is a major and common pain point
> > > for users. It is made worse as many users in this environment aren't
> > > programmers by trade and aren't domain experts in version control. I
> > > also believe this environment is becoming a non-trivial portion of the
> > > Git userbase and it would be worthwhile looking into resolving.
> > >
> > > Solution Request
> > > It would be fantastic if we could tell Git to stop caring about EOL
> > > changes on a per-repo basis, with the effective output being that git
> > > status shouldn't show changes for files with differing EOLs.
> > >
> > > I'm experienced with Git, though I am not expert enough to consider
> > > creating such a change myself - It is unclear to me just how
> > > complicated a change may be. However maybe I could look into it if it
> > > was made clear that this improvement is possible and has no serious
> > > side effects.
> >
> > Hej Scott,
> > I think that you problem can be solved.
> > For each repository, you can tell Git to ignore the line endings,
> > CRLF vs LF.
> >
> > If you start with a fresh repo,
> > you can do something like:
> >
> > echo "* text=auto" >.gitattributes
> > git add .gitattributes
> > git commit -m "Add .gitattributes"
> >
> > For existing repos, we need to take another step:
> >
> > echo "* text=auto" >.gitattributes
> > git add .gitattributes
> > git add  --renormlize .
> > git commit -m "Add .gitattributes"
> >
> > More information can be found e.g. here:
> > https://git-scm.com/docs/git-add
> >
> > Once you done that, you can merge branches
> > into the master branch with help of the renormalize
> >
> > git merge -X renormalze branch-name
> >
> > See even here:
> > https://git-scm.com/docs/git-merge
> >
> >
> > This is just a (too) short introduction, I hope that it
> > helps and that you find the time to dig somewhat deeper into
> > the documentation.
> >
> > Other developers have that problem as well, you are not alone.
> >
> > If you have a public repo, I could help with one example.
> >
> >
> > >
> > > Regards,
> > >
> > > Scott Richmond.
> > >   Director, Producer, Programmer
> > >   Brightrock Games
> > >   T: 07480795661
On Wed, Dec 18, 2019 at 09:33:32PM +0000, Scott Richmond wrote:
> Hey Torsten,
>
> Thanks for the reply!
>
> Correct me if am wrong, but those steps don't tell git to "ignore"
> line endings. That just causes git to check all text files in and out
> with a specific EOL type (Either automatically chosen, or not). If an
> app in the dev env changes a files' EOL to something else, git will
> notice the change locally.
>
> Regards,
>
> Scott Richmond.
>   Director, Producer, Programmer
>   Brightrock Games
>   T: 07480795661
>

Hej Scott,

I am not sure whether I understand your question correctly.
So I set up a little test, to illustrate things better.


user@linux:/tmp/EOLtest $ git init
Initialized empty Git repository in /tmp/EOLtest/.git/
user@linux:/tmp/EOLtest $ printf "Line1\r\nLine2\r\n" >file
user@linux:/tmp/EOLtest $ git add file
user@linux:/tmp/EOLtest $ git commit -m "add file with CRLF"
[master (root-commit) 0e4d1df] add file with CRLF
 1 file changed, 2 insertions(+)
  create mode 100644 file

user@linux:/tmp/EOLtest $ printf "Line1\nLine2\n" >file
user@linux:/tmp/EOLtest $ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
    (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file

no changes added to commit (use "git add" and/or "git commit -a")
user@linux:/tmp/EOLtest $ git diff
diff --git a/file b/file
index 4aa551d..ac2dd81 100644
--- a/file
+++ b/file
@@ -1,2 +1,2 @@
-Line1
-Line2
+Line1
+Line2

user@linux:/tmp/EOLtest $ echo "* text=auto" >.gitattributes
user@linux:/tmp/EOLtest $ git add --renormalize .
user@linux:/tmp/EOLtest $ git commit -m "Normalize the repo"
[master b41136d] Normalize the repo
 1 file changed, 2 insertions(+), 2 deletions(-)
 user@linux:/tmp/EOLtest $

user@linux:/tmp/EOLtest $ printf "Line1\r\nLine2\r\n" >file
user@linux:/tmp/EOLtest $ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
    (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   file

user@linux:/tmp/EOLtest $ git diff
warning: CRLF will be replaced by LF in file.
The file will have its original line endings in your working directory

#############################
No, at this point, I am surprised myself.
"file" is reported as "modified", but it should not be modified, right?
Is that the problem, that you have ?


We can fix it, by running:

user@linux:/tmp/EOLtest $ git add file
user@linux:/tmp/EOLtest $ git status
On branch master
nothing to commit, working tree clean

But I still think, that this "modified" is unexpected,
and a (possible) bug in Git, thanks for reporting.

Which Git version do you use ?
Is this test script a description of you problem ?

Re: Ability to ignore EOL changes for certain projects

[Scott Richmond]

> Scott, please avoid top-posting in this list, see my answers at the end.
Sure! Sorry I rarely use mail lists.

> But I still think, that this "modified" is unexpected,
> and a (possible) bug in Git, thanks for reporting.
>
> Which Git version do you use ?
> Is this test script a description of you problem ?

Yes this appears to be a correct repro of the kind of scenario that is
common in many dev environments like ours.
I'm not yet sure if this is actually a bug (I'd be happy if it is!).
Reading the docs I think this is intended behaviour - All the text
CRLF options relate to what Git should do on check in/out. It is
correctly reporting the file change, even if it is just EOL.
What Philip is describing with the sha1/iod check sounds to me what
the root cause "problem" is - Any file change will show up on git.
Philip appears to imply that there is no way to get git to do further
introspection on the file at this time. But maybe that can be added?
Re the Git version - I try to stay on the latest at all times. This
issue has been present for years, for as long as I can remember.

Regards,

Scott Richmond.
  Director, Producer, Programmer
  Brightrock Games
  T: 07480795661

On Thu, Dec 19, 2019 at 3:12 AM Torsten Bögershausen <tboegi@web.de> wrote:
>
> Scott, please avoid top-posting in this list, see my answers at the end.
>
> > On Wed, Dec 18, 2019 at 7:27 PM Torsten Bögershausen <tboegi@web.de> wrote:
> > >
> > > On Wed, Dec 18, 2019 at 11:10:27AM +0000, Scott Richmond wrote:
> > > > The Problem Domain
> > > > In certain dev environments (Unity3D projects) there is (AFAIK) an
> > > > unsolvable problem where some files are often modified with line
> > > > endings that aren't the native system or not the committed line
> > > > endings for that file. Secondarily, in this case line endings don't
> > > > matter - Nothing in the dev environment "cares" which kind of line
> > > > ending is used.
> > > >
> > > > The Problem
> > > > Git always cares about EOL. Git has options to transparently modify
> > > > EOLs when files are checked in or out. However it is not possible to
> > > > tell Git to ignore EOLs in other commands:
> > > > Git status shows the file modified.
> > > > Merging/Pulling has to care because it can't merge with a modified
> > > > working tree. Which means the user has to care - They have to either
> > > > stash the EOL changes or wipe them out. Sometimes, if the user has a
> > > > particular app running, it may automatically reload that file and
> > > > recreate the modified EOL changes, causing an endless loop. This
> > > > problem is often made unclear to the user how to solve, especially if
> > > > they aren't domain experts.
> > > >
> > > > To be clear, in this particular dev environment, I can say with
> > > > confidence that this particular issue is a major and common pain point
> > > > for users. It is made worse as many users in this environment aren't
> > > > programmers by trade and aren't domain experts in version control. I
> > > > also believe this environment is becoming a non-trivial portion of the
> > > > Git userbase and it would be worthwhile looking into resolving.
> > > >
> > > > Solution Request
> > > > It would be fantastic if we could tell Git to stop caring about EOL
> > > > changes on a per-repo basis, with the effective output being that git
> > > > status shouldn't show changes for files with differing EOLs.
> > > >
> > > > I'm experienced with Git, though I am not expert enough to consider
> > > > creating such a change myself - It is unclear to me just how
> > > > complicated a change may be. However maybe I could look into it if it
> > > > was made clear that this improvement is possible and has no serious
> > > > side effects.
> > >
> > > Hej Scott,
> > > I think that you problem can be solved.
> > > For each repository, you can tell Git to ignore the line endings,
> > > CRLF vs LF.
> > >
> > > If you start with a fresh repo,
> > > you can do something like:
> > >
> > > echo "* text=auto" >.gitattributes
> > > git add .gitattributes
> > > git commit -m "Add .gitattributes"
> > >
> > > For existing repos, we need to take another step:
> > >
> > > echo "* text=auto" >.gitattributes
> > > git add .gitattributes
> > > git add  --renormlize .
> > > git commit -m "Add .gitattributes"
> > >
> > > More information can be found e.g. here:
> > > https://git-scm.com/docs/git-add
> > >
> > > Once you done that, you can merge branches
> > > into the master branch with help of the renormalize
> > >
> > > git merge -X renormalze branch-name
> > >
> > > See even here:
> > > https://git-scm.com/docs/git-merge
> > >
> > >
> > > This is just a (too) short introduction, I hope that it
> > > helps and that you find the time to dig somewhat deeper into
> > > the documentation.
> > >
> > > Other developers have that problem as well, you are not alone.
> > >
> > > If you have a public repo, I could help with one example.
> > >
> > >
> > > >
> > > > Regards,
> > > >
> > > > Scott Richmond.
> > > >   Director, Producer, Programmer
> > > >   Brightrock Games
> > > >   T: 07480795661
> On Wed, Dec 18, 2019 at 09:33:32PM +0000, Scott Richmond wrote:
> > Hey Torsten,
> >
> > Thanks for the reply!
> >
> > Correct me if am wrong, but those steps don't tell git to "ignore"
> > line endings. That just causes git to check all text files in and out
> > with a specific EOL type (Either automatically chosen, or not). If an
> > app in the dev env changes a files' EOL to something else, git will
> > notice the change locally.
> >
> > Regards,
> >
> > Scott Richmond.
> >   Director, Producer, Programmer
> >   Brightrock Games
> >   T: 07480795661
> >
>
> Hej Scott,
>
> I am not sure whether I understand your question correctly.
> So I set up a little test, to illustrate things better.
>
>
> user@linux:/tmp/EOLtest $ git init
> Initialized empty Git repository in /tmp/EOLtest/.git/
> user@linux:/tmp/EOLtest $ printf "Line1\r\nLine2\r\n" >file
> user@linux:/tmp/EOLtest $ git add file
> user@linux:/tmp/EOLtest $ git commit -m "add file with CRLF"
> [master (root-commit) 0e4d1df] add file with CRLF
>  1 file changed, 2 insertions(+)
>   create mode 100644 file
>
> user@linux:/tmp/EOLtest $ printf "Line1\nLine2\n" >file
> user@linux:/tmp/EOLtest $ git status
> On branch master
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>     (use "git checkout -- <file>..." to discard changes in working directory)
>
>         modified:   file
>
> no changes added to commit (use "git add" and/or "git commit -a")
> user@linux:/tmp/EOLtest $ git diff
> diff --git a/file b/file
> index 4aa551d..ac2dd81 100644
> --- a/file
> +++ b/file
> @@ -1,2 +1,2 @@
> -Line1
> -Line2
> +Line1
> +Line2
>
> user@linux:/tmp/EOLtest $ echo "* text=auto" >.gitattributes
> user@linux:/tmp/EOLtest $ git add --renormalize .
> user@linux:/tmp/EOLtest $ git commit -m "Normalize the repo"
> [master b41136d] Normalize the repo
>  1 file changed, 2 insertions(+), 2 deletions(-)
>  user@linux:/tmp/EOLtest $
>
> user@linux:/tmp/EOLtest $ printf "Line1\r\nLine2\r\n" >file
> user@linux:/tmp/EOLtest $ git status
> On branch master
> Changes not staged for commit:
>   (use "git add <file>..." to update what will be committed)
>     (use "git checkout -- <file>..." to discard changes in working directory)
>
>         modified:   file
>
> user@linux:/tmp/EOLtest $ git diff
> warning: CRLF will be replaced by LF in file.
> The file will have its original line endings in your working directory
>
> #############################
> No, at this point, I am surprised myself.
> "file" is reported as "modified", but it should not be modified, right?
> Is that the problem, that you have ?
>
>
> We can fix it, by running:
>
> user@linux:/tmp/EOLtest $ git add file
> user@linux:/tmp/EOLtest $ git status
> On branch master
> nothing to commit, working tree clean
>
> But I still think, that this "modified" is unexpected,
> and a (possible) bug in Git, thanks for reporting.
>
> Which Git version do you use ?
> Is this test script a description of you problem ?
>

Re: Ability to ignore EOL changes for certain projects

[Scott Richmond]

> If I recall previous discussions, part of the issue is determining what
the alternate abuse cases are, such as lone CRs, or alternate character
sets to the 'pure' utf-8.
In this dev env, I would be surprised if any CR type makes a
difference. But I could be wrong.
If that is indeed one of the reasons why previous discussions broke
down, maybe that could be a configurable option in the feature
improvement.

> Does the environment (on any of the OS's) change the files in the background, such that the file time stamps will indicate a modification?
I would imagine the file timestamps do get updated yes. Though I'd
have to do some research to confirm it.

If indeed this isn't a bug (As Torsten suspects), then I suppose what
I am asking for is a feature improvement to perform a further check
after sha1/iod to confirm if the changes are only between CR types?

Regards,

Scott Richmond.
  Director, Producer, Programmer
  Brightrock Games
  T: 07480795661
On Wed, Dec 18, 2019 at 10:35 PM Philip Oakley <philipoakley@iee.email> wrote:
>
> On 18/12/2019 21:33, Scott Richmond wrote:
> > Hey Torsten,
> >
> > Thanks for the reply!
> >
> > Correct me if am wrong, but those steps don't tell git to "ignore"
> > line endings. That just causes git to check all text files in and out
> > with a specific EOL type (Either automatically chosen, or not). If an
> > app in the dev env changes a files' EOL to something else, git will
> > notice the change locally.
>
> A broader question is "how does the dev environment handle lone `CR`
> characters? are they also considered as EOLs?"
>
> If I recall previous discussions, part of the issue is determining what
> the alternate abuse cases are, such as lone CRs, or alternate character
> sets to the 'pure' utf-8.
>
> You will still have the difficulty of how `identicality` is determined,
> which currently uses the sha1/oid value. In essence you need a way of
> ensuring that all checkins are always of one defined EOL, but then need
> git to have a broader allowance of changed EOL values (including no lone
> CRs that are not EOL, etc).
>
> Does the environment (on any of the OS's) change the files in the
> background, such that the file time stamps will indicate a modification?
>
> Philip
>
> > Regards,
> >
> > Scott Richmond.
> >   Director, Producer, Programmer
> >   Brightrock Games
> >   T: 07480795661
> >
> > On Wed, Dec 18, 2019 at 7:27 PM Torsten Bögershausen <tboegi@web.de> wrote:
> >> On Wed, Dec 18, 2019 at 11:10:27AM +0000, Scott Richmond wrote:
> >>> The Problem Domain
> >>> In certain dev environments (Unity3D projects) there is (AFAIK) an
> >>> unsolvable problem where some files are often modified with line
> >>> endings that aren't the native system or not the committed line
> >>> endings for that file. Secondarily, in this case line endings don't
> >>> matter - Nothing in the dev environment "cares" which kind of line
> >>> ending is used.
> >>>
> >>> The Problem
> >>> Git always cares about EOL. Git has options to transparently modify
> >>> EOLs when files are checked in or out. However it is not possible to
> >>> tell Git to ignore EOLs in other commands:
> >>> Git status shows the file modified.
> >>> Merging/Pulling has to care because it can't merge with a modified
> >>> working tree. Which means the user has to care - They have to either
> >>> stash the EOL changes or wipe them out. Sometimes, if the user has a
> >>> particular app running, it may automatically reload that file and
> >>> recreate the modified EOL changes, causing an endless loop. This
> >>> problem is often made unclear to the user how to solve, especially if
> >>> they aren't domain experts.
> >>>
> >>> To be clear, in this particular dev environment, I can say with
> >>> confidence that this particular issue is a major and common pain point
> >>> for users. It is made worse as many users in this environment aren't
> >>> programmers by trade and aren't domain experts in version control. I
> >>> also believe this environment is becoming a non-trivial portion of the
> >>> Git userbase and it would be worthwhile looking into resolving.
> >>>
> >>> Solution Request
> >>> It would be fantastic if we could tell Git to stop caring about EOL
> >>> changes on a per-repo basis, with the effective output being that git
> >>> status shouldn't show changes for files with differing EOLs.
> >>>
> >>> I'm experienced with Git, though I am not expert enough to consider
> >>> creating such a change myself - It is unclear to me just how
> >>> complicated a change may be. However maybe I could look into it if it
> >>> was made clear that this improvement is possible and has no serious
> >>> side effects.
> >> Hej Scott,
> >> I think that you problem can be solved.
> >> For each repository, you can tell Git to ignore the line endings,
> >> CRLF vs LF.
> >>
> >> If you start with a fresh repo,
> >> you can do something like:
> >>
> >> echo "* text=auto" >.gitattributes
> >> git add .gitattributes
> >> git commit -m "Add .gitattributes"
> >>
> >> For existing repos, we need to take another step:
> >>
> >> echo "* text=auto" >.gitattributes
> >> git add .gitattributes
> >> git add  --renormlize .
> >> git commit -m "Add .gitattributes"
> >>
> >> More information can be found e.g. here:
> >> https://git-scm.com/docs/git-add
> >>
> >> Once you done that, you can merge branches
> >> into the master branch with help of the renormalize
> >>
> >> git merge -X renormalze branch-name
> >>
> >> See even here:
> >> https://git-scm.com/docs/git-merge
> >>
> >>
> >> This is just a (too) short introduction, I hope that it
> >> helps and that you find the time to dig somewhat deeper into
> >> the documentation.
> >>
> >> Other developers have that problem as well, you are not alone.
> >>
> >> If you have a public repo, I could help with one example.
> >>
> >>
> >>> Regards,
> >>>
> >>> Scott Richmond.
> >>>   Director, Producer, Programmer
> >>>   Brightrock Games
> >>>   T: 07480795661
>

Re: Ability to ignore EOL changes for certain projects

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 4270 bytes --]

On 2019-12-18 at 11:10:27, Scott Richmond wrote:
> The Problem Domain
> In certain dev environments (Unity3D projects) there is (AFAIK) an
> unsolvable problem where some files are often modified with line
> endings that aren't the native system or not the committed line
> endings for that file. Secondarily, in this case line endings don't
> matter - Nothing in the dev environment "cares" which kind of line
> ending is used.

I'm not sure this syncs up with your statements below.  Typically,
programs that do editing of text files either (a) write the line endings
that the file started out with or (b) write the system line endings.  If
you set the appropriate files with the `text` attribute in
`.gitattributes` (or use `* text=auto` if appropriate), then Git will
always write in the system line endings and convert internally to LF,
and programs that do either (a) or (b) will work.

Yet it sounds like you have development tools that don't do either (a)
or (b): they write in some fixed line ending, and therefore care very
much which line ending is being used.  In such a case, you could
write (for example) `*.yaml text eol=crlf` (or `eol=lf`) if you always
want them to do that and Git will convert to those line endings and
store in LF.

If the problem is text editor settings, you can use a `.editorconfig`
file, which is a cross-platform text editor configuration that can
specify line endings.  Most text editors can be configured to honor such
settings, although it may require a plugin.

If your problem is a shared Windows / Linux environment like WSL, you
can set `core.eol` to `crlf` in the repository and things will work.  If
you need settings like this, you can even set them up appropriately for
the system using a make target or bootstrap script so you don't need to
do that by hand on each system.

If you do this, then your tools will write the same line endings as are
checked out, and files won't appear modified.  You can see how Git
itself uses this to set up files appropriately for different systems.

The only case this wouldn't work is if the tools wrote some random line
endings depending on an attribute other than the OS they're on, or if
you had multiple tools doing different things.  If that's really your
problem, then yes, you'd need a new Git feature.  It is of course
possible to use a filter to strip out all carriage returns, but that
doesn't prevent Git from showing the file as being modified.

What I've proposed, of course, requires some setup work and
configuration.  It isn't trivial, but it does work for a lot of projects
already.

> Solution Request
> It would be fantastic if we could tell Git to stop caring about EOL
> changes on a per-repo basis, with the effective output being that git
> status shouldn't show changes for files with differing EOLs.
> 
> I'm experienced with Git, though I am not expert enough to consider
> creating such a change myself - It is unclear to me just how
> complicated a change may be. However maybe I could look into it if it
> was made clear that this improvement is possible and has no serious
> side effects.

I'm not sure how such a feature would interact with how Git operates
when it re-reads the index.  It isn't 100% clear to me when data is
filtered through various text filters such as EOL filters.  All the
filtering I've used is one-to-one, and therefore any modification of the
file contents necessarily means that the indexed contents have changed.

If Git does apply such filters when refreshing the index (such as
happens before `git status`), then such a feature would be relatively
easy to implement, although you'd incur a performance penalty when
changing the EOL of a file, even if the file were otherwise identical.
I suspect it would make the most sense as an additional value for
`core.autocrlf`.

If Git doesn't apply those filters, then there really isn't a way to do
what you want without fundamentally changing the characteristics of how
Git operates on the index, since it would still show files as modified.

Maybe someone else can comment on the feasibility of this better than I
can.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]

Re: Ability to ignore EOL changes for certain projects

[Scott Richmond]

> I'm not sure this syncs up with your statements below...[snip]

Unfortunately it is the latter case where certain dev apps, or plugins
within dev apps, that can occasionally write incorrect line endings.
We don't have configurable control over these things unfortunately.
If I widen the scope to all the scenarios you presented though - My
feature request could still have value as another way to alleviate the
pressure EOL changes create on a git repo. I would agree that it would
be best to solve the problem at its source where possible, but it is
sometimes not possible.

> I'm not sure how such a feature would interact with how Git operates when it re-reads the index...

AFAIK git status does not filter the results, as that would push
writes to files (I think?) when the user wouldn't expect it to. So it
makes sense that actions such as git status are unaffected by
text/crlf options.
I agree the additional cost to run a filter or similar action with git
status would be a cause for concern, but maybe an optional and
acceptable trade-off for the pain it alleviates elsewhere.

I do not know the underlying way git works either, but I suspect it
would have to be a 3rd test after sha1 & oid against the files that
come out of git status. I'm guessing that is an area of git that has
little configurability and therefore very difficult to make changes
to. :/

Regards,

Scott Richmond.
  Director, Producer, Programmer
  Brightrock Games
  T: 07480795661

On Fri, Dec 20, 2019 at 3:03 AM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2019-12-18 at 11:10:27, Scott Richmond wrote:
> > The Problem Domain
> > In certain dev environments (Unity3D projects) there is (AFAIK) an
> > unsolvable problem where some files are often modified with line
> > endings that aren't the native system or not the committed line
> > endings for that file. Secondarily, in this case line endings don't
> > matter - Nothing in the dev environment "cares" which kind of line
> > ending is used.
>
> I'm not sure this syncs up with your statements below.  Typically,
> programs that do editing of text files either (a) write the line endings
> that the file started out with or (b) write the system line endings.  If
> you set the appropriate files with the `text` attribute in
> `.gitattributes` (or use `* text=auto` if appropriate), then Git will
> always write in the system line endings and convert internally to LF,
> and programs that do either (a) or (b) will work.
>
> Yet it sounds like you have development tools that don't do either (a)
> or (b): they write in some fixed line ending, and therefore care very
> much which line ending is being used.  In such a case, you could
> write (for example) `*.yaml text eol=crlf` (or `eol=lf`) if you always
> want them to do that and Git will convert to those line endings and
> store in LF.
>
> If the problem is text editor settings, you can use a `.editorconfig`
> file, which is a cross-platform text editor configuration that can
> specify line endings.  Most text editors can be configured to honor such
> settings, although it may require a plugin.
>
> If your problem is a shared Windows / Linux environment like WSL, you
> can set `core.eol` to `crlf` in the repository and things will work.  If
> you need settings like this, you can even set them up appropriately for
> the system using a make target or bootstrap script so you don't need to
> do that by hand on each system.
>
> If you do this, then your tools will write the same line endings as are
> checked out, and files won't appear modified.  You can see how Git
> itself uses this to set up files appropriately for different systems.
>
> The only case this wouldn't work is if the tools wrote some random line
> endings depending on an attribute other than the OS they're on, or if
> you had multiple tools doing different things.  If that's really your
> problem, then yes, you'd need a new Git feature.  It is of course
> possible to use a filter to strip out all carriage returns, but that
> doesn't prevent Git from showing the file as being modified.
>
> What I've proposed, of course, requires some setup work and
> configuration.  It isn't trivial, but it does work for a lot of projects
> already.
>
> > Solution Request
> > It would be fantastic if we could tell Git to stop caring about EOL
> > changes on a per-repo basis, with the effective output being that git
> > status shouldn't show changes for files with differing EOLs.
> >
> > I'm experienced with Git, though I am not expert enough to consider
> > creating such a change myself - It is unclear to me just how
> > complicated a change may be. However maybe I could look into it if it
> > was made clear that this improvement is possible and has no serious
> > side effects.
>
> I'm not sure how such a feature would interact with how Git operates
> when it re-reads the index.  It isn't 100% clear to me when data is
> filtered through various text filters such as EOL filters.  All the
> filtering I've used is one-to-one, and therefore any modification of the
> file contents necessarily means that the indexed contents have changed.
>
> If Git does apply such filters when refreshing the index (such as
> happens before `git status`), then such a feature would be relatively
> easy to implement, although you'd incur a performance penalty when
> changing the EOL of a file, even if the file were otherwise identical.
> I suspect it would make the most sense as an additional value for
> `core.autocrlf`.
>
> If Git doesn't apply those filters, then there really isn't a way to do
> what you want without fundamentally changing the characteristics of how
> Git operates on the index, since it would still show files as modified.
>
> Maybe someone else can comment on the feasibility of this better than I
> can.
> --
> brian m. carlson: Houston, Texas, US
> OpenPGP: https://keybase.io/bk2204