[BUG] rebase --interactive silently overwrites ignored files

[BUG] rebase --interactive silently overwrites ignored files

[wh]

I'm using git 2.20.1 from Debian. Git is usually careful not to
overwrite untracked files, including ignored files. But interactive
rebase doesn't detect this (non-interactive rebase works fine).

Reproduction:
-----

#!/bin/sh
mkdir upstream
cd upstream
git init
echo 1 >feature-1
git add feature-1
git commit -m "feature 1"

cd ..
git clone upstream local
cd local
# write some tools for our own convenience
echo ours >tools
echo /tools >>.git/info/exclude
# start working on a feature
git checkout -b f2
echo wip >feature-2
git add feature-2
git commit -m "wip"

cd ../upstream
# official tools are available
echo theirs >tools
git add tools
git commit -m "tools"

cd ../local
git fetch ../upstream master

# this would be okay
#git rebase FETCH_HEAD

# problem: overwrites tools silently
GIT_EDITOR=true git rebase -i FETCH_HEAD

cat tools

-----

Expected: `git rebase -i` fails because it would have to overwrite the
untracked "tools" file. Contents of tools file remains `ours`.

Actual: Contents of tools file becomes `theirs`.

Re: [BUG] rebase --interactive silently overwrites ignored files

[Phillip Wood]

Hi

On 12/04/2019 00:56, wh wrote:
> I'm using git 2.20.1 from Debian. Git is usually careful not to
> overwrite untracked files, including ignored files.

Git normally overwrites ignored files, so I think in your example rebase 
-i is working as expected, I'm surprised that the am based rebase does 
not overwrite the ignored file. There has been some discussion about 
introducing 'precious' files that are ignored but protected in the same 
way as untracked files [1].

Best Wishes

Phillip

[1] https://public-inbox.org/git/20190409102649.22115-1-pclouds@gmail.com/

  But interactive
> rebase doesn't detect this (non-interactive rebase works fine).
> 
> Reproduction:
> -----
> 
> #!/bin/sh
> mkdir upstream
> cd upstream
> git init
> echo 1 >feature-1
> git add feature-1
> git commit -m "feature 1"
> 
> cd ..
> git clone upstream local
> cd local
> # write some tools for our own convenience
> echo ours >tools
> echo /tools >>.git/info/exclude
> # start working on a feature
> git checkout -b f2
> echo wip >feature-2
> git add feature-2
> git commit -m "wip"
> 
> cd ../upstream
> # official tools are available
> echo theirs >tools
> git add tools
> git commit -m "tools"
> 
> cd ../local
> git fetch ../upstream master
> 
> # this would be okay
> #git rebase FETCH_HEAD
> 
> # problem: overwrites tools silently
> GIT_EDITOR=true git rebase -i FETCH_HEAD
> 
> cat tools
> 
> -----
> 
> Expected: `git rebase -i` fails because it would have to overwrite the
> untracked "tools" file. Contents of tools file remains `ours`.
> 
> Actual: Contents of tools file becomes `theirs`.
> 

Re: [BUG] rebase --interactive silently overwrites ignored files

[wh]

Thanks for the info about the upcoming "precious" attribute. Looks useful.

I didn't get the impression that Git normally overwrites ignored
files. I ran some more experiments:

git rebase FETCH_HEAD        # bails
git rebase -i FETCH_HEAD     # overwrites
git merge FETCH_HEAD         # bails
git reset --keep FETCH_HEAD  # bails
git reset --merge FETCH_HEAD # bails
git checkout FETCH_HEAD      # overwrites
# without feature 2 locally:
git merge FETCH_HEAD         # overwrites, fast-forwards
git merge --no-ff FETCH_HEAD # bails

On Fri, Apr 12, 2019 at 9:30 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
>
> Hi
>
> On 12/04/2019 00:56, wh wrote:
> > I'm using git 2.20.1 from Debian. Git is usually careful not to
> > overwrite untracked files, including ignored files.
>
> Git normally overwrites ignored files, so I think in your example rebase
> -i is working as expected, I'm surprised that the am based rebase does
> not overwrite the ignored file. There has been some discussion about
> introducing 'precious' files that are ignored but protected in the same
> way as untracked files [1].
>
> Best Wishes
>
> Phillip
>
> [1] https://public-inbox.org/git/20190409102649.22115-1-pclouds@gmail.com/
>
>   But interactive
> > rebase doesn't detect this (non-interactive rebase works fine).
> >
> > Reproduction:
> > -----
> >
> > #!/bin/sh
> > mkdir upstream
> > cd upstream
> > git init
> > echo 1 >feature-1
> > git add feature-1
> > git commit -m "feature 1"
> >
> > cd ..
> > git clone upstream local
> > cd local
> > # write some tools for our own convenience
> > echo ours >tools
> > echo /tools >>.git/info/exclude
> > # start working on a feature
> > git checkout -b f2
> > echo wip >feature-2
> > git add feature-2
> > git commit -m "wip"
> >
> > cd ../upstream
> > # official tools are available
> > echo theirs >tools
> > git add tools
> > git commit -m "tools"
> >
> > cd ../local
> > git fetch ../upstream master
> >
> > # this would be okay
> > #git rebase FETCH_HEAD
> >
> > # problem: overwrites tools silently
> > GIT_EDITOR=true git rebase -i FETCH_HEAD
> >
> > cat tools
> >
> > -----
> >
> > Expected: `git rebase -i` fails because it would have to overwrite the
> > untracked "tools" file. Contents of tools file remains `ours`.
> >
> > Actual: Contents of tools file becomes `theirs`.
> >

Re: [BUG] rebase --interactive silently overwrites ignored files

[Phillip Wood]

On 14/04/2019 02:59, wh wrote:
> Thanks for the info about the upcoming "precious" attribute. Looks useful.

Hmm, unfortunately it's not looking so hopeful now [1]

> I didn't get the impression that Git normally overwrites ignored
> files. I ran some more experiments:
> 
> git rebase FETCH_HEAD        # bails
> git rebase -i FETCH_HEAD     # overwrites
> git merge FETCH_HEAD         # bails
> git reset --keep FETCH_HEAD  # bails
> git reset --merge FETCH_HEAD # bails
> git checkout FETCH_HEAD      # overwrites
> # without feature 2 locally:
> git merge FETCH_HEAD         # overwrites, fast-forwards
> git merge --no-ff FETCH_HEAD # bails

Thanks for sharing that, I was assuming the other commands behaved like 
checkout, I'm intrigued that merge behaves differently to rebase -i as 
they use the same underlying machinery, I'm a bit snowed under at the 
moment but I'll try and take a look at what's going sometime in the not 
too distant future.

Best Wishes

Phillip

[1] 
https://public-inbox.org/git/CACsJy8AEZ-Lz6zgEsuNukvphB9TTa9FAC1gK05fhnie2xtfc9w@mail.gmail.com/T/#u
> On Fri, Apr 12, 2019 at 9:30 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
>>
>> Hi
>>
>> On 12/04/2019 00:56, wh wrote:
>>> I'm using git 2.20.1 from Debian. Git is usually careful not to
>>> overwrite untracked files, including ignored files.
>>
>> Git normally overwrites ignored files, so I think in your example rebase
>> -i is working as expected, I'm surprised that the am based rebase does
>> not overwrite the ignored file. There has been some discussion about
>> introducing 'precious' files that are ignored but protected in the same
>> way as untracked files [1].
>>
>> Best Wishes
>>
>> Phillip
>>
>> [1] https://public-inbox.org/git/20190409102649.22115-1-pclouds@gmail.com/
>>
>>    But interactive
>>> rebase doesn't detect this (non-interactive rebase works fine).
>>>
>>> Reproduction:
>>> -----
>>>
>>> #!/bin/sh
>>> mkdir upstream
>>> cd upstream
>>> git init
>>> echo 1 >feature-1
>>> git add feature-1
>>> git commit -m "feature 1"
>>>
>>> cd ..
>>> git clone upstream local
>>> cd local
>>> # write some tools for our own convenience
>>> echo ours >tools
>>> echo /tools >>.git/info/exclude
>>> # start working on a feature
>>> git checkout -b f2
>>> echo wip >feature-2
>>> git add feature-2
>>> git commit -m "wip"
>>>
>>> cd ../upstream
>>> # official tools are available
>>> echo theirs >tools
>>> git add tools
>>> git commit -m "tools"
>>>
>>> cd ../local
>>> git fetch ../upstream master
>>>
>>> # this would be okay
>>> #git rebase FETCH_HEAD
>>>
>>> # problem: overwrites tools silently
>>> GIT_EDITOR=true git rebase -i FETCH_HEAD
>>>
>>> cat tools
>>>
>>> -----
>>>
>>> Expected: `git rebase -i` fails because it would have to overwrite the
>>> untracked "tools" file. Contents of tools file remains `ours`.
>>>
>>> Actual: Contents of tools file becomes `theirs`.
>>>

Re: [BUG] rebase --interactive silently overwrites ignored files

[Phillip Wood]

Hi wh

On 14/04/2019 02:59, wh wrote:
> Thanks for the info about the upcoming "precious" attribute. Looks useful.
> 
> I didn't get the impression that Git normally overwrites ignored
> files. I ran some more experiments:
> 
> git rebase FETCH_HEAD        # bails
> git rebase -i FETCH_HEAD     # overwrites
> git merge FETCH_HEAD         # bails
> git reset --keep FETCH_HEAD  # bails
> git reset --merge FETCH_HEAD # bails
> git checkout FETCH_HEAD      # overwrites
> # without feature 2 locally:
> git merge FETCH_HEAD         # overwrites, fast-forwards
> git merge --no-ff FETCH_HEAD # bails

I've had a look at the rebase -i code and I think it only overwrites 
ignored files when it is fast-forwarding. This matches what merge does 
when fast-forwarding but I'm not convinced either of them should be 
doing this by default (I think checkout doing it is probably asking for 
trouble), as you point out most operations preserve ignored files. When 
pull fast-forwards it does not overwrite ignored files.

Both checkout and merge have an undocumented --overwrite-ignore option 
which is on by default. Checkout and fast-forward merges overwriting 
ignored files dates back a long way to commit 1127148089 ("Loosen 
"working file will be lost" check in Porcelain-ish", 2006-12-04). There 
does not seem to have been much discussion of changing the semantics of 
ignored files around the patches that implemented this [1,2]. There is a 
brief mention of doing it in this thread [3] but no one seems to comment 
on the idea as far as I can see. Before this change ignored files were 
just ignored by status and add but otherwise treated like any other 
untracked file.

I'll put a patch together to fix rebase -i, I'd like to see the defaults 
for merge and checkout changed but I'm not sure that would be popular. 
It does seem like surprising behavior though when most operations try to 
preserve untracked files.

Best Wishes

Phillip

[1] 
https://public-inbox.org/git/7vlklnkv39.fsf@assigned-by-dhcp.cox.net/T/#u
[2] 
https://public-inbox.org/git/7vbqmjkuzd.fsf@assigned-by-dhcp.cox.net/T/#u
[3]https://public-inbox.org/git/Pine.LNX.4.64.0610081657400.3952@g5.osdl.org/

> On Fri, Apr 12, 2019 at 9:30 AM Phillip Wood <phillip.wood123@gmail.com> wrote:
>>
>> Hi
>>
>> On 12/04/2019 00:56, wh wrote:
>>> I'm using git 2.20.1 from Debian. Git is usually careful not to
>>> overwrite untracked files, including ignored files.
>>
>> Git normally overwrites ignored files, so I think in your example rebase
>> -i is working as expected, I'm surprised that the am based rebase does
>> not overwrite the ignored file. There has been some discussion about
>> introducing 'precious' files that are ignored but protected in the same
>> way as untracked files [1].
>>
>> Best Wishes
>>
>> Phillip
>>
>> [1] https://public-inbox.org/git/20190409102649.22115-1-pclouds@gmail.com/
>>
>>    But interactive
>>> rebase doesn't detect this (non-interactive rebase works fine).
>>>
>>> Reproduction:
>>> -----
>>>
>>> #!/bin/sh
>>> mkdir upstream
>>> cd upstream
>>> git init
>>> echo 1 >feature-1
>>> git add feature-1
>>> git commit -m "feature 1"
>>>
>>> cd ..
>>> git clone upstream local
>>> cd local
>>> # write some tools for our own convenience
>>> echo ours >tools
>>> echo /tools >>.git/info/exclude
>>> # start working on a feature
>>> git checkout -b f2
>>> echo wip >feature-2
>>> git add feature-2
>>> git commit -m "wip"
>>>
>>> cd ../upstream
>>> # official tools are available
>>> echo theirs >tools
>>> git add tools
>>> git commit -m "tools"
>>>
>>> cd ../local
>>> git fetch ../upstream master
>>>
>>> # this would be okay
>>> #git rebase FETCH_HEAD
>>>
>>> # problem: overwrites tools silently
>>> GIT_EDITOR=true git rebase -i FETCH_HEAD
>>>
>>> cat tools
>>>
>>> -----
>>>
>>> Expected: `git rebase -i` fails because it would have to overwrite the
>>> untracked "tools" file. Contents of tools file remains `ours`.
>>>
>>> Actual: Contents of tools file becomes `theirs`.
>>>

Re: [BUG] rebase --interactive silently overwrites ignored files

[Duy Nguyen]

On Thu, May 2, 2019 at 10:45 PM Phillip Wood <phillip.wood123@gmail.com> wrote:
>
> Hi wh
>
> On 14/04/2019 02:59, wh wrote:
> > Thanks for the info about the upcoming "precious" attribute. Looks useful.
> >
> > I didn't get the impression that Git normally overwrites ignored
> > files.

Please do not include me in these discussions again. Thank you.
-- 
Duy

Re: [BUG] rebase --interactive silently overwrites ignored files

[Junio C Hamano]

Phillip Wood <phillip.wood123@gmail.com> writes:

> I've had a look at the rebase -i code and I think it only overwrites
> ignored files when it is fast-forwarding. This matches what merge does
> when fast-forwarding but I'm not convinced either of them should be
> doing this by default (I think checkout doing it is probably asking
> for trouble)...

> I'll put a patch together to fix rebase -i, I'd like to see the
> defaults for merge and checkout changed but I'm not sure that would be
> popular.

I suspect that such a change in behaviour for "rebase -i" is not a
fix but a regression.  Shouldn't it be consistent with checkout and
merge?

> It does seem like surprising behavior though when most
> operations try to preserve untracked files.

Are you conflating untracked and ignored?

Because we haven't adopted 'precious' (or whatever the final name
would be), which is "ignored but not expendable", ignored files are
by definition "ignored and expendable".  When checkout (and merge
that is its more general form) needs to match the working tree to
the index contents and an ingored file is in the way, it should
overwrite it.  

Until we introduce "ignored but not expendable" class, that is.

Re: [BUG] rebase --interactive silently overwrites ignored files

[Phillip Wood]

Hi Junio

On 05/05/2019 05:02, Junio C Hamano wrote:
> Phillip Wood <phillip.wood123@gmail.com> writes:
> 
>> I've had a look at the rebase -i code and I think it only overwrites
>> ignored files when it is fast-forwarding. This matches what merge does
>> when fast-forwarding but I'm not convinced either of them should be
>> doing this by default (I think checkout doing it is probably asking
>> for trouble)...
> 
>> I'll put a patch together to fix rebase -i, I'd like to see the
>> defaults for merge and checkout changed but I'm not sure that would be
>> popular.
> 
> I suspect that such a change in behaviour for "rebase -i" is not a
> fix but a regression.  Shouldn't it be consistent with checkout and
> merge?

My problem is that I don't really understand why some operations should 
overwrite ignored files and others shouldn't. Having nearly all rebase 
operations preserve ignored files and only overwriting them for a 
fast-forward seems confusing and probably surprising to users. Is it a 
bug that pull does not overwrite ignored files when fast-forwarding but 
merge does? Why is it ok to overwrite ignored files when merge 
fast-forwards but not otherwise?

>> It does seem like surprising behavior though when most
>> operations try to preserve untracked files.
> 
> Are you conflating untracked and ignored?
> 
> Because we haven't adopted 'precious' (or whatever the final name
> would be), which is "ignored but not expendable", ignored files are
> by definition "ignored and expendable".  

My confusion is that they are only "ignored and expendable" in certain 
circumstances. Many git operations actually fail if they are going to 
overwrite an ignored file [1] - I don't understand if that is 
intentional or not.

Best Wishes

Phillip

[1] 
https://public-inbox.org/git/CAL_tzDFQQtDYMStN+RDVYN_TzJmO+kufMhG9PGHwvsUWREpgWQ@mail.gmail.com/

> When checkout (and merge
> that is its more general form) needs to match the working tree to
> the index contents and an ingored file is in the way, it should
> overwrite it.
> 
> Until we introduce "ignored but not expendable" class, that is.
>