'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Yuri]

In the FreeBSD ports repository I resurrected the directory (that was 
removed a long time ago) with the command:

 > $ git checkout {hash}~1 -- math/polymake


I made local changes to this directory and called 'git add math/polymake'.

Then 'git pull' complained:

 > $ git pull
 > error: Your local changes to the following files would be overwritten 
by merge:
 >   math/polymake/Makefile math/polymake/distinfo 
math/polymake/files/patch-Makefile 
math/polymake/files/patch-support_install.pl math/polymake/pkg-descr 
math/polymake/pkg-plist


No incoming changes affect math/polymake. Nobody has created this 
directory simultaneously with me. There is no intersection with incoming 
changes.


Why does 'git pull' complain then?



Yuri

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Jeff King]

On Tue, Aug 24, 2021 at 09:41:49AM -0700, Yuri wrote:

> In the FreeBSD ports repository I resurrected the directory (that was
> removed a long time ago) with the command:
> 
> > $ git checkout {hash}~1 -- math/polymake
> 
> 
> I made local changes to this directory and called 'git add math/polymake'.
> 
> Then 'git pull' complained:
> 
> > $ git pull
> > error: Your local changes to the following files would be overwritten by
> merge:
> >   math/polymake/Makefile math/polymake/distinfo
> math/polymake/files/patch-Makefile
> math/polymake/files/patch-support_install.pl math/polymake/pkg-descr
> math/polymake/pkg-plist
> 
> 
> No incoming changes affect math/polymake. Nobody has created this directory
> simultaneously with me. There is no intersection with incoming changes.
> 
> 
> Why does 'git pull' complain then?

It's hard to say without seeing a full example. The merge machinery is
supposed to handle this situation (and indeed, in a simple reproduction
example it does). So presumably it thinks the other side may have
touched those files for some reason. Perhaps it's confused by rename
detection?

Is it possible to give us a more complete example, including:

  - a url for the repository
  - the commit at HEAD when you ran "git checkout"
  - the {hash} commit from which you rescued the files
  - the state of the remote branch (i.e., what we attempted to merge
    with "git pull")

?

-Peff

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Bagas Sanjaya]

On 24/08/21 23.41, Yuri wrote:
> In the FreeBSD ports repository I resurrected the directory (that was 
> removed a long time ago) with the command:
> 
>  > $ git checkout {hash}~1 -- math/polymake
> 
> 
> I made local changes to this directory and called 'git add math/polymake'.
> 
> Then 'git pull' complained:
> 
>  > $ git pull
>  > error: Your local changes to the following files would be overwritten 
> by merge:
>  >   math/polymake/Makefile math/polymake/distinfo 
> math/polymake/files/patch-Makefile 
> math/polymake/files/patch-support_install.pl math/polymake/pkg-descr 
> math/polymake/pkg-plist
> 
> 
> No incoming changes affect math/polymake. Nobody has created this 
> directory simultaneously with me. There is no intersection with incoming 
> changes.

Have you committed or stashed such changes?

-- 
An old man doll... just what I always wanted! - Clara

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Yuri]

On 8/24/21 10:43 PM, Bagas Sanjaya wrote:
> Have you committed or stashed such changes? 


No, they were uncommitted and unstashed.


Yuri

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Elijah Newren]

On Tue, Aug 24, 2021 at 6:09 PM Jeff King <peff@peff.net> wrote:
>
> On Tue, Aug 24, 2021 at 09:41:49AM -0700, Yuri wrote:
>
> > In the FreeBSD ports repository I resurrected the directory (that was
> > removed a long time ago) with the command:
> >
> > > $ git checkout {hash}~1 -- math/polymake
> >
> >
> > I made local changes to this directory and called 'git add math/polymake'.
> >
> > Then 'git pull' complained:
> >
> > > $ git pull
> > > error: Your local changes to the following files would be overwritten by
> > merge:
> > >   math/polymake/Makefile math/polymake/distinfo
> > math/polymake/files/patch-Makefile
> > math/polymake/files/patch-support_install.pl math/polymake/pkg-descr
> > math/polymake/pkg-plist
> >
> >
> > No incoming changes affect math/polymake. Nobody has created this directory
> > simultaneously with me. There is no intersection with incoming changes.
> >
> >
> > Why does 'git pull' complain then?
>
> It's hard to say without seeing a full example. The merge machinery is
> supposed to handle this situation (and indeed, in a simple reproduction
> example it does). So presumably it thinks the other side may have
> touched those files for some reason. Perhaps it's confused by rename
> detection?
>
> Is it possible to give us a more complete example, including:
>
>   - a url for the repository
>   - the commit at HEAD when you ran "git checkout"
>   - the {hash} commit from which you rescued the files
>   - the state of the remote branch (i.e., what we attempted to merge
>     with "git pull")
>
> ?

The `git checkout {hash}~1 -- math/polymake` is enough to highlight
that Yuri doesn't just have local changes (which the merge machinery
should allow if the incoming changes don't touch the same files), but
local *staged* changes.  As per the merge manpage:

"""
To avoid recording unrelated changes in the merge commit, git pull and
git merge will also abort if there are any changes registered in the
index relative to the HEAD commit.
"""

While this particular example could theoretically be handled by the
merge machinery without requiring the index match HEAD, at least with
the new ort backend (it'd be way too complex with the recursive
backend, see below), in practice I'm so sick of index != HEAD bugs
that I have zero motivation to even consider it and I might even NAK
patches from others who attempted it.  For why, let me quote from the
commit message of 9822175d2b ("Ensure index matches head before
invoking merge machinery, round N", 2019-08-17), which also references
another commit that I'm tempted to also quote:

"""
Ensure index matches head before invoking merge machinery, round N

This is the bug that just won't die; there always seems to be another
form of it somewhere.  See the commit message of 55f39cf7551b ("merge:
fix misleading pre-merge check documentation", 2018-06-30) for a more
detailed explanation), but in short:

<quick summary>

builtin/merge.c contains this important requirement for merge
strategies:

    ...the index must be in sync with the head commit.  The strategies are
    responsible to ensure this.

This condition is important to enforce because there are two likely
failure cases when the index isn't in sync with the head commit:

  * we silently throw away changes the user had staged before the merge

  * we accidentally (and silently) include changes in the merge that
    were not part of either of the branches/trees being merged

Discarding users' work and mis-merging are both bad outcomes, especially
when done silently, so naturally this rule was stated sternly -- but,
unfortunately totally ignored in practice unless and until actual bugs
were found.  But, fear not: the bugs from this were fixed in commit
  ee6566e8d70d ("[PATCH] Rewrite read-tree", 2005-09-05)
through a rewrite of read-tree (again, commit 55f39cf7551b has a more
detailed explanation of how this affected merge).  And it was fixed
again in commit
  160252f81626 ("git-merge-ours: make sure our index matches HEAD", 2005-11-03)
...and it was fixed again in commit
  3ec62ad9ffba ("merge-octopus: abort if index does not match HEAD", 2016-04-09)
...and again in commit
  65170c07d466 ("merge-recursive: avoid incorporating uncommitted
changes in a merge", 2017-12-21)
...and again in commit
  eddd1a411d93 ("merge-recursive: enforce rule that index matches head
before merging", 2018-06-30)

...with multiple testcases added to the testsuite that could be
enumerated in even more commits.

Then, finally, in a patch in the same series as the last fix above, the
documentation about this requirement was fixed in commit 55f39cf7551b
("merge: fix misleading pre-merge check documentation", 2018-06-30), and
we all lived happily ever after...

</quick summary>

Unfortunately, "ever after" apparently denotes a limited time and it
expired today.
"""

and then the commit message goes on to explain multiple other bugs
found and fixed due to not strictly enforcing that the index needs to
match HEAD before starting a merge.

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Jeff King]

On Wed, Aug 25, 2021 at 08:42:09AM -0700, Elijah Newren wrote:

> > Is it possible to give us a more complete example, including:
> >
> >   - a url for the repository
> >   - the commit at HEAD when you ran "git checkout"
> >   - the {hash} commit from which you rescued the files
> >   - the state of the remote branch (i.e., what we attempted to merge
> >     with "git pull")
> >
> > ?
> 
> The `git checkout {hash}~1 -- math/polymake` is enough to highlight
> that Yuri doesn't just have local changes (which the merge machinery
> should allow if the incoming changes don't touch the same files), but
> local *staged* changes.  As per the merge manpage:
> 
> """
> To avoid recording unrelated changes in the merge commit, git pull and
> git merge will also abort if there are any changes registered in the
> index relative to the HEAD commit.
> """

Doh, you're right. I did a simple reproduction test, but in my example,
the merge was a fast-forward, which we do allow. Picking a branch more
carefully using:

  git for-each-ref --no-contains HEAD --no-merged HEAD refs/remotes/origin

shows the error. Sorry for the confusion.

-Peff

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> ... allow if the incoming changes don't touch the same files), but
> local *staged* changes.  As per the merge manpage:
>
> """
> To avoid recording unrelated changes in the merge commit, git pull and
> git merge will also abort if there are any changes registered in the
> index relative to the HEAD commit.
> """
>
> While this particular example could theoretically be handled by the
> merge machinery without requiring the index match HEAD,...

While I do not mind seeing a patch that loosens the condition ONLY
when the merge will cleanly auto-resolve without end-user
interaction, when any paths conflict and require editing by the
end-user, it is pretty much essential to require that the index
matches HEAD to keep "git merge" usable. 

This is because the final step to conclude such an "automated
procedure cannot cleanly resolve, so the end user helps resolving
with the editor and mark the resolved paths with 'git add' or 'git
rm'" session will become very error prone if we did not have the
requirement.  Not just the user MUST remember not to use "commit -a"
or "git add" a path that was already dirty in the working tree
before the merge started (which is the consequence of the current
requirement, which allows local changes to the unrelated working
tree files), they must MUST remember to somehow EXCLUDE the changes
already registered for unrelted paths from the concluded merge.

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Jeff King]

On Wed, Aug 25, 2021 at 02:19:15PM -0700, Junio C Hamano wrote:

> Elijah Newren <newren@gmail.com> writes:
> 
> > ... allow if the incoming changes don't touch the same files), but
> > local *staged* changes.  As per the merge manpage:
> >
> > """
> > To avoid recording unrelated changes in the merge commit, git pull and
> > git merge will also abort if there are any changes registered in the
> > index relative to the HEAD commit.
> > """
> >
> > While this particular example could theoretically be handled by the
> > merge machinery without requiring the index match HEAD,...
> 
> While I do not mind seeing a patch that loosens the condition ONLY
> when the merge will cleanly auto-resolve without end-user
> interaction, when any paths conflict and require editing by the
> end-user, it is pretty much essential to require that the index
> matches HEAD to keep "git merge" usable. 
> 
> This is because the final step to conclude such an "automated
> procedure cannot cleanly resolve, so the end user helps resolving
> with the editor and mark the resolved paths with 'git add' or 'git
> rm'" session will become very error prone if we did not have the
> requirement.  Not just the user MUST remember not to use "commit -a"
> or "git add" a path that was already dirty in the working tree
> before the merge started (which is the consequence of the current
> requirement, which allows local changes to the unrelated working
> tree files), they must MUST remember to somehow EXCLUDE the changes
> already registered for unrelted paths from the concluded merge.

Good point.

In theory we could reject the merge only after finding that there were
conflicts. That lets the happy path continue even with unrelated changes
(just like a fast-forward does). I suspect we'd need to change the
interface to the merge-backends, though (to say "do the merge, and bail
on conflicts, but _don't_ write out conflict markers or touch any other
state in that case").

(I am just thinking out loud, though. My personal opinion is that if you
have a bunch of staged changes and want to do any non-trivial merging,
you should considering committing or stashing those changes).

-Peff

Re: 'git pull' complains that a locally resurrected directory would be overwritten by merge when no pulled changes are affecting that directory

[Elijah Newren]

On Thu, Aug 26, 2021 at 6:05 PM Jeff King <peff@peff.net> wrote:
>
> On Wed, Aug 25, 2021 at 02:19:15PM -0700, Junio C Hamano wrote:
>
> > Elijah Newren <newren@gmail.com> writes:
> >
> > > ... allow if the incoming changes don't touch the same files), but
> > > local *staged* changes.  As per the merge manpage:
> > >
> > > """
> > > To avoid recording unrelated changes in the merge commit, git pull and
> > > git merge will also abort if there are any changes registered in the
> > > index relative to the HEAD commit.
> > > """
> > >
> > > While this particular example could theoretically be handled by the
> > > merge machinery without requiring the index match HEAD,...
> >
> > While I do not mind seeing a patch that loosens the condition ONLY
> > when the merge will cleanly auto-resolve without end-user
> > interaction, when any paths conflict and require editing by the
> > end-user, it is pretty much essential to require that the index
> > matches HEAD to keep "git merge" usable.
> >
> > This is because the final step to conclude such an "automated
> > procedure cannot cleanly resolve, so the end user helps resolving
> > with the editor and mark the resolved paths with 'git add' or 'git
> > rm'" session will become very error prone if we did not have the
> > requirement.  Not just the user MUST remember not to use "commit -a"
> > or "git add" a path that was already dirty in the working tree
> > before the merge started (which is the consequence of the current
> > requirement, which allows local changes to the unrelated working
> > tree files), they must MUST remember to somehow EXCLUDE the changes
> > already registered for unrelted paths from the concluded merge.
>
> Good point.
>
> In theory we could reject the merge only after finding that there were
> conflicts. That lets the happy path continue even with unrelated changes
> (just like a fast-forward does). I suspect we'd need to change the
> interface to the merge-backends, though (to say "do the merge, and bail
> on conflicts, but _don't_ write out conflict markers or touch any other
> state in that case").

We actually wouldn't need to change the interface.  One of the reasons
there were so many annoying bugs with index != HEAD, was that
builtin/merge.c stated that the merge backends were responsible for
enforcing that condition.  So every single merge backend had to be
individually fixed -- ours, octopus, resolve, recursive.  And then
some of those, like recursive, had multiple affected code paths, and
each one had to be fixed, and I didn't catch them all the first time.

That doesn't mean I'm in favor of the change, just pointing out that
one backend could decide to do this on its own even if the other
backends didn't support it.

> (I am just thinking out loud, though. My personal opinion is that if you
> have a bunch of staged changes and want to do any non-trivial merging,
> you should considering committing or stashing those changes).

Agreed.

Though perhaps I should have pointed out to Yuri that he wouldn't have
run into these problems if he would have used
    git restore --source={hash}~1 -- math/polymake
instead of
    git checkout {hash}~1 -- math/polymake

(since the former defaults to updating just the working directory,
while the later updates both the index and the working directory.)