git merge deletes my changes

git merge deletes my changes

[Eduard Egorov]

Hello Git experts,

A week ago, I've reset a state of 'ceph-ansible' folder in %current% branch with code from corresponding branch (that tracks an upstream from github):

# git read-tree --prefix=ceph-ansible/ -u ceph_ansible

Then I've committed several changes, including:

1. Renamed file and commited:
# git mv site.yml.sample site.yml

2. Made some changes and committed

3. Pulled updates from original branch by:
# git merge -s subtree --squash ceph_ansible

It said:
    Auto-merging ceph-ansible/site.yml.sample
    blablabla
    Squash commit -- not updating HEAD
    Automatic merge went well; stopped before committing as requested

I can see that "my" ceph-ansible/site.yml is deleted (as well as few new files added by me), ceph-ansible/site.yml.sample is restored and doesn't contain my changes (it's just restored to the state before my changes).
`git log ceph_ansible site.yml.sample` shows the latest change on ceph_ansible branch done from 26th of August, 2016, my changes in %current% branch was this October, so it shouldn't be any conflict either (even if it was).
I believe there is some obvious explanation for this behavior. Could you help me please? What information can  I provide in order to troubleshoot this issue? 

A post on SO: http://stackoverflow.com/questions/39954265/git-merge-deletes-my-changes

With best regards
Eduard Egorov

RE: git merge deletes my changes

[Eduard Egorov]

Hello again,

I've noticed that my git is quite old (1.8.3) and built it from source tarball (2.10.1). Now the output is:

# ~/gitbuild/git-2.10.1/git merge -s subtree --squash ceph_ansible
fatal: refusing to merge unrelated histories

I've checked my command history again:
  738  git rm -rf ceph-ansible/
  739  ll
  740  ll ceph-ansible/
  741  rm ceph-ansible/purge_ceph_cluster.yml 
  742  git read-tree --prefix=ceph-ansible/ -u ceph_ansible
  743  gs
  744  gc "ceph-ansible: reset repo state"

A quick googling showed (http://stackoverflow.com/questions/37937984/git-refusing-to-merge-unrelated-histories ) that the default behavior is changed. Adding ' --allow-unrelated-histories' clears the error message but the merge itself is still wrong (my changes are lost). 

This error message might explain why git can't merge it correctly (since these repos doesn't has any relations, right). Can somebody confirm this please? Doesn't "merge -s subtree" really merges branches?

With best regards
Eduard Egorov

-----Original Message-----
From: Eduard Egorov 
Sent: Monday, October 10, 2016 12:39 PM
To: 'git@vger.kernel.org' <git@vger.kernel.org>
Subject: git merge deletes my changes

Hello Git experts,

A week ago, I've reset a state of 'ceph-ansible' folder in %current% branch with code from corresponding branch (that tracks an upstream from github):

# git read-tree --prefix=ceph-ansible/ -u ceph_ansible

Then I've committed several changes, including:

1. Renamed file and commited:
# git mv site.yml.sample site.yml

2. Made some changes and committed

3. Pulled updates from original branch by:
# git merge -s subtree --squash ceph_ansible

It said:
    Auto-merging ceph-ansible/site.yml.sample
    blablabla
    Squash commit -- not updating HEAD
    Automatic merge went well; stopped before committing as requested

I can see that "my" ceph-ansible/site.yml is deleted (as well as few new files added by me), ceph-ansible/site.yml.sample is restored and doesn't contain my changes (it's just restored to the state before my changes).
`git log ceph_ansible site.yml.sample` shows the latest change on ceph_ansible branch done from 26th of August, 2016, my changes in %current% branch was this October, so it shouldn't be any conflict either (even if it was).
I believe there is some obvious explanation for this behavior. Could you help me please? What information can  I provide in order to troubleshoot this issue? 

A post on SO: http://stackoverflow.com/questions/39954265/git-merge-deletes-my-changes

With best regards
Eduard Egorov

Re: git merge deletes my changes

[Jeff King]

On Mon, Oct 10, 2016 at 09:39:13AM +0000, Eduard Egorov wrote:

> A week ago, I've reset a state of 'ceph-ansible' folder in %current%
> branch with code from corresponding branch (that tracks an upstream
> from github):
> 
> # git read-tree --prefix=ceph-ansible/ -u ceph_ansible

This pulls in the subtree files, but there's no actual relationship with
the commit history in ceph_ansible.

So later...

> Then I've committed several changes, including:
> 
> 1. Renamed file and commited:
> # git mv site.yml.sample site.yml
> 
> 2. Made some changes and committed
> 
> 3. Pulled updates from original branch by:
> # git merge -s subtree --squash ceph_ansible
> 
> It said:
>     Auto-merging ceph-ansible/site.yml.sample
>     blablabla
>     Squash commit -- not updating HEAD
>     Automatic merge went well; stopped before committing as requested

When you merge from ceph_ansible, there is no shared history, and git
uses the empty tree as a common ancestor. It looks like the other side
added site.yml.sample, for instance, because that is a change from the
empty tree.

> A post on SO: http://stackoverflow.com/questions/39954265/git-merge-deletes-my-changes

As you noted on SO, modern git disallows merges of unrelated history by
default, because it's usually a mistake to do so.

If you are doing repeated merges into the subtree, you need to somehow
tell git how the histories are related. The obvious answer is to do a
"git merge -s ours ceph_ansible" after your initial read-tree, so that
git knows you've pulled in the changes up to that point. But I'd guess
from your use of "--squash" that you don't want to carry the
ceph_ansible history in your project.

So you need to record the original upstream commit somewhere (probably
in the commit message when you commit the read-tree result), and then
ask git to use that as the merge-base during subsequent merges (which
will require using plumbing codes, as git-merge wants to compute the
merge base itself).  I believe the git-subtree command (in
contrib/subtree of git.git) handles this use case, but I haven't used it
myself.

-Peff

Re: git merge deletes my changes

[Paul Smith]

On Mon, 2016-10-10 at 10:19 +0000, Eduard Egorov wrote:
> # ~/gitbuild/git-2.10.1/git merge -s subtree --squash ceph_ansible
> 
> Can somebody confirm this please? Doesn't "merge -s subtree" really
> merges branches?

I think possibly you're not fully understanding what the --squash flag
does... that's what's causing your issue here, not the "-s" option.

A squash merge takes the commits that would be merged from the origin
branch and squashes them into a single patch and applies them to the
current branch as a new commit... but this new commit is not a merge
commit (that is, when you look at it with "git show" etc. the commit
will have only one parent, not two--or more--parents like a normal merge
commit).

Basically, it's syntactic sugar for a diff plus patch operation plus
some Git goodness wrapped around it to make it easier to use.

But ultimately once you're done, Git has no idea that this new commit
has any relationship whatsoever to the origin branch.  So the next time
you merge, Git doesn't know that there was a previous merge and it will
try to merge everything from scratch rather than starting at the
previous common merge point.

So either you'll have to use a normal, non-squash merge, or else you'll
have to tell Git by hand what the previous common merge point was (as
Jeff King's excellent email suggests).  Or else, you'll have to live
with this behavior.

RE: git merge deletes my changes

[Eduard Egorov]

Hello Jeff, Paul

Thanks a lot for sharing your time!

As you correctly stated, I used  "--squash" for preventing my project history being flooded with the ceph_ansible's commits and  I didn't really understand that this is critical for having relationship history with other repository.

Also I realized that made another big mistake - I should use another branch (that is directly branched from 'ceph-ansible' one) for storing my ceph-ansible's code updates. And, periodically, this branch will be merged into my main repo with '-s subtree --squash' without worrying about merge conflicts since this folder would never be changed in main repo's branches.

Again - thank you very much for your help!

P.S. Please feel free to copy-paste this thread as an answer on SO (http://stackoverflow.com/questions/39954265/git-merge-s-subtree-works-incorrectly ) - I will accept it with great pleasure ;-)

With best regards
Eduard Egorov

-----Original Message-----
From: Paul Smith [mailto:paul@mad-scientist.net] 
Sent: Monday, October 10, 2016 8:52 PM
To: Eduard Egorov <Eduard.Egorov@icl-services.com>; 'git@vger.kernel.org' <git@vger.kernel.org>
Subject: Re: git merge deletes my changes

On Mon, 2016-10-10 at 10:19 +0000, Eduard Egorov wrote:
> # ~/gitbuild/git-2.10.1/git merge -s subtree --squash ceph_ansible
> 
> Can somebody confirm this please? Doesn't "merge -s subtree" really 
> merges branches?

I think possibly you're not fully understanding what the --squash flag does... that's what's causing your issue here, not the "-s" option.

A squash merge takes the commits that would be merged from the origin branch and squashes them into a single patch and applies them to the current branch as a new commit... but this new commit is not a merge commit (that is, when you look at it with "git show" etc. the commit will have only one parent, not two--or more--parents like a normal merge commit).

Basically, it's syntactic sugar for a diff plus patch operation plus some Git goodness wrapped around it to make it easier to use.

But ultimately once you're done, Git has no idea that this new commit has any relationship whatsoever to the origin branch.  So the next time you merge, Git doesn't know that there was a previous merge and it will try to merge everything from scratch rather than starting at the previous common merge point.

So either you'll have to use a normal, non-squash merge, or else you'll have to tell Git by hand what the previous common merge point was (as Jeff King's excellent email suggests).  Or else, you'll have to live with this behavior.

-----Original Message-----
From: Jeff King [mailto:peff@peff.net] 
Sent: Monday, October 10, 2016 6:26 PM
To: Eduard Egorov <Eduard.Egorov@icl-services.com>
Cc: 'git@vger.kernel.org' <git@vger.kernel.org>
Subject: Re: git merge deletes my changes

On Mon, Oct 10, 2016 at 09:39:13AM +0000, Eduard Egorov wrote:

> A week ago, I've reset a state of 'ceph-ansible' folder in %current% 
> branch with code from corresponding branch (that tracks an upstream 
> from github):
> 
> # git read-tree --prefix=ceph-ansible/ -u ceph_ansible

This pulls in the subtree files, but there's no actual relationship with the commit history in ceph_ansible.

So later...

> Then I've committed several changes, including:
> 
> 1. Renamed file and commited:
> # git mv site.yml.sample site.yml
> 
> 2. Made some changes and committed
> 
> 3. Pulled updates from original branch by:
> # git merge -s subtree --squash ceph_ansible
> 
> It said:
>     Auto-merging ceph-ansible/site.yml.sample
>     blablabla
>     Squash commit -- not updating HEAD
>     Automatic merge went well; stopped before committing as requested

When you merge from ceph_ansible, there is no shared history, and git uses the empty tree as a common ancestor. It looks like the other side added site.yml.sample, for instance, because that is a change from the empty tree.

> A post on SO: 
> http://stackoverflow.com/questions/39954265/git-merge-deletes-my-chang
> es

As you noted on SO, modern git disallows merges of unrelated history by default, because it's usually a mistake to do so.

If you are doing repeated merges into the subtree, you need to somehow tell git how the histories are related. The obvious answer is to do a "git merge -s ours ceph_ansible" after your initial read-tree, so that git knows you've pulled in the changes up to that point. But I'd guess from your use of "--squash" that you don't want to carry the ceph_ansible history in your project.

So you need to record the original upstream commit somewhere (probably in the commit message when you commit the read-tree result), and then ask git to use that as the merge-base during subsequent merges (which will require using plumbing codes, as git-merge wants to compute the merge base itself).  I believe the git-subtree command (in contrib/subtree of git.git) handles this use case, but I haven't used it myself.

-Peff

Re: git merge deletes my changes

[Jakub Narębski]

W dniu 10.10.2016 o 19:52, Paul Smith pisze:
> On Mon, 2016-10-10 at 10:19 +0000, Eduard Egorov wrote:
>> # ~/gitbuild/git-2.10.1/git merge -s subtree --squash ceph_ansible
>>
>> Can somebody confirm this please? Doesn't "merge -s subtree" really
>> merges branches?
> 
> I think possibly you're not fully understanding what the --squash flag
> does... that's what's causing your issue here, not the "-s" option.
> 
> A squash merge takes the commits that would be merged from the origin
> branch and squashes them into a single patch and applies them to the
> current branch as a new commit... but this new commit is not a merge
> commit (that is, when you look at it with "git show" etc. the commit
> will have only one parent, not two--or more--parents like a normal merge
> commit).
> 
> Basically, it's syntactic sugar for a diff plus patch operation plus
> some Git goodness wrapped around it to make it easier to use.

Actually this is full merge + commit surgery (as if you did merge with
--no-commit, then deleted MERGE_HEAD); the state of worktree is as if
it were after a merge.

> 
> But ultimately once you're done, Git has no idea that this new commit
> has any relationship whatsoever to the origin branch.  So the next time
> you merge, Git doesn't know that there was a previous merge and it will
> try to merge everything from scratch rather than starting at the
> previous common merge point.
> 
> So either you'll have to use a normal, non-squash merge, or else you'll
> have to tell Git by hand what the previous common merge point was (as
> Jeff King's excellent email suggests).  Or else, you'll have to live
> with this behavior.

The `git subtree` command (from contrib) allows yet another way: it
squashes *history* of merged subproject (as if with interactive rebase
'squash'), then merges this squash commit.

Now I know why this feature is here...
-- 
Jakub Narębski

RE: git merge deletes my changes

[Eduard Egorov]

Hello Jakub,

Thank you for addition. Eventually, I've merged your explanations into single answer post (http://stackoverflow.com/questions/39954265/git-merge-s-subtree-works-incorrectly/ ). I hope this will prevent other confused people from disturbing you by similar emails on this mailing list.

This is another time I can evidence the power and flexibility the git provides, thank you all for your great work!

With best regards
Eduard Egorov

-----Original Message-----
From: Jakub Narębski [mailto:jnareb@gmail.com] 
Sent: Tuesday, October 11, 2016 6:57 PM
To: Paul Smith; Eduard Egorov; 'git@vger.kernel.org'
Subject: Re: git merge deletes my changes

W dniu 10.10.2016 o 19:52, Paul Smith pisze:
> On Mon, 2016-10-10 at 10:19 +0000, Eduard Egorov wrote:
>> # ~/gitbuild/git-2.10.1/git merge -s subtree --squash ceph_ansible
>>
>> Can somebody confirm this please? Doesn't "merge -s subtree" really 
>> merges branches?
> 
> I think possibly you're not fully understanding what the --squash flag 
> does... that's what's causing your issue here, not the "-s" option.
> 
> A squash merge takes the commits that would be merged from the origin 
> branch and squashes them into a single patch and applies them to the 
> current branch as a new commit... but this new commit is not a merge 
> commit (that is, when you look at it with "git show" etc. the commit 
> will have only one parent, not two--or more--parents like a normal 
> merge commit).
> 
> Basically, it's syntactic sugar for a diff plus patch operation plus 
> some Git goodness wrapped around it to make it easier to use.

Actually this is full merge + commit surgery (as if you did merge with --no-commit, then deleted MERGE_HEAD); the state of worktree is as if it were after a merge.

> 
> But ultimately once you're done, Git has no idea that this new commit 
> has any relationship whatsoever to the origin branch.  So the next 
> time you merge, Git doesn't know that there was a previous merge and 
> it will try to merge everything from scratch rather than starting at 
> the previous common merge point.
> 
> So either you'll have to use a normal, non-squash merge, or else 
> you'll have to tell Git by hand what the previous common merge point 
> was (as Jeff King's excellent email suggests).  Or else, you'll have 
> to live with this behavior.

The `git subtree` command (from contrib) allows yet another way: it squashes *history* of merged subproject (as if with interactive rebase 'squash'), then merges this squash commit.

Now I know why this feature is here...
--
Jakub Narębski