"git subtree --squash" interacts poorly with revert, merge, and rebase

"git subtree --squash" interacts poorly with revert, merge, and rebase

[Matt McCutchen]

I'm the lead developer of a research software application (https://bitb
ucket.org/objsheets/objsheets) that uses modified versions of two
third-party libraries, which we need to version and distribute along
with our application.  For better or for worse, we haven't made it a
priority to upstream our changes, so for now we just want to optimize
for ease of (1) making and reviewing changes and (2) upgrading to newer
upstream versions.

We've been using git submodules, but that's a pain for several reasons:
- We have to run "git submodule update" manually.
- We have to make separate commits and manage corresponding topic
branches for the superproject and subprojects.
- A diff of the superproject doesn't include the content of
subprojects.

Recently I looked into switching to the "git subtree" contrib tool in
the --squash mode, but I identified a few drawbacks compared to
submodules:

1. The upstream commit on which the subtree is based is assumed to be
given by the latest squash commit in "git log".  This means that (i) a
change to a different upstream commit can't be reverted with "git
revert" and (ii) a "git merge" of two superproject branches based on
different upstream commits may successfully merge the content of the
upstream commits but leave the tool thinking the subtree is based on an
arbitrary one of the two commits.

2. Rebasing messes up the merge commits generated by "git subtree --
squash".  --preserve-merges worked in a simple test but supposedly
doesn't work if there are conflicts or I want to reorder commits with
--interactive.

Maybe we would never hit any of these problems in practice, but they
give me a bad enough feeling that I'm planning to write my own tool
that tracks the upstream commit ID in a file (like a submodule) and
doesn't generate any extra commits.  Without generating extra commits,
the only place to store the upstream content in the superproject would
be in another subtree, which would take up disk space in every working
tree unless developers manually set skip-worktree.  I think I prefer to
not store the upstream content and just have the tool fetch it from a
local subproject repository each time it's needed.

I'll of course post the tool on the web and would be happy to see it
integrated into "git subtree" if that makes sense, but I don't know how
much time I'd be willing to put into making that happen.

Any advice?

Thanks,
Matt

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Stefan Beller]

On Wed, Oct 26, 2016 at 4:07 PM, Matt McCutchen <matt@mattmccutchen.net> wrote:
> I'm the lead developer of a research software application (https://bitb
> ucket.org/objsheets/objsheets) that uses modified versions of two
> third-party libraries, which we need to version and distribute along
> with our application.  For better or for worse, we haven't made it a
> priority to upstream our changes, so for now we just want to optimize
> for ease of (1) making and reviewing changes and (2) upgrading to newer
> upstream versions.
>
> We've been using git submodules, but that's a pain for several reasons:
> - We have to run "git submodule update" manually.

That is true for now. :( But there are plans to revive a patch series
to include updating the submodule into git-checkout.

https://github.com/stefanbeller/git/commits/submodule-co

> - We have to make separate commits and manage corresponding topic
> branches for the superproject and subprojects.

Well yeah, that is how submodule work on a conceptual level.
While having multiple commits may seem like overhead, note
the subtle difference for these commits. One if deep down in the
stack patching one of the submodules, the other is a high level
commit advancing the submodule pointer.

Note that the target audience of these two commit messages
might be vastly different, hence can be worded differently.
(The submodule describing how you fixed e.g. a memleak or race condition
and the superproject describes on why you needed to include that submodule,
e.g. because you switched your toplevel application to use threads.)

> - A diff of the superproject doesn't include the content of
> subprojects.

A recent patch series by Jacob Keller (jk/diff-submodule-diff-inline)
taught git diff to show the diff in the submodule as well, see
commits that are merged in 305d7f133956a5f43c94d938beabbfbb0ac1753c
or:
https://kernel.googlesource.com/pub/scm/git/git/+/fd47ae6a5b9cc0cfc56c1f7c43db612d26ca4b75%5E%21/#F1

Although this is just Git, you probably also have a code review system that
would need that change as well. Also it is not part of any release yet, but
soon will be.

Is there anything else besides these 3 points that encourages you to
switch away from submodules?

Thanks,
Stefan

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

>> - We have to make separate commits and manage corresponding topic
>> branches for the superproject and subprojects.
>
> Well yeah, that is how submodule work on a conceptual level.
> While having multiple commits may seem like overhead, note
> the subtle difference for these commits. One if deep down in the
> stack patching one of the submodules, the other is a high level
> commit advancing the submodule pointer.
>
> Note that the target audience of these two commit messages
> might be vastly different, hence can be worded differently.
> (The submodule describing how you fixed e.g. a memleak or race condition
> and the superproject describes on why you needed to include that submodule,
> e.g. because you switched your toplevel application to use threads.)

Both good points. 

Another thing to keep in mind is that in a well-organized project,
it is expected that you would have multiple commits in a submodule,
solving one single issue that is needed by the superproject in a
finer grained way, before the resulting submodule tip is recorded in
the tree of the superproject in one commit.  IOW, between the time
the superproject's history moves by one commit, the submodule may
have multiple commits in order for the submodule to become ready to
be consumed by the superproject.

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Matt McCutchen]

Hi Stefan,

I appreciate the effort to remove obstacles to the use of submodules!
 It looks like a custom tool is probably still our best option at this
time, though we can always switch back to submodules later.

On Wed, 2016-10-26 at 16:23 -0700, Stefan Beller wrote:
> On Wed, Oct 26, 2016 at 4:07 PM, Matt McCutchen <matt@mattmccutchen.net> wrote:
> > - We have to make separate commits and manage corresponding topic
> > branches for the superproject and subprojects.
> 
> Well yeah, that is how submodule work on a conceptual level.
> While having multiple commits may seem like overhead, note
> the subtle difference for these commits. One if deep down in the
> stack patching one of the submodules, the other is a high level
> commit advancing the submodule pointer.
> 
> Note that the target audience of these two commit messages
> might be vastly different, hence can be worded differently.
> (The submodule describing how you fixed e.g. a memleak or race condition
> and the superproject describes on why you needed to include that submodule,
> e.g. because you switched your toplevel application to use threads.)

I understand one can adhere to that philosophy, but I don't see the
practical benefit of doing so in our case compared to using a "git
subtree"-like approach and making a single commit.  It would just be us
looking at both commits.  If we do upstream any of the library changes,
we'll probably have to rework them anyway.

> > - A diff of the superproject doesn't include the content of
> > subprojects.

> Although this is just Git, you probably also have a code review system that
> would need that change as well.

Indeed.  We currently use Bitbucket.  I'd be open to switching, though
maybe not just for this.

> Is there anything else besides these 3 points that encourages you to
> switch away from submodules?

Those 3 are the ongoing pain points I can think of.  There are a few
other drawbacks compared to "git subtree" that come up less often:

1b. On another project, I was working with a teammate who was new to
version control and not very careful, who forgot to run "git submodule
update" and ended up committing back the old submodule pointer.
 Thankfully, this hasn't happened yet on my current project.

4. I pushed several dangling submodule pointers before I learned I
could set push.recurseSubmodules = check.  This isn't the default; each
developer has to do it manually.  (In theory, I could put such things
in a setup script for them to run if they trust me.)

5. Stashing changes to both the superproject and the subproject takes
more steps.

6. I use multiple worktrees (with "git worktree") because of #5 and
also so that I can run two versions of the application at the same time
and compare the behavior.  Using "git worktree" with submodules is
officially unsupported, though I was able to get things working by
manually editing some files.

7. We have to set up a repository for each subproject on our hosting
service.  Anyone who forks our application and modifies a subproject
has to set up a subproject repository and carry a change to .gitmodules
to point to their repository.  If we use relative URLs in .gitmodules,
it's even worse: anyone who forks our application has to set up
repositories for all the subprojects, even if they don't modify them.

Matt

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Stefan Beller]

On Wed, Oct 26, 2016 at 6:52 PM, Matt McCutchen <matt@mattmccutchen.net> wrote:
> Hi Stefan,
>
> I appreciate the effort to remove obstacles to the use of submodules!
>  It looks like a custom tool is probably still our best option at this
> time, though we can always switch back to submodules later.
>
> On Wed, 2016-10-26 at 16:23 -0700, Stefan Beller wrote:
>> On Wed, Oct 26, 2016 at 4:07 PM, Matt McCutchen <matt@mattmccutchen.net> wrote:
>> > - We have to make separate commits and manage corresponding topic
>> > branches for the superproject and subprojects.
>>
>> Well yeah, that is how submodule work on a conceptual level.
>> While having multiple commits may seem like overhead, note
>> the subtle difference for these commits. One if deep down in the
>> stack patching one of the submodules, the other is a high level
>> commit advancing the submodule pointer.
>>
>> Note that the target audience of these two commit messages
>> might be vastly different, hence can be worded differently.
>> (The submodule describing how you fixed e.g. a memleak or race condition
>> and the superproject describes on why you needed to include that submodule,
>> e.g. because you switched your toplevel application to use threads.)
>
> I understand one can adhere to that philosophy, but I don't see the
> practical benefit of doing so in our case compared to using a "git
> subtree"-like approach and making a single commit.  It would just be us
> looking at both commits.  If we do upstream any of the library changes,
> we'll probably have to rework them anyway.
>
>> > - A diff of the superproject doesn't include the content of
>> > subprojects.
>
>> Although this is just Git, you probably also have a code review system that
>> would need that change as well.
>
> Indeed.  We currently use Bitbucket.  I'd be open to switching, though
> maybe not just for this.
>
>> Is there anything else besides these 3 points that encourages you to
>> switch away from submodules?
>
> Those 3 are the ongoing pain points I can think of.  There are a few
> other drawbacks compared to "git subtree" that come up less often:
>
> 1b. On another project, I was working with a teammate who was new to
> version control and not very careful, who forgot to run "git submodule
> update" and ended up committing back the old submodule pointer.
>  Thankfully, this hasn't happened yet on my current project.
>
> 4. I pushed several dangling submodule pointers before I learned I
> could set push.recurseSubmodules = check.  This isn't the default; each
> developer has to do it manually.  (In theory, I could put such things
> in a setup script for them to run if they trust me.)

There is a current series in flight/for review that makes "check" default.
(It is blocked as check has some performance issues when having lots
of commits to be pushed, so it may take a while and not show up in the
next release)

>
> 5. Stashing changes to both the superproject and the subproject takes
> more steps.

True, so you'd want to have a `git stash --recurse-submodules={yes,no}`
where the command line option is configurable, so you don't have to type
it all the time?

>
> 6. I use multiple worktrees (with "git worktree") because of #5 and
> also so that I can run two versions of the application at the same time
> and compare the behavior.  Using "git worktree" with submodules is
> officially unsupported, though I was able to get things working by
> manually editing some files.

Heh, true. I made an attempt on fixing git worktree a few weeks ago, but that
did not go anywhere, but it's still on the TODO list.

You can use git clone --reference with --recurse-submodule though when
having bandwidth concerns. It doesn't deliver the full worktree experience
though as it would be separate clones with shared object store.

>
> 7. We have to set up a repository for each subproject on our hosting
> service.  Anyone who forks our application and modifies a subproject
> has to set up a subproject repository and carry a change to .gitmodules
> to point to their repository.  If we use relative URLs in .gitmodules,
> it's even worse: anyone who forks our application has to set up
> repositories for all the subprojects, even if they don't modify them.
>
> Matt

Yeah the model of referencing in gitmodules is conceptually broken.
I don't even claim it is on my todo list. ;)

Thanks for pointing out the issues though. they align to what
we plan on doing for submodules, so ... the plan actually makes
sense :)

Thanks,
Stefan

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Matt McCutchen]

On Wed, 2016-10-26 at 19:03 -0700, Stefan Beller wrote:
> On Wed, Oct 26, 2016 at 6:52 PM, Matt McCutchen <matt@mattmccutchen.net> wrote:
> > 4. I pushed several dangling submodule pointers before I learned I
> > could set push.recurseSubmodules = check.  This isn't the default; each
> > developer has to do it manually.  (In theory, I could put such things
> > in a setup script for them to run if they trust me.)
> 
> There is a current series in flight/for review that makes "check" default.
> (It is blocked as check has some performance issues when having lots
> of commits to be pushed, so it may take a while and not show up in the
> next release)

Great!  One other thing: IIRC, "check" does not distinguish between
different remotes.  For example, suppose I fork a project that already
has a submodule and I have a pair of repositories that pull from the
"upstream" repositories and push to "origin" repositories for my
project.  Suppose I upgrade to a new upstream version and find that I'm
(temporarily) able to use the upstream submodule without modifications.
 The "check" feature won't stop me from pushing a pointer into the
"origin" superproject that points to a commit that exists in the
"upstream" subproject but not the "origin" subproject.

> > 5. Stashing changes to both the superproject and the subproject takes
> > more steps.
> 
> True, so you'd want to have a `git stash --recurse-submodules={yes,no}`
> where the command line option is configurable, so you don't have to type
> it all the time?

Sounds good.  I'm sure you realize this is not just a matter of running
"git stash" in each submodule because there are many ways the stash
stacks could get out of lockstep.  The submodule content needs to be
incorporated into the superproject stash.

> Thanks for pointing out the issues though. they align to what
> we plan on doing for submodules, so ... the plan actually makes
> sense :)

Again, I'm thrilled you're working on this, even if I don't use it on
my current project.

Matt

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Peter Williams]

On 27/10/16 09:59, Junio C Hamano wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>>> - We have to make separate commits and manage corresponding topic
>>> branches for the superproject and subprojects.
>>
>> Well yeah, that is how submodule work on a conceptual level.
>> While having multiple commits may seem like overhead, note
>> the subtle difference for these commits. One if deep down in the
>> stack patching one of the submodules, the other is a high level
>> commit advancing the submodule pointer.
>>
>> Note that the target audience of these two commit messages
>> might be vastly different, hence can be worded differently.
>> (The submodule describing how you fixed e.g. a memleak or race condition
>> and the superproject describes on why you needed to include that submodule,
>> e.g. because you switched your toplevel application to use threads.)
>
> Both good points.
>
> Another thing to keep in mind is that in a well-organized project,
> it is expected that you would have multiple commits in a submodule,
> solving one single issue that is needed by the superproject in a
> finer grained way, before the resulting submodule tip is recorded in
> the tree of the superproject in one commit.  IOW, between the time
> the superproject's history moves by one commit, the submodule may
> have multiple commits in order for the submodule to become ready to
> be consumed by the superproject.
>
>

I'm a relatively new user of submodules and I quite like them (having 
tried a few other strategies for sharing common code between multiple 
projects and found them quite painful) and find them fairly easy to use. 
  I especially like the fact that the submodule command isn't very 
complicated and that the best method for managing commits, etc in the 
submodule is to cd into their root directory and then treat them like 
any other git repository (greatly reducing the amount of new stuff that 
you have to learn in order to use them).  Also, from my experience so 
far, I see three different types of work going on within my workspaces 
that include submodules:

1. I'm working on changes to the submodule and using the superproject 
that it's checked out in to test those changes in which case most of the 
change is occurring in the submodule with changes in the superproject 
usually being small one related to API changes in the submodule.

2. I'm working on changes in the superproject and the only changes that 
get made in the submodules are to fix bugs uncovered by the work in the 
superproject.

3. I'm modifying a superproject to accommodate changes to a submodule 
that's changed as a result of having changes pulled from another repository.

In none of these cases do I feel the desire/need to commit the changes 
to the superproject and submodule(s) with a single commit command which 
more or less agrees with your points.

However, for git commands such as diff/status whose job is to display 
information it would be nice if they had a --recursive option to 
override the default submodule diff/status and show details of the 
changes in the submodules.  Sometimes you want to see the big picture in 
detail.

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Junio C Hamano]

Peter Williams <pwil3058@bigpond.net.au> writes:

> However, for git commands such as diff/status whose job is to display
> information it would be nice if they had a --recursive option to
> override the default submodule diff/status and show details of the
> changes in the submodules.  Sometimes you want to see the big picture
> in detail.

I won't disagree. My comment was only on this part from the original:

>> - We have to make separate commits and manage corresponding topic
>> branches for the superproject and subprojects.

and on this point, we seem to be in agreement.

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Peter Williams <pwil3058@bigpond.net.au> writes:
>
>> However, for git commands such as diff/status whose job is to display
>> information it would be nice if they had a --recursive option to
>> override the default submodule diff/status and show details of the
>> changes in the submodules.  Sometimes you want to see the big picture
>> in detail.
>
> I won't disagree. My comment was only on this part from the original:
>
>>> - We have to make separate commits and manage corresponding topic
>>> branches for the superproject and subprojects.
>
> and on this point, we seem to be in agreement.

Oh, and as Stefan mentioned, a "git diff" that recurses into the
submodules to give you detailed big picture has been in 'next'
(perhaps aready in 'master' as of tonight, but I am not sure
offhand) to be tested, together with many other fixes and
enhancements that all are waiting to be included in future releases.

The more people try and give feedback to these branches early, the
more solid release with better support for more goodies you'd want
we will be able to give you.  Early adopters are always appreciated
but especially in time like this before the feature freeze for the
upcoming release (see tinyurl.com/gitCal for the schedule), they are
of great help.

Start by cloning from any one of these places

  git://git.kernel.org/pub/scm/git/git.git/
  https://kernel.googlesource.com/pub/scm/git/git
  git://repo.or.cz/alt-git.git/
  https://github.com/git/git/

and then

  $ git checkout -b next origin/next
  : read INSTALL to figure out if any custom options are needed
  : in the following 'make' invocations for your environment
  $ make && make install
  $ PATH=$HOME/bin:$PATH

to join the fun.

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Matt McCutchen]

On Wed, 2016-10-26 at 19:07 -0400, Matt McCutchen wrote:
> Maybe we would never hit any of these problems in practice, but they
> give me a bad enough feeling that I'm planning to write my own tool
> that tracks the upstream commit ID in a file (like a submodule) and
> doesn't generate any extra commits.  Without generating extra commits,
> the only place to store the upstream content in the superproject would
> be in another subtree, which would take up disk space in every working
> tree unless developers manually set skip-worktree.  I think I prefer to
> not store the upstream content and just have the tool fetch it from a
> local subproject repository each time it's needed.
> 
> I'll of course post the tool on the web and would be happy to see it
> integrated into "git subtree" if that makes sense, but I don't know how
> much time I'd be willing to put into making that happen.

I have named my tool "git subtree-lite" and posted it here:

https://mattmccutchen.net/utils/git-subtree-lite.git/

For now, please email any bug reports, enhancement requests, or
proposed patches to me.  I have philosophical concerns about hosting my
own projects on services I don't control and practical concerns about a
 few "forge" apps that I looked into installing on my own web site, but
if people are seriously interested in collaborating on this, I'll work
something out.

Matt

Re: "git subtree --squash" interacts poorly with revert, merge, and rebase

[Matt McCutchen]

On Thu, 2016-11-10 at 16:53 -0500, Matt McCutchen wrote:
> On Wed, 2016-10-26 at 19:07 -0400, Matt McCutchen wrote:
> > 
> > Maybe we would never hit any of these problems in practice, but they
> > give me a bad enough feeling that I'm planning to write my own tool
> > that tracks the upstream commit ID in a file (like a submodule) and
> > doesn't generate any extra commits.  Without generating extra commits,
> > the only place to store the upstream content in the superproject would
> > be in another subtree, which would take up disk space in every working
> > tree unless developers manually set skip-worktree.  I think I prefer to
> > not store the upstream content and just have the tool fetch it from a
> > local subproject repository each time it's needed.
> > 
> > I'll of course post the tool on the web and would be happy to see it
> > integrated into "git subtree" if that makes sense, but I don't know how
> > much time I'd be willing to put into making that happen.
> 
> I have named my tool "git subtree-lite" and posted it here:
> 
> https://mattmccutchen.net/utils/git-subtree-lite.git/

As I was doing additional research in preparation for adding git-
subtree-lite to the tools page
(https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools),
by chance I found an existing tool, Braid
(http://cristibalan.github.io/braid/), whose design meets my
requirements.  I have a few minor concerns, but assuming I'm able to
fix them without too much work and upstream accepts my patches, I plan
to switch to Braid.

I've made a properly marked section on the tools page for subproject
management tools:

https://git.wiki.kernel.org/index.php/Interfaces,_frontends,_and_tools#Subprojects_or_sets_of_repositories

in the hope that the next person with the same requirements as me finds
Braid.  (I unfortunately didn't check that page before starting, but I
will the next time I need something.)

Matt