[RFC/PATCH] add update to branch support for "floating submodules"

[RFC/PATCH] add update to branch support for "floating submodules"

[Heiko Voigt]

This adds the capability to configure a branch which submodule update
will use to checkout the tips sha1 instead of the registered one.

It will first attempt to read the configuration directly from the
currently checked out .gitmodules file from the key
submodule.$name.branch.  This configuration can be overridden by local
user configuration values. The parameter --branch can be used to
specify/override the branch using the commandline. The parameter
--checkout can be used to switch to the exact model for all submodules.

Such a thing is helpful if a user wants to follow a defined branches tip
in the submodule. Image such a branch is the stable branch for some
central library or similar.

When the newly checked out tip will not match the registered sha1 in the
superproject it will show up as a change as usual. You can imagine this
as a configuration which lets the upstream project tell a user the
branch it usually updates to. The usual revision control is still in
place.
---

This is almost ready but I would like to know what users of the
"floating submodule" think about this.

 Documentation/git-submodule.txt |   26 +++++++++--
 git-submodule.sh                |   47 ++++++++++++++++++++
 t/t7406-submodule-update.sh     |   93 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 162 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 6ec3fef..b8affa3 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -133,9 +133,11 @@ init::
 update::
 	Update the registered submodules, i.e. clone missing submodules and
 	checkout the commit specified in the index of the containing repository.
-	This will make the submodules HEAD be detached unless `--rebase` or
-	`--merge` is specified or the key `submodule.$name.update` is set to
-	`rebase`, `merge` or `none`.
+	This will make the submodules HEAD be detached. This will not
+	happen if `--rebase`, `--merge` or `--branch` are specified.
+	Also if the key `submodule.$name.update` is set to `rebase`,
+	`merge` or `none`. If `submodule.$name.branch` is set to some
+	local branch this will also not happen.
 +
 If the submodule is not yet initialized, and you just want to use the
 setting as stored in .gitmodules, you can automatically initialize the
@@ -146,7 +148,16 @@ registered submodules, and update any nested submodules within.
 +
 If the configuration key `submodule.$name.update` is set to `none` the
 submodule with name `$name` will not be updated by default. This can be
-overriden by adding `--checkout` to the command.
+overriden by adding `--checkout` to the command. `--checkout` can also
+be used to enforce exact checkout of submodule sha1's.
++
+If the configuration key `submodule.$name.branch` is set to some valid
+branch in the submodule named by `$name` the submodule will be updated
+to the tip of that branch instead of the registered sha1. This option
+can either be set in .gitmodules or via git's configuration. Gits local
+configuration takes precedence over .gitmodules. If you want to override
+the branch checkout you can use the value `HEAD` to tell git to checkout
+exactly the registered sha1.
 
 summary::
 	Show commit summary between the given commit (defaults to HEAD) and
@@ -252,6 +263,13 @@ OPTIONS
 	If the key `submodule.$name.update` is set to `rebase`, this option is
 	implicit.
 
+--branch::
+	This option is only valid for the update command. You can use
+	this parameter to specify which branch you want to update all
+	submodules to. This is helpful if you want to update all
+	submodules to the tip of a certain branch or need to work on a
+	branch for all submodules for some time.
+
 --init::
 	This option is only valid for the update command.
 	Initialize all submodules for which "git submodule init" has not been
diff --git a/git-submodule.sh b/git-submodule.sh
index 3adab93..a4b117b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -465,6 +465,14 @@ cmd_update()
 		--checkout)
 			update="checkout"
 			;;
+		--branch=*)
+			case "$1" in
+			*=*)
+				update="checkout"
+				branch=`expr "z$1" : 'z--[^=]*=\(.*\)'` ;;
+			*)
+				usage ;;
+			esac ;;
 		--)
 			shift
 			break
@@ -504,6 +512,45 @@ cmd_update()
 			update_module=$(git config submodule."$name".update)
 		fi
 
+		if ! test -z "$branch"
+		then
+			branch_module=$branch
+		else
+			if test "$update" != "checkout"
+			then
+				branch_module=$(git config submodule."$name".branch)
+				if test -z "$branch_module"
+				then
+					branch_module=$(git config -f .gitmodules --get submodule."$name".branch)
+				fi
+			fi
+		fi
+
+		if test "$branch_module" = "HEAD"
+		then
+			branch_module=
+		fi
+
+		if ! test -z "$branch_module"
+		then
+			(clear_local_git_env; cd "$path" &&
+			 if test ! $nofetch
+			 then
+				git-fetch --all >/dev/null 2>/dev/null || exit 1
+			 fi) ||
+			die "$(eval_gettext "Unable to fetch submodule in path '\$path'")"
+
+			sha1=$(clear_local_git_env; cd "$path" &&
+				git rev-parse $branch_module) ||
+			say "$(eval_gettext "Unable to find branch '\$branch_module' in submodule path '\$path'")"
+		fi
+
+		if test "$branch" -a "$update" != "checkout"
+		then
+			die "$(eval_gettext "You can not set update='\$update' and
+use a branch for submodule '\$path'")"
+		fi
+
 		if test "$update_module" = "none"
 		then
 			echo "Skipping submodule '$path'"
diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 33b292b..517ed83 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -611,4 +611,97 @@ test_expect_success 'submodule update places git-dir in superprojects git-dir re
 	)
 '
 
+test_expect_success '--branch updates follow the given branch' '
+	git clone . branch &&
+	(cd branch &&
+		git submodule add ./submodule submodule1 &&
+		git submodule add ./submodule submodule2 &&
+		(cd submodule1 &&
+			git rev-parse HEAD >../expected1 &&
+			git checkout HEAD^) &&
+		(cd submodule2 &&
+			git rev-parse HEAD >../expected2 &&
+			git checkout HEAD^) &&
+		git add submodule1 &&
+		git add submodule2 &&
+		git commit -m "add submodule1 and submodule2" &&
+		git submodule update --branch=origin/master &&
+		(cd submodule1 && git rev-parse HEAD >../actual1) &&
+		(cd submodule2 && git rev-parse HEAD >../actual2) &&
+		test_cmp expected1 actual1 &&
+		test_cmp expected2 actual2
+	)
+'
+
+cat >branch/expect_status <<EOF
+ M submodule1
+EOF
+
+check_submodule_one_follows()
+{
+	(cd submodule1 &&
+		git rev-parse origin/master >../expected1 &&
+		git rev-parse HEAD >../actual1) &&
+	(cd submodule2 &&
+		git rev-parse origin/master^ >../expected2 &&
+		git rev-parse HEAD >../actual2) &&
+	test_cmp expected1 actual1 &&
+	test_cmp expected2 actual2 &&
+	git status --porcelain --untracked-files=no >actual_status &&
+	test_cmp expect_status actual_status
+}
+
+check_both_submodules_exact()
+{
+	(cd submodule1 &&
+		git rev-parse origin/master^ >../expected1 &&
+		git rev-parse HEAD >../actual1) &&
+	(cd submodule2 &&
+		git rev-parse origin/master^ >../expected2 &&
+		git rev-parse HEAD >../actual2) &&
+	test_cmp expected1 actual1 &&
+	test_cmp expected2 actual2 &&
+	test -z "$(git status --porcelain --untracked-files=no)"
+}
+
+test_expect_success 'local branch configuration follows branch' '
+	(cd branch &&
+		git submodule update &&
+		check_both_submodules_exact &&
+		git config submodule.submodule1.branch origin/master &&
+		git submodule update &&
+		check_submodule_one_follows
+	)
+'
+
+test_expect_success '.gitmodules branch configuration follows branch' '
+	(cd branch &&
+		git config --unset submodule.submodule1.branch &&
+		git submodule update &&
+		check_both_submodules_exact &&
+		git config -f .gitmodules submodule.submodule1.branch origin/master &&
+		git add .gitmodules &&
+		git commit -m ".gitmodules follows branch" &&
+		git submodule update &&
+		check_submodule_one_follows
+	)
+'
+
+test_expect_success '--checkout commandline overrides branch config' '
+	(cd branch &&
+		git submodule update --checkout &&
+		check_both_submodules_exact
+	)
+'
+
+test_expect_success 'local config overrides .gitmodules branch config' '
+	(cd branch &&
+		git submodule update &&
+		check_submodule_one_follows &&
+		git config submodule.submodule1.branch HEAD &&
+		git submodule update &&
+		check_both_submodules_exact
+	)
+'
+
 test_done
-- 
1.7.7.433.gcf1e7

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Junio C Hamano]

Heiko Voigt <hvoigt@hvoigt.net> writes:

> This is almost ready but I would like to know what users of the
> "floating submodule" think about this.

Thanks for working on this.

I do like to hear from potential users as well, because the general
impression we got was that floating submodules is not a real need of
anybody, but it is merely an inertia of people who (perhaps mistakenly)
thought svn externals that are not anchored to a particular revision is a
feature when it is just a limitation in reality. During the GitTogether'11
we learned that Android that uses floating model does not really have to.

Re: Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Heiko Voigt]

Hi,

On Wed, Nov 09, 2011 at 10:01:33AM -0800, Junio C Hamano wrote:
> Heiko Voigt <hvoigt@hvoigt.net> writes:
> 
> > This is almost ready but I would like to know what users of the
> > "floating submodule" think about this.
> 
> Thanks for working on this.
> 
> I do like to hear from potential users as well, because the general
> impression we got was that floating submodules is not a real need of
> anybody, but it is merely an inertia of people who (perhaps mistakenly)
> thought svn externals that are not anchored to a particular revision is a
> feature when it is just a limitation in reality. During the GitTogether'11
> we learned that Android that uses floating model does not really have to.

Since we did not get any reply from potential floating submodule users I
do not mind to drop this patch for now. It is archived in the mailing list
and it should be easy to revive once there is real world need for it.

Once we have the "exact" model support for checkout and friends this
might be a handy tool to update submodules before releases and such. But
currently I would like to focus on the "exact" front first.

Cheers Heiko

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Leif Gruenwoldt]

Heiko Voigt <hvoigt <at> hvoigt.net> writes:

> 
> Hi,
> 
> On Wed, Nov 09, 2011 at 10:01:33AM -0800, Junio C Hamano wrote:
> > Heiko Voigt <hvoigt <at> hvoigt.net> writes:
> > 
> > > This is almost ready but I would like to know what users of the
> > > "floating submodule" think about this.
> > 
> > Thanks for working on this.
> > 
> > I do like to hear from potential users as well, because the general
> > impression we got was that floating submodules is not a real need of
> > anybody, but it is merely an inertia of people who (perhaps mistakenly)
> > thought svn externals that are not anchored to a particular revision is a
> > feature when it is just a limitation in reality. During the GitTogether'11
> > we learned that Android that uses floating model does not really have to.
> 
> Since we did not get any reply from potential floating submodule users I
> do not mind to drop this patch for now. It is archived in the mailing list
> and it should be easy to revive once there is real world need for it.
> 
> Once we have the "exact" model support for checkout and friends this
> might be a handy tool to update submodules before releases and such. But
> currently I would like to focus on the "exact" front first.
> 
> Cheers Heiko
> 

If I understand the description of "floating submodules", it's something I have 
been wanting for a while now! The lack of it is currently a deal breaker for 
using submodules within my organisation.

Our use case is as follows. We have several repositories for our common code 
(commonA.git, commonB.git, etc) and a few different products that leverage these 
common repos (productA.git, productB.git, etc). When one of the products is in 
heavy development we often need to do a lot of work in the common repos. Having 
to increment the sha1 of the submodules to track the latest tip would be overly 
arduous. (Obviously when development of the product stabilizes we would want to 
change to anchoring to a specific sha1 in the common repos).

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Jonathan Nieder]

(restoring cc list)
Hi Leif,

Leif Gruenwoldt wrote:

> If I understand the description of "floating submodules", it's something I have 
> been wanting for a while now! The lack of it is currently a deal breaker for 
> using submodules within my organisation.
>
> Our use case is as follows.
[...]
>                                                 When one of the products is in 
> heavy development we often need to do a lot of work in the common repos. Having 
> to increment the sha1 of the submodules to track the latest tip would be overly 
> arduous.

What happens when a bug was introduced in this period of heavy development
and someone wants to look back in the development history and build each
version to find which introduced the bug?

If I were part of such a project, I would be tempted to follow one of two
rules.  Either

 A. Each commit of productA strives to work with the latest version of
    the common code possible.  Which version of the common code that was
    tested against gets recorded (perhaps by some record-submodule-versions-
    and-commit script, or even a pre-commit hook) so others can
    reproduce the results.

or

 B. Occasionally (e.g., daily or weekly) the "baseline" version of the
    common code that can be relied on gets bumped, and each commit of
    productA should work with that version and all later versions for a
    while.  Everyday development might typically happen with the tip
    version of the common code which may be faster, have more
    bugfixes, and otherwise be more pleasant to work with, but commits
    should work against the baseline version as well.  When it is time
    to bump the baseline, that fact gets recorded (in a separate
    commit).

    For this, the '[submodule "<name>"] ignore' setting described in
    gitmodules(5) might be helpful.

Though of course other variations are possible.

Would you be able to try out using Heiko's patch for a while, adapt it
to your needs as necessary, and let us know how it goes?

Thanks very much, and good luck,
Jonathan

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Junio C Hamano]

Leif Gruenwoldt <leifer@gmail.com> writes:

> Our use case is as follows. We have several repositories for our common code 
> (commonA.git, commonB.git, etc) and a few different products that leverage these 
> common repos (productA.git, productB.git, etc). When one of the products is in 
> heavy development we often need to do a lot of work in the common repos. Having 
> to increment the sha1 of the submodules to track the latest tip would be overly 
> arduous. (Obviously when development of the product stabilizes we would want to 
> change to anchoring to a specific sha1 in the common repos).

Nobody forces you to update the commit in the submodule bound to the
superproject tree every time you update areas that are unrelated to or
independent from that frequently updated submodule.

During the period the submodule is so often updated that you feel "having
to increment ... would be overly arduous", it does not matter which exact
commit in that submodule is used in the tree for your other modules and
the superproject. Otherwise you _would_ want to say something like "for
this entire tree state from the top-level superproject to correctly work,
we absolutely need to have this commit, not any commit that is older and
is known to be broken, from this submodule", and cannot afford to use
floating.

Which means by definition anybody who wants floating can instead let such
an often updated submodule stay somewhat stale by not running "submodule
update" for it unnecessarily.  In a well-modularized set of projects, the
interface to the busy submodule may be stable and I can imagine that kind
of arrangement would well be not just possible but practical, and probably
yours may be such a project.

So that use case does not sound like a good rationale to require addition
of floating submodules.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Gioele Barabucci]

On 09/11/2011 18:01, Junio C Hamano wrote:
>> This is almost ready but I would like to know what users of the
>> "floating submodule" think about this.
 >
> I do like to hear from potential users as well, because the general
> impression we got was that floating submodules is not a real need of
> anybody,

Floating modules are something much sought after by those who use Git 
for non-development purposes, like those who have most of their $HOME 
versioned with Git [1]. For example, part of what Joey Hess's `mr` tool 
[2] does is to simulate floating submodules for Git-versioned $HOMEs.

In the context of versioned $HOMEs, or with backups in general, precise 
tracking of submodules updates is not that important. To quote [3]: 
«Last, change tracking is a bit more lenient with home directories. I 
may shuffle some stuff around, and I don't need to explain the changes 
to anyone else.». In my case, I want my ~/Documents dir (that is in a 
different repo from $HOME) to be always updated; I would prefer not to 
deal with submodule updates, merges and detached HEADs.

Bye,

[1] http://vcs-home.branchable.com/
[2] http://kitenet.net/~joey/code/mr/
[3] http://joshcarter.com/productivity/svn_hg_git_for_home_directory

--
Gioele Barabucci <gioele@svario.it>

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Leif Gruenwoldt]

On Sat, Dec 10, 2011 at 1:30 AM, Junio C Hamano <gitster@pobox.com> wrote:

> So that use case does not sound like a good rationale to require addition
> of floating submodules.

Ok I will try another scenario :)

Imagine again products A, B and C and a common library. The products are in
a stable state of development and track a stable branch of the common lib.
Then imagine an important security fix gets made to the common library. On
the next pull of products A, B, and C they get this fix for free
because they were
floating. They didn't need to communicate with the maintainer of the common
repo to know this. In fact they don't really care. They just want the
latest stable
code for that release branch.

This is how package management on many linux systems works. Dependencies
get updated and all products reap the benefit (or catastrophe) automatically.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Andreas T.Auer]

On 10.12.2011 16:27 Leif Gruenwoldt wrote:
>  On Sat, Dec 10, 2011 at 1:30 AM, Junio C Hamano <gitster@pobox.com>
>  wrote:
>
> > So that use case does not sound like a good rationale to require
> > addition of floating submodules.
>
>  Ok I will try another scenario :)
>
>  Imagine again products A, B and C and a common library. The products
>   are in a stable state of development and track a stable branch of
>  the common lib. Then imagine an important security fix gets made to
>  the common library. On the next pull of products A, B, and C they get
>  this fix for free because they were floating. They didn't need to
>  communicate with the maintainer of the common repo to know this. In
>  fact they don't really care. They just want the latest stable code
>  for that release branch.

So you don't want to have a stale submodule as Junio suggested, which is 
older than the gitlinked commit in the superproject, but you want to 
have the newest stable version, which is not yet gitlinked in the 
superproject, right?

Wouldn't  ( cd commonlib ; git pull stable ) instead of
git submodule update commonlib
work as you want?

To be able to configure this update behavior in .gitmodules for _some_ 
submodules, could be helpful in this case.

So you don't want to add a new commit to the products A, B and C repos 
whenever the stable branch of the submodule changes, but on the other 
hand when you commit changes to the products it would still make sense 
to update the gitlink to the current commonlib version together with 
your changes,  too, right?


>  This is how package management on many linux systems works.
>  Dependencies get updated and all products reap the benefit (or
>  catastrophe) automatically.
If I have e.g. the Debian testing distro, which is more floating than 
the most other Linux distro releases, then I still get only those 
versions of the packages that are referenced by this "Debian testing" 
superproject, unless I specify a different superproject (e.g. "Debian 
unstable") to get a newer version, but they are still tracked in some 
superproject. I'm not aware of a way to get the newest version of a 
package before it is in some "superproject", except downloading it 
explictily somewhere else. But I don't think this is what you want.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Leif Gruenwoldt]

On Mon, Dec 12, 2011 at 10:34 AM, Andreas T.Auer
<andreas.t.auer_gtml_37453@ursus.ath.cx> wrote:

> So you don't want to have a stale submodule as Junio suggested, which is
> older than the gitlinked commit in the superproject, but you want to have
> the newest stable version, which is not yet gitlinked in the superproject,
> right?

Right.

> Wouldn't  ( cd commonlib ; git pull stable ) instead of
> git submodule update commonlib
> work as you want?

Yes that's how we perform business now.

> To be able to configure this update behavior in .gitmodules for _some_
> submodules, could be helpful in this case.

Yes my thoughts exactly.


> So you don't want to add a new commit to the products A, B and C repos
> whenever the stable branch of the submodule changes, but on the other hand
> when you commit changes to the products it would still make sense to update
> the gitlink to the current commonlib version together with your changes,
>  too, right?

Hmm I supose that does make sense. If the commonlib version was auto recorded
during a commit of the product it would be nice. Then if/when the user
reconfigured
the submodule from "floating" to "strict" mode it would then have a
submodule sha1
reference. I like how this sounds.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Andreas T.Auer]

On 12.12.2011 19:04 Leif Gruenwoldt wrote:
>  On Mon, Dec 12, 2011 at 10:34 AM, Andreas T.Auer
>  <andreas.t.auer_gtml_37453@ursus.ath.cx> wrote:
>
> > So you don't want to add a new commit to the products A, B and C
> > repos whenever the stable branch of the submodule changes, but on
> > the other hand when you commit changes to the products it would
> > still make sense to update the gitlink to the current commonlib
> > version together with your changes, too, right?
>
>  Hmm I supose that does make sense. If the commonlib version was auto
>  recorded during a commit of the product it would be nice. Then
>  if/when the user reconfigured the submodule from "floating" to
>  "strict" mode it would then have a submodule sha1 reference. I like
>  how this sounds.

The next question is: Wouldn't you like to have the new stable branch 
only pulled in, when the projectX (as the superproject) is currently on 
that new development branch (maybe master)?

But if you checkout that fixed released version 1.2.9.8, wouldn't it be 
better that in that case the gitlinked version of the submodule is 
checked out instead of some unrelated new version? I mean, when the 
gitlinks are tracked with the projectX commits, this should work well.

And what about a maintenance branch, which is not a fixed version but a 
quite stable branch which should only have bugfixes. Shouldn't the 
auto-pull be disabled in that case, too?

I think the "auto-pull" behavior should depend on the currently checked 
out branch. So the configuration options should allow the definition of 
one or more mappings.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Leif Gruenwoldt]

On Mon, Dec 12, 2011 at 1:42 PM, Andreas T.Auer
<andreas.t.auer_gtml_37453@ursus.ath.cx> wrote:

> The next question is: Wouldn't you like to have the new stable branch only
> pulled in, when the projectX (as the superproject) is currently on that new
> development branch (maybe master)?
>
> But if you checkout that fixed released version 1.2.9.8, wouldn't it be
> better that in that case the gitlinked version of the submodule is checked
> out instead of some unrelated new version? I mean, when the gitlinks are
> tracked with the projectX commits, this should work well.
>
> And what about a maintenance branch, which is not a fixed version but a
> quite stable branch which should only have bugfixes. Shouldn't the auto-pull
> be disabled in that case, too?
>
> I think the "auto-pull" behavior should depend on the currently checked out
> branch. So the configuration options should allow the definition of one or
> more mappings.

Yes. I think you nailed it. The floating behaviour would best be
configured per branch.

An aside. Would this mean a "git pull" on the product repo would
automatically do a pull (git submodule update) on the submodule too?

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Junio C Hamano]

Leif Gruenwoldt <leifer@gmail.com> writes:

> On Sat, Dec 10, 2011 at 1:30 AM, Junio C Hamano <gitster@pobox.com> wrote:
>
>> So that use case does not sound like a good rationale to require addition
>> of floating submodules.
>
> Ok I will try another scenario :)
>
> Imagine again products A, B and C and a common library. The products are in
> a stable state of development and track a stable branch of the common lib.
> Then imagine an important security fix gets made to the common library. On
> the next pull of products A, B, and C they get this fix for free
> because they were
> floating. They didn't need to communicate with the maintainer of the common
> repo to know this. In fact they don't really care. They just want the
> latest stable
> code for that release branch.
>
> This is how package management on many linux systems works. Dependencies
> get updated and all products reap the benefit (or catastrophe) automatically.

Distro package dependency tracking is a poor analogy for many reasons, but
I'll only touch a few.

If you have a common library L and application packages A, B and C, first
of all, you do not build distro package of A from the sources of A and
L. Instead, package A has a build dependency on package L-devel (in other
words, "in order to build A from the source, you need L-devel package
installed in your build environment"), build A from its source, link it
against L's binary without having the source of L. So the source code
arrangement is very different from the typical submodule case, in that you
do not even need to have A and L appear in the same working tree, let
alone bound in a same superproject as two submodules.

A library L may have two API versions, and application A and B may be
written for its v1.0 and v2.0 API, respectively. Distro packaging makes
the binary package A and B _know_ about their own dependency requirements
by recording "A depends on L (v1.0<=,<v2.0)", "B depends on L (v2.0<=)",
etc.

Naively, one might think that two branches, branch-1.0 and branch-2.0, can
be defined in the repository of L, tell somebody (like "superproject that
covers all the packages in the distro") that A wants branch-1.0 and B
wants branch-2.0 of L respectively, to emulate this, but if one thinks
further, one would realize that it is insufficient. For one thing, it is
unclear what should happen when both A and B are asked to be checked out,
but more importantly, in dependency requirements on real distro packaging,
the application C could say "I want v1.0 API but v1.4 is broken and not
compatible with me", which won't fit on the two-branches model. A
workaround to add more branches to L could be devised but any workaround
cannot be a good solution that allows a random application C among 47
others to dictate how the branch structure of L project should look like.

Fortunately, the dependency management is a solved problem by distro
package management and build systems, and they do so without using
anything from submodules. There is no point reinventing these logic in git
submodules and emulating poorly.

The only remotely plausible analogy around distro packaging would be a
superproject full of all the packages in a distro as its submodules, and
records exact versions of each and every package that goes on a release
CD (or DVD). In that case, you do want to have a central registry that
records what exact version of each package is used to cut the CD and the
mother of all modules superproject could be one way to implement it. But
that is not an example of floating, but is a direct opposite.

This exchange convinced me further that anybody who wishes to use
"floating" is better off either by doing one or both of the following:

 - using "exact" but not updating religiously, as the interdepency
   requirement in their project is not strict; or

 - not using submodules at all, but merely keeping these unrelated A, B, C
   and L as standalone repositories next to each other in the directory
   structure.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Jens Lehmann]

Am 12.12.2011 20:13, schrieb Leif Gruenwoldt:
> On Mon, Dec 12, 2011 at 1:42 PM, Andreas T.Auer
> <andreas.t.auer_gtml_37453@ursus.ath.cx> wrote:
> 
>> The next question is: Wouldn't you like to have the new stable branch only
>> pulled in, when the projectX (as the superproject) is currently on that new
>> development branch (maybe master)?
>>
>> But if you checkout that fixed released version 1.2.9.8, wouldn't it be
>> better that in that case the gitlinked version of the submodule is checked
>> out instead of some unrelated new version? I mean, when the gitlinks are
>> tracked with the projectX commits, this should work well.
>>
>> And what about a maintenance branch, which is not a fixed version but a
>> quite stable branch which should only have bugfixes. Shouldn't the auto-pull
>> be disabled in that case, too?
>>
>> I think the "auto-pull" behavior should depend on the currently checked out
>> branch. So the configuration options should allow the definition of one or
>> more mappings.
> 
> Yes. I think you nailed it. The floating behaviour would best be
> configured per branch.

Why not use .gitmodules and make the "branch" setting work without having to
sync it?

> An aside. Would this mean a "git pull" on the product repo would
> automatically do a pull (git submodule update) on the submodule too?

Me thinks that only "git submodule update" should do that. Or what should
happen for people not using pull but just doing fetch and checkout?

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Phil Hord]

On Mon, Dec 12, 2011 at 2:13 PM, Leif Gruenwoldt <leifer@gmail.com> wrote:
> On Mon, Dec 12, 2011 at 1:42 PM, Andreas T.Auer
> <andreas.t.auer_gtml_37453@ursus.ath.cx> wrote:
>
>> The next question is: Wouldn't you like to have the new stable branch only
>> pulled in, when the projectX (as the superproject) is currently on that new
>> development branch (maybe master)?
>>
>> But if you checkout that fixed released version 1.2.9.8, wouldn't it be
>> better that in that case the gitlinked version of the submodule is checked
>> out instead of some unrelated new version? I mean, when the gitlinks are
>> tracked with the projectX commits, this should work well.
>>
>> And what about a maintenance branch, which is not a fixed version but a
>> quite stable branch which should only have bugfixes. Shouldn't the auto-pull
>> be disabled in that case, too?
>>
>> I think the "auto-pull" behavior should depend on the currently checked out
>> branch. So the configuration options should allow the definition of one or
>> more mappings.
>
> Yes. I think you nailed it. The floating behaviour would best be
> configured per branch.

Yes, I think you nailed it too.  I've been thinking the same thing for
a while now, but I didn't know how to express it completely.  Some of
the discussion on here last week gelled the last bits in my mind.

To wit, I think I would want something like this in my project:

Use gitlinks when the superproject HEAD is one of these:
    refs/heads/maint/*
    refs/heads/svn/*     (historic branches)
    refs/tags/*
    <SHA1> (detached)

Float on the rest, using the branch given in .gitmodules (which may be
* to mean "use the same branch as the superproject".)

But maybe it is foolish of me to keep branches where I really want
lightweight tags.  If so, I could get away with this:

   Float if .git/HEAD begins with "refs/heads"
   Else, use the SHA1.


> An aside. Would this mean a "git pull" on the product repo would
> automatically do a pull (git submodule update) on the submodule too?

Good question.

I want to say "eventually, but not yet."  But someone else may
disagree.  "git pull --recurse-submodules=yes" does not do this yet.
A separate git-submodule-update is still required.  But I think this
is a separate issue from floating submodules.

Phil

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Brandon Casey]

On Sat, Dec 10, 2011 at 9:27 AM, Leif Gruenwoldt <leifer@gmail.com> wrote:
> On Sat, Dec 10, 2011 at 1:30 AM, Junio C Hamano <gitster@pobox.com> wrote:
>
>> So that use case does not sound like a good rationale to require addition
>> of floating submodules.
>
> Ok I will try another scenario :)
>
> Imagine again products A, B and C and a common library. The products are in
> a stable state of development and track a stable branch of the common lib.
> Then imagine an important security fix gets made to the common library. On
> the next pull of products A, B, and C they get this fix for free
> because they were
> floating. They didn't need to communicate with the maintainer of the common
> repo to know this. In fact they don't really care. They just want the
> latest stable
> code for that release branch.
>
> This is how package management on many linux systems works. Dependencies
> get updated and all products reap the benefit (or catastrophe) automatically.

What happens if the update to the floating submodule introduces a bug
that prevents A, B, and/or C from building/running correctly?  If the
submodule states are not recorded, how would the previously working
submodule version be restored so that development on A, B, and C,
could proceed?  I guess each developer could manually checkout @{1} in
the submodule in their working directory, though that wouldn't work
for a new clone, and it's not very elegant.

I presume that if A, B, and C, do not care to know exactly what was
fixed in the common library, they probably do not care to investigate
the breakage in that repo either.  Or, they may not have the
expertise.

-Brandon

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Phil Hord]

On Mon, Dec 12, 2011 at 2:36 PM, Junio C Hamano <gitster@pobox.com> wrote:
[...]
> Distro package dependency tracking is a poor analogy for many reasons, but
> I'll only touch a few.
[...]
> Naively, one might think that two branches, branch-1.0 and branch-2.0, can
> be defined in the repository of L, tell somebody (like "superproject that
> covers all the packages in the distro") that A wants branch-1.0 and B
> wants branch-2.0 of L respectively, to emulate this, but if one thinks
> further, one would realize that it is insufficient. For one thing, it is
> unclear what should happen when both A and B are asked to be checked out,
> but more importantly, in dependency requirements on real distro packaging,
> the application C could say "I want v1.0 API but v1.4 is broken and not
> compatible with me", which won't fit on the two-branches model. A
> workaround to add more branches to L could be devised but any workaround
> cannot be a good solution that allows a random application C among 47
> others to dictate how the branch structure of L project should look like.
>
> Fortunately, the dependency management is a solved problem by distro
> package management and build systems, and they do so without using
> anything from submodules. There is no point reinventing these logic in git
> submodules and emulating poorly.
>
> The only remotely plausible analogy around distro packaging would be a
> superproject full of all the packages in a distro as its submodules, and
> records exact versions of each and every package that goes on a release
> CD (or DVD). In that case, you do want to have a central registry that
> records what exact version of each package is used to cut the CD and the
> mother of all modules superproject could be one way to implement it. But
> that is not an example of floating, but is a direct opposite.
>
> This exchange convinced me further that anybody who wishes to use
> "floating" is better off either by doing one or both of the following:
>
>  - using "exact" but not updating religiously, as the interdepency
>   requirement in their project is not strict; or
>
>  - not using submodules at all, but merely keeping these unrelated A, B, C
>   and L as standalone repositories next to each other in the directory
>   structure.

My interdependency requirements are not so cut-and-dry.  We use
submodules to isolate controlled regions of code.  We may need to
share our project with a contractor who is allowed to see code
pertaining to "vendorA" but not that for "vendorB" or "VendorN".  But
our in-house developers want to have all the vendor code in one place
for convenient integration. Submodules do this nicely for us.  We can
give the contractor just the main modules and the VendorA modules and
he'll be fine.  In-house devs get all the submodules (using the
vendor-ALL superproject).

But this necessarily means there is too much coupling for comfort
between our submodules.   For example, when an API changes in the main
submodule, each of the vendor submodules is affected because they each
implement that API in a custom method.  Some of those vendor modules
belong to different people.  Submodule synchronization becomes a real
chore.

Floating would help, I think.  Instead I do this:

  git pull origin topic_foo && git submodule foreach 'git pull origin topic_foo'

  git submodule foreach 'git push origin topic_foo' && git push origin topic_foo

But not all my developers are git-gurus yet, and they sometimes mess
up their ad hoc scripts or miss important changes they forgot to push
in one submodule or another.  Or worse, their pull or push fails and
they can't see the problem for all the noise.  So they email it to me.

On my git server, I have a hook that automatically propagates each
push to "master" from the submodules into the superproject.  But this
is tedious and limited.  And it relies on a centralized server.

You may say this itch is all in my head, but it sure seems real to me.

Phil

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Marc Branchaud]

On 11-12-12 05:56 PM, Phil Hord wrote:
> On Mon, Dec 12, 2011 at 2:13 PM, Leif Gruenwoldt <leifer@gmail.com> wrote:
>> On Mon, Dec 12, 2011 at 1:42 PM, Andreas T.Auer
>> <andreas.t.auer_gtml_37453@ursus.ath.cx> wrote:
>>
>>> The next question is: Wouldn't you like to have the new stable branch only
>>> pulled in, when the projectX (as the superproject) is currently on that new
>>> development branch (maybe master)?
>>>
>>> But if you checkout that fixed released version 1.2.9.8, wouldn't it be
>>> better that in that case the gitlinked version of the submodule is checked
>>> out instead of some unrelated new version? I mean, when the gitlinks are
>>> tracked with the projectX commits, this should work well.
>>>
>>> And what about a maintenance branch, which is not a fixed version but a
>>> quite stable branch which should only have bugfixes. Shouldn't the auto-pull
>>> be disabled in that case, too?
>>>
>>> I think the "auto-pull" behavior should depend on the currently checked out
>>> branch. So the configuration options should allow the definition of one or
>>> more mappings.
>>
>> Yes. I think you nailed it. The floating behaviour would best be
>> configured per branch.
> 
> Yes, I think you nailed it too.  I've been thinking the same thing for
> a while now, but I didn't know how to express it completely.  Some of
> the discussion on here last week gelled the last bits in my mind.
> 
> To wit, I think I would want something like this in my project:
> 
> Use gitlinks when the superproject HEAD is one of these:
>     refs/heads/maint/*
>     refs/heads/svn/*     (historic branches)
>     refs/tags/*
>     <SHA1> (detached)
> 
> Float on the rest, using the branch given in .gitmodules (which may be
> * to mean "use the same branch as the superproject".)
> 
> But maybe it is foolish of me to keep branches where I really want
> lightweight tags.  If so, I could get away with this:
> 
>    Float if .git/HEAD begins with "refs/heads"
>    Else, use the SHA1.

Wouldn't this break creating a bugfix topic branch based on an earlier
revision of the repo?  I wouldn't want such a branch to automatically give me
the latest submodules.

I'd prefer to have floating be explicitly configured on a per-branch (or
per-branch-glob) basis.  So in addition to what Jens described yesterday [1]
to configure an individual submodule's floating branch, I suggest there also
be a new section in the .gitmodules file for configuring the super-repo's
floating branches, e.g.

	[super]
		floaters = refs/heads/master refs/heads/dev*

	[submodule "Sub1"]
		path = foo/bar
		branch = maint
		url = ...

	[submodule "Sub2"]
		path = other/place
		url = ...

This would mean that whenever the super-repo checks out either the "master"
branch or a branch whose name starts with "dev" (assuming recursive checkouts
are on):

  * The Sub1 submodule automatically checks out the tip of its
    "maint" branch.

  * The Sub2 submodule (lacking a "branch" variable) would not float
    and would check out the commit recorded in the super-repo.

A super-repo recursive-checkout that doesn't match a floaters pattern would
work in the regular, non-floating way.

		M.

[1] http://article.gmane.org/gmane.comp.version-control.git/186969

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Jens Lehmann]

Am 13.12.2011 15:17, schrieb Phil Hord:
> On Mon, Dec 12, 2011 at 2:36 PM, Junio C Hamano <gitster@pobox.com> wrote:
> [...]
>> Distro package dependency tracking is a poor analogy for many reasons, but
>> I'll only touch a few.
> [...]
>> Naively, one might think that two branches, branch-1.0 and branch-2.0, can
>> be defined in the repository of L, tell somebody (like "superproject that
>> covers all the packages in the distro") that A wants branch-1.0 and B
>> wants branch-2.0 of L respectively, to emulate this, but if one thinks
>> further, one would realize that it is insufficient. For one thing, it is
>> unclear what should happen when both A and B are asked to be checked out,
>> but more importantly, in dependency requirements on real distro packaging,
>> the application C could say "I want v1.0 API but v1.4 is broken and not
>> compatible with me", which won't fit on the two-branches model. A
>> workaround to add more branches to L could be devised but any workaround
>> cannot be a good solution that allows a random application C among 47
>> others to dictate how the branch structure of L project should look like.
>>
>> Fortunately, the dependency management is a solved problem by distro
>> package management and build systems, and they do so without using
>> anything from submodules. There is no point reinventing these logic in git
>> submodules and emulating poorly.
>>
>> The only remotely plausible analogy around distro packaging would be a
>> superproject full of all the packages in a distro as its submodules, and
>> records exact versions of each and every package that goes on a release
>> CD (or DVD). In that case, you do want to have a central registry that
>> records what exact version of each package is used to cut the CD and the
>> mother of all modules superproject could be one way to implement it. But
>> that is not an example of floating, but is a direct opposite.
>>
>> This exchange convinced me further that anybody who wishes to use
>> "floating" is better off either by doing one or both of the following:
>>
>>  - using "exact" but not updating religiously, as the interdepency
>>   requirement in their project is not strict; or
>>
>>  - not using submodules at all, but merely keeping these unrelated A, B, C
>>   and L as standalone repositories next to each other in the directory
>>   structure.
> 
> My interdependency requirements are not so cut-and-dry.  We use
> submodules to isolate controlled regions of code.  We may need to
> share our project with a contractor who is allowed to see code
> pertaining to "vendorA" but not that for "vendorB" or "VendorN".  But
> our in-house developers want to have all the vendor code in one place
> for convenient integration. Submodules do this nicely for us.  We can
> give the contractor just the main modules and the VendorA modules and
> he'll be fine.  In-house devs get all the submodules (using the
> vendor-ALL superproject).
> 
> But this necessarily means there is too much coupling for comfort
> between our submodules.   For example, when an API changes in the main
> submodule, each of the vendor submodules is affected because they each
> implement that API in a custom method.  Some of those vendor modules
> belong to different people.  Submodule synchronization becomes a real
> chore.

Hmm, maybe having vendor-specific branches in the superproject would
help here. But that is hard to tell without knowing more details about
your setup. But I suspect your vendor-ALL superproject is exactly the
right spot to deal with these kind of problems (and if that isn't easy
that might be a result of the difficulty of the problem you are trying
to solve here, keeping different vendors in sync with your API ;-).

> Floating would help, I think.  Instead I do this:
> 
>   git pull origin topic_foo && git submodule foreach 'git pull origin topic_foo'
> 
>   git submodule foreach 'git push origin topic_foo' && git push origin topic_foo

This sounds to me like you would need the "--recurse-submodules" option
implemented for "git pull" and "git push", no? And I miss to see how
floating would help when the tips of some submodules are not ready to
work with other submodules tips ...

> But not all my developers are git-gurus yet, and they sometimes mess
> up their ad hoc scripts or miss important changes they forgot to push
> in one submodule or another.

Sure, even though current git should help you some by showing changes
in the submodules.

>  Or worse, their pull or push fails and
> they can't see the problem for all the noise.  So they email it to me.

We circumvent that by not pulling, but fetching and merging in the
submodule first and after that in the superproject. You have much more
control about what is going wrong where (and can have more
git-experienced people help with - or even do - the merges).

> On my git server, I have a hook that automatically propagates each
> push to "master" from the submodules into the superproject.  But this
> is tedious and limited.  And it relies on a centralized server.

But for closely related stuff that is a good option. Our continuous
integration server shows us quite some breakage between submodules
before they hit a superproject, which is really helpful.

> You may say this itch is all in my head, but it sure seems real to me.

This definitely is a real problem. Lets see how far git can help you
here...

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Jens Lehmann]

Am 13.12.2011 16:35, schrieb Marc Branchaud:
> I'd prefer to have floating be explicitly configured on a per-branch (or
> per-branch-glob) basis.  So in addition to what Jens described yesterday [1]
> to configure an individual submodule's floating branch, I suggest there also
> be a new section in the .gitmodules file for configuring the super-repo's
> floating branches, e.g.
> 
> 	[super]
> 		floaters = refs/heads/master refs/heads/dev*
> 
> 	[submodule "Sub1"]
> 		path = foo/bar
> 		branch = maint
> 		url = ...
> 
> 	[submodule "Sub2"]
> 		path = other/place
> 		url = ...

Hmm, but you can have different .gitmodules files in different branches of
the superproject, no? Why not just have the "branch = maint" setting for
"Sub1" in the master and the dev branches .gitmodules file and drop it in
the other branches?

> This would mean that whenever the super-repo checks out either the "master"
> branch or a branch whose name starts with "dev" (assuming recursive checkouts
> are on):
> 
>   * The Sub1 submodule automatically checks out the tip of its
>     "maint" branch.
> 
>   * The Sub2 submodule (lacking a "branch" variable) would not float
>     and would check out the commit recorded in the super-repo.
> 
> A super-repo recursive-checkout that doesn't match a floaters pattern would
> work in the regular, non-floating way.

Which would just work with my proposal too if git would honor the
.gitmodules file of the currently checked out branch.

> [1] http://article.gmane.org/gmane.comp.version-control.git/186969

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Marc Branchaud]

On 11-12-13 04:19 PM, Jens Lehmann wrote:
> Am 13.12.2011 16:35, schrieb Marc Branchaud:
>> I'd prefer to have floating be explicitly configured on a per-branch (or
>> per-branch-glob) basis.  So in addition to what Jens described yesterday [1]
>> to configure an individual submodule's floating branch, I suggest there also
>> be a new section in the .gitmodules file for configuring the super-repo's
>> floating branches, e.g.
>>
>> 	[super]
>> 		floaters = refs/heads/master refs/heads/dev*
>>
>> 	[submodule "Sub1"]
>> 		path = foo/bar
>> 		branch = maint
>> 		url = ...
>>
>> 	[submodule "Sub2"]
>> 		path = other/place
>> 		url = ...
> 
> Hmm, but you can have different .gitmodules files in different branches of
> the superproject, no?

Yes.  I'm not sure I see that as a problem though.

> Why not just have the "branch = maint" setting for
> "Sub1" in the master and the dev branches .gitmodules file and drop it in
> the other branches?

Because I think that's an error-prone approach.

If the user creates a topic branch off (an ancestor) of master, git doesn't
know if the user wants floating submodules or not.  If this is a bugfix
topic, the user would have to edit .gitmodules to turn off floating.  But
that modified .gitmodules is too easily committed to the branch, and once it
gets merged back into master suddenly master loses its "floating" feature.

What's more, less-sophisticated users would be wary of editing an "internal"
file like .gitmodules.

Instead I think it's more intuitive for the repository to define which
branches get floating submodules and which don't, and IMO a list of
names/globs is a good way to do that.  The repo's users would need to be
aware of what the magic branch names are, but I think that's easily
communicated in the floating-submodule scenarios I've seen posted.  Git could
also help by telling the user, when a branch is created or it's name is
displayed, whether or not it's got floating submodules.

		M.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Phil Hord]

I lost my grip on this thread over the holidays...

On Tue, Dec 13, 2011 at 4:09 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 13.12.2011 15:17, schrieb Phil Hord:
>> On Mon, Dec 12, 2011 at 2:36 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> [...]
>>> Distro package dependency tracking is a poor analogy for many reasons, but
>>> I'll only touch a few.
>> [...]
>>> Naively, one might think that two branches, branch-1.0 and branch-2.0, can
>>> be defined in the repository of L, tell somebody (like "superproject that
>>> covers all the packages in the distro") that A wants branch-1.0 and B
>>> wants branch-2.0 of L respectively, to emulate this, but if one thinks
>>> further, one would realize that it is insufficient. For one thing, it is
>>> unclear what should happen when both A and B are asked to be checked out,
>>> but more importantly, in dependency requirements on real distro packaging,
>>> the application C could say "I want v1.0 API but v1.4 is broken and not
>>> compatible with me", which won't fit on the two-branches model. A
>>> workaround to add more branches to L could be devised but any workaround
>>> cannot be a good solution that allows a random application C among 47
>>> others to dictate how the branch structure of L project should look like.
>>>
>>> Fortunately, the dependency management is a solved problem by distro
>>> package management and build systems, and they do so without using
>>> anything from submodules. There is no point reinventing these logic in git
>>> submodules and emulating poorly.
>>>
>>> The only remotely plausible analogy around distro packaging would be a
>>> superproject full of all the packages in a distro as its submodules, and
>>> records exact versions of each and every package that goes on a release
>>> CD (or DVD). In that case, you do want to have a central registry that
>>> records what exact version of each package is used to cut the CD and the
>>> mother of all modules superproject could be one way to implement it. But
>>> that is not an example of floating, but is a direct opposite.
>>>
>>> This exchange convinced me further that anybody who wishes to use
>>> "floating" is better off either by doing one or both of the following:
>>>
>>>  - using "exact" but not updating religiously, as the interdepency
>>>   requirement in their project is not strict; or
>>>
>>>  - not using submodules at all, but merely keeping these unrelated A, B, C
>>>   and L as standalone repositories next to each other in the directory
>>>   structure.
>>
>> My interdependency requirements are not so cut-and-dry.  We use
>> submodules to isolate controlled regions of code.  We may need to
>> share our project with a contractor who is allowed to see code
>> pertaining to "vendorA" but not that for "vendorB" or "VendorN".  But
>> our in-house developers want to have all the vendor code in one place
>> for convenient integration. Submodules do this nicely for us.  We can
>> give the contractor just the main modules and the VendorA modules and
>> he'll be fine.  In-house devs get all the submodules (using the
>> vendor-ALL superproject).
>>
>> But this necessarily means there is too much coupling for comfort
>> between our submodules.   For example, when an API changes in the main
>> submodule, each of the vendor submodules is affected because they each
>> implement that API in a custom method.  Some of those vendor modules
>> belong to different people.  Submodule synchronization becomes a real
>> chore.
>
> Hmm, maybe having vendor-specific branches in the superproject would
> help here. But that is hard to tell without knowing more details about
> your setup. But I suspect your vendor-ALL superproject is exactly the
> right spot to deal with these kind of problems (and if that isn't easy
> that might be a result of the difficulty of the problem you are trying
> to solve here, keeping different vendors in sync with your API ;-).
>
>> Floating would help, I think.  Instead I do this:
>>
>>   git pull origin topic_foo && git submodule foreach 'git pull origin topic_foo'
>>
>>   git submodule foreach 'git push origin topic_foo' && git push origin topic_foo
>
> This sounds to me like you would need the "--recurse-submodules" option
> implemented for "git pull" and "git push", no?

Only if I have nested submodules, but yes, we do use --recurs* in our scripts.

> And I miss to see how
> floating would help when the tips of some submodules are not ready to
> work with other submodules tips ...

By project policy, for any branch, all submodules' tips of the
same-named branch should be interoperable.  The CI server looks after
this, as much as he can.

I think of branch names as sticky notes (extra-lightweight tags,
sometimes).  We have linear history in many of our vendor submodules,
but multiple "branches" indicate where each superproject branch has
presumably finished integration.

>> But not all my developers are git-gurus yet, and they sometimes mess
>> up their ad hoc scripts or miss important changes they forgot to push
>> in one submodule or another.
>
> Sure, even though current git should help you some by showing changes
> in the submodules.

Real newbies may not even remember to use 'git status' strategically.

>>  Or worse, their pull or push fails and
>> they can't see the problem for all the noise.  So they email it to me.
>
> We circumvent that by not pulling, but fetching and merging in the
> submodule first and after that in the superproject. You have much more
> control about what is going wrong where (and can have more
> git-experienced people help with - or even do - the merges).

I do that, too, and I wish I didn't have to.  I wish I could safely
and sanely recover from a conflicted "git pull --recurse-submodules"
pull from the superproject.  That is, I wish doing so were as
straightforward as recovering from the same condition would be if all
my code were in one repository instead of in submodules.

Which is the gist -- I wish submodules did not make git more
complicated than it already is.

Phil

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Jens Lehmann]

Am 30.01.2012 22:15, schrieb Phil Hord:
> I lost my grip on this thread over the holidays...
> 
> On Tue, Dec 13, 2011 at 4:09 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Am 13.12.2011 15:17, schrieb Phil Hord:
>>>   git pull origin topic_foo && git submodule foreach 'git pull origin topic_foo'
>>>
>>>   git submodule foreach 'git push origin topic_foo' && git push origin topic_foo
>>
>> This sounds to me like you would need the "--recurse-submodules" option
>> implemented for "git pull" and "git push", no?
> 
> Only if I have nested submodules, but yes, we do use --recurs* in our scripts.

I'm confused, push doesn't know the "--recurse-submodules" option
at all yet while pull only does a deep fetch when it is given, the
submodule work trees are not updated to the merge result right now.

>> And I miss to see how
>> floating would help when the tips of some submodules are not ready to
>> work with other submodules tips ...
> 
> By project policy, for any branch, all submodules' tips of the
> same-named branch should be interoperable.  The CI server looks after
> this, as much as he can.

We do the same thing on our CI server, but it can only test some
combinations (even though that tends to show most problems pretty
early). But in the end every superproject is responsible to use a
working set of submodule commits, and I would rather bet on a
combination the CI server tested than on what happens to be on the
current tips.

> I think of branch names as sticky notes (extra-lightweight tags,
> sometimes).  We have linear history in many of our vendor submodules,
> but multiple "branches" indicate where each superproject branch has
> presumably finished integration.

We also add a branch in submodules every time a superproject needs
to move away from the submodules master (so the commits won't get
lost by accident).

>>> But not all my developers are git-gurus yet, and they sometimes mess
>>> up their ad hoc scripts or miss important changes they forgot to push
>>> in one submodule or another.
>>
>> Sure, even though current git should help you some by showing changes
>> in the submodules.
> 
> Real newbies may not even remember to use 'git status' strategically.

Hmm, but then they will screw up things in the superproject too, no?

>>>  Or worse, their pull or push fails and
>>> they can't see the problem for all the noise.  So they email it to me.
>>
>> We circumvent that by not pulling, but fetching and merging in the
>> submodule first and after that in the superproject. You have much more
>> control about what is going wrong where (and can have more
>> git-experienced people help with - or even do - the merges).
> 
> I do that, too, and I wish I didn't have to.  I wish I could safely
> and sanely recover from a conflicted "git pull --recurse-submodules"
> pull from the superproject.

That's what my recursive checkout work is aiming at. Me thinks after
that we will also need some good ideas on how to present and help
solving submodule merge conflicts.

> That is, I wish doing so were as
> straightforward as recovering from the same condition would be if all
> my code were in one repository instead of in submodules.
>
> Which is the gist -- I wish submodules did not make git more
> complicated than it already is.

I think we can make working with submodule much easier than it is
now, the next step being updating all submodule work trees as git
updates the superproject's work tree. Even though I suspect that in
the long run submodules will always be a bit more complicated than
having everything in one repository, I'm confident that will be by
far outweighed by the advantages they bring.

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Phil Hord]

On Tue, Jan 31, 2012 at 3:55 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 30.01.2012 22:15, schrieb Phil Hord:
>> I lost my grip on this thread over the holidays...
>>
>> On Tue, Dec 13, 2011 at 4:09 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>> Am 13.12.2011 15:17, schrieb Phil Hord:
>>>>   git pull origin topic_foo && git submodule foreach 'git pull origin topic_foo'
>>>>
>>>>   git submodule foreach 'git push origin topic_foo' && git push origin topic_foo
>>>
>>> This sounds to me like you would need the "--recurse-submodules" option
>>> implemented for "git pull" and "git push", no?
>>
>> Only if I have nested submodules, but yes, we do use --recurs* in our scripts.
>
> I'm confused, push doesn't know the "--recurse-submodules" option
> at all yet while pull only does a deep fetch when it is given, the
> submodule work trees are not updated to the merge result right now.

Sorry.  I type faster than I think sometimes.

I meant that I do use something like this:
    git submodule foreach --recursive 'git checkout master' && git
checkout master

And I expect to use something like this:
    git submodule foreach --recursive 'git push origin topic_foo' &&
git push origin topic_foo

>>> And I miss to see how
>>> floating would help when the tips of some submodules are not ready to
>>> work with other submodules tips ...
>>
>> By project policy, for any branch, all submodules' tips of the
>> same-named branch should be interoperable.  The CI server looks after
>> this, as much as he can.
>
> We do the same thing on our CI server, but it can only test some
> combinations (even though that tends to show most problems pretty
> early). But in the end every superproject is responsible to use a
> working set of submodule commits, and I would rather bet on a
> combination the CI server tested than on what happens to be on the
> current tips.

Yes, I see what you mean.  In our case, what "happens to be on the
tips" should also have been vetted by our Gerrit+Jenkins code review
dance (for each branch * each superproject), and so it should always
be good.  If it's not, we have the Jenkins-maintained gitlink history
we can use to bisect.

>> I think of branch names as sticky notes (extra-lightweight tags,
>> sometimes).  We have linear history in many of our vendor submodules,
>> but multiple "branches" indicate where each superproject branch has
>> presumably finished integration.
>
> We also add a branch in submodules every time a superproject needs
> to move away from the submodules master (so the commits won't get
> lost by accident).

Good idea.  Gerrit sees to this for us also, but we do share code on
ancillary git servers and we follow the same practice.

>>>> But not all my developers are git-gurus yet, and they sometimes mess
>>>> up their ad hoc scripts or miss important changes they forgot to push
>>>> in one submodule or another.
>>>
>>> Sure, even though current git should help you some by showing changes
>>> in the submodules.
>>
>> Real newbies may not even remember to use 'git status' strategically.
>
> Hmm, but then they will screw up things in the superproject too, no?

What I mean is that a developer may be completely focused on one
particular submodule (his domain).  He does his work in this module,
and when it's ready he commits and pushes to the server.  'git status'
shows him that his directory is clean.  But this is only because he
doesn't really know where the submodules top-directories are, so he
doesn't realize that he has changes in another submodule that he has
not committed.  He has to know to run 'git status' from somewhere in
the superproject (ostensibly in the root directory of that
superproject).  But he may forget since 'git status' already assured
him he was done.

Like this:

#-- Setup
mkdir super && cd super && git init
mkdir A && touch A/foo
git add .
git submodule add gerrit:iptv/iptv_scripts B
git commit -m "Initial commit"

#-- Work flow
cd A && echo changes > foo
cd ../B && echo changes > foo
git add . && git commit -m "Made some changes"
git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
nothing to commit (working directory clean)

So git just told my new-to-git developer that his workdir is clean
because he happened to run it from super/B.  But if he ran it from
super/A (or super) it would tell him otherwise.

I guess what would help here is something like the opposite of 'git
status' showing the status of descendant submodules;  it would help if
it showed the status of sibling submodules and the superproject as
well.

>>>>  Or worse, their pull or push fails and
>>>> they can't see the problem for all the noise.  So they email it to me.
>>>
>>> We circumvent that by not pulling, but fetching and merging in the
>>> submodule first and after that in the superproject. You have much more
>>> control about what is going wrong where (and can have more
>>> git-experienced people help with - or even do - the merges).
>>
>> I do that, too, and I wish I didn't have to.  I wish I could safely
>> and sanely recover from a conflicted "git pull --recurse-submodules"
>> pull from the superproject.
>
> That's what my recursive checkout work is aiming at. Me thinks after
> that we will also need some good ideas on how to present and help
> solving submodule merge conflicts.
>
>> That is, I wish doing so were as
>> straightforward as recovering from the same condition would be if all
>> my code were in one repository instead of in submodules.
>>
>> Which is the gist -- I wish submodules did not make git more
>> complicated than it already is.
>
> I think we can make working with submodule much easier than it is
> now, the next step being updating all submodule work trees as git
> updates the superproject's work tree. Even though I suspect that in
> the long run submodules will always be a bit more complicated than
> having everything in one repository, I'm confident that will be by
> far outweighed by the advantages they bring.

I'm really looking forward to it.

Thanks,
Phil

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Jens Lehmann]

Am 31.01.2012 23:50, schrieb Phil Hord:
> What I mean is that a developer may be completely focused on one
> particular submodule (his domain).  He does his work in this module,
> and when it's ready he commits and pushes to the server.  'git status'
> shows him that his directory is clean.  But this is only because he
> doesn't really know where the submodules top-directories are, so he
> doesn't realize that he has changes in another submodule that he has
> not committed.  He has to know to run 'git status' from somewhere in
> the superproject (ostensibly in the root directory of that
> superproject).  But he may forget since 'git status' already assured
> him he was done.
<snip>
> I guess what would help here is something like the opposite of 'git
> status' showing the status of descendant submodules;  it would help if
> it showed the status of sibling submodules and the superproject as
> well.

Hmm, I really think the fact that submodules are unaware that they
are part of a superproject is a feature. I'd prefer seeing that kind
of problem being tackled by the CI server and/or user education. Or
maybe a pre-commit hook which issues a warning in that case?

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Phil Hord]

On Wed, Feb 1, 2012 at 5:37 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 31.01.2012 23:50, schrieb Phil Hord:
>> What I mean is that a developer may be completely focused on one
>> particular submodule (his domain).  He does his work in this module,
>> and when it's ready he commits and pushes to the server.  'git status'
>> shows him that his directory is clean.  But this is only because he
>> doesn't really know where the submodules top-directories are, so he
>> doesn't realize that he has changes in another submodule that he has
>> not committed.  He has to know to run 'git status' from somewhere in
>> the superproject (ostensibly in the root directory of that
>> superproject).  But he may forget since 'git status' already assured
>> him he was done.
> <snip>
>> I guess what would help here is something like the opposite of 'git
>> status' showing the status of descendant submodules;  it would help if
>> it showed the status of sibling submodules and the superproject as
>> well.
>
> Hmm, I really think the fact that submodules are unaware that they
> are part of a superproject is a feature. I'd prefer seeing that kind
> of problem being tackled by the CI server and/or user education. Or
> maybe a pre-commit hook which issues a warning in that case?

I agree that submodule isolation is a feature essential to the
architecture of git and the submodules implementation.  But it is also
a limitation, not just of this example.  A pre-commit hook is a nice
idea, but it doesn't help 'git status' (which is the standard go-to
answer point for "where am I").

This has me thinking more about recursing siblings now, though. I find
myself typing something like this quite a lot:
    git submodule foreach 'git grep "someFunction" || :'

Or worse (in that the UI is more unwieldy):
    git submodule foreach 'git log --oneline "-SsomeFunction" || :'

But what I want is this:
    git --git-dir=${TOP}/../.git grep --recurse-submodules "someFunction"

But not really, because I am lazy and that is too much typing.
    git grep --include-siblings "someFunction"

Maybe I can add a "sib" macro to get this:
    git sib grep "someFunction"

But now I've really wandered off-topic.
Phil

Re: [RFC/PATCH] add update to branch support for "floating submodules"

[Jens Lehmann]

Am 06.02.2012 18:31, schrieb Phil Hord:
> On Wed, Feb 1, 2012 at 5:37 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Hmm, I really think the fact that submodules are unaware that they
>> are part of a superproject is a feature. I'd prefer seeing that kind
>> of problem being tackled by the CI server and/or user education. Or
>> maybe a pre-commit hook which issues a warning in that case?
> 
> I agree that submodule isolation is a feature essential to the
> architecture of git and the submodules implementation.  But it is also
> a limitation, not just of this example.  A pre-commit hook is a nice
> idea, but it doesn't help 'git status' (which is the standard go-to
> answer point for "where am I").

Yes, this feature also is a limitation. To put it in other words: I want
each submodule to be a full fledged repo of its own. IMO it must always
be possible to just clone a submodule without any superproject and run
any git command in it. And the other way around: each superproject must
be usable as a submodule in another superproject. That makes adding some
kind of "superproject awareness" in a sane way rather difficult, as a
repo can't say "I live in a superproject" or "I am the topmost project".

> This has me thinking more about recursing siblings now, though. I find
> myself typing something like this quite a lot:
>     git submodule foreach 'git grep "someFunction" || :'

There was an attempt to teach git grep the --recurse-submodules option,
but unfortunately it looks like this didn't lead anywhere so far.

> Or worse (in that the UI is more unwieldy):
>     git submodule foreach 'git log --oneline "-SsomeFunction" || :'

This could also be done by teaching git log the --recurse-submodules
option. Me thinks in the long run a lot of git commands should learn
that option to make it easy to optionally include submodules in
whatever they are doing. But my focus is on recursive checkout for the
next time, so I have no idea when I find some time to do that.

> But what I want is this:
>     git --git-dir=${TOP}/../.git grep --recurse-submodules "someFunction"
> 
> But not really, because I am lazy and that is too much typing.
>     git grep --include-siblings "someFunction"
> 
> Maybe I can add a "sib" macro to get this:
>     git sib grep "someFunction"

And from what you where saying earlier a "git sib status" would be nice
too? What about using alias commands for that functionality? They could
point to a script which searches the topmost repo and calls "git submodule
foreach 'git <command> "$@" || :'" from there ...