Updating a submodule with a compatible version from another submodule version using the parent meta-repository

Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

I am using git submodule in one of my professional projects and I am
facing an issue when I am using git bisect in one of the submodules.

Basically I have a meta repository which I will call A and two
submodules B and C. Sometimes I use git bisect in B but it is
dependent on C so when I go back too much in the history of B, C needs
to change its version to a compatible one. Doing this manually is
really time consuming for me and I guess a lot of people have this
issue so I was a little bit surprise to not find easily anything on
the net that permits to do this automatically.

Is there anything existing to do that and I just didn't find it yet?
If not I think I might have an implementation idea I would like to try
out.

Thanks for your time,

Julian Ibarz

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Jens Lehmann]

Am 26.01.2011 19:32, schrieb Julian Ibarz:
> I am using git submodule in one of my professional projects and I am
> facing an issue when I am using git bisect in one of the submodules.
> 
> Basically I have a meta repository which I will call A and two
> submodules B and C. Sometimes I use git bisect in B but it is
> dependent on C so when I go back too much in the history of B, C needs
> to change its version to a compatible one. Doing this manually is
> really time consuming for me and I guess a lot of people have this
> issue so I was a little bit surprise to not find easily anything on
> the net that permits to do this automatically.

What about bisecting in A (doing "git submodule update" after every
step) to bisect to a smaller range of commits in B (which are then
not dependent on your submodule C anymore and can be bisected inside
B)? This of course assumes A properly records the dependencies
between B and C.

> Is there anything existing to do that and I just didn't find it yet?
> If not I think I might have an implementation idea I would like to try
> out.

The call to "git submodule update" after each bisect step in the
superproject will be obsolete as soon as the recursive checkout
I am currently working on is done, but that is not here yet.

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

On Wed, Jan 26, 2011 at 2:06 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 26.01.2011 19:32, schrieb Julian Ibarz:
>> I am using git submodule in one of my professional projects and I am
>> facing an issue when I am using git bisect in one of the submodules.
>>
>> Basically I have a meta repository which I will call A and two
>> submodules B and C. Sometimes I use git bisect in B but it is
>> dependent on C so when I go back too much in the history of B, C needs
>> to change its version to a compatible one. Doing this manually is
>> really time consuming for me and I guess a lot of people have this
>> issue so I was a little bit surprise to not find easily anything on
>> the net that permits to do this automatically.
>
> What about bisecting in A (doing "git submodule update" after every
> step) to bisect to a smaller range of commits in B (which are then
> not dependent on your submodule C anymore and can be bisected inside
> B)? This of course assumes A properly records the dependencies
> between B and C.

Yes but actually my real use case that made me write this mail was
more I have a feature done in an old branch and to try it I never to
revert back to this version. In this case, I have to find out the
corresponding good version in A and C. In this case I cannot start
like what you propose in A to find out the good version in B and C, I
already know the version I want in B.

>> Is there anything existing to do that and I just didn't find it yet?
>> If not I think I might have an implementation idea I would like to try
>> out.
>
> The call to "git submodule update" after each bisect step in the
> superproject will be obsolete as soon as the recursive checkout
> I am currently working on is done, but that is not here yet.

Can you be more detailed about your recursive checkout feature? Is it
what I proposed?

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Jens Lehmann]

Am 26.01.2011 20:10, schrieb Julian Ibarz:
> On Wed, Jan 26, 2011 at 2:06 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Am 26.01.2011 19:32, schrieb Julian Ibarz:
>>> I am using git submodule in one of my professional projects and I am
>>> facing an issue when I am using git bisect in one of the submodules.
>>>
>>> Basically I have a meta repository which I will call A and two
>>> submodules B and C. Sometimes I use git bisect in B but it is
>>> dependent on C so when I go back too much in the history of B, C needs
>>> to change its version to a compatible one. Doing this manually is
>>> really time consuming for me and I guess a lot of people have this
>>> issue so I was a little bit surprise to not find easily anything on
>>> the net that permits to do this automatically.
>>
>> What about bisecting in A (doing "git submodule update" after every
>> step) to bisect to a smaller range of commits in B (which are then
>> not dependent on your submodule C anymore and can be bisected inside
>> B)? This of course assumes A properly records the dependencies
>> between B and C.
> 
> Yes but actually my real use case that made me write this mail was
> more I have a feature done in an old branch and to try it I never to
> revert back to this version. In this case, I have to find out the
> corresponding good version in A and C. In this case I cannot start
> like what you propose in A to find out the good version in B and C, I
> already know the version I want in B.

Hmm, looks like I lost you here ... you want to bisect in B although
you know what commit you want there? Care to explain a bit more?

>>> Is there anything existing to do that and I just didn't find it yet?
>>> If not I think I might have an implementation idea I would like to try
>>> out.
>>
>> The call to "git submodule update" after each bisect step in the
>> superproject will be obsolete as soon as the recursive checkout
>> I am currently working on is done, but that is not here yet.
> 
> Can you be more detailed about your recursive checkout feature? Is it
> what I proposed?

I don't think so, that will just get rid of the extra call to "git
submodule update" when bisecting in A.

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

> Am 26.01.2011 20:10, schrieb Julian Ibarz:
>> On Wed, Jan 26, 2011 at 2:06 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>> Am 26.01.2011 19:32, schrieb Julian Ibarz:
>>>> I am using git submodule in one of my professional projects and I am
>>>> facing an issue when I am using git bisect in one of the submodules.
>>>>
>>>> Basically I have a meta repository which I will call A and two
>>>> submodules B and C. Sometimes I use git bisect in B but it is
>>>> dependent on C so when I go back too much in the history of B, C needs
>>>> to change its version to a compatible one. Doing this manually is
>>>> really time consuming for me and I guess a lot of people have this
>>>> issue so I was a little bit surprise to not find easily anything on
>>>> the net that permits to do this automatically.
>>>
>>> What about bisecting in A (doing "git submodule update" after every
>>> step) to bisect to a smaller range of commits in B (which are then
>>> not dependent on your submodule C anymore and can be bisected inside
>>> B)? This of course assumes A properly records the dependencies
>>> between B and C.
>>
>> Yes but actually my real use case that made me write this mail was
>> more I have a feature done in an old branch and to try it I never to
>> revert back to this version. In this case, I have to find out the
>> corresponding good version in A and C. In this case I cannot start
>> like what you propose in A to find out the good version in B and C, I
>> already know the version I want in B.
>
> Hmm, looks like I lost you here ... you want to bisect in B although
> you know what commit you want there? Care to explain a bit more?

In B I have a feature to integrate in master branch. This feature is
in branch old_feature. But this branch is really old. To try this
feature I need to rebuild it at this version. To make the build
success I need also to revert back the submodule C because B is
dependent on it. But finding the good version of C that match
old_feature version is a pain... Is it clear?

>>>> Is there anything existing to do that and I just didn't find it yet?
>>>> If not I think I might have an implementation idea I would like to try
>>>> out.
>>>
>>> The call to "git submodule update" after each bisect step in the
>>> superproject will be obsolete as soon as the recursive checkout
>>> I am currently working on is done, but that is not here yet.
>>
>> Can you be more detailed about your recursive checkout feature? Is it
>> what I proposed?
>
> I don't think so, that will just get rid of the extra call to "git
> submodule update" when bisecting in A.

Basically my feature would work like this:

in B:
git submodule checkout some_version

This will checkout B but also change A and C so that it is compatible
with some_version of B. Basically it will find the commit in A that
has the closest parent commit of some_version in B. When this is done
it just does git submodule udate on other submodules.

I see in gitk that there is a feature that has a common implementation
for what I want to do:

For every commits you can see Follows and Precedes which lists the
closest label before this release and after. What I need is the same
thing: instead of finding a closest labeled commit, I need to find a
closest commit referenced by A that precedes current HEAD of B. When
this is done I know which commit A has to be and then just have to
call git submodule update in A (update every other submodules except
for B).

Julian

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Jens Lehmann]

Am 26.01.2011 20:48, schrieb Julian Ibarz:
> Basically my feature would work like this:
> 
> in B:
> git submodule checkout some_version
> 
> This will checkout B but also change A and C so that it is compatible
> with some_version of B. Basically it will find the commit in A that
> has the closest parent commit of some_version in B. When this is done
> it just does git submodule udate on other submodules.

Thanks, now I understand what you are trying to achieve.

> I see in gitk that there is a feature that has a common implementation
> for what I want to do:
> 
> For every commits you can see Follows and Precedes which lists the
> closest label before this release and after. What I need is the same
> thing: instead of finding a closest labeled commit, I need to find a
> closest commit referenced by A that precedes current HEAD of B. When
> this is done I know which commit A has to be and then just have to
> call git submodule update in A (update every other submodules except
> for B).

I am not aware of something like that in current Git, But I see that
such functionality would be helpful. Care to share your implementation
idea?

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

> Am 26.01.2011 20:48, schrieb Julian Ibarz:
>> Basically my feature would work like this:
>>
>> in B:
>> git submodule checkout some_version
>>
>> This will checkout B but also change A and C so that it is compatible
>> with some_version of B. Basically it will find the commit in A that
>> has the closest parent commit of some_version in B. When this is done
>> it just does git submodule udate on other submodules.
>
> Thanks, now I understand what you are trying to achieve.
>
>> I see in gitk that there is a feature that has a common implementation
>> for what I want to do:
>>
>> For every commits you can see Follows and Precedes which lists the
>> closest label before this release and after. What I need is the same
>> thing: instead of finding a closest labeled commit, I need to find a
>> closest commit referenced by A that precedes current HEAD of B. When
>> this is done I know which commit A has to be and then just have to
>> call git submodule update in A (update every other submodules except
>> for B).
>
> I am not aware of something like that in current Git, But I see that
> such functionality would be helpful. Care to share your implementation
> idea?

Well actually the paragraph just above is what is my implementation
idea. I recognize it is really high level but it is still a start ;).
Basically the "complex" part is to find the precedent commit in B that
is referenced by A. Since gitk has this already but for labels, we
just need to reuse/copy their function that does this but instead as
input having a list of commit labels and find the closest precedent it
is instead the list of commits referenced by A. By the way, anyone
reading this knows where it is in the code of gitk? Or if there is
something like that already in the git code? I never looked into both
code projects.

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Junio C Hamano]

Julian Ibarz <julian.ibarz@gmail.com> writes:

>> Hmm, looks like I lost you here ... you want to bisect in B although
>> you know what commit you want there? Care to explain a bit more?
>
> In B I have a feature to integrate in master branch. This feature is
> in branch old_feature. But this branch is really old. To try this
> feature I need to rebuild it at this version. To make the build
> success I need also to revert back the submodule C because B is
> dependent on it. But finding the good version of C that match
> old_feature version is a pain... Is it clear?

That sounds like bisecting in C with a frozen checkout of B to see which
version in C works well with that target commit in B you know you want to
use.  Why do you need to bisect in B???

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

On Wed, Jan 26, 2011 at 3:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Julian Ibarz <julian.ibarz@gmail.com> writes:
>
>>> Hmm, looks like I lost you here ... you want to bisect in B although
>>> you know what commit you want there? Care to explain a bit more?
>>
>> In B I have a feature to integrate in master branch. This feature is
>> in branch old_feature. But this branch is really old. To try this
>> feature I need to rebuild it at this version. To make the build
>> success I need also to revert back the submodule C because B is
>> dependent on it. But finding the good version of C that match
>> old_feature version is a pain... Is it clear?
>
> That sounds like bisecting in C with a frozen checkout of B to see which
> version in C works well with that target commit in B you know you want to
> use.  Why do you need to bisect in B???
>

Forget about bisect. This is a different use case where I need the
feature I am talking about: checkout an old version in B and
automatically having A and C switch to a compatible version (the cause
can be because of a bisect or just because I want to try an old
feature not yet integrated into master or whatever the reason I want
to be on this old version).

Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Junio C Hamano]

Julian Ibarz <julian.ibarz@gmail.com> writes:

> On Wed, Jan 26, 2011 at 3:41 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Julian Ibarz <julian.ibarz@gmail.com> writes:
>>
>>>> Hmm, looks like I lost you here ... you want to bisect in B although
>>>> you know what commit you want there? Care to explain a bit more?
>>>
>>> In B I have a feature to integrate in master branch. This feature is
>>> in branch old_feature. But this branch is really old. To try this
>>> feature I need to rebuild it at this version. To make the build
>>> success I need also to revert back the submodule C because B is
>>> dependent on it. But finding the good version of C that match
>>> old_feature version is a pain... Is it clear?
>>
>> That sounds like bisecting in C with a frozen checkout of B to see which
>> version in C works well with that target commit in B you know you want to
>> use.  Why do you need to bisect in B???
>
> Forget about bisect. This is a different use case where I need the
> feature I am talking about: checkout an old version in B and
> automatically having A and C switch to a compatible version (the cause
> can be because of a bisect or just because I want to try an old
> feature not yet integrated into master or whatever the reason I want
> to be on this old version).

If that version of submodule B is explicitly bound to a commit in the
superproject A, you know which version of A and C were recorded, and the
problem is solved.

But otherwise, you are wishing for a miracle, I am afraid, without giving
your tool (git or your own tool you write on top of it) some hint to say
how "similar" commits in B are, as there is no guarantee that anybody even
tried a commit that is not directly referenced from the superproject
together with other parts of the system.

If you are confident that you didn't introduce different kind of
dependency to other submodules while developing your "old_feature" branch
in submodule B, one strategy may be to find an ancestor, preferrably the
fork point, of your "old_feature" branch that is bound to the superproject
A.  Then at that point at least you know whoever made that commit in A
tested the combination of what was recorded in that commit, together with
the version of B and C, and you can go forward from there, replaying the
changes you made to the "old_feature" branch in submodule B.

Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Heiko Voigt]

Hi,

On Wed, Jan 26, 2011 at 02:05:43PM -0800, Junio C Hamano wrote:
> If that version of submodule B is explicitly bound to a commit in the
> superproject A, you know which version of A and C were recorded, and the
> problem is solved.
> 
[...]
> 
> If you are confident that you didn't introduce different kind of
> dependency to other submodules while developing your "old_feature" branch
> in submodule B, one strategy may be to find an ancestor, preferrably the
> fork point, of your "old_feature" branch that is bound to the superproject
> A.  Then at that point at least you know whoever made that commit in A
> tested the combination of what was recorded in that commit, together with
> the version of B and C, and you can go forward from there, replaying the
> changes you made to the "old_feature" branch in submodule B.

Lets extend your explanation a little further and maybe demonstrate the problem
Julian is having a little more. I think what Julian searches for is a tool in
git that does the lookup for you which is AFAIK not that easy currently. It
seems to be a quite useful feature. Here what I understand Julian wants:

1. Find the most recent superproject commit X'' in A that records a submodule
   commit X' in B which contains the commit X in B you are searching for.

   For this we would need use something similar to git describe --contains
   but instead of using the list of existing tags in B it should use the list
   of commits in B which are recorded in A.

   Here a drawing to explain (linear history for simplicity):

   superproject A:

      O---O---X''---O
               \
   submodule B: \
                 \
      O---X---O---X'---O---O

2. Look up the commit of C which is recorded in X'' of A and check it
   out.

Step 2 is easy but for Step 1 the lookup of X' is missing for the commandline.
Is there already anything that implements git describe --contains for a defined
list of commits instead of refs?

Cheers Heiko

Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

Today I have started to implement a proof of concept in C (I know a
script would be better but I am really not good in sh so...). I
struggle with the manipulation of the git API. I have pushed my work
here:

http://gitorious.org/julian_ibarz_git/julian_ibarz_git

in branch submodule_checkout

My work is in:

builtin/submodulecheckout.c

And my questions are prepended by the keyword QUESTION (two questions
for now only).

Any help is welcome.

Thanks,

Julian Ibarz

On Sat, Jan 29, 2011 at 6:08 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> Hi,
>
> On Wed, Jan 26, 2011 at 02:05:43PM -0800, Junio C Hamano wrote:
>> If that version of submodule B is explicitly bound to a commit in the
>> superproject A, you know which version of A and C were recorded, and the
>> problem is solved.
>>
> [...]
>>
>> If you are confident that you didn't introduce different kind of
>> dependency to other submodules while developing your "old_feature" branch
>> in submodule B, one strategy may be to find an ancestor, preferrably the
>> fork point, of your "old_feature" branch that is bound to the superproject
>> A.  Then at that point at least you know whoever made that commit in A
>> tested the combination of what was recorded in that commit, together with
>> the version of B and C, and you can go forward from there, replaying the
>> changes you made to the "old_feature" branch in submodule B.
>
> Lets extend your explanation a little further and maybe demonstrate the problem
> Julian is having a little more. I think what Julian searches for is a tool in
> git that does the lookup for you which is AFAIK not that easy currently. It
> seems to be a quite useful feature. Here what I understand Julian wants:
>
> 1. Find the most recent superproject commit X'' in A that records a submodule
>   commit X' in B which contains the commit X in B you are searching for.
>
>   For this we would need use something similar to git describe --contains
>   but instead of using the list of existing tags in B it should use the list
>   of commits in B which are recorded in A.
>
>   Here a drawing to explain (linear history for simplicity):
>
>   superproject A:
>
>      O---O---X''---O
>               \
>   submodule B: \
>                 \
>      O---X---O---X'---O---O
>
> 2. Look up the commit of C which is recorded in X'' of A and check it
>   out.
>
> Step 2 is easy but for Step 1 the lookup of X' is missing for the commandline.
> Is there already anything that implements git describe --contains for a defined
> list of commits instead of refs?
>
> Cheers Heiko
>

Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

Hi,

After a couple of hours, I have finally succeeded to go through the
entire history of a repository and I can detect the submodules in a
tree of a commit. For those interested here is the source code:

http://gitorious.org/julian_ibarz_git/julian_ibarz_git/blobs/submodule_checkout/builtin/submodulecheckout.c

Next time I will build the list of commits of a submodule in the
entire history. Then I will need to find the distance of each commit
in the list compared to the current  commit of a submodule and keep
the closest one (which has to be a parent so an algebric distance
would be good to have).

So now my two questions that could save me some time are:
- is there a function that gives the distance between two commits? I'm
sure there is something like this coded somewhere but I didn't find it
yet
- is the struct decorate a hash map and if yes could someone explain
me how to use it or point me to a location where it is used?

Except from that I think I have all the information I need.

Best regards,
Julian Ibarz

On Sun, Jan 30, 2011 at 4:44 AM, Julian Ibarz <julian.ibarz@gmail.com> wrote:
> Today I have started to implement a proof of concept in C (I know a
> script would be better but I am really not good in sh so...). I
> struggle with the manipulation of the git API. I have pushed my work
> here:
>
> http://gitorious.org/julian_ibarz_git/julian_ibarz_git
>
> in branch submodule_checkout
>
> My work is in:
>
> builtin/submodulecheckout.c
>
> And my questions are prepended by the keyword QUESTION (two questions
> for now only).
>
> Any help is welcome.
>
> Thanks,
>
> Julian Ibarz
>
> On Sat, Jan 29, 2011 at 6:08 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>> Hi,
>>
>> On Wed, Jan 26, 2011 at 02:05:43PM -0800, Junio C Hamano wrote:
>>> If that version of submodule B is explicitly bound to a commit in the
>>> superproject A, you know which version of A and C were recorded, and the
>>> problem is solved.
>>>
>> [...]
>>>
>>> If you are confident that you didn't introduce different kind of
>>> dependency to other submodules while developing your "old_feature" branch
>>> in submodule B, one strategy may be to find an ancestor, preferrably the
>>> fork point, of your "old_feature" branch that is bound to the superproject
>>> A.  Then at that point at least you know whoever made that commit in A
>>> tested the combination of what was recorded in that commit, together with
>>> the version of B and C, and you can go forward from there, replaying the
>>> changes you made to the "old_feature" branch in submodule B.
>>
>> Lets extend your explanation a little further and maybe demonstrate the problem
>> Julian is having a little more. I think what Julian searches for is a tool in
>> git that does the lookup for you which is AFAIK not that easy currently. It
>> seems to be a quite useful feature. Here what I understand Julian wants:
>>
>> 1. Find the most recent superproject commit X'' in A that records a submodule
>>   commit X' in B which contains the commit X in B you are searching for.
>>
>>   For this we would need use something similar to git describe --contains
>>   but instead of using the list of existing tags in B it should use the list
>>   of commits in B which are recorded in A.
>>
>>   Here a drawing to explain (linear history for simplicity):
>>
>>   superproject A:
>>
>>      O---O---X''---O
>>               \
>>   submodule B: \
>>                 \
>>      O---X---O---X'---O---O
>>
>> 2. Look up the commit of C which is recorded in X'' of A and check it
>>   out.
>>
>> Step 2 is easy but for Step 1 the lookup of X' is missing for the commandline.
>> Is there already anything that implements git describe --contains for a defined
>> list of commits instead of refs?
>>
>> Cheers Heiko
>>
>

Re: Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Heiko Voigt]

Hi Julian,

interesting work so far.

One thing regarding the mailing list: Please do not top post and try to cut non
relevant text (the one you are not replying to) from your replies.

On Wed, Feb 02, 2011 at 11:31:40PM -0500, Julian Ibarz wrote:
> After a couple of hours, I have finally succeeded to go through the
> entire history of a repository and I can detect the submodules in a
> tree of a commit. For those interested here is the source code:
> 
> http://gitorious.org/julian_ibarz_git/julian_ibarz_git/blobs/submodule_checkout/builtin/submodulecheckout.c
> 
> Next time I will build the list of commits of a submodule in the
> entire history.

Sounds great. That will be one tool you need. Here is a small script
snippet which already does that as an example for the msysgit repository
(git://repo.or.cz/msysgit.git):

export sub=git
git log --pretty="%h" -- $sub | \
    while read i; do git ls-tree $i -- $sub; done | \
        cut -f3 -d" " | cut -f1

It outputs all the hashes that have been registered for the git
submodule. Maybe thats a good starting point to find example code in C.
You can use it for other submodules also just change the name in the variable
sub.

> Then I will need to find the distance of each commit
> in the list compared to the current  commit of a submodule and keep
> the closest one (which has to be a parent so an algebric distance
> would be good to have).

Have a look at the codepath that is used for

  git branch --contains <sha1>

It essentially does the thing you want (find all refs that contain a
commit). An approach could be to extend this mechanism not to iterate
through all refs but e.g. take a list of sha1's for iteration.

> So now my two questions that could save me some time are:
> - is there a function that gives the distance between two commits? I'm
> sure there is something like this coded somewhere but I didn't find it
> yet
> - is the struct decorate a hash map and if yes could someone explain
> me how to use it or point me to a location where it is used?

Please see the tips above.

> Except from that I think I have all the information I need.

There are a few points I would like to mention:

 * Citing my drawing:
> >>   superproject A:
> >>
> >>      O---O---X''---O
> >>               \
> >>   submodule B: \
> >>                 \
> >>      O---X---O---X'---O---O

   There can be multiple X' which contain X. These are all valid
   candidates. The same applies to X'' where there can be multiple
   candidates that record a specific X'. IMO, all of them need to be
   considered. If in the end they all reduce to one commit in submodule C
   you can check that out. If not printing a list of all candidates is
   probably the best option.

 * If we want your code to go upstream it probably needs to be
   integrated as a subcommand into the current submodule script. I can
   offer to help you on the bash scripting side. I think once the tools
   to find the appropriate commits are ready that will not be a big
   issue.

 * I think for protoyping the tools you can later use in the submodule
   script its fine to implement your own command. Keep in mind that you
   later on you should probably integrate your code into the appropriate
   existing commands. Here some suggestions.

   To support proper bash scripting two parts come into my mind:

   1. A tool to output all of the submodules commits which are
      candidates for X'' like the script snippet above.

   2. A tool that takes the list from 1. and then reduces that list to
      a list of commits that actually register the possible commits X'.
      If that is one commit X'' we can directly checkout. If not we
      print the list in a nice format and let the user choose.

Looking forward to your work.

Cheers Heiko

Re: Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Heiko Voigt]

Hi Julian,

On Wed, Feb 02, 2011 at 11:31:40PM -0500, Julian Ibarz wrote:
> So now my two questions that could save me some time are:
> - is there a function that gives the distance between two commits? I'm
> sure there is something like this coded somewhere but I didn't find it
> yet

One thing which now came to my mind is that you might be interested in
is to use merge bases to find the earliest commit that contains a
certain revision.

Have a look at how I implemented the submodule merge strategy that might
help you. You can find it in commit 68d03e4a. I basically do a search
for the earliest commit that contains the commits from both sides. You
could use a similar strategy but limit the possible candidates to a
restricted list of commits which are contained in the supermodule.

Cheers Heiko

Re: Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

Hi Heiko,

Thanks for all your feedbacks. Before your first mail I had already
implemented command 1 you are talking about and almost finished
command 2. Today I could spend some time again on this side project
and I have finished command 2. Here is what you do:

git submodulecheckout --find-versions <submodule> > <some_temp_file>
cd <submodule>
git submodulecheckout --find-closest < <some_temp_file>

This will output the version you need to checkout in the meta
repository. I don't handle the case where there is multiple solutions
but it is a good start I think. I think with this you can already
start the integration in the submodule bash script to make it
recursive and be more user-friendly? Are you still interested Heiko?

I have pushed this to my repository so you can check it out if you want.

Both commands have a suboptimal implementation but it can be improved
without changing the current API so I think it is a minor issue. I
will probably try this at work today on msysgit. Currently I am  doing
this at home on my linux using some open source meta-repository, if
you have a big one to try this feature let me know (I am currently
using git://git.assembla.com/luadocsuperforest.git and it is not that
big).

Best regards,

Julian Ibarz

On Wed, Feb 9, 2011 at 2:36 PM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> Hi Julian,
>
> On Wed, Feb 02, 2011 at 11:31:40PM -0500, Julian Ibarz wrote:
>> So now my two questions that could save me some time are:
>> - is there a function that gives the distance between two commits? I'm
>> sure there is something like this coded somewhere but I didn't find it
>> yet
>
> One thing which now came to my mind is that you might be interested in
> is to use merge bases to find the earliest commit that contains a
> certain revision.
>
> Have a look at how I implemented the submodule merge strategy that might
> help you. You can find it in commit 68d03e4a. I basically do a search
> for the earliest commit that contains the commits from both sides. You
> could use a similar strategy but limit the possible candidates to a
> restricted list of commits which are contained in the supermodule.
>
> Cheers Heiko
>

Re: Re: Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Heiko Voigt]

Hi Julian,

On Sat, Feb 12, 2011 at 03:32:43PM -0500, Julian Ibarz wrote:
> Thanks for all your feedbacks. Before your first mail I had already
> implemented command 1 you are talking about and almost finished
> command 2. Today I could spend some time again on this side project
> and I have finished command 2. Here is what you do:
> 
> git submodulecheckout --find-versions <submodule> > <some_temp_file>
> cd <submodule>
> git submodulecheckout --find-closest < <some_temp_file>

I do not quite understand how this is used. I just tried it on the
msysgit repository. Which is a good candidate for testing since you can
easily verify the results because for each git version there is a
corresponding tag in the superproject.

You can find it on repo: http://repo.or.cz/w/msysgit.git

As far as I understand the sha1 you are searching for is determined by
the currently checked out revision. So here we go:

cd msysgit/git
git checkout v1.7.2.3.msysgit.0~3
cd ..
git submodulecheckout --find-versions git > git-versions.txt
cd git
git submodulecheckout --find-closest < ../git-versions.txt
> submodulecheckout: no commit found

Ok lets not be so hard:

git checkout v1.7.2.3.msysgit.0
cd ..
git submodulecheckout --find-versions git > git-versions.txt
cd git
git submodulecheckout --find-closest < ../git-versions.txt
> 820fd48475eb72cdd4f2

Close but not quite the right commit. See the output of git log:
820fd48 Chomp CRLF before feeding to tar when building netinstaller.
83e164c Avoid quoting environment variable values.
daafc61 (tag: Git-1.7.2.3-preview20100911) Updated git and git documents and ReleaseNotes to v1.7.2.3

Whats strange is that this commit did not even change the submodule.

I think currently it does not make sense to implement the scripting
since the interface is not yet cleaned up enough. It should be possible
to specify a commit as parameter and the path should probably go after
that like the other commands do as well.

The other issue is that we need to change the working directory. IMO,
everything should be possible to do from superproject. Have a look at
add_submodule_odb() in submodule.c it lets you load the database of a
submodule. This way you can run the --find-closest from the superproject.

So what I would be looking for is something like this:

# This looks up all commits of <submodule> ever registered in this
# project
git submodulecheckout --find-versions <submodule> > out.tmp

# This reads the previous list and outputs a list of of all the commits
# that contain <treeish>. Function in_merge_bases(treeish, i&commit, 1)
# can be of help here. Afterwards you have to prune all later commits
# that contain earlier ones. I did this in find_first_merges(). I
# suggest to refactor that code into a function so that you can use it.
git submodulecheckout --find-closest <treeish> -- <submodule> < out.tmp > out2.tmp

# And now we can search for the commit in the other submodule C
git submodulecheckout --list-commits <other submodule> < out2.tmp > out3.tmp

And if out3.tmp just contained one line we could checkout that one. I am
still not sure in which command these options could go. Junio do you
maybe have a suggestion? Since its all about submodules it would make
sense to put it in 'git submodule' but thats already the script.

Maybe a new 'git-submodule-tool' ?

I suggest that you start implementing some basic tests if you found the
right commits using the testsuite. Basically you build up a sample
repository where you know which commits of the submodules are tied
together with a sequence of commands. Then you check whether your
commands find the correct one. Maybe have a look at the submodule
tests Jens is implementing on his branch:

https://github.com/jlehmann/git-submod-enhancements

Hope that helps.

Cheers Heiko

Re: Re: Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

On Sun, Feb 13, 2011 at 8:30 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> Hi Julian,
>
> On Sat, Feb 12, 2011 at 03:32:43PM -0500, Julian Ibarz wrote:
>> Thanks for all your feedbacks. Before your first mail I had already
>> implemented command 1 you are talking about and almost finished
>> command 2. Today I could spend some time again on this side project
>> and I have finished command 2. Here is what you do:
>>
>> git submodulecheckout --find-versions <submodule> > <some_temp_file>
>> cd <submodule>
>> git submodulecheckout --find-closest < <some_temp_file>
>
> I do not quite understand how this is used. I just tried it on the
> msysgit repository. Which is a good candidate for testing since you can
> easily verify the results because for each git version there is a
> corresponding tag in the superproject.
>
> You can find it on repo: http://repo.or.cz/w/msysgit.git
>
> As far as I understand the sha1 you are searching for is determined by
> the currently checked out revision. So here we go:
>
> cd msysgit/git
> git checkout v1.7.2.3.msysgit.0~3
> cd ..
> git submodulecheckout --find-versions git > git-versions.txt
> cd git
> git submodulecheckout --find-closest < ../git-versions.txt
>> submodulecheckout: no commit found

There was multiple bugs in the version you tried and I just pushed the
fix. It is working on my repository at work but on msysgit, just
forget it, it is too slow. I still need to optimize it, it doesn't
scale at all lol. But it shouldn't be a problem. The two optimizations
I want to do:
* use a real hash map to know if I already visited a commit
* go through the history only once and finding the closest commit in a
list instead of going through the entire history for each commit of
the least

I also need to send the list of possible commits in the
meta-repository instead of the first one. For msysgit sometime some
solutions have more than 20 possible solutions!

> Ok lets not be so hard:
>
> git checkout v1.7.2.3.msysgit.0
> cd ..
> git submodulecheckout --find-versions git > git-versions.txt
> cd git
> git submodulecheckout --find-closest < ../git-versions.txt
>> 820fd48475eb72cdd4f2
>
> Close but not quite the right commit. See the output of git log:
> 820fd48 Chomp CRLF before feeding to tar when building netinstaller.
> 83e164c Avoid quoting environment variable values.
> daafc61 (tag: Git-1.7.2.3-preview20100911) Updated git and git documents and ReleaseNotes to v1.7.2.3
>
> Whats strange is that this commit did not even change the submodule.

Well because it is _exactly_ the commit that point to the current
submodule version you have checked out :) So it _is_ the good version.
Which version were you thinking was good?

> I think currently it does not make sense to implement the scripting
> since the interface is not yet cleaned up enough. It should be possible
> to specify a commit as parameter and the path should probably go after
> that like the other commands do as well.
>
> The other issue is that we need to change the working directory. IMO,
> everything should be possible to do from superproject. Have a look at
> add_submodule_odb() in submodule.c it lets you load the database of a
> submodule. This way you can run the --find-closest from the superproject.

I don't think we want to do that. The usual scenario is:

* you are in a submodule
* you checkout a specific version
* you want the meta-repo and side submodules to be updated accordingly

And in this scenario at the beginning you are at the submodule
directory, not the meta repository so... What do you think? Also even
if in these low-level commands you need to change directory, if it is
hided in an integration in the sh submodule script, it doesn't matter
right?

> So what I would be looking for is something like this:
>
> # This looks up all commits of <submodule> ever registered in this
> # project
> git submodulecheckout --find-versions <submodule> > out.tmp
>
> # This reads the previous list and outputs a list of of all the commits
> # that contain <treeish>. Function in_merge_bases(treeish, i&commit, 1)
> # can be of help here. Afterwards you have to prune all later commits
> # that contain earlier ones. I did this in find_first_merges(). I
> # suggest to refactor that code into a function so that you can use it.
> git submodulecheckout --find-closest <treeish> -- <submodule> < out.tmp > out2.tmp
>
> # And now we can search for the commit in the other submodule C
> git submodulecheckout --list-commits <other submodule> < out2.tmp > out3.tmp
>
> And if out3.tmp just contained one line we could checkout that one. I am
> still not sure in which command these options could go. Junio do you
> maybe have a suggestion? Since its all about submodules it would make
> sense to put it in 'git submodule' but thats already the script.
>
> Maybe a new 'git-submodule-tool' ?

Before I wanted to name it submodule-checkout but I think I hade a
naming conflict issue with submodule... not sure if submodule-tool
will work...

> I suggest that you start implementing some basic tests if you found the
> right commits using the testsuite. Basically you build up a sample
> repository where you know which commits of the submodules are tied
> together with a sequence of commands. Then you check whether your
> commands find the correct one. Maybe have a look at the submodule
> tests Jens is implementing on his branch:
>
> https://github.com/jlehmann/git-submod-enhancements

I will take a look into this.

Thanks,
Julian

> Hope that helps.
>
> Cheers Heiko
>

Re: Re: Re: Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Heiko Voigt]

Hi,

On Sun, Feb 13, 2011 at 01:59:09PM -0500, Julian Ibarz wrote:
> On Sun, Feb 13, 2011 at 8:30 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> > Ok lets not be so hard:
> >
> > git checkout v1.7.2.3.msysgit.0
> > cd ..
> > git submodulecheckout --find-versions git > git-versions.txt
> > cd git
> > git submodulecheckout --find-closest < ../git-versions.txt
> >> 820fd48475eb72cdd4f2
> >
> > Close but not quite the right commit. See the output of git log:
> > 820fd48 Chomp CRLF before feeding to tar when building netinstaller.
> > 83e164c Avoid quoting environment variable values.
> > daafc61 (tag: Git-1.7.2.3-preview20100911) Updated git and git documents and ReleaseNotes to v1.7.2.3
> >
> > Whats strange is that this commit did not even change the submodule.
> 
> Well because it is _exactly_ the commit that point to the current
> submodule version you have checked out :) So it _is_ the good version.
> Which version were you thinking was good?

Well I think the interface was/is still confusing me. Once you change to
one where I can specify the hash I want to search for I will give it
another try.

> > I think currently it does not make sense to implement the scripting
> > since the interface is not yet cleaned up enough. It should be possible
> > to specify a commit as parameter and the path should probably go after
> > that like the other commands do as well.
> >
> > The other issue is that we need to change the working directory. IMO,
> > everything should be possible to do from superproject. Have a look at
> > add_submodule_odb() in submodule.c it lets you load the database of a
> > submodule. This way you can run the --find-closest from the superproject.
> 
> I don't think we want to do that. The usual scenario is:
> 
> * you are in a submodule
> * you checkout a specific version
> * you want the meta-repo and side submodules to be updated accordingly
> 
> And in this scenario at the beginning you are at the submodule
> directory, not the meta repository so... What do you think? Also even
> if in these low-level commands you need to change directory, if it is
> hided in an integration in the sh submodule script, it doesn't matter
> right?

Well I think the interface should be useable by hand also. Lots of tools
in git were invented by plugging the tools together differently than the
original developer even had in his mind. So a nice "do only one thing
but that very good" kind of interface makes sense.

I think a nice thing would be if you could stay in one directory and it
should not be too hard since we have add_submodule_odb(). Implementation
would be easier if we started this command from the supermodule since
searching for a supermodule from a submodule is still missing.

> > So what I would be looking for is something like this:
> >
> > # This looks up all commits of <submodule> ever registered in this
> > # project
> > git submodulecheckout --find-versions <submodule> > out.tmp
> >
> > # This reads the previous list and outputs a list of of all the commits
> > # that contain <treeish>. Function in_merge_bases(treeish, i&commit, 1)
> > # can be of help here. Afterwards you have to prune all later commits
> > # that contain earlier ones. I did this in find_first_merges(). I
> > # suggest to refactor that code into a function so that you can use it.
> > git submodulecheckout --find-closest <treeish> -- <submodule> < out.tmp > out2.tmp
> >
> > # And now we can search for the commit in the other submodule C
> > git submodulecheckout --list-commits <other submodule> < out2.tmp > out3.tmp
> >
> > And if out3.tmp just contained one line we could checkout that one. I am
> > still not sure in which command these options could go. Junio do you
> > maybe have a suggestion? Since its all about submodules it would make
> > sense to put it in 'git submodule' but thats already the script.
> >
> > Maybe a new 'git-submodule-tool' ?
> 
> Before I wanted to name it submodule-checkout but I think I hade a
> naming conflict issue with submodule... not sure if submodule-tool
> will work...

Since the submodule script might be obsoleted partly and get rewritten
in C at some point I was thinking of this as a kind of intermediate
internal submodule helper tool.

Cheers Heiko

Re: Re: Re: Re: Re: Updating a submodule with a compatible version from another submodule version using the parent meta-repository

[Julian Ibarz]

On Mon, Feb 14, 2011 at 4:13 PM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
> Hi,
>
> On Sun, Feb 13, 2011 at 01:59:09PM -0500, Julian Ibarz wrote:
>> On Sun, Feb 13, 2011 at 8:30 AM, Heiko Voigt <hvoigt@hvoigt.net> wrote:
>> > Ok lets not be so hard:
>> >
>> > git checkout v1.7.2.3.msysgit.0
>> > cd ..
>> > git submodulecheckout --find-versions git > git-versions.txt
>> > cd git
>> > git submodulecheckout --find-closest < ../git-versions.txt
>> >> 820fd48475eb72cdd4f2
>> >
>> > Close but not quite the right commit. See the output of git log:
>> > 820fd48 Chomp CRLF before feeding to tar when building netinstaller.
>> > 83e164c Avoid quoting environment variable values.
>> > daafc61 (tag: Git-1.7.2.3-preview20100911) Updated git and git documents and ReleaseNotes to v1.7.2.3
>> >
>> > Whats strange is that this commit did not even change the submodule.
>>
>> Well because it is _exactly_ the commit that point to the current
>> submodule version you have checked out :) So it _is_ the good version.
>> Which version were you thinking was good?
>
> Well I think the interface was/is still confusing me. Once you change to
> one where I can specify the hash I want to search for I will give it
> another try.
>> > I think currently it does not make sense to implement the scripting
>> > since the interface is not yet cleaned up enough. It should be possible
>> > to specify a commit as parameter and the path should probably go after
>> > that like the other commands do as well.
>> >
>> > The other issue is that we need to change the working directory. IMO,
>> > everything should be possible to do from superproject. Have a look at
>> > add_submodule_odb() in submodule.c it lets you load the database of a
>> > submodule. This way you can run the --find-closest from the superproject.
>>
>> I don't think we want to do that. The usual scenario is:
>>
>> * you are in a submodule
>> * you checkout a specific version
>> * you want the meta-repo and side submodules to be updated accordingly
>>
>> And in this scenario at the beginning you are at the submodule
>> directory, not the meta repository so... What do you think? Also even
>> if in these low-level commands you need to change directory, if it is
>> hided in an integration in the sh submodule script, it doesn't matter
>> right?
>
> Well I think the interface should be useable by hand also. Lots of tools
> in git were invented by plugging the tools together differently than the
> original developer even had in his mind. So a nice "do only one thing
> but that very good" kind of interface makes sense.
>
> I think a nice thing would be if you could stay in one directory and it
> should not be too hard since we have add_submodule_odb(). Implementation
> would be easier if we started this command from the supermodule since
> searching for a supermodule from a submodule is still missing.
>
>> > So what I would be looking for is something like this:
>> >
>> > # This looks up all commits of <submodule> ever registered in this
>> > # project
>> > git submodulecheckout --find-versions <submodule> > out.tmp
>> >
>> > # This reads the previous list and outputs a list of of all the commits
>> > # that contain <treeish>. Function in_merge_bases(treeish, i&commit, 1)
>> > # can be of help here. Afterwards you have to prune all later commits
>> > # that contain earlier ones. I did this in find_first_merges(). I
>> > # suggest to refactor that code into a function so that you can use it.
>> > git submodulecheckout --find-closest <treeish> -- <submodule> < out.tmp > out2.tmp
>> >
>> > # And now we can search for the commit in the other submodule C
>> > git submodulecheckout --list-commits <other submodule> < out2.tmp > out3.tmp
>> >
>> > And if out3.tmp just contained one line we could checkout that one. I am
>> > still not sure in which command these options could go. Junio do you
>> > maybe have a suggestion? Since its all about submodules it would make
>> > sense to put it in 'git submodule' but thats already the script.
>> >
>> > Maybe a new 'git-submodule-tool' ?
>>
>> Before I wanted to name it submodule-checkout but I think I hade a
>> naming conflict issue with submodule... not sure if submodule-tool
>> will work...
>
> Since the submodule script might be obsoleted partly and get rewritten
> in C at some point I was thinking of this as a kind of intermediate
> internal submodule helper tool.
>
> Cheers Heiko
>

Today I have added the possibility to provide a hash to find-closest
feature. I have also implemented the two optimizations I was talking
about and now it is pretty fast. By the way I am using the binary tree
from string-list.h. An example of execution:

git submodulecheckout --find-closest v1.7.3.1.msysgit.0 < ../git-versions.txt
279f4c28cad4bd97dc06129118314b0a36f193db

468fabbff85c10e5e226a2ea22699cd4186536f7
fe61e4c51c9f3577fcb85951f72d3c65afce251d

The first commit is the closest referenced commit found starting from
v1.7.3.1.msysgit.0 and then the last two are the commits in the parent
repository that reference this commit. If we integrate this in
git-submodule.sh, the script could take those hash and instead display
them + first line of commit message and the tags if there is one.

As for your third command --list-commits I don't think it matters. As
soon as the user has decided which commit in the parent repository he
wants to checkout, then we just need to git submodule update
everything except the submodule that helped us find it. Basically from
the example above. if we say that
468fabbff85c10e5e226a2ea22699cd4186536f7 is the commit we were looking
for for msysgit then we can do git submodule update on everything
except submodule git.

Next on my TODO list:
* Optimize find-versions using string-list
* Run find-closest in the meta-repository instead
* Implement a test example

Let me know if I am missing something.

Best regards,
Julian