Restoring detached HEADs after Git operations

Restoring detached HEADs after Git operations

[Patrick Lehmann]

Hello,

I wrote a Bash script to recover branch names after Git operations have create detached HEADs in a Git repository containing lots of Git submodules. The script works recursively.

I would like to see:
a) that script or algorithm being integrated into Git by default
b) that as a default behavior for all Git operations creating detached HEADs

That's the command:
--------------------------------
git submodule foreach --recursive  'HEAD=$(git branch --list | head -n 1); if [[ "$HEAD" == *HEAD* ]]; then REF=$(git rev-parse HEAD); FOUND=0; for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do if [[ "$(git rev-parse "$Branch")" == $REF ]]; then echo -e "  \e[36mCheckout $Branch...\e[0m"; git checkout $Branch; FOUND=1; break; fi done; if [[ $FOUND -eq 0 ]]; then echo -e "  \e[31mNo matching branch found.\e[0m"; fi else echo -e "  \e[36mNothing to do.\e[0m"; fi'
--------------------------------

How does it work:
1. It uses git submodule foreach to dive into each Git submodule and execute a series of Bash commands.
2. It's reading the list of branches and checks if the submodule is in detached mode. The first line contains the string HEAD.
3. Retrieve the hash of the detached HEAD
4. Iterate all local branches and get their hashes
5. Compare the branch hashes with the detached HEAD's hash. If it matches do a checkout.
6. Report if no branch name was found or if a HEAD was not in detached mode.

The Bash code with line breaks and indentation:
--------------------------------
HEAD=$(git branch --list | head -n 1)
if [[ "$HEAD" == *HEAD* ]]; then
  REF=$(git rev-parse HEAD)
  FOUND=0
  for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do
    if [[ "$(git rev-parse "$Branch")" == $REF ]]; then
      echo -e "  \e[36mCheckout $Branch...\e[0m"
      git checkout $Branch
      FOUND=1
      break
    fi
  done
  if [[ $FOUND -eq 0 ]]; then
    echo -e "  \e[31mNo matching branch found.\e[0m"
  fi
else
  echo -e "  \e[36mNothing to do.\e[0m"
fi
--------------------------------

Are their any chances to get it integrated into Git?

I tried to register that code as a Git alias, but git config complains about quote problem not showing where. It neither specifies if it's a single or double quote problem. Any advice on how to register that piece of code as an alias?

If wished, I think I could expand the script to also recover hash values to Git tags if no branch was found.

Kind regards
    Patrick Lehmann

Re: Restoring detached HEADs after Git operations

[Lars Schneider]

> On 19 Jun 2017, at 10:46, Patrick Lehmann <Patrick.Lehmann@plc2.de> wrote:
> 
> Hello,
> 
> I wrote a Bash script to recover branch names after Git operations have create detached HEADs in a Git repository containing lots of Git submodules. The script works recursively.

I did run into this situation myself and therefore
I understand your motivation. I've CC'ed Stefan as
he is a Submodule expert!


> I would like to see:
> a) that script or algorithm being integrated into Git by default
> b) that as a default behavior for all Git operations creating detached HEADs
> 
> That's the command:
> --------------------------------
> git submodule foreach --recursive  'HEAD=$(git branch --list | head -n 1); if [[ "$HEAD" == *HEAD* ]]; then REF=$(git rev-parse HEAD); FOUND=0; for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do if [[ "$(git rev-parse "$Branch")" == $REF ]]; then echo -e "  \e[36mCheckout $Branch...\e[0m"; git checkout $Branch; FOUND=1; break; fi done; if [[ $FOUND -eq 0 ]]; then echo -e "  \e[31mNo matching branch found.\e[0m"; fi else echo -e "  \e[36mNothing to do.\e[0m"; fi'
> --------------------------------
> 
> How does it work:
> 1. It uses git submodule foreach to dive into each Git submodule and execute a series of Bash commands.
> 2. It's reading the list of branches and checks if the submodule is in detached mode. The first line contains the string HEAD.
> 3. Retrieve the hash of the detached HEAD
> 4. Iterate all local branches and get their hashes
> 5. Compare the branch hashes with the detached HEAD's hash. If it matches do a checkout.

If there are multiple branches with the same hash then
your script would pick the first one. Can you imagine a
situation where this would be a problem?

Plus, you are looking only at local branches. Wouldn't it
make sense to look at remote branches, too?


> 6. Report if no branch name was found or if a HEAD was not in detached mode.
> 
> The Bash code with line breaks and indentation:
> --------------------------------
> HEAD=$(git branch --list | head -n 1)
> if [[ "$HEAD" == *HEAD* ]]; then
>  REF=$(git rev-parse HEAD)
>  FOUND=0
>  for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do

There is a convenient "git for-each-ref" function to iterate over
branches in scripts. See here an example:
https://github.com/larsxschneider/scotty/blob/master/admin/oss-fork.sh#L88


>    if [[ "$(git rev-parse "$Branch")" == $REF ]]; then
>      echo -e "  \e[36mCheckout $Branch...\e[0m"
>      git checkout $Branch
>      FOUND=1
>      break
>    fi
>  done
>  if [[ $FOUND -eq 0 ]]; then
>    echo -e "  \e[31mNo matching branch found.\e[0m"
>  fi
> else
>  echo -e "  \e[36mNothing to do.\e[0m"
> fi
> --------------------------------
> 
> Are their any chances to get it integrated into Git?
> 
> I tried to register that code as a Git alias, but git config complains about quote problem not showing where. It neither specifies if it's a single or double quote problem. Any advice on how to register that piece of code as an alias?

Try to escape ". See here for an example:
https://github.com/Autodesk/enterprise-config-for-git/blob/master/config.include#L76-L94


> If wished, I think I could expand the script to also recover hash values to Git tags if no branch was found.

It would be indeed nice to see the tagged version on my prompt.

--

Submodule processing is already quite slow if you have many of them.
I wonder how much this approach would affect the performance.

- Lars

AW: Restoring detached HEADs after Git operations

[Patrick Lehmann]

Hello Lars,

for your questions:
> If there are multiple branches with the same hash then your script would pick the first one. Can you imagine a situation where this would be a problem?

I can't think of a good solution to resolve it automatically. Maybe a script could print that there are multiple possibilities and it choose the first branch in the list.


> Plus, you are looking only at local branches. Wouldn't it make sense to look at remote branches, too?

This is also related to restoring tags. If we go this way, we should have this priority list:
- local branches
- remote branches
- tags


> Submodule processing is already quite slow if you have many of them. I wonder how much this approach would affect the performance.

Yes. It takes a few seconds to iterate all the submodules. It could be improved if the processing wouldn't be based on slow Bash scripts spawning lot's of sub-shells to execute multiple Git commands.



Is there a way to avoid detached DEADs at the beginning?
Many submodules are attached to a reference and get detached to a hash of the same reference. It would be better, if they never get detached when the current and new hash are the same.


Kind regards
    Patrick

________________________________________
Von: git-owner@vger.kernel.org [git-owner@vger.kernel.org]" im Auftrag von "Lars Schneider [larsxschneider@gmail.com]
Gesendet: Montag, 19. Juni 2017 11:30
Bis: Patrick Lehmann
Cc: Git Mailinglist; Stefan Beller
Betreff: Re: Restoring detached HEADs after Git operations

> On 19 Jun 2017, at 10:46, Patrick Lehmann <Patrick.Lehmann@plc2.de> wrote:
>
> Hello,
>
> I wrote a Bash script to recover branch names after Git operations have create detached HEADs in a Git repository containing lots of Git submodules. The script works recursively.

I did run into this situation myself and therefore
I understand your motivation. I've CC'ed Stefan as
he is a Submodule expert!


> I would like to see:
> a) that script or algorithm being integrated into Git by default
> b) that as a default behavior for all Git operations creating detached HEADs
>
> That's the command:
> --------------------------------
> git submodule foreach --recursive  'HEAD=$(git branch --list | head -n 1); if [[ "$HEAD" == *HEAD* ]]; then REF=$(git rev-parse HEAD); FOUND=0; for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do if [[ "$(git rev-parse "$Branch")" == $REF ]]; then echo -e "  \e[36mCheckout $Branch...\e[0m"; git checkout $Branch; FOUND=1; break; fi done; if [[ $FOUND -eq 0 ]]; then echo -e "  \e[31mNo matching branch found.\e[0m"; fi else echo -e "  \e[36mNothing to do.\e[0m"; fi'
> --------------------------------
>
> How does it work:
> 1. It uses git submodule foreach to dive into each Git submodule and execute a series of Bash commands.
> 2. It's reading the list of branches and checks if the submodule is in detached mode. The first line contains the string HEAD.
> 3. Retrieve the hash of the detached HEAD
> 4. Iterate all local branches and get their hashes
> 5. Compare the branch hashes with the detached HEAD's hash. If it matches do a checkout.

If there are multiple branches with the same hash then
your script would pick the first one. Can you imagine a
situation where this would be a problem?

Plus, you are looking only at local branches. Wouldn't it
make sense to look at remote branches, too?


> 6. Report if no branch name was found or if a HEAD was not in detached mode.
>
> The Bash code with line breaks and indentation:
> --------------------------------
> HEAD=$(git branch --list | head -n 1)
> if [[ "$HEAD" == *HEAD* ]]; then
>  REF=$(git rev-parse HEAD)
>  FOUND=0
>  for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do

There is a convenient "git for-each-ref" function to iterate over
branches in scripts. See here an example:
https://github.com/larsxschneider/scotty/blob/master/admin/oss-fork.sh#L88


>    if [[ "$(git rev-parse "$Branch")" == $REF ]]; then
>      echo -e "  \e[36mCheckout $Branch...\e[0m"
>      git checkout $Branch
>      FOUND=1
>      break
>    fi
>  done
>  if [[ $FOUND -eq 0 ]]; then
>    echo -e "  \e[31mNo matching branch found.\e[0m"
>  fi
> else
>  echo -e "  \e[36mNothing to do.\e[0m"
> fi
> --------------------------------
>
> Are their any chances to get it integrated into Git?
>
> I tried to register that code as a Git alias, but git config complains about quote problem not showing where. It neither specifies if it's a single or double quote problem. Any advice on how to register that piece of code as an alias?

Try to escape ". See here for an example:
https://github.com/Autodesk/enterprise-config-for-git/blob/master/config.include#L76-L94


> If wished, I think I could expand the script to also recover hash values to Git tags if no branch was found.

It would be indeed nice to see the tagged version on my prompt.

--

Submodule processing is already quite slow if you have many of them.
I wonder how much this approach would affect the performance.

- Lars

Re: Restoring detached HEADs after Git operations

[Stefan Beller]

On Mon, Jun 19, 2017 at 2:52 AM, Patrick Lehmann
<Patrick.Lehmann@plc2.de> wrote:
> Hello Lars,
>
> for your questions:
>> If there are multiple branches with the same hash then your script would pick the first one. Can you imagine a situation where this would be a problem?
>
> I can't think of a good solution to resolve it automatically. Maybe a script could print that there are multiple possibilities and it choose the first branch in the list.
>
>
>> Plus, you are looking only at local branches. Wouldn't it make sense to look at remote branches, too?
>
> This is also related to restoring tags. If we go this way, we should have this priority list:
> - local branches
> - remote branches

For remote branches you would create a local branch of the same name
(if such a branch would not exist, possibly setting it up to track that remote
branch)?

> - tags

as said in the other email and similar to remote branches, we'd not want to have
HEAD pointing to them directly but somehow have a local branch.

>> Submodule processing is already quite slow if you have many of them. I wonder how much this approach would affect the performance.
>
> Yes. It takes a few seconds to iterate all the submodules. It could be improved if the processing wouldn't be based on slow Bash scripts spawning lot's of sub-shells to execute multiple Git commands.

How many submodules are we talking about? (Are you on Windows to make
shell even more fun?)

AW: Restoring detached HEADs after Git operations

[Patrick Lehmann]

Hello,

I'm just an advanced Git user, not a Git developer. So I might find some time to improve the suggested script, which I provided with the hints given on the mailing list, but I have no time to do a complete feature release in your patch based Git flow.

I'm currently involved in 8 other open source projects. One can't improve the world alone by supplying patches to any open source project one is using...

I have no experience with other shells then Bash. So if you rely on a Bash with less features, please port the syntax to such a shell system. (I personally do not support legacy programs or out-date programs).

------
We are talking about circa 50 submodules in total with a maximum depth of 4. The platforms are:
- Mint OS with Git in Bash
- Windows 7 with Git-Bash
- Windows 10 with Git-Bash
- Windows 10 with Posh-Git


Kind regards
    Patrick

________________________________________
Von: Stefan Beller [sbeller@google.com]
Gesendet: Montag, 19. Juni 2017 18:37
Bis: Patrick Lehmann
Cc: Lars Schneider; Git Mailinglist
Betreff: Re: Restoring detached HEADs after Git operations

On Mon, Jun 19, 2017 at 2:52 AM, Patrick Lehmann
<Patrick.Lehmann@plc2.de> wrote:
> Hello Lars,
>
> for your questions:
>> If there are multiple branches with the same hash then your script would pick the first one. Can you imagine a situation where this would be a problem?
>
> I can't think of a good solution to resolve it automatically. Maybe a script could print that there are multiple possibilities and it choose the first branch in the list.
>
>
>> Plus, you are looking only at local branches. Wouldn't it make sense to look at remote branches, too?
>
> This is also related to restoring tags. If we go this way, we should have this priority list:
> - local branches
> - remote branches

For remote branches you would create a local branch of the same name
(if such a branch would not exist, possibly setting it up to track that remote
branch)?

> - tags

as said in the other email and similar to remote branches, we'd not want to have
HEAD pointing to them directly but somehow have a local branch.

>> Submodule processing is already quite slow if you have many of them. I wonder how much this approach would affect the performance.
>
> Yes. It takes a few seconds to iterate all the submodules. It could be improved if the processing wouldn't be based on slow Bash scripts spawning lot's of sub-shells to execute multiple Git commands.

How many submodules are we talking about? (Are you on Windows to make
shell even more fun?)

Re: Restoring detached HEADs after Git operations

[Stefan Beller]

On Mon, Jun 19, 2017 at 10:34 AM, Patrick Lehmann
<Patrick.Lehmann@plc2.de> wrote:
> Hello,
>
> I'm just an advanced Git user, not a Git developer. So I might find some time to improve the suggested script, which I provided with the hints given on the mailing list, but I have no time to do a complete feature release in your patch based Git flow.

ok, thanks for letting us know. I may re-prioritize the "reattach
HEAD" patches that I referenced earlier.
I would have hoped that additionally to the shell lines you'd have
given a good use case/summary.

> I have no experience with other shells then Bash. So if you rely on a Bash with less features, please port the syntax to such a shell system. (I personally do not support legacy programs or out-date programs).
>
> ------
> We are talking about circa 50 submodules in total with a maximum depth of 4. The platforms are:
> - Mint OS with Git in Bash
> - Windows 7 with Git-Bash
> - Windows 10 with Git-Bash
> - Windows 10 with Posh-Git

Thanks,
Stefan

AW: Restoring detached HEADs after Git operations

[Patrick Lehmann]

Hello Stefan,

the use case is as follows:

The projects consists of circa 18 IP cores. Each IP core is represented by a Git repository. Think of an IP core as of a lonestanding DLL or SO file project. Each IP core references 2 submodules, which bring the verification environments for testing the IP core standalone.

These 18 IP cores are grouped to bigger IP cores, referencing the low-level IP cores and each again the 2 verification submodules. Finally, the main project references the bigger IP cores and again the 2 verification cores.

TOPLEVEL
  o- IP1
       o- UVVM
       o- VUnit
  o- IP2
       o- UVVM
       o- VUnit
  o- IP3
       o- UVVM
       o- VUnit
  o- IP4
       o- UVVM
       o- VUnit
       o- IP5
           o- UVVM
           o- VUnit
       o- IP6
           o- UVVM
           o- VUnit
       o- IP7
           o- UVVM
           o- VUnit
  o- IP8
       o- UVVM
       o- VUnit
       o- IP9
           o- UVVM
           o- VUnit
       o- IP10
           o- UVVM
           o- VUnit
  o- IP11
       o- UVVM
       o- VUnit
       o- IP9
           o- UVVM
           o- VUnit
       o- IP12
           o- UVVM
           o- VUnit
   o- UVVM
   o- VUnit

That's the simplified structure. I can't write more, because it's a closed source project. You can find other usecases e.g. in my other open source projects. E.g. The PoC-Library or The PicoBlaze-Library and the corresponding PoC-Examples repository.

Example: PoC
Pile of Cores includes 4 Git submodules and is itself an IP core library.
So PoC-Examples again references PoC. This looks like this tree:

PoC-Examples
  |- lib/
       o- PoC
            |- lib
                o- Cocotb
                o- OSVVM
                o- VUnit
                     o- .... OSVVM
                o- UVVM

The library VUnit itself already includes OSVVM as a library.

----------------------
Forcast:
I'll write a new question / idea about multiple equal submodules and the memory footprint soon...
Here is my original question posted on StackOverflow: https://stackoverflow.com/questions/44585425/how-to-reduce-the-memory-footprint-for-multiple-submodules-of-the-same-source
----------------------

Do you need more use cases?


Kind regards
    Patrick
________________________________________
Von: git-owner@vger.kernel.org [git-owner@vger.kernel.org]" im Auftrag von "Stefan Beller [sbeller@google.com]
Gesendet: Montag, 19. Juni 2017 19:47
Bis: Patrick Lehmann
Cc: Lars Schneider; Git Mailinglist
Betreff: Re: Restoring detached HEADs after Git operations

On Mon, Jun 19, 2017 at 10:34 AM, Patrick Lehmann
<Patrick.Lehmann@plc2.de> wrote:
> Hello,
>
> I'm just an advanced Git user, not a Git developer. So I might find some time to improve the suggested script, which I provided with the hints given on the mailing list, but I have no time to do a complete feature release in your patch based Git flow.

ok, thanks for letting us know. I may re-prioritize the "reattach
HEAD" patches that I referenced earlier.
I would have hoped that additionally to the shell lines you'd have
given a good use case/summary.

> I have no experience with other shells then Bash. So if you rely on a Bash with less features, please port the syntax to such a shell system. (I personally do not support legacy programs or out-date programs).
>
> ------
> We are talking about circa 50 submodules in total with a maximum depth of 4. The platforms are:
> - Mint OS with Git in Bash
> - Windows 7 with Git-Bash
> - Windows 10 with Git-Bash
> - Windows 10 with Posh-Git

Thanks,
Stefan

Re: Restoring detached HEADs after Git operations

[Stefan Beller]

On Mon, Jun 19, 2017 at 11:09 AM, Patrick Lehmann
<Patrick.Lehmann@plc2.de> wrote:
> Hello Stefan,
>
> the use case is as follows:
>
> The projects consists of circa 18 IP cores. Each IP core is represented by a Git repository. Think of an IP core as of a lonestanding DLL or SO file project. Each IP core references 2 submodules, which bring the verification environments for testing the IP core standalone.

So phrased differently: You are using submodules to avoid "DLL hell"
(sharing a lib, with ease of versioning as the submodules in the different IP
cores may be pointing at different versions).

>
> These 18 IP cores are grouped to bigger IP cores, referencing the low-level IP cores and each again the 2 verification submodules. Finally, the main project references the bigger IP cores and again the 2 verification cores.
>
> TOPLEVEL
>   o- IP1
>        o- UVVM
>        o- VUnit
>   o- IP2
>        o- UVVM
>        o- VUnit
>   o- IP3
>        o- UVVM
>        o- VUnit
>   o- IP4
>        o- UVVM
>        o- VUnit
>        o- IP5
>            o- UVVM
>            o- VUnit
>        o- IP6
>            o- UVVM
>            o- VUnit
>        o- IP7
>            o- UVVM
>            o- VUnit
>   o- IP8
>        o- UVVM
>        o- VUnit
>        o- IP9
>            o- UVVM
>            o- VUnit
>        o- IP10
>            o- UVVM
>            o- VUnit
>   o- IP11
>        o- UVVM
>        o- VUnit
>        o- IP9
>            o- UVVM
>            o- VUnit
>        o- IP12
>            o- UVVM
>            o- VUnit
>    o- UVVM
>    o- VUnit
>
> That's the simplified structure. I can't write more, because it's a closed source project. You can find other usecases e.g. in my other open source projects. E.g. The PoC-Library or The PicoBlaze-Library and the corresponding PoC-Examples repository.
>
> Example: PoC
> Pile of Cores includes 4 Git submodules and is itself an IP core library.
> So PoC-Examples again references PoC. This looks like this tree:
>
> PoC-Examples
>   |- lib/
>        o- PoC
>             |- lib
>                 o- Cocotb
>                 o- OSVVM
>                 o- VUnit
>                      o- .... OSVVM
>                 o- UVVM
>
> The library VUnit itself already includes OSVVM as a library.
>
> ----------------------
> Forcast:
> I'll write a new question / idea about multiple equal submodules and the memory footprint soon...
> Here is my original question posted on StackOverflow: https://stackoverflow.com/questions/44585425/how-to-reduce-the-memory-footprint-for-multiple-submodules-of-the-same-source
> ----------------------
>
> Do you need more use cases?
>

Well this use case points out a different issue than I hoped for. ;)
From the stackoverflow post and from looking at the layout here,
one of the major questions is how to deduplicate the submodule
object store for example.

By use case I rather meant a sales pitch for your initial email:

    I use this bash script because it fits in my workflow because
    I need branches instead of detached HEADS, because $REASONS

and I'd be interested in these $REASONS, which I assumed to be
* easier to work with branches than detached HEADS (it aids the workflow)
* we're not challenging the underlying mental model of tracking sha1s in
  the superproject rather than branches.

At least I gave these reasons in the "reattach HEAD" stuff that I wrote,
but maybe there are others? (I know the code base of submodules very
well, but I do not work with submodules on a day-to-day basis myself...)

AW: Restoring detached HEADs after Git operations

[Patrick Lehmann]

Hello Stefan,

I never have tapped into the DLL Hell trap. That's maybe I never did C++ development or I started with VB .NET / C# as .NET solved major parts of the DLL Hell :). That doesn't mean my new beloved language Python doesn't have a similar problem ...


Thinking about DLL Hell is a thinking in big version numbers like 1.0, 2.0 oder even 2.1, 2.2, ...
We are here talking about revisions in the build numbers which need to be synchronized between the parent repository and the sub modules (IP cores). Both sides are under heavy development and interfaces evolving from day to day because hardware design can't be planned as easy as software design.

So by using Git submodules a developer - responsible for a submodule / IP core - can after he finished interface level 1 now go on and implement interface level 2. The parent project can finish it's integration and testing of the level 1 interface before proceeding with level 2. More over if the same IP core is used multiple time in different sub IP cores, it's possible to update one usage place to interface level 2 by a second developer so he can finish his IP core at level 2, which other usage places can still use the level 1 interface.

Start situation:
--------------------------------------
TOPLEVEL (developer A)
  o- IP_1 @level1 (developer B)
       o- IP_2 @level1 (developer C)
  o- IP_3 @level1 (developer D)
       o- IP_2 @level1


Developer C creates interface level 2, but all instances use level1 of IP_2:
--------------------------------------
TOPLEVEL (developer A)
  o- IP_1 @level1 (developer B)
       o- IP_2 @level1 (developer C)
  o- IP_3 @level1 (developer D)
       o- IP_2 @level1


Developer D updates instance of IP_2 to level 2 and completes level 2 of IP_3:
--------------------------------------
TOPLEVEL (developer A)
  o- IP_1 @level1 (developer B)
       o- IP_2 @level1 (developer C)
  o- IP_3 @level1 (developer D)
       o- IP_2 @level2

Developer A updates instance of IP_3 to level 2:
--------------------------------------
TOPLEVEL (developer A)
  o- IP_1 @level1 (developer B)
       o- IP_2 @level1 (developer C)
  o- IP_3 @level2 (developer D)
       o- IP_2 @level2

Developer B has finished his testing for IP_1 and can now update the instance if IP_2:
--------------------------------------
TOPLEVEL (developer A)
  o- IP_1 @level1 (developer B)
       o- IP_2 @level2 (developer C)
  o- IP_3 @level2 (developer D)
       o- IP_2 @level2


So now imaging 8 developers, whereof 6 are working remote on the project. There is one responsible developer per IP core (maintainer) and an overall maintainer overseeing all integration merges and test results (CI).


Kind regards
    Patrick

________________________________________
Von: Stefan Beller [sbeller@google.com]
Gesendet: Montag, 19. Juni 2017 21:21
Bis: Patrick Lehmann
Cc: Lars Schneider; Git Mailinglist
Betreff: Re: Restoring detached HEADs after Git operations

On Mon, Jun 19, 2017 at 11:09 AM, Patrick Lehmann
<Patrick.Lehmann@plc2.de> wrote:
> Hello Stefan,
>
> the use case is as follows:
>
> The projects consists of circa 18 IP cores. Each IP core is represented by a Git repository. Think of an IP core as of a lonestanding DLL or SO file project. Each IP core references 2 submodules, which bring the verification environments for testing the IP core standalone.

So phrased differently: You are using submodules to avoid "DLL hell"
(sharing a lib, with ease of versioning as the submodules in the different IP
cores may be pointing at different versions).

>
> These 18 IP cores are grouped to bigger IP cores, referencing the low-level IP cores and each again the 2 verification submodules. Finally, the main project references the bigger IP cores and again the 2 verification cores.
>
> TOPLEVEL
>   o- IP1
>        o- UVVM
>        o- VUnit
>   o- IP2
>        o- UVVM
>        o- VUnit
>   o- IP3
>        o- UVVM
>        o- VUnit
>   o- IP4
>        o- UVVM
>        o- VUnit
>        o- IP5
>            o- UVVM
>            o- VUnit
>        o- IP6
>            o- UVVM
>            o- VUnit
>        o- IP7
>            o- UVVM
>            o- VUnit
>   o- IP8
>        o- UVVM
>        o- VUnit
>        o- IP9
>            o- UVVM
>            o- VUnit
>        o- IP10
>            o- UVVM
>            o- VUnit
>   o- IP11
>        o- UVVM
>        o- VUnit
>        o- IP9
>            o- UVVM
>            o- VUnit
>        o- IP12
>            o- UVVM
>            o- VUnit
>    o- UVVM
>    o- VUnit
>
> That's the simplified structure. I can't write more, because it's a closed source project. You can find other usecases e.g. in my other open source projects. E.g. The PoC-Library or The PicoBlaze-Library and the corresponding PoC-Examples repository.
>
> Example: PoC
> Pile of Cores includes 4 Git submodules and is itself an IP core library.
> So PoC-Examples again references PoC. This looks like this tree:
>
> PoC-Examples
>   |- lib/
>        o- PoC
>             |- lib
>                 o- Cocotb
>                 o- OSVVM
>                 o- VUnit
>                      o- .... OSVVM
>                 o- UVVM
>
> The library VUnit itself already includes OSVVM as a library.
>
> ----------------------
> Forcast:
> I'll write a new question / idea about multiple equal submodules and the memory footprint soon...
> Here is my original question posted on StackOverflow: https://stackoverflow.com/questions/44585425/how-to-reduce-the-memory-footprint-for-multiple-submodules-of-the-same-source
> ----------------------
>
> Do you need more use cases?
>

Well this use case points out a different issue than I hoped for. ;)
From the stackoverflow post and from looking at the layout here,
one of the major questions is how to deduplicate the submodule
object store for example.

By use case I rather meant a sales pitch for your initial email:

    I use this bash script because it fits in my workflow because
    I need branches instead of detached HEADS, because $REASONS

and I'd be interested in these $REASONS, which I assumed to be
* easier to work with branches than detached HEADS (it aids the workflow)
* we're not challenging the underlying mental model of tracking sha1s in
  the superproject rather than branches.

At least I gave these reasons in the "reattach HEAD" stuff that I wrote,
but maybe there are others? (I know the code base of submodules very
well, but I do not work with submodules on a day-to-day basis myself...)

Re: Restoring detached HEADs after Git operations

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Mon, Jun 19, 2017 at 2:52 AM, Patrick Lehmann
> <Patrick.Lehmann@plc2.de> wrote:
>> Hello Lars,
>>
>> for your questions:
>>> If there are multiple branches with the same hash then your script would pick the first one. Can you imagine a situation where this would be a problem?
>>
>> I can't think of a good solution to resolve it automatically. Maybe a script could print that there are multiple possibilities and it choose the first branch in the list.
>>
>>
>>> Plus, you are looking only at local branches. Wouldn't it make sense to look at remote branches, too?
>>
>> This is also related to restoring tags. If we go this way, we should have this priority list:
>> - local branches
>> - remote branches
>
> For remote branches you would create a local branch of the same name
> (if such a branch would not exist, possibly setting it up to track that remote
> branch)?
>
>> - tags
>
> as said in the other email and similar to remote branches, we'd not want to have
> HEAD pointing to them directly but somehow have a local branch.

Let's step back a bit.  We detach the HEAD for a good reason, no?
Why is it a good idea to move them back on to a branch picked among
multiple ones that all happen to be pointing at the same commit?

The user may build on a history of a submodule, and then may push
the result out to a particular branch at the other side; that is
when being on a named branch in the submodule becomes useful, but
even then I do not think randomly picking one branch and be on it
is a good thing to do.

I would understand the workflow would go more like so:

 - You do something at the superproject (e.g. create a new branch X
   from an existing commit and check it out), which results in
   submodules' HEADs getting detached at the commits bound to the
   superproject's tree.

 - Because you want to make changes to both submodules and the
   superproject in a consistent way, you'd want to commit changes to
   all of these repositories and the push the result out in an
   atomic way.

 - Hence you tell "Hey, Git, I want all the submodules that I
   modified to be on branch X" from the superproject.

   - This may succeed in a submodule where X is a new name, or the
     current tip of branch X is an ancestor of the detached HEAD.

   - This may fail in a submodule where there is branch X that does
     not want to move to the detached HEAD's state.  In this latter
     case, the user needs to deal with the situation (perhaps the
     old X is expendable; perhaps the HEAD's commit may need to be
     merged to old X; perhaps there are other cases).

though.

Re: Restoring detached HEADs after Git operations

[Stefan Beller]

On Mon, Jun 19, 2017 at 10:55 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> On Mon, Jun 19, 2017 at 2:52 AM, Patrick Lehmann
>> <Patrick.Lehmann@plc2.de> wrote:
>>> Hello Lars,
>>>
>>> for your questions:
>>>> If there are multiple branches with the same hash then your script would pick the first one. Can you imagine a situation where this would be a problem?
>>>
>>> I can't think of a good solution to resolve it automatically. Maybe a script could print that there are multiple possibilities and it choose the first branch in the list.
>>>
>>>
>>>> Plus, you are looking only at local branches. Wouldn't it make sense to look at remote branches, too?
>>>
>>> This is also related to restoring tags. If we go this way, we should have this priority list:
>>> - local branches
>>> - remote branches
>>
>> For remote branches you would create a local branch of the same name
>> (if such a branch would not exist, possibly setting it up to track that remote
>> branch)?
>>
>>> - tags
>>
>> as said in the other email and similar to remote branches, we'd not want to have
>> HEAD pointing to them directly but somehow have a local branch.
>
> Let's step back a bit.  We detach the HEAD for a good reason, no?

And the 'good reason' being that at the time git-submodule was written
we did not know what would be best, and having a detached HEAD
would be (a) easy to implement, and (b) removing one moving thing
from the whole construction, hence making it a bit safer,
(c) it sort of follows the mental model:

    the superproject said it had the submodule at X
        (and not at branch Y!)
    the submodule itself is a whole repo on its own
        (it doesn't need to be aware of the superproject)

so in this world detaching at X is the best we can do.

> Why is it a good idea to move them back on to a branch picked among
> multiple ones that all happen to be pointing at the same commit?

This (rhetorical?) question reads like 2 questions actually:
(a) "Why is it a good idea to move them back on to a branch?"
It makes working easier as the submodule is not detached,
but on a proper branch
(b) "picked among multiple ones that all ..."
I think this is a bad idea and we'd rather want to follow
some configuration instead of wild-guessing by Git.

> The user may build on a history of a submodule, and then may push
> the result out to a particular branch at the other side; that is
> when being on a named branch in the submodule becomes useful, but
> even then I do not think randomly picking one branch and be on it
> is a good thing to do.

so you provide one reason why it is useful, but then claiming it is
'not a good thing' (yet). Can you give a reason why this is a 'bad thing'?

> I would understand the workflow would go more like so:
>
>  - You do something at the superproject (e.g. create a new branch X
>    from an existing commit and check it out), which results in
>    submodules' HEADs getting detached at the commits bound to the
>    superproject's tree.

And here we'd want to discuss if we *really* want to detach the HEADs
or rather have a symbolic ref "following the superproject".

>  - Because you want to make changes to both submodules and the
>    superproject in a consistent way, you'd want to commit changes to
>    all of these repositories and the push the result out in an
>    atomic way.

Committing and pushing are different things. You should not care if
I commit atomically as you (in the general "upstream" sense)
cannot observe my local commits.

For pushing we would want to have an atomic push, but that is
not the scope of this discussion. (As a Gerrit user, we implemented
the submodule atomicity serverside, but in plain Git server you'd
not need the atomicity either:

    git commit -a -m "update submodule pointers"
    git submodule foreach git push
    git push

should be fine w.r.t. any non-atomic race condition.)

>  - Hence you tell "Hey, Git, I want all the submodules that I
>    modified to be on branch X" from the superproject.
>
>    - This may succeed in a submodule where X is a new name, or the
>      current tip of branch X is an ancestor of the detached HEAD.

so we'd allow fast forward for X. This seems arbitrary to me. I could
also say "If X exists I allow a merge to be made between old X and
the object name given by the superproject". (maybe as a config option)

>    - This may fail in a submodule where there is branch X that does
>      not want to move to the detached HEAD's state.  In this latter
>      case, the user needs to deal with the situation (perhaps the
>      old X is expendable; perhaps the HEAD's commit may need to be
>      merged to old X; perhaps there are other cases).

makes sense.

>
> though.

Re: Restoring detached HEADs after Git operations

[Stefan Beller]

On Mon, Jun 19, 2017 at 1:46 AM, Patrick Lehmann
<Patrick.Lehmann@plc2.de> wrote:
> Hello,
>
> I wrote a Bash script to recover branch names after Git operations have create detached HEADs in a Git repository containing lots of Git submodules. The script works recursively.

Cool. :)

You may also like
https://public-inbox.org/git/20170501180058.8063-5-sbeller@google.com/
https://public-inbox.org/git/20170501180058.8063-6-sbeller@google.com/

These patches are still on my plate, they are not landed yet as I had issues
coming up with a good convincing commit message.

They are essentially putting submodules back on a branch (if configured).
Let's see how this differs from your solution.


> I would like to see:
> a) that script or algorithm being integrated into Git by default

For that you'd want to send a patch, see Documentation/SubmittingPatches.
We'd want to discuss if this command is an independent command
("git submodule reattachHEADs", name subject to bikeshedding ;) )
or if it is a configurable option that is obeyed by anything that touches
submodules (which I would prefer, as this mode seems to be the
"correct default". When having it as a mode we can switch the default
eventually such that submodules are always on a branch).

> b) that as a default behavior for all Git operations creating detached HEADs

changing defaults is hard. Let's go with a) first and then people will
report how
awesome the new mode/command is and then it is easier to see how this
may be a good default. :)

>
> That's the command:
> --------------------------------
(reformatted for readability:)

git submodule foreach --recursive
  'HEAD=$(git branch --list | head -n 1);
    if [[ "$HEAD" == *HEAD* ]]; then
      REF=$(git rev-parse HEAD);
      FOUND=0;
      for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" );
      do
        if [[ "$(git rev-parse "$Branch")" == $REF ]]; then
          echo -e "  \e[36mCheckout $Branch...\e[0m";
          git checkout $Branch;
          FOUND=1;
          break;
        fi
      done;
      if [[ $FOUND -eq 0 ]]; then
        echo -e "  \e[31mNo matching branch found.\e[0m";
      fi
    else
      echo -e "  \e[36mNothing to do.\e[0m";
    fi'

> --------------------------------
>
> How does it work:
> 1. It uses git submodule foreach to dive into each Git submodule and execute a series of Bash commands.

If you want to see it upstream eventually, we'd make it shell commands.
There are some subtle differences between shell and bash,
one of them is the way conditions are written. I think plain shell
does not support [[ ]], so that would become

  if test $FOUND -eq 0
  then
    echo ...

Maybe look at git-submodule.sh for coding style suggestions.

> 2. It's reading the list of branches and checks if the submodule is in detached mode. The first line contains the string HEAD.

This works for you but some crazy person may have a branch containing
HEAD in their branch name. ;)
("git checkout -b notADetachedHEAD")

I think that check can be improved via

    if test $(git symbolic-ref HEAD 2>/dev/null >/dev/null) -eq 128
    then
      # detached HEAD
    else
      # on a branch
    fi

so if the output of symbolic-ref starts with ref then it is on a
branch. In detached HEAD

> 3. Retrieve the hash of the detached HEAD
> 4. Iterate all local branches and get their hashes

  What happens (/should happen) when multiple branches have the same sha1?
  With this implementation the first wins? Is this 'lazy guessing' desired?
  The patches referenced above assumed you'd have submodule.NAME.branch
  set and we'd reattach to that branch only (if matching hashes)

> 5. Compare the branch hashes with the detached HEAD's hash. If it matches do a checkout.

Speaking of checkout: checkout --recurse-submodules is a
thing in the latest version of Git, but it also detaches HEADs.

I'd like to have reattaching HEADs in there and then combined with
"git config submodule.recurse true", which is in master but no release
a plain "git checkout <branch>" in the superproject would put the submodules
on branches.

Using checkout within git submodule-foreach works of course just as fine.
Note: Currently Prathamesh Chavan converts git-submodule-foreach to C
https://public-inbox.org/git/CAME+mvUrzVxpRdPDvA1ZyatNm2R27QGJVjSB3=KX85CEedMaRQ@mail.gmail.com/
so it will be faster. In the process of doing so, we surfaced a couple
of bugs, but
they would not impact this script AFAICT.


> 6. Report if no branch name was found or if a HEAD was not in detached mode.

... and it is colored unconditionally in red. Maybe have a look at
    git config --get-color[bool]
which can help in figuring out if we want to print color codes.

> The Bash code with line breaks and indentation:
> --------------------------------
> HEAD=$(git branch --list | head -n 1)
> if [[ "$HEAD" == *HEAD* ]]; then
>   REF=$(git rev-parse HEAD)
>   FOUND=0
>   for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do
>     if [[ "$(git rev-parse "$Branch")" == $REF ]]; then
>       echo -e "  \e[36mCheckout $Branch...\e[0m"
>       git checkout $Branch
>       FOUND=1
>       break
>     fi
>   done
>   if [[ $FOUND -eq 0 ]]; then
>     echo -e "  \e[31mNo matching branch found.\e[0m"
>   fi
> else
>   echo -e "  \e[36mNothing to do.\e[0m"
> fi
> --------------------------------
>
> Are their any chances to get it integrated into Git?

I like the idea and I'd be happy to review patches. :)
Also you may want to look at the C version that I provided
above and tell me why yours is better. ;)
(Maybe the chosen defaults are saner, or such?)

>
> I tried to register that code as a Git alias, but git config complains about quote problem not showing where. It neither specifies if it's a single or double quote problem. Any advice on how to register that piece of code as an alias?

(a) not using the alias system for everything:
* You can define this as a (ba)sh function in e.g. .bashrc and then
just call the shell function from the alias.
* or you can put the code into an executable script "git-NAME" and
then the alias would be just "git submodule foreach --recursive git
NAME"
(b) define the function inside the alias, cf.
https://www.atlassian.com/blog/git/advanced-git-aliases

> If wished, I think I could expand the script to also recover hash values to Git tags if no branch was found.

Personally I do not think we should attach a HEAD to a tag in that case.
tags are just like branches with special meaning, i.e. they are also
in the refs/* hierarchy.  Note how git-checkout <tag> detaches from
the tag such that you do not modify the tag by default.

>
> Kind regards
>     Patrick Lehmann

Re: Restoring detached HEADs after Git operations

[Jeff King]

On Mon, Jun 19, 2017 at 08:46:45AM +0000, Patrick Lehmann wrote:

>   for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do
>     if [[ "$(git rev-parse "$Branch")" == $REF ]]; then
>       echo -e "  \e[36mCheckout $Branch...\e[0m"
>       git checkout $Branch
>       FOUND=1
>       break
>     fi
>   done

I see Lars pointed you at for-each-ref, which is the preferred way for
scripts to enumerate branches. But you can also use --points-at to skip
the inner part of your loop entirely, like:

  git for-each-ref --points-at=HEAD --format='%(refname:short)' refs/heads

That should run much faster, as it only has to spawn one process rather
than one for each branch (it will print all of them, of course. You can
pipe through head -n 1 to get the first, and check out --sort if you
want to prioritize by recency or similar).

-Peff

Re: Restoring detached HEADs after Git operations

[Ævar Arnfjörð Bjarmason]

On Mon, Jun 19 2017, Patrick Lehmann jotted:

> Hello,
>
> I wrote a Bash script to recover branch names after Git operations have create detached HEADs in a Git repository containing lots of Git submodules. The script works recursively.
>
> I would like to see:
> a) that script or algorithm being integrated into Git by default
> b) that as a default behavior for all Git operations creating detached HEADs
>
> That's the command:
> --------------------------------
> git submodule foreach --recursive  'HEAD=$(git branch --list | head -n 1); if [[ "$HEAD" == *HEAD* ]]; then REF=$(git rev-parse HEAD); FOUND=0; for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do if [[ "$(git rev-parse "$Branch")" == $REF ]]; then echo -e "  \e[36mCheckout $Branch...\e[0m"; git checkout $Branch; FOUND=1; break; fi done; if [[ $FOUND -eq 0 ]]; then echo -e "  \e[31mNo matching branch found.\e[0m"; fi else echo -e "  \e[36mNothing to do.\e[0m"; fi'
> --------------------------------
>
> How does it work:
> 1. It uses git submodule foreach to dive into each Git submodule and execute a series of Bash commands.
> 2. It's reading the list of branches and checks if the submodule is in detached mode. The first line contains the string HEAD.
> 3. Retrieve the hash of the detached HEAD
> 4. Iterate all local branches and get their hashes
> 5. Compare the branch hashes with the detached HEAD's hash. If it matches do a checkout.
> 6. Report if no branch name was found or if a HEAD was not in detached mode.
>
> The Bash code with line breaks and indentation:
> --------------------------------
> HEAD=$(git branch --list | head -n 1)
> if [[ "$HEAD" == *HEAD* ]]; then
>   REF=$(git rev-parse HEAD)
>   FOUND=0
>   for Branch in $(git branch --list | grep "^  " | sed -e "s/  //" ); do
>     if [[ "$(git rev-parse "$Branch")" == $REF ]]; then
>       echo -e "  \e[36mCheckout $Branch...\e[0m"
>       git checkout $Branch
>       FOUND=1
>       break
>     fi
>   done
>   if [[ $FOUND -eq 0 ]]; then
>     echo -e "  \e[31mNo matching branch found.\e[0m"
>   fi
> else
>   echo -e "  \e[36mNothing to do.\e[0m"
> fi
> --------------------------------
>
> Are their any chances to get it integrated into Git?
>
> I tried to register that code as a Git alias, but git config complains about quote problem not showing where. It neither specifies if it's a single or double quote problem. Any advice on how to register that piece of code as an alias?
>
> If wished, I think I could expand the script to also recover hash values to Git tags if no branch was found.

I have something similar to this, this written before git-submodule
learned --branch (or at least before I knew about it):

    $ git config alias.sm-pull-all
    !git submodule foreach 'git checkout $(NAME=$name git sm-mainbranch) && git pull'
    $ git config alias.sm-mainbranch
    !git config --file ../.gitmodules submodule.$NAME.branch || git describe --all --always | sed 's!^heads/!!'

So with this I can run `git sm-pull-all` to update all the submodules in
a superproject to update them all. This relies on the branch name being
in the .gitmodules config.a

Now if you add a module with e.g. 'git submodule add -b master ...'
you'll have it checked out at the master branch, but I'm not familiar
enough with all the workflows around them to say if there are any holes
in that implementation.

But that seems like a fundimentally better approach to me than what
you're suggesting. Why would we try to work our way back from the SHA-1
and guess what branch it lives on when we can just encode the branch
name we'd like to be on in the superproject?