Merging submodules

Merging submodules

[Brian Gernhardt]

This message got eaten by a syntax error somewhere.  This is a re-send, sorry for any duplicate messages.

On Jul 30, 2008, at 9:58 AM, H.Merijn Brand wrote:

> One (very) big disadvantage of  SCCS  is that commits are on a per-file
> basis, and only in a single directory. This drawback still haunts me in
> git, as my first attempts to convert were successful in a single folder
> and git cannot merge folders into a single project.
> 
> Say I now have
> 
> /work/src/project/.git
> /work/src/project/module_a/.git
> /work/src/project/module_b/.git
> /work/src/project/module_c/.git
> 
> Which are all converted repos from SCCS, I'd like to merge the three
> module_# repos into the top level repo.

Following the example of Linus, the following is completely untested.

First you fetch all of the heads/tags/etc into the superproject with commands like

git fetch module_a refs/heads/*:refs/remotes/module_a/*
git fetch module_b
refs/heads/*:refs/remotes/module_b/*
git fetch module_c
refs/heads/*:refs/remotes/module_c/*

Then you do something like:

rm -rf module_{a,b,c}/.git # Do this in a test repository, obviously...
git add module_a module_b module_c
git commit # Needed because '-s ours' uses current HEAD, not index
git merge --no-commit -s ours module_a/master module_b/master module_c/master
git commit --amend

>From this point on, the project repository has a merged history of the sub-projects, and if anyone doesn't catch up and still makes a commit on a subproject you can use "git merge -s subtree" to merge it in anyway.

You may need to "git rm --cached" some files after the "git add" step if your .gitignore files aren't perfect.

~~ Brian

Re: Merging submodules (was Re: Feature suggestion: git-hist)

[Brian Gernhardt]

This message got eaten by a syntax error somewhere.  This is a re-send, sorry for any duplicate messages.

On Jul 30, 2008, at 12:26 PM, H.Merijn Brand wrote:

> On Wed, 30 Jul 2008 11:15:55 -0400, Brian Gernhardt
> <benji@silverinsanity.com> wrote:
> 
> > Then you do something like:
> > 
> > rm -rf module_{a,b,c}/.git # Do this in a test repository, obviously...
> > git add module_a module_b module_c
> > git commit # Needed because '-s ours' uses current HEAD, not index
> 
> So far so good.
> 
> > git merge --no-commit -s ours module_a/master module_b/master module_c/master
> 
> $ git merge --no-commit -s ours fnc/master i00f000/master
> i99f000/master include/master l00m000/master l01f000/master
> l02f000/master l03f000/master l06f000/master l90z000/master
> leerpl/master mutbev/master prtabel/master rpt/master tabellen/master
> zoomen/master Automatic merge went well; stopped before committing as
> requested
> 
> > git commit --amend
> 
> $ git commit --amend
> fatal: You are in the middle of a merge -- cannot amend.

Hm.  I did mention this was completely untested, yes?  The problem comes
from the fact that '-s ours' wants to use HEAD, not the index.  But you
can't amend a normal commit into a merge, apparently.  And I don't think
you want a commit that adds the files and a commit that "does the merge"
as two separate steps.

Well, I don't know how to make the porcelain do this then. But the
plumbing can definitely do it.  Hopefully someone more used to doing
strange things like this can give a simpler recipe, but this should
work.

# First reset to the commit you made with all the modules added.
vim commit-message # Create a merge message
commit=$(git commit-tree HEAD: -p HEAD^ -p module_a/master -p
		module_b/master -p module_c/master < commit-message)
git update-ref HEAD $commit  # Update your current ref

~~ Brian

Re: Merging submodules (was Re: Feature suggestion: git-hist)

[H.Merijn Brand]

[-- Attachment #1: Type: text/plain, Size: 5759 bytes --]

On Wed, 30 Jul 2008 19:03:37 -0400, Brian Gernhardt
<benji@silverinsanity.com> wrote:

> This message got eaten by a syntax error somewhere.  This is a re-send, sorry for any duplicate messages.
> 
> On Jul 30, 2008, at 12:26 PM, H.Merijn Brand wrote:
> 
> > On Wed, 30 Jul 2008 11:15:55 -0400, Brian Gernhardt
> > <benji@silverinsanity.com> wrote:
> > 
> > > Then you do something like:
> > > 
> > > rm -rf module_{a,b,c}/.git # Do this in a test repository, obviously...
> > > git add module_a module_b module_c
> > > git commit # Needed because '-s ours' uses current HEAD, not index
> > 
> > So far so good.
> > 
> > > git merge --no-commit -s ours module_a/master module_b/master module_c/master
> > 
> > $ git merge --no-commit -s ours fnc/master i00f000/master
> > i99f000/master include/master l00m000/master l01f000/master
> > l02f000/master l03f000/master l06f000/master l90z000/master
> > leerpl/master mutbev/master prtabel/master rpt/master tabellen/master
> > zoomen/master Automatic merge went well; stopped before committing as
> > requested
> > 
> > > git commit --amend
> > 
> > $ git commit --amend
> > fatal: You are in the middle of a merge -- cannot amend.
> 
> Hm.  I did mention this was completely untested, yes?  The problem comes
> from the fact that '-s ours' wants to use HEAD, not the index.  But you
> can't amend a normal commit into a merge, apparently.  And I don't think
> you want a commit that adds the files and a commit that "does the merge"
> as two separate steps.
> 
> Well, I don't know how to make the porcelain do this then. But the
> plumbing can definitely do it.  Hopefully someone more used to doing
> strange things like this can give a simpler recipe, but this should
> work.
> 
> # First reset to the commit you made with all the modules added.
> vim commit-message # Create a merge message
> commit=$(git commit-tree HEAD: -p HEAD^ -p module_a/master -p
                                 ^^^^^^^^
had to remove that part

> 		module_b/master -p module_c/master < commit-message)
> git update-ref HEAD $commit  # Update your current ref

Some history
---
I'm aware I started at the wrong end of being a git user. I had to move
from SCCS to `something better', and at that point only git, svn, and
hq seemed to be likely candidates.

hq being python, and our company not using python, but perl, made that
an easy drop. I gave up compiling svn on HP-UX in 64bit mode after
about a week, mainly because it depended on way too many things, and
the new VCS has to run on this platform, as it is our main development
system. I got git up and running in two days (compile in less than two
hours, but then I got to chase HP-UX and 64bit oddities).

By the I knew a lot about the git source code, make files, and test
scripts, but still had no idea about the whole plumbing/porcelain
approach. The plan was to make that someone else's job.

Once it was up and running, I had to create a way to convert all our
SCCS repo's to git, so we could get started and test if it met our
needs. That part went smooth, and with a little help from Sam Villain
to get some speed into the conversions using git-fast-import, it is now
available to the public on CPAN as VCS::SCCS, with git2sccs in the
examples folder.

Using git-gui and gitk my users were enthousiastic, and they saw all
the advantages of using git over SCCS. Of course, with every change
there are a few (serious) drawbacks, but we have to live with those.

Being a perl5 porter/maintainer, I was used to p4v (perforce) and still
wonder why there are two GUI's instead of just one, and why they don't
offer the functionality I love in p4v. Not that I think perforce is
better than git, but their GUI certainly is.
---

So, back to this merging issue. Now you might understand why I have all
those `silly' questions and have (still) no good idea of what all these
commands do. (The person that were to do all that never came into the
picture). I'm learning.

I'm VERY happy and thankful for the help I get from you here, and I get
the impression that my feedback on getting git running in our somewhat
different environment to you is also appreciated.

I had to cut down my number of modules to merge, as I got an error that
the maximum number of merges was 16. I had 18.

I will now be playing with the results a bit. I have attached the
script, in case you might want to use it in documentation or examples.
For now, all the mods are hardcoded. No arguments and so on.

Again, Thanks!

$ bash git-merge-mods.sh
Re-initialize GIT repo
Initialized empty Git repository in /work/lep/4gl/.git/
Recovering original module repo's
Fetching for i00f000
remote: Counting objects: 24, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 24 (delta 9), reused 24 (delta 9)
Unpacking objects: 100% (24/24), done.
From i00f000
 * [new branch]      master     -> i00f000/master
Fetching for i99f000
:
:
Receiving objects: 100% (356/356), 139.05 KiB, done.
Resolving deltas: 100% (180/180), done.
From rpt
 * [new branch]      master     -> rpt/master
Removing module repo's
Adding modules
Commit
Merge
Automatic merge went well; stopped before committing as requested
Commit
=========
53229f046c5d85d11bbd500cf04b468fd3f62c08
=========
Update
$




-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

[-- Attachment #2: git-merge-mods.sh --]
[-- Type: application/x-shellscript, Size: 1057 bytes --]

Re: Merging submodules

[H.Merijn Brand]

On Thu, 31 Jul 2008 09:21:04 +0200, "H.Merijn Brand"
<h.m.brand@xs4all.nl> wrote:

> I will now be playing with the results a bit. I have attached the
> script, in case you might want to use it in documentation or examples.
> For now, all the mods are hardcoded. No arguments and so on.
> 
> Again, Thanks!

There is a slight problem with this merging approach. The path names
are as they are/were in the submodules. In module_a, foo.pl was without
a leading module_a/ path, and now after integration, it still is. Is it
possible to rethink this whole process that integrates/merges the
several git repo's in subfolders into the current folder, as-if they
would have been in this folder in the first place?

Short summary: History made me have the current situation

	4gl
	4gl/fnc
	4gl/fnc/.git
	4gl/foo
	4gl/foo/.git
	:
	:
	4gl/zlork
	4gl/zlork/.git

Where I want

	4gl
	4gl/.git
	4gl/fnc
	4gl/foo
	:
	4gl/zlork

-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[Petr Baudis]

On Thu, Jul 31, 2008 at 02:39:55PM +0200, H.Merijn Brand wrote:
> On Thu, 31 Jul 2008 09:21:04 +0200, "H.Merijn Brand"
> <h.m.brand@xs4all.nl> wrote:
> 
> > I will now be playing with the results a bit. I have attached the
> > script, in case you might want to use it in documentation or examples.
> > For now, all the mods are hardcoded. No arguments and so on.
> > 
> > Again, Thanks!
> 
> There is a slight problem with this merging approach. The path names
> are as they are/were in the submodules. In module_a, foo.pl was without
> a leading module_a/ path, and now after integration, it still is. Is it
> possible to rethink this whole process that integrates/merges the
> several git repo's in subfolders into the current folder, as-if they
> would have been in this folder in the first place?

I would suggest re-reading Santi's suggestions:

> You have, basically, two possibilities:
> 
> 1) Add the module_# as submodules:
>   http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
>   http://git.or.cz/gitwiki/GitSubmoduleTutorial
> 2) Add the submodules as subtrees (as gitk and git-gui in git.git)
>   http://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html

I think the latter is specifically what you want.

-- 
				Petr "Pasky" Baudis
As in certain cults it is possible to kill a process if you know
its true name.  -- Ken Thompson and Dennis M. Ritchie

Re: Merging submodules

[Santi Béjar]

On Thu, Jul 31, 2008 at 14:39, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
>
> There is a slight problem with this merging approach. The path names
> are as they are/were in the submodules. In module_a, foo.pl was without
> a leading module_a/ path, and now after integration, it still is. Is it
> possible to rethink this whole process that integrates/merges the
> several git repo's in subfolders into the current folder, as-if they
> would have been in this folder in the first place?

If you want to have module_a/foo.pl in the old commits you will need
to rewrite the history, either with "git filter-branch" or
fast-export/import, or importing again.

For the new commits it depends on how you would do them. If you modify
and commit without these module subfolders (without the module_a/.git
as your git-merge-mods does) then the new commits will have
module_a/foo.pl. So at the end you will have a merge of all the mods
(using the subtree strategy), and after that you will have a single
project.

All depends in a few factors, if you have published the history (for
the rewriting), if you want to have the module_# as independent
projects (this way you can merge them in other projects),...

Santi

Re: Merging submodules

[H.Merijn Brand]

On Thu, 31 Jul 2008 15:06:26 +0200, Petr Baudis <pasky@suse.cz> wrote:

> On Thu, Jul 31, 2008 at 02:39:55PM +0200, H.Merijn Brand wrote:
> > On Thu, 31 Jul 2008 09:21:04 +0200, "H.Merijn Brand"
> > <h.m.brand@xs4all.nl> wrote:
> > 
> > > I will now be playing with the results a bit. I have attached the
> > > script, in case you might want to use it in documentation or examples.
> > > For now, all the mods are hardcoded. No arguments and so on.
> > > 
> > > Again, Thanks!
> > 
> > There is a slight problem with this merging approach. The path names
> > are as they are/were in the submodules. In module_a, foo.pl was without
> > a leading module_a/ path, and now after integration, it still is. Is it
> > possible to rethink this whole process that integrates/merges the
> > several git repo's in subfolders into the current folder, as-if they
> > would have been in this folder in the first place?
> 
> I would suggest re-reading Santi's suggestions:
> 
> > You have, basically, two possibilities:
> > 
> > 1) Add the module_# as submodules:
> >   http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
> >   http://git.or.cz/gitwiki/GitSubmoduleTutorial
> > 2) Add the submodules as subtrees (as gitk and git-gui in git.git)
> >   http://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html
> 
> I think the latter is specifically what you want.

I got stuck in that process, as the new repo thinks all the files from
the sub-folders belong to the top-folder: After this process, there are
a lot of files from subfolder include in the "new" top-level folder.

--8<--- git-join.sh
#!/bin/sh

export MODS
MODS="include fnc"
MODS="$MODS i00f000 i99f000"
MODS="$MODS l00m000 l01f000 l02f000 l03f000 l06f000 l90z000"
MODS="$MODS rpt"
MODS="$MODS leerpl mutbev prtabel tabellen zoomen"

echo "Creating merge environment"
rm -rf new
mkdir new
cd new

echo "Recovering original repo's"
tbz -s -x ../lep4gl-git

echo "Initializing new repo"
git init

for mod in $MODS ; do
    echo "Merging $mod ..."
    git remote add -f B$mod $mod
    git merge -s ours --no-commit B$mod/master
    git read-tree --prefix=$mod/ -u B$mod/master
    git commit -m "Merge $mod as our subdirectory"
    git pull -s subtree B$mod master
    done

echo Done
-->8---



-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[Santi Béjar]

On Thu, Jul 31, 2008 at 17:01, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
> On Thu, 31 Jul 2008 15:06:26 +0200, Petr Baudis <pasky@suse.cz> wrote:
>
>> On Thu, Jul 31, 2008 at 02:39:55PM +0200, H.Merijn Brand wrote:
>> >
>> > There is a slight problem with this merging approach. The path names
>> > are as they are/were in the submodules. In module_a, foo.pl was without
>> > a leading module_a/ path, and now after integration, it still is. Is it
>> > possible to rethink this whole process that integrates/merges the
>> > several git repo's in subfolders into the current folder, as-if they
>> > would have been in this folder in the first place?
>>
>> I would suggest re-reading Santi's suggestions:
>>
>> > You have, basically, two possibilities:
>> >
>> > 1) Add the module_# as submodules:
>> >   http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
>> >   http://git.or.cz/gitwiki/GitSubmoduleTutorial
>> > 2) Add the submodules as subtrees (as gitk and git-gui in git.git)
>> >   http://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html
>>
>> I think the latter is specifically what you want.
>
> I got stuck in that process, as the new repo thinks all the files from
> the sub-folders belong to the top-folder: After this process, there are
> a lot of files from subfolder include in the "new" top-level folder.
>

I see all OK. Can you provide a self consistent simple testcase that
shows what is wrong?

# prepare the project and the modules
mkdir project
cd project
git init
...
mkdir module_a
cd module_a
git init
...
# your script
# show what you get and what you expect, for example:

$ git show HEAD:   # or something equivalent
$ ls
$ git status
...

> --8<--- git-join.sh
> #!/bin/sh
>
> export MODS
> MODS="include fnc"
> MODS="$MODS i00f000 i99f000"
> MODS="$MODS l00m000 l01f000 l02f000 l03f000 l06f000 l90z000"
> MODS="$MODS rpt"
> MODS="$MODS leerpl mutbev prtabel tabellen zoomen"
>
> echo "Creating merge environment"
> rm -rf new
> mkdir new
> cd new
>
> echo "Recovering original repo's"
> tbz -s -x ../lep4gl-git
>
> echo "Initializing new repo"
> git init
>
> for mod in $MODS ; do
>    echo "Merging $mod ..."
>    git remote add -f B$mod $mod
>    git merge -s ours --no-commit B$mod/master
>    git read-tree --prefix=$mod/ -u B$mod/master
>    git commit -m "Merge $mod as our subdirectory"


>    git pull -s subtree B$mod master

This is only needed when updating the subtree if the module has been
updated (but it should do nothing in you case).

>    done
>
> echo Done
> -->8---

Santi

Re: Merging submodules

[H.Merijn Brand]

On Thu, 31 Jul 2008 17:24:40 +0200, "Santi Béjar" <sbejar@gmail.com>
wrote:

> On Thu, Jul 31, 2008 at 17:01, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
> > On Thu, 31 Jul 2008 15:06:26 +0200, Petr Baudis <pasky@suse.cz> wrote:
> >
> >> On Thu, Jul 31, 2008 at 02:39:55PM +0200, H.Merijn Brand wrote:
> >> >
> >> > There is a slight problem with this merging approach. The path names
> >> > are as they are/were in the submodules. In module_a, foo.pl was without
> >> > a leading module_a/ path, and now after integration, it still is. Is it
> >> > possible to rethink this whole process that integrates/merges the
> >> > several git repo's in subfolders into the current folder, as-if they
> >> > would have been in this folder in the first place?
> >>
> >> I would suggest re-reading Santi's suggestions:
> >>
> >> > You have, basically, two possibilities:
> >> >
> >> > 1) Add the module_# as submodules:
> >> >   http://www.kernel.org/pub/software/scm/git/docs/git-submodule.html
> >> >   http://git.or.cz/gitwiki/GitSubmoduleTutorial
> >> > 2) Add the submodules as subtrees (as gitk and git-gui in git.git)
> >> >   http://www.kernel.org/pub/software/scm/git/docs/howto/using-merge-subtree.html
> >>
> >> I think the latter is specifically what you want.
> >
> > I got stuck in that process, as the new repo thinks all the files from
> > the sub-folders belong to the top-folder: After this process, there are
> > a lot of files from subfolder include in the "new" top-level folder.
> >
> 
> I see all OK. Can you provide a self consistent simple testcase that
> shows what is wrong?

Yes. was rather easy.
http://www.xs4all.nl/~hmbrand/testcase.tgz

test 133 > ll *
23875989 -rwxr-xr-x 1 merijn users   486 2008-07-31 20:08 git-join.sh
23875990 -rw-rw-rw- 1 merijn users 14385 2008-07-31 20:07 test-git.tgz

4gl:
total 16
23871493 drwxrwxrwx 4 merijn users 4096 2008-07-31 20:01 .
23871492 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:08 ..
23871494 drwxrwxrwx 3 merijn users 4096 2008-07-31 19:56 fnc
23875966 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:04 include
test 134 > bash git-join.sh
Creating merge environment
Recovering original repo's
Initializing new repo
Initialized empty Git repository in /work/lep/test/new/.git/
Merging include ...
Updating Binclude
remote: Counting objects: 9, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 9 (delta 1), reused 6 (delta 1)
Unpacking objects: 100% (9/9), done.
From include
 * [new branch]      master     -> Binclude/master
Created commit c2c491e: Merge include as our subdirectory
 1 files changed, 4 insertions(+), 0 deletions(-)
 create mode 100644 include/ini0.h
From include
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Merging fnc ...
Updating Bfnc
warning: no common commits
remote: Counting objects: 12, done.
remote: Compressing objects: 100% (4/4), done.
Unpacking objects: 100% (12/12), done.
remote: Total 12 (delta 0), reused 9 (delta 0)
From fnc
 * [new branch]      master     -> Bfnc/master
Automatic merge went well; stopped before committing as requested
Created commit d7da4d6: Merge fnc as our subdirectory
From fnc
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Done
test 135 > ll new
total 24
23887892 drwxrwxrwx 5 merijn users 4096 2008-07-31 20:11 .
23871492 drwxrwxrwx 4 merijn users 4096 2008-07-31 20:11 ..
23887973 drwxrwxrwx 8 merijn users 4096 2008-07-31 20:11 .git
23887893 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:11 fnc
23887933 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:11 include
23888029 -rw-rw-rw- 1 merijn users   56 2008-07-31 20:11 ini0.h
test 136 > ll new/include/
total 16
23887933 drwxrwxrwx 3 merijn users 4096 2008-07-31 20:11 .
23887892 drwxrwxrwx 5 merijn users 4096 2008-07-31 20:11 ..
23887934 drwxrwxrwx 8 merijn users 4096 2008-07-31 20:04 .git
23888031 -rw-rw-rw- 1 merijn users   56 2008-07-31 20:11 ini0.h
test 137 >

What is ini0.h doing in new/ ?

test 137 > cd new
test/new 138 > git-ls-files
fnc/foo.txt
include/ini0.h
ini0.h
test/new 139 >


-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[Santi Béjar]

On Thu, Jul 31, 2008 at 20:15, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
> On Thu, 31 Jul 2008 17:24:40 +0200, "Santi Béjar" <sbejar@gmail.com>
> wrote:
>
>> I see all OK. Can you provide a self consistent simple testcase that
>> shows what is wrong?
>
> Yes. was rather easy.
> http://www.xs4all.nl/~hmbrand/testcase.tgz
>

It is because you cannot merge a branch with an empty branch. So, or
you create an initial commit in the "superproject"  or you create a
commit just moving the files of the first module as in:

http://article.gmane.org/gmane.comp.version-control.git/79887

Santi

Re: Merging submodules

[H.Merijn Brand]

On Thu, 31 Jul 2008 21:03:26 +0200, "Santi Béjar" <sbejar@gmail.com>
wrote:

> On Thu, Jul 31, 2008 at 20:15, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
> > On Thu, 31 Jul 2008 17:24:40 +0200, "Santi Béjar" <sbejar@gmail.com>
> > wrote:
> >
> >> I see all OK. Can you provide a self consistent simple testcase that
> >> shows what is wrong?
> >
> > Yes. was rather easy.
> > http://www.xs4all.nl/~hmbrand/testcase.tgz
> >
> 
> It is because you cannot merge a branch with an empty branch.

Super. So I start with a .gitignore and continue this process. It
worked!

> So, or you create an initial commit in the "superproject" or you
> create a commit just moving the files of the first module as in:
> 
> http://article.gmane.org/gmane.comp.version-control.git/79887

Thanks again!

-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[H.Merijn Brand]

On Thu, 31 Jul 2008 21:03:26 +0200, "Santi Béjar" <sbejar@gmail.com>
wrote:

> On Thu, Jul 31, 2008 at 20:15, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
> > On Thu, 31 Jul 2008 17:24:40 +0200, "Santi Béjar" <sbejar@gmail.com> wrote:
> >
> >> I see all OK. Can you provide a self consistent simple testcase that
> >> shows what is wrong?
> >
> > Yes. was rather easy.
> > http://www.xs4all.nl/~hmbrand/testcase.tgz
> 
> It is because you cannot merge a branch with an empty branch. So, or
> you create an initial commit in the "superproject"  or you create a
> commit just moving the files of the first module as in:
> 
> http://article.gmane.org/gmane.comp.version-control.git/79887

Almost perfect now.

4gl/new 121 > git-ls-files | grep fnc_declare
include/fnc_declare.h
4gl/new 122 > git-show d971ec91a3dbc9f1f27a96e4f9b95366babd036c
commit d971ec91a3dbc9f1f27a96e4f9b95366babd036c
Author: H.M. Brand <merijn@a5.(none)>
Date:   Tue Jul 29 16:45:43 2008 +0200

    Backward comp functie voor PV indicatie

diff --git a/fnc_declare.h b/fnc_declare.h
index acd686c..25d9a73 100644
--- a/fnc_declare.h
+++ b/fnc_declare.h
@@ -136,6 +136,11 @@ eaf_N Indicaties    ($x1);
 eaf_N pv_datum      ($x1, $x2, $x3);
 #endif

+#ifndef PV_INDICATIE
+#define PV_INDICATIE
+eaf_S pv_indicatie  ($x1);
+#endif
+
 #undef eaf_B
 #undef eaf_N
 #undef eaf_S
4gl/new 123 >

all the diffs are still showing as if they were against the folder they
were originally committed in. Is that correct? I would now have expected
something like

commit d971ec91a3dbc9f1f27a96e4f9b95366babd036c
Author: H.M. Brand <merijn@a5.(none)>
Date:   Tue Jul 29 16:45:43 2008 +0200

    Backward comp functie voor PV indicatie

diff --git a/include/fnc_declare.h b/include/fnc_declare.h
index acd686c..25d9a73 100644
--- a/include/fnc_declare.h
+++ b/include/fnc_declare.h
@@ -136,6 +136,11 @@ eaf_N Indicaties    ($x1);
 eaf_N pv_datum      ($x1, $x2, $x3);
 #endif

+#ifndef PV_INDICATIE
+#define PV_INDICATIE
+eaf_S pv_indicatie  ($x1);
+#endif
+
 #undef eaf_B
 #undef eaf_N
 #undef eaf_S

-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[Santi Béjar]

On Fri, Aug 1, 2008 at 09:04, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
>
> Almost perfect now.
>
> 4gl/new 121 > git-ls-files | grep fnc_declare
> include/fnc_declare.h
> 4gl/new 122 > git-show d971ec91a3dbc9f1f27a96e4f9b95366babd036c
> commit d971ec91a3dbc9f1f27a96e4f9b95366babd036c
> Author: H.M. Brand <merijn@a5.(none)>
> Date:   Tue Jul 29 16:45:43 2008 +0200
>
>    Backward comp functie voor PV indicatie
>
> diff --git a/fnc_declare.h b/fnc_declare.h
> index acd686c..25d9a73 100644
> --- a/fnc_declare.h
> +++ b/fnc_declare.h
> @@ -136,6 +136,11 @@ eaf_N Indicaties    ($x1);
>  eaf_N pv_datum      ($x1, $x2, $x3);
>  #endif
>
> +#ifndef PV_INDICATIE
> +#define PV_INDICATIE
> +eaf_S pv_indicatie  ($x1);
> +#endif
> +
>  #undef eaf_B
>  #undef eaf_N
>  #undef eaf_S
> 4gl/new 123 >
>
> all the diffs are still showing as if they were against the folder they
> were originally committed in. Is that correct? I would now have expected
> something like

It is correct for old commits, when you look at an old commit it
doesn't know that later it is merge in a subfolder. For new commits it
depends on how you made them, in the modules and doing a subtree merge
or directly in the "superproject".

Santi


>
> commit d971ec91a3dbc9f1f27a96e4f9b95366babd036c
> Author: H.M. Brand <merijn@a5.(none)>
> Date:   Tue Jul 29 16:45:43 2008 +0200
>
>    Backward comp functie voor PV indicatie
>
> diff --git a/include/fnc_declare.h b/include/fnc_declare.h
> index acd686c..25d9a73 100644
> --- a/include/fnc_declare.h
> +++ b/include/fnc_declare.h
> @@ -136,6 +136,11 @@ eaf_N Indicaties    ($x1);
>  eaf_N pv_datum      ($x1, $x2, $x3);
>  #endif
>
> +#ifndef PV_INDICATIE
> +#define PV_INDICATIE
> +eaf_S pv_indicatie  ($x1);
> +#endif
> +
>  #undef eaf_B
>  #undef eaf_N
>  #undef eaf_S
>
> --
> H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
> using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
> 11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
> http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
> http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/
>

Re: Merging submodules

[H.Merijn Brand]

On Fri, 1 Aug 2008 11:52:08 +0200, "Santi Béjar" <sbejar@gmail.com>
wrote:

> On Fri, Aug 1, 2008 at 09:04, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
> >
> > Almost perfect now.
> >
> > 4gl/new 121 > git-ls-files | grep fnc_declare
> > include/fnc_declare.h
> > 4gl/new 122 > git-show d971ec91a3dbc9f1f27a96e4f9b95366babd036c
> > commit d971ec91a3dbc9f1f27a96e4f9b95366babd036c
> > Author: H.M. Brand <merijn@a5.(none)>
> > Date:   Tue Jul 29 16:45:43 2008 +0200
> >
> >    Backward comp functie voor PV indicatie
> >
> > diff --git a/fnc_declare.h b/fnc_declare.h
> > index acd686c..25d9a73 100644
> > --- a/fnc_declare.h
> > +++ b/fnc_declare.h
> > @@ -136,6 +136,11 @@ eaf_N Indicaties    ($x1);
> > :
> >
> > all the diffs are still showing as if they were against the folder they
> > were originally committed in. Is that correct? I would now have expected
> > something like
> 
> It is correct for old commits, when you look at an old commit it
> doesn't know that later it is merge in a subfolder. For new commits it
> depends on how you made them, in the modules and doing a subtree merge
> or directly in the "superproject".

After we join/merge these into the super-project, we're going to remove
the sup-repos, so all new commits will be made in the super-repo.

What I'm a bit worried about is that if people are looking for a change
in "include/inih001.h", they will not be able to find the commits on
that path, as the old commits have stored it without the path element.

Will that be a problem, or is git/gitk clever enough to still find
commits to both?

-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[Santi Béjar]

On Fri, Aug 1, 2008 at 12:35, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:

>
> After we join/merge these into the super-project, we're going to remove
> the sup-repos, so all new commits will be made in the super-repo.

If you'll remove the subrepos, the best thing would be to rewrite the
history on those subrepos just moving all the path to the
corresponding subfolder (with git-filter-branch, and you have exactly
what you need at the end of the example section in the manpage). And
then just do a normal merge. Or, even, you could try to create a
project with everything there in the correct order and location, I
don't know if git-filter-branch or git-fast-import/export (you have
some examples in git.git in contrib/fast-import) can do it, but if you
get it, please, post it here because it can be useful for others).

Santi

Re: Merging submodules

[H.Merijn Brand]

On Fri, 1 Aug 2008 13:34:42 +0200, "Santi Béjar" <sbejar@gmail.com>
wrote:

> On Fri, Aug 1, 2008 at 12:35, H.Merijn Brand <h.m.brand@xs4all.nl> wrote:
> 
> >
> > After we join/merge these into the super-project, we're going to remove
> > the sup-repos, so all new commits will be made in the super-repo.
> 
> If you'll remove the subrepos, the best thing would be to rewrite the
> history on those subrepos just moving all the path to the
> corresponding subfolder (with git-filter-branch, and you have exactly
> what you need at the end of the example section in the manpage). And
> then just do a normal merge. Or, even, you could try to create a
> project with everything there in the correct order and location, I
> don't know if git-filter-branch or git-fast-import/export (you have
> some examples in git.git in contrib/fast-import) can do it, but if you
> get it, please, post it here because it can be useful for others).

I gave up, even thought I'm not satisfied with the result: Merging with
subtrees works, but completely ruins my view over the history, cause
gitk shows the branches as a whole, and not `merged' by date. I did
change the merging process to merge the repo with the least recent
change date first, so that it would show up at the bottom and the
newest on top. This is workable, but far from perfect.

The fast-export/fast import with renaming started out fine, but
fast-import does not merge. For the latter I (tried to) use the
following approach:

--8<---
#!/pro/bin/perl

use strict;
use warnings;

sub usage
{
    my $err = shift and select STDERR;
    print "usage: $0 dir|repo.tar\n";
    @_ and print join "\n", @_, "";
    exit $err;
    } # usage

-d "new" and die "Dir new already exists\n";

use Cwd;
use File::Find;
use Getopt::Long qw(:config bundling nopermute);
my $opt_v = 1;
GetOptions (
    "help|?"    => sub { usage (0); },

    "v:2"       => \$opt_v,
    ) or usage (1);

@ARGV == 1 or usage (1);

my $tmp_archive = "/tmp/git-join-$$.tgz";
END { unlink $tmp_archive };

my $archive = shift;
if (-d $archive) {
    my $cwd = getcwd;
    my @dir;
    chdir $archive or die "$archive: $!\n";
    find (sub {
        $_ eq ".git" && -d $_ and push @dir, $File::Find::name;
        }, ".");
    qx{ tar czf $tmp_archive @dir };
    $archive = $tmp_archive;
    chdir $cwd;
    }

sub pr_time
{
    my @d = @_;
    sprintf "%4d-%02d-%02d %02d:%02d:%02d", 1900 + $d[5], ++$d[4], @d[3,2,1,0];
    } # pr_time

-f $archive && -s _ or usage (1, "Archive is not a file");

my @cmd =
    $archive =~ m/\.tar$/               ? qw( tar xf  )         :
    $archive =~ m/\.t(ar\.)?gz$/        ? qw( tar xzf )         :
    $archive =~ m/\.t(ar\.)?bz2?$/      ? qw( tar xhf )         :
    usage (1, "$archive is not a recognized archive type");

print STDERR "Creating merge environment\n";
mkdir "new", 0777;
chdir "new" or die "Canot use folder new\n";

print STDERR "Recovering original repo's\n";
system @cmd, $archive;

my %modules;
find (sub {
    (my $f = $File::Find::name) =~ s{^\./}{};
    $f =~ s{/\.git$}{};
    $_ eq ".git" && -d $_ && !$modules{$f} or return;
    print "Found $f\n";
    system "git-log '--pretty=format:%ct' | head -1";
    chomp ($modules{$f} = `git-log '--pretty=format:%ct' | head -1`);
    }, ".");
my @modules = sort { $modules{$a} <=> $modules{$b} } keys %modules;

sub git
{
    system "git", @_;
    } # git

my $top = getcwd;

print STDERR "Initializing new repo\n";
git "init";

foreach my $mod (@modules) {
    print STDERR "Merging ", pr_time (localtime $modules{$mod}), " $mod ...\n";
    chdir $mod;

    git "checkout", "-f";
    git "filter-branch", "--index-filter",
        qq{git ls-files -s | sed "s-\t-&$mod/-" | }.
         q{GIT_INDEX_FILE=$GIT_INDEX_FILE.new }.
         q{git update-index --index-info && }.
         q{mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE}, "HEAD";
    git "fast-export", "--all", ">/tmp/git-export-$$.bin";

    chdir $top;
    git "fast-import", "</tmp/git-export-$$.bin";
    }

print STDERR "Done\n";
-->8---

Which looks perfect after the first import, but fails on every next

Using --force for fast-import removes every previous import, so it is
useless for this process.

I do have to work with the repo, and that is more important than having
a perfect repo.

-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[Petr Baudis]

On Mon, Aug 04, 2008 at 03:24:43PM +0200, H.Merijn Brand wrote:
> I do have to work with the repo, and that is more important than having
> a perfect repo.

You might consider adopting a (relatively?) common strategy when
importing historical projects: Actually start the history from scratch
(git init && git add . && git commit -m"Initial commit") and fine-tune
your historical import in a separate repository. Then, provide a script
that people interested in the old history can run and it will graft the
imported history to your pure-git history.

Conceptually, it should be pretty simple:

	git fetch git://perl-company.nl/sccs-import.git
	echo initial_git_commit_sha1 last_imported_commit_sha1 \
		>>$(git rev-parse --git-dir)/info/grafts

Example of a fine-tuned script:

	http://repo.or.cz/w/elinks.git?a=blob;f=contrib/grafthistory.sh

If you find out that the import is not perfect later on, you can just
redo it, refetch and rewrite the info/grafts line.

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates

Re: Merging submodules

[H.Merijn Brand]

On Mon, 4 Aug 2008 15:40:53 +0200, Petr Baudis <pasky@suse.cz> wrote:

> On Mon, Aug 04, 2008 at 03:24:43PM +0200, H.Merijn Brand wrote:
> > I do have to work with the repo, and that is more important than having
> > a perfect repo.
> 
> You might consider adopting a (relatively?) common strategy when
> importing historical projects: Actually start the history from scratch
> (git init && git add . && git commit -m"Initial commit") and fine-tune
> your historical import in a separate repository. Then, provide a script
> that people interested in the old history can run and it will graft the
> imported history to your pure-git history.

We already changed the approach for converting projects from SCCS to
create a recursive repo from all SCCS repo's. And that *is* perfect.

Problem with the current join/merge is that there already have been
commits to the git repo after the SCCS convert that I do not want to
loose.

> Conceptually, it should be pretty simple:
> 
> 	git fetch git://perl-company.nl/sccs-import.git
> 	echo initial_git_commit_sha1 last_imported_commit_sha1 \
> 		>>$(git rev-parse --git-dir)/info/grafts
> 
> Example of a fine-tuned script:
> 
> 	http://repo.or.cz/w/elinks.git?a=blob;f=contrib/grafthistory.sh
> 
> If you find out that the import is not perfect later on, you can just
> redo it, refetch and rewrite the info/grafts line.

I'll have a look at this anyway.

-- 
H.Merijn Brand          Amsterdam Perl Mongers  http://amsterdam.pm.org/
using & porting perl 5.6.2, 5.8.x, 5.10.x, 5.11.x on HP-UX 10.20, 11.00,
11.11, 11.23, and 11.31, SuSE 10.1, 10.2, and 10.3, AIX 5.2, and Cygwin.
http://mirrors.develooper.com/hpux/           http://www.test-smoke.org/
http://qa.perl.org      http://www.goldmark.org/jeff/stupid-disclaimers/

Re: Merging submodules

[Petr Baudis]

On Mon, Aug 04, 2008 at 03:57:33PM +0200, H.Merijn Brand wrote:
> We already changed the approach for converting projects from SCCS to
> create a recursive repo from all SCCS repo's. And that *is* perfect.

Yes, but that's orthogonal to whether to separate the imported history
from the pure history? (Not that I'd want to force it upon you. If you
have a lot of nested repositories, the grafting can get a bit obnoxious
if you don't automate it well, I suppose.)

> Problem with the current join/merge is that there already have been
> commits to the git repo after the SCCS convert that I do not want to
> loose.

You could rebase these to the new initial commit. But if you already use
Git intensively and rewriting history would create big headache for you
at this point already, it's too late, I guess.

-- 
				Petr "Pasky" Baudis
The next generation of interesting software will be done
on the Macintosh, not the IBM PC.  -- Bill Gates