Best way to merge two repos with same content, different history

Best way to merge two repos with same content, different history

[Kelly F. Hickel]

Hi all,
	We're converting out of CVS after 10 years... The cvs2git
conversion takes around 4-5 days, and there doesn't seem to be any way
to speed that up.  So, our current plan is to take the tip of the
branches that we need for the next week, and import each of those
branches into its own clean git repo (top-skim).  Then we'll work out of
those repos while the conversion is in process (presumably creating
branches in those repos as needed).  Once the conversion is finished,
we'll need to get all the work done in the top-skimmed repos merged into
the converted repo so that we end up with little to no developer down
time, and all of our history, pre and post conversion in one repo.
	I'm testing all of this in advance, of course, and the tricky
part at the moment is how to "stitch" the commits from the top-skim
repos back onto the converted repo when the conversion is done.  The
file content of the initial commit for the skimmed repo is identical to
the last commit for the respective branch in the converted repo, but the
SHA1s are different, presumably because the history of the content is
different.

	Stated another way, I have two repositories, "new" and "old",
where the files in the initial commit on branch "B1" in "new" have
exactly the same content as the last commit on branch "B1" in "old".
There also exist various branches in "new" based on "B1".  I'd like to
merge all the commits from "new" into "old", but the SHA1s are
different, presumably because the history leading up to those points are
different.

	Other than using manually format-patch on every branch in new,
then applying the patches (presumably with regular old patch, since the
ancestor commit IDs won't match), is there any "good" way to merge "new"
into "old"?

Thanks,	


--

Kelly F. Hickel
Senior Product Architect
MQSoftware, Inc.
952-345-8677 Office
952-345-8721 Fax
kfh@mqsoftware.com
www.mqsoftware.com
Certified IBM SOA Specialty
Your Full Service Provider for IBM WebSphere
Learn more at www.mqsoftware.com 

Re: Best way to merge two repos with same content, different history

[Rostislav Svoboda]

On Fri, Jun 5, 2009 at 18:30, Kelly F. Hickel<kfh@mqsoftware.com> wrote:
>        We're converting out of CVS after 10 years... The cvs2git
> conversion takes around 4-5 days, and there doesn't seem to be any way
> to speed that up.

try google:
"cvs2git migration - cloning CVS repository"

Bost

RE: Best way to merge two repos with same content, different history

[Kelly F. Hickel]

> -----Original Message-----
> From: Rostislav Svoboda [mailto:rostislav.svoboda@gmail.com]
> Sent: Friday, June 05, 2009 11:54 AM
> To: Kelly F. Hickel
> Cc: git@vger.kernel.org
> Subject: Re: Best way to merge two repos with same content, different
> history
> 
> On Fri, Jun 5, 2009 at 18:30, Kelly F. Hickel<kfh@mqsoftware.com>
> wrote:
> >        We're converting out of CVS after 10 years... The cvs2git
> > conversion takes around 4-5 days, and there doesn't seem to be any
> way
> > to speed that up.
> 
> try google:
> "cvs2git migration - cloning CVS repository"
> 
> Bost

Bost, 
	Thanks, but I'm already working with a local copy of the CVS repo.  I've corresponded with Michael Haggerty about the time this takes, and there just doesn't seem to be any way to improve the speed, without making some fairly drastic changes to cvs2git.

Thanks,
Kelly

Re: Best way to merge two repos with same content, different history

[Rostislav Svoboda]

On Fri, Jun 5, 2009 at 19:10, Kelly F. Hickel<kfh@mqsoftware.com> wrote:
> Bost,
>        Thanks, but I'm already working with a local copy of the CVS repo.  I've corresponded with Michael Haggerty about the time this takes, and there just doesn't seem to be any way to improve the speed, without making some fairly drastic changes to cvs2git.

uhm, see below

Bost

On Tue, Feb 3, 2009 at 14:55, Johannes
Schindelin<Johannes.Schindelin@gmx.de> wrote:
>> If you do not have filesystem access to your CVS repository, you might
>> be able to clone it using CVSSuck [2,3].
>
> A substantially faster option would be to go with cvsclone:
>
>        http://samba.org/ftp/tridge/rtc/cvsclone.l
>
> (in my case, cvsclone was not only faster, but it actually worked, too,
> which is more than I could say of CVSSuck).

Re: Best way to merge two repos with same content, different history

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

On Fri, Jun 05, 2009 at 12:10:30PM -0500, Kelly F. Hickel wrote:
> Bost, 
> 	Thanks, but I'm already working with a local copy of the CVS repo.
> 	I've corresponded with Michael Haggerty about the time this takes,
> 	and there just doesn't seem to be any way to improve the speed,
> 	without making some fairly drastic changes to cvs2git.
I've been working with mhagger lately as it also pertains to the Gentoo
conversion. We've made some very good progress.

A couple of comments in that regard:
- Make really sure your box is not short of RAM. Throw some measurement
  tools onto there to see it. A couple of GiB is worthwhile. After we
  found this early on, and switched boxes, we dropped from our initial
  multiple days to 20 hours.
- His latest ExternalBlobGenerator code (_NOT_ available in SVN yet)
  reduced our pass1 time from 36204 seconds to 1598 seconds, with
  a potential to be much faster now, as parallelization of part of that
  is now trivial.
- pass9 is still the remaining large time-eater for us. I've started to
  look at it, but I haven't made any actual developments yet.

Would you mind posting your cvs2svn stats like these?
http://archives.gentoo.org/gentoo-scm/msg_b69b2f6ecee0ec7bb402d31b372b945b.xml

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

RE: Best way to merge two repos with same content, differenthistory

[Kelly F. Hickel]

> -----Original Message-----
> From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On
> Behalf Of Robin H. Johnson
> Sent: Friday, June 05, 2009 1:46 PM
> To: Git Mailing List
> Subject: Re: Best way to merge two repos with same content,
> differenthistory
> 
> On Fri, Jun 05, 2009 at 12:10:30PM -0500, Kelly F. Hickel wrote:
> > Bost,
> > 	Thanks, but I'm already working with a local copy of the CVS
> repo.
> > 	I've corresponded with Michael Haggerty about the time this
> takes,
> > 	and there just doesn't seem to be any way to improve the speed,
> > 	without making some fairly drastic changes to cvs2git.
> I've been working with mhagger lately as it also pertains to the
Gentoo
> conversion. We've made some very good progress.
> 
> A couple of comments in that regard:
> - Make really sure your box is not short of RAM. Throw some
measurement
>   tools onto there to see it. A couple of GiB is worthwhile. After we
>   found this early on, and switched boxes, we dropped from our initial
>   multiple days to 20 hours.
> - His latest ExternalBlobGenerator code (_NOT_ available in SVN yet)
>   reduced our pass1 time from 36204 seconds to 1598 seconds, with
>   a potential to be much faster now, as parallelization of part of
that
>   is now trivial.
> - pass9 is still the remaining large time-eater for us. I've started
to
>   look at it, but I haven't made any actual developments yet.
> 
> Would you mind posting your cvs2svn stats like these?
> http://archives.gentoo.org/gentoo-
> scm/msg_b69b2f6ecee0ec7bb402d31b372b945b.xml
> 
> --
> Robin Hugh Johnson
> Gentoo Linux Developer & Infra Guy
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Robin, 
	That's all good news, I have an 8 way box with 32gb of ram
running a 64 bit Linux, a box with 4 gb of ram panics during the
conversion.

My conversion data is below...

Thanks,
Kelly


cvs2svn Statistics:
------------------
Total CVS Files:             18488
Total CVS Revisions:        225208
Total CVS Branches:       15203751
Total CVS Tags:           39079236
Total Unique Tags:           11453
Total Unique Branches:        4364
CVS Repos Size in KB:      3355895
Total SVN Commits:           49967
First Revision Date:    Mon Nov  8 02:26:51 1999
Last Revision Date:     Wed Apr 22 17:59:44 2009
------------------
Timings (seconds):
------------------
251546   pass1    CollectRevsPass
     4   pass2    CleanMetadataPass
   142   pass3    CollateSymbolsPass
 53491   pass4    FilterSymbolsPass
     4   pass5    SortRevisionSummaryPass
   163   pass6    SortSymbolSummaryPass
  8825   pass7    InitializeChangesetsPass
   418   pass8    BreakRevisionChangesetCyclesPass
   418   pass9    RevisionTopologicalSortPass
  4256   pass10   BreakSymbolChangesetCyclesPass
  4914   pass11   BreakAllChangesetCyclesPass
  4575   pass12   TopologicalSortPass
  3111   pass13   CreateRevsPass
   270   pass14   SortSymbolsPass
   154   pass15   IndexSymbolsPass
  5517   pass16   OutputPass
337808   total
251783.89user 80800.42system 93:50:11elapsed 98%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (3major+3132264023minor)pagefaults 0swaps


git-fast-import statistics: 
---------------------------------------------------------------------
Alloc'd objects:     415000
Total objects:       410079 (   2628078 duplicates                  )
      blobs  :       152002 (     68715 duplicates     135125 deltas)
      trees  :       213636 (   2559363 duplicates     164052 deltas)
      commits:        44441 (         0 duplicates          0 deltas)
      tags   :            0 (         0 duplicates          0 deltas)
Total branches:       15822 (      6184 loads     )
      marks:     1073741824 (    265158 unique    )
      atoms:          14807
Memory total:         32402 KiB
       pools:         16192 KiB
     objects:         16210 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =     311525
pack_report: pack_mmap_calls          =      13303
pack_report: pack_open_windows        =          1 /          1
pack_report: pack_mapped              =  403041230 /  403041230
---------------------------------------------------------------------


git repack -a -d -f --depth=4000 --window=4000 && git pack-refs --all
Counting objects: 409458, done.5/119582)   
Compressing objects: 100% (119582/119582), done.
Writing objects: 100% (128713/128713), done.
Total 128713 (delta 82330), reused 0 (delta 0)
Compressing objects: 100% (384171/384171), done.
Writing objects: 100% (409458/409458), done.
Total 409458 (delta 309214), reused 0 (delta 0)

Re: Best way to merge two repos with same content, differenthistory

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1710 bytes --]

On Fri, Jun 05, 2009 at 02:06:25PM -0500, Kelly F. Hickel wrote:
> Robin, 
> 	That's all good news, I have an 8 way box with 32gb of ram
> running a 64 bit Linux, a box with 4 gb of ram panics during the
> conversion.
Thanks for your data.

For comparison, our conversion box is also 8-way, but only 16GiB RAM.

I'm surprised at how long pass1 is for you, especially since you've got
a lot less CVS Files and CVS Revisions than the Gentoo repo (I do deduce
that your individual revisions are larger, averaging at 15KiB vs. our
711 bytes).

I think there's something odd in the total CVS branches/tags count
however, as the counts there imply an average of 67 branches and 173
tags per CVS revision. You might want to dig into that part manually and
see about it (not sure of your Python skills). That would probably cut
down both your pass1 and pass4 times significantly.

Hopefully mhagger will get the external blob stuff committed soon, I was
working on validating it's results. 

In doing so discovered a testcase where RCSRevisionReader and
CVSRevisionReader gave different output themselves, the latter (which is
documented as more accurate otherwise) missing the contents of an entire
file. It's on the cvs2svn-dev mailing list now. Tracing that first,
thereafter comparing it to the new Git side.

> git repack -a -d -f --depth=4000 --window=4000 && git pack-refs --all
Did those extreme depth/window values actually help size much? The
Gentoo ones actually didn't improve significantly over depth=window=50.

-- 
Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 330 bytes --]

RE: Best way to merge two repos with same content, differenthistory

[Kelly F. Hickel]

> -----Original Message-----
> From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On
> Behalf Of Robin H. Johnson
> Sent: Friday, June 05, 2009 3:02 PM
> To: Git Mailing List
> Subject: Re: Best way to merge two repos with same content,
> differenthistory
> 
> On Fri, Jun 05, 2009 at 02:06:25PM -0500, Kelly F. Hickel wrote:
> > Robin,
> > 	That's all good news, I have an 8 way box with 32gb of ram
> running a
> > 64 bit Linux, a box with 4 gb of ram panics during the conversion.
> Thanks for your data.
> 
> For comparison, our conversion box is also 8-way, but only 16GiB RAM.
> 
> I'm surprised at how long pass1 is for you, especially since you've
got
> a lot less CVS Files and CVS Revisions than the Gentoo repo (I do
> deduce
> that your individual revisions are larger, averaging at 15KiB vs. our
> 711 bytes).
> 
> I think there's something odd in the total CVS branches/tags count
> however, as the counts there imply an average of 67 branches and 173
> tags per CVS revision. You might want to dig into that part manually
> and
> see about it (not sure of your Python skills). That would probably cut
> down both your pass1 and pass4 times significantly.

Robin, I'm not much with python, so haven't dug into the code much at
all. The numbers are high, although we do create a lot of branches (had
to contribute a fix a year or two to CVS to get the branching time down
from the 2.5 hours it was taking).  At one point I carefully examined
the symbol file that cvs2git was outputting and convinced myself that it
was doing the right thing, but that was awhile ago.

> 
> Hopefully mhagger will get the external blob stuff committed soon, I
> was
> working on validating it's results.
> 
> In doing so discovered a testcase where RCSRevisionReader and
> CVSRevisionReader gave different output themselves, the latter (which
> is
> documented as more accurate otherwise) missing the contents of an
> entire
> file. It's on the cvs2svn-dev mailing list now. Tracing that first,
> thereafter comparing it to the new Git side.
> 
> > git repack -a -d -f --depth=4000 --window=4000 && git pack-refs
--all
> Did those extreme depth/window values actually help size much? The
> Gentoo ones actually didn't improve significantly over
depth=window=50.

I know that they were still (apparently) improving after the 200 mark,
it took long enough at 200 that I just decided to crank the numbers way
up and let it run over the weekend.

> 
> --
> Robin Hugh Johnson
> Gentoo Linux Developer & Infra Guy
> E-Mail     : robbat2@gentoo.org
> GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

I'll be looking forward to a newer faster cvs2git, although I did just
get the graft idea working, so not sure if we'll wait that long or not
(would be nice not to have to muck around with it though).

Thanks,
Kelly

Re: Best way to merge two repos with same content, differenthistory

[Michael Haggerty]

Robin H. Johnson wrote:
> [Regarding cvs2git] In doing so discovered a testcase where RCSRevisionReader and
> CVSRevisionReader gave different output themselves, the latter (which is
> documented as more accurate otherwise) missing the contents of an entire
> file. It's on the cvs2svn-dev mailing list now. Tracing that first,
> thereafter comparing it to the new Git side.

In case anybody is following this, the issue that Robin found was in
release 2.2.0 of cvs2svn/cvs2git, but had already been fixed in the
current trunk version...

Conclusion 1: Please use the trunk version of cvs2git, not release
2.2.0.  Trunk is usually the most stable version available, especially
for 2git conversions.

Conclusion 2: I've really got to get around to making a new release :-/

Michael

Re: Best way to merge two repos with same content, different history

[Avery Pennarun]

On Fri, Jun 5, 2009 at 12:30 PM, Kelly F. Hickel <kfh@mqsoftware.com> wrote:
>        Stated another way, I have two repositories, "new" and "old",
> where the files in the initial commit on branch "B1" in "new" have
> exactly the same content as the last commit on branch "B1" in "old".
> There also exist various branches in "new" based on "B1".  I'd like to
> merge all the commits from "new" into "old", but the SHA1s are
> different, presumably because the history leading up to those points are
> different.
>
>        Other than using manually format-patch on every branch in new,
> then applying the patches (presumably with regular old patch, since the
> ancestor commit IDs won't match), is there any "good" way to merge "new"
> into "old"?

The usual replacement for "manually using format-patch" is to use "git
rebase."  It does pretty much exactly what you're describing, assuming
you don't do too many complicated merges in the meantime.

Another option is to use the .git/info/grafts file.  Here's a brief
intro: http://git.or.cz/gitwiki/GraftPoint

You'd use that to pretend the parent of your top-skimmed branch is
actually the equivalent commit in your new branch.  Then could run
"git filter-branch" to make the graft permanent, and get all your
users to switch to the new repository.

Or you could skip the filter-branch stuff and keep the really hold
history somewhere else, available for use if someone installs the
graft in their local repo.  This would lead to a smaller repository in
the general case.  (I gather that's what the Linux kernel does for
per-2.6.11 versions.)

Have fun,

Avery

RE: Best way to merge two repos with same content, different history

[Kelly F. Hickel]

> -----Original Message-----
> From: Avery Pennarun [mailto:apenwarr@gmail.com]
> Sent: Friday, June 05, 2009 12:01 PM
> To: Kelly F. Hickel
> Cc: git@vger.kernel.org
> Subject: Re: Best way to merge two repos with same content, different
> history
> 
> On Fri, Jun 5, 2009 at 12:30 PM, Kelly F. Hickel <kfh@mqsoftware.com>
> wrote:
> >        Stated another way, I have two repositories, "new" and "old",
> > where the files in the initial commit on branch "B1" in "new" have
> > exactly the same content as the last commit on branch "B1" in "old".
> > There also exist various branches in "new" based on "B1".  I'd like
> to
> > merge all the commits from "new" into "old", but the SHA1s are
> > different, presumably because the history leading up to those points
> are
> > different.
> >
> >        Other than using manually format-patch on every branch in new,
> > then applying the patches (presumably with regular old patch, since
> the
> > ancestor commit IDs won't match), is there any "good" way to merge
> "new"
> > into "old"?
> 
> The usual replacement for "manually using format-patch" is to use "git
> rebase."  It does pretty much exactly what you're describing, assuming
> you don't do too many complicated merges in the meantime.
> 
> Another option is to use the .git/info/grafts file.  Here's a brief
> intro: http://git.or.cz/gitwiki/GraftPoint
> 
> You'd use that to pretend the parent of your top-skimmed branch is
> actually the equivalent commit in your new branch.  Then could run
> "git filter-branch" to make the graft permanent, and get all your
> users to switch to the new repository.
> 
> Or you could skip the filter-branch stuff and keep the really hold
> history somewhere else, available for use if someone installs the
> graft in their local repo.  This would lead to a smaller repository in
> the general case.  (I gather that's what the Linux kernel does for
> per-2.6.11 versions.)
> 
> Have fun,
> 
> Avery

Thanks Avery,
	This appears to be just what I was looking for!  I'll fiddle with it a bit to see if I can convince it to work for me.


Thanks,
	Kelly

Re: Best way to merge two repos with same content, different history

[Markus Heidelberg]

Kelly F. Hickel, 05.06.2009:
> 	Other than using manually format-patch on every branch in new,
> then applying the patches (presumably with regular old patch, since the
> ancestor commit IDs won't match), is there any "good" way to merge "new"
> into "old"?

If rebasing 'new' on top of 'old' isn't an option, then you could try:

    $ git checkout new
    $ git merge -s ours old

It's the other way round (not merging 'new' into 'old', but vice versa),
but there is now merge strategy "theirs".

Markus