Can I Insert some Ancient History into a repo?

Can I Insert some Ancient History into a repo?

[Kerry, Richard]

 
The Story
- - - - - - - -
My project, "L", has two branches "L1" and "L2".  It was developed as L1 for a long period, then a branch, L2, was made and further development done on that.  Since the branch there has been further development on L1.
 
A-B-C-D-E-F-G-H
             \
              J-K-L-M
 
(That's supposed to show J branching off D.  I'm writing this with proportional spacing so it may look different on receipt)


The Reality
- - - - - - - - -
L was developed in CVS as its trunk.  That is what I am now referring to as L1.
A-B-C-D-E-F-G-H
 
At some point a snapshot was taken and put into Subversion.  That is what I am now referring to as L2.
It is either 
D-J-K-L-M
Or, more likely, just
J-K-L-M
(If some work was done on the snapshot before it was used to start the new project in SVN, which I think it was)
 
L2 has been converted to git using git's built-in Subversion handling.
I have a backup of the CVS repository for L1 and I can easily convert it to git using cvs2git.
 
What I would like to do is to merge these two branches into a single git repo.  Specifically, to insert L1 into the existing git repo which is currently only L2.

(The subversion server no longer exists, and I don't have easy access to the repository backup, and there may have been work done since the import.  I have a full backup of the CVS repository and I can easily convert it to git using cvs2git, with any extra parameters required.)

I've read up on merging repositories and I can easily get L1 into the same repo as L2, as a separate line of development.  However, merging it is not the right thing to do, as that leaves me with L1 and L2 having separate starting points, with a merge leading to there being only one head.  That's the opposite of what I want - I want to keep one starting point (that of L1), and branch L2 off it (with L2 ending up as master).

cvs2git works by generating fast-import scripts, so I can intercept the result of that and edit the import scripts if necessary.  I can even dump L2 and rebuild everything using fast-import if that's the best option.  Maybe L2 needs to be imported into L1 after generating a tag in L1 to indicate where the branch happens?  That seems to be a more complex work-flow for this task - I would hope to be able to do minor manipulations of branches and only one fast-import (ie L1).


So, I'm still looking to see if I can find any other way of wrangling branches to make this happen, but I'd appreciate anyone else's advice on how to achieve my merger.


Regards,
Richard.

Re: Can I Insert some Ancient History into a repo?

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 3117 bytes --]

Hi Richard,

On Wed, 2 Dec 2020, Kerry, Richard wrote:

> The Story
> - - - - - - - -
> My project, "L", has two branches "L1" and "L2".  It was developed as L1
> for a long period, then a branch, L2, was made and further development
> done on that.  Since the branch there has been further development on
> L1.
>  
> A-B-C-D-E-F-G-H
>              \
>               J-K-L-M
>  
> (That's supposed to show J branching off D.  I'm writing this with
> proportional spacing so it may look different on receipt)
>
>
> The Reality
> - - - - - - - - -
> L was developed in CVS as its trunk.  That is what I am now referring to as L1.
> A-B-C-D-E-F-G-H
>  
> At some point a snapshot was taken and put into Subversion.  That is what I am now referring to as L2.
> It is either
> D-J-K-L-M
> Or, more likely, just
> J-K-L-M
> (If some work was done on the snapshot before it was used to start the
> new project in SVN, which I think it was)
>  
> L2 has been converted to git using git's built-in Subversion handling.
> I have a backup of the CVS repository for L1 and I can easily convert it
> to git using cvs2git.
>  
> What I would like to do is to merge these two branches into a single git
> repo.  Specifically, to insert L1 into the existing git repo which is
> currently only L2.
>
> (The subversion server no longer exists, and I don't have easy access to
> the repository backup, and there may have been work done since the
> import.  I have a full backup of the CVS repository and I can easily
> convert it to git using cvs2git, with any extra parameters required.)
>
> I've read up on merging repositories and I can easily get L1 into the
> same repo as L2, as a separate line of development.  However, merging it
> is not the right thing to do, as that leaves me with L1 and L2 having
> separate starting points, with a merge leading to there being only one
> head.  That's the opposite of what I want - I want to keep one starting
> point (that of L1), and branch L2 off it (with L2 ending up as master).
>
> cvs2git works by generating fast-import scripts, so I can intercept the
> result of that and edit the import scripts if necessary.  I can even
> dump L2 and rebuild everything using fast-import if that's the best
> option.  Maybe L2 needs to be imported into L1 after generating a tag in
> L1 to indicate where the branch happens?  That seems to be a more
> complex work-flow for this task - I would hope to be able to do minor
> manipulations of branches and only one fast-import (ie L1).
>
>
> So, I'm still looking to see if I can find any other way of wrangling
> branches to make this happen, but I'd appreciate anyone else's advice on
> how to achieve my merger.

If I were you, I would use `git replace` to pretend that the `D` in your
`L2` history has the `C` in your `L1` history as parent:

	git replace --graft <D-as-in-L2> <C>

Note that this is a local-only operation. Even if you push, the remote
repository won't know that you grafted L2 onto L1 in your local worktree.

Ciao,
Johannes

Re: Can I Insert some Ancient History into a repo?

[Ævar Arnfjörð Bjarmason]

On Wed, Dec 02 2020, Kerry, Richard wrote:

> I've read up on merging repositories and I can easily get L1 into the
> same repo as L2, as a separate line of development.  However, merging
> it is not the right thing to do, as that leaves me with L1 and L2
> having separate starting points, with a merge leading to there being
> only one head.  That's the opposite of what I want - I want to keep
> one starting point (that of L1), and branch L2 off it (with L2 ending
> up as master).

I've brought in existing repos into other repos like what you describe
you shouldn't be doing, "git merge --allow-unrelated-histories" exists
for a reason, you can just use it.

I do think you're starting from the wrong starting point though, maybe
not in your thinking, but your post just says you want to combine these,
and how to do it.

The reason I've done this in the past is for some practical
reason. E.g. you might want to arrange it so a "git blame" or "git log"
on a file traverses down the right path & history, or to be able to
refer to commits by SHA1.

Depending on what you actually want to with the end result merging or
using grafts might not be the best thing to do. E.g. a perfectly OK
thing in some cases is to just have a side-repo somewhere to keep the
old history, you can add it as a remote to do some ad-hoc inspection,
but never need to push a merger of the two to the current active repo.

RE: Can I Insert some Ancient History into a repo?

[Kerry, Richard]

I've got back onto this issue after a couple of months on other matters,

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com> 
Sent: 02 December 2020 15:50

>On Wed, Dec 02 2020, Kerry, Richard wrote:
>
>> I've read up on merging repositories and I can easily get L1 into the 
>> same repo as L2, as a separate line of development.  However, merging 
>> it is not the right thing to do, as that leaves me with L1 and L2 
>> having separate starting points, with a merge leading to there being 
>> only one head.  That's the opposite of what I want - I want to keep 
>> one starting point (that of L1), and branch L2 off it (with L2 ending 
>> up as master).
>
>I've brought in existing repos into other repos like what you describe you shouldn't be doing, "git merge --allow-unrelated-histories" exists for a reason, you can >just use it.

I don't think I'm saying I shouldn't be doing it, just that doing it and leaving it as disjoint history doesn't seem like what I'm looking for for future clarity of history and for some work I need to do quite soon.
And I have discovered --allow-unrelated-histories, so I knew that was possible.

>I do think you're starting from the wrong starting point though, maybe not in your thinking, but your post just says you want to combine these, and how to do it.
>
>The reason I've done this in the past is for some practical reason. E.g. you might want to arrange it so a "git blame" or "git log"
>on a file traverses down the right path & history, or to be able to refer to commits by SHA1.

As you say, I should be doing it for a clearer reason, which I should have been clearer about when I first asked.  My reasons do include the reasons you give, so blame and log should than work, but more significant is that I want to merge some of the work done on the L2 branch into the L1 main line of development.

>Depending on what you actually want to with the end result merging or using grafts might not be the best thing to do. E.g. a perfectly OK thing in some cases is >to just have a side-repo somewhere to keep the old history, you can add it as a remote to do some ad-hoc inspection, but never need to push a merger of the >two to the current active repo.


From: Johannes Schindelin <Johannes.Schindelin@gmx.de> 
Sent: 02 December 2020 11:15

>If I were you, I would use `git replace` to pretend that the `D` in your `L2` history has the `C` in your `L1` history as parent:
>
>	git replace --graft <D-as-in-L2> <C>
>
>Note that this is a local-only operation. Even if you push, the remote repository won't know that you grafted L2 onto L1 in your local 
>worktree.

I don't think this would work for me as you say it is local-only so it won't get pushed to my remotes, which wouldn't be satisfactory.
Though I have to admit I don't really understand what it would be doing to git's structures, so maybe it would be useful.


Giving it some thought during the intervening time it seemed that to do this much reorganization of existing data I'd have to start from scratch and rebuild a repo importing the various exports, in whole or in part, in the right order and with right references edited in.  I've split the L1 export at the point where the L2 branch starts and generated master up to that point, then imported L2 so master is now A-B-C-D-J-K-L-M, as required.
Then I've taken the remainder of the L1 export and edited the export to set the base revision and the branch and imported the result, so L1 is now A-B-C-D-E-F-G-H.  That gives me a structure exactly as I want, with master being the same as the original L2 but with the appropriate additional ancient history, and L1 being as required.

However, I'm now not able to write the results back to BitBucket.  I've used "remote add" to add the reference to the existing remote repo where L2 came from.  But when I try "git push origin" it tells me :

fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use
    git push --set-upstream origin master

Why does it do that?
If I'm setting up a more conventional local repo to push to a remote it is usually sufficient to do "git remote add origin" and it'll then push (ie one set-up using a simple fast-import from cvs2git, with the instance on the remote server created there).
Presumably my current position of generating the local repo from scratch from multiple imports means it's missing something vital.  What might it be missing?  Is it just that it is aware that it did not originate from a fetch and the above set-upstream is the best way to fix it?  But other repos that originate from CVS exports are happy with just "remote add".
Does git know that the repo of the same name on the remote server is somehow "different" from my hand-crafted repo?


Regards,
Richard.

RE: Can I Insert some Ancient History into a repo?

[Kerry, Richard]

And...
Just after I sent that email I thought "is that right?  I'd better check", and no it isn't right.
So to answer my own question.....

From: Kerry, Richard
Sent: 12 March 2021 15:37

>
>However, I'm now not able to write the results back to BitBucket.  I've used "remote add" to add the reference to the existing remote >repo where L2 came from.  But when I try "git push origin" it tells me :
>
>fatal: The current branch master has no upstream branch.
>To push the current branch and set the remote as upstream, use
>    git push --set-upstream origin master
>
>Why does it do that?
>If I'm setting up a more conventional local repo to push to a remote it is usually sufficient to do "git remote add origin" and it'll then >push (ie one set-up using a simple fast-import from cvs2git, with the instance on the remote server created there).
>Presumably my current position of generating the local repo from scratch from multiple imports means it's missing something vital.  >What might it be missing?  Is it just that it is aware that it did not originate from a fetch and the above set-upstream is the best way to >fix it?  But other repos that originate from CVS exports are happy with just "remote add".
>Does git know that the repo of the same name on the remote server is somehow "different" from my hand-crafted repo?

I misremembered the details of adding a new item to BitBucket and in that case the required push command does include "-u", which is "--set-upstream".
So, sorry for the noise there.

So I've done that and now I'm getting, as possibly expected, a message from BB about it containing work that I don't have locally.  I'll get back onto that next week.  Again presumably a side-effect of building a repo from scratch that doesn't really share history with the remote one.

Regards,
Richard.