git-p4: Clone p4 path with bidirectional integrations

git-p4: Clone p4 path with bidirectional integrations

[Aaron Miller]

Hi all,

Is it possible to `git p4 clone --detect-branches` from a Perforce
path which contains bidirectional integrations?

I've tried a bunch of things to get this to work, but here's an
example which hopefully illustrates what I'm trying to accomplish
and the issue I'm having.

Perforce setup, assuming PWD is mapped to //depot/... in your client spec:

  1. mkdir -p testing/master
  2. touch testing/master/test1 && p4 add testing/master/test1 && p4 submit
  3. p4 integrate //depot/testing/master/...
//depot/testing/staging/... && p4 submit
  3. touch testing/staging/test2 && p4 add testing/staging/test2 && p4 submit
  4. p4 integrate //depot/testing/staging/...
//depot/testing/master/... && p4 submit

Now try to clone with git-p4:

  1. git init p4_git_test && cd p4_git_test
  2. git config git-p4.branchList master:staging
  3. git config --add git-p4.branchList staging:master
  4. git p4 clone //depot/testing/...@all --detect-branches .

You end up with a failure like:

  Importing from //depot/testing/...@all into .
  Reinitialized existing Git repository in /home/amiller/p4_git_test/.git/
  Importing revision 1205832 (25%)
      Importing new branch testing/master

      Resuming with change 1205832
  fatal: ambiguous argument 'refs/remotes/p4/testing/staging': unknown
revision or path not in the working tree.
  Use '--' to separate paths from revisions, like this:
  'git <command> [<revision>...] -- [<file>...]'
  Command failed: ['git', 'rev-list', '--reverse', '--no-merges',
'refs/remotes/p4/testing/staging']

I'm using Git 2.22.1.

Thanks,
Aaron

Re: git-p4: Clone p4 path with bidirectional integrations

[Luke Diamand]

On Mon, 19 Aug 2019 at 18:30, Aaron Miller <aaronkmiller@gmail.com> wrote:
>
> Hi all,
>
> Is it possible to `git p4 clone --detect-branches` from a Perforce
> path which contains bidirectional integrations?
>
> I've tried a bunch of things to get this to work, but here's an
> example which hopefully illustrates what I'm trying to accomplish
> and the issue I'm having.

I have to admit I don't use the detect-branches code myself.

It's possible that running with "-v" might give a bit more information.

Can you write a test case, or even just a shell script, that might
help figure out what's going on.

Unfortunately Perforce doesn't really know about branches, so git-p4
has to make a guess!

>
> Perforce setup, assuming PWD is mapped to //depot/... in your client spec:
>
>   1. mkdir -p testing/master
>   2. touch testing/master/test1 && p4 add testing/master/test1 && p4 submit
>   3. p4 integrate //depot/testing/master/...
> //depot/testing/staging/... && p4 submit
>   3. touch testing/staging/test2 && p4 add testing/staging/test2 && p4 submit
>   4. p4 integrate //depot/testing/staging/...
> //depot/testing/master/... && p4 submit
>
> Now try to clone with git-p4:
>
>   1. git init p4_git_test && cd p4_git_test
>   2. git config git-p4.branchList master:staging
>   3. git config --add git-p4.branchList staging:master
>   4. git p4 clone //depot/testing/...@all --detect-branches .
>
> You end up with a failure like:
>
>   Importing from //depot/testing/...@all into .
>   Reinitialized existing Git repository in /home/amiller/p4_git_test/.git/
>   Importing revision 1205832 (25%)
>       Importing new branch testing/master
>
>       Resuming with change 1205832
>   fatal: ambiguous argument 'refs/remotes/p4/testing/staging': unknown
> revision or path not in the working tree.
>   Use '--' to separate paths from revisions, like this:
>   'git <command> [<revision>...] -- [<file>...]'
>   Command failed: ['git', 'rev-list', '--reverse', '--no-merges',
> 'refs/remotes/p4/testing/staging']
>
> I'm using Git 2.22.1.
>
> Thanks,
> Aaron

Re: git-p4: Clone p4 path with bidirectional integrations

[Andrey]

19.08.2019, 13:30, "Aaron Miller" <aaronkmiller@gmail.com>:
> Hi all,
>
> Is it possible to `git p4 clone --detect-branches` from a Perforce
> path which contains bidirectional integrations?

Yes, but it would require some manual work most likely.

First of all, git-p4 should normally take only one direction from bidirectional integrations on its own.
Do you see "p4 branch <branchABC> defines a mapping from <path1> to <path2>, but there exists another mapping from <path2> to <path1> already!"?
If you do, it means that git-p4 will ignore <branchABC> mapping.

Also, just FYI, as far as I know, git-p4 doesn't create "merge" commits,
so bidirectional integrations won't look different from ordinary commits in git commit graph.

> I've tried a bunch of things to get this to work, but here's an
> example which hopefully illustrates what I'm trying to accomplish
> and the issue I'm having.
>
> Perforce setup, assuming PWD is mapped to //depot/... in your client spec:
>
>   1. mkdir -p testing/master
>   2. touch testing/master/test1 && p4 add testing/master/test1 && p4 submit
>   3. p4 integrate //depot/testing/master/...
> //depot/testing/staging/... && p4 submit
>   3. touch testing/staging/test2 && p4 add testing/staging/test2 && p4 submit
>   4. p4 integrate //depot/testing/staging/...
> //depot/testing/master/... && p4 submit
>
> Now try to clone with git-p4:
>
>   1. git init p4_git_test && cd p4_git_test
>   2. git config git-p4.branchList master:staging
>   3. git config --add git-p4.branchList staging:master
>   4. git p4 clone //depot/testing/...@all --detect-branches .
>
> You end up with a failure like:
>
>   Importing from //depot/testing/...@all into .
>   Reinitialized existing Git repository in /home/amiller/p4_git_test/.git/
>   Importing revision 1205832 (25%)

Uh-oh, 5M commits!
But given that it fails at 25% instead of 1%,
you've got some luck. :)

>       Importing new branch testing/master
>
>       Resuming with change 1205832
>   fatal: ambiguous argument 'refs/remotes/p4/testing/staging': unknown
> revision or path not in the working tree.
>   Use '--' to separate paths from revisions, like this:
>   'git <command> [<revision>...] -- [<file>...]'
>   Command failed: ['git', 'rev-list', '--reverse', '--no-merges',
> 'refs/remotes/p4/testing/staging']

This might not be just because of bidirectional integrations per se.
This error may happen if, say, there's a P4 branch mapping from staging to master,
but master was actually created before staging.
git-p4 tries to find a "parent branch" for master, but it doesn't exist yet,
so git-p4 fails in an ugly way.


One way to filter out troublesome P4 branch mappings is to set git-p4.branchUser to a particular user.
But most likely, this won't help you because different people created different branch mappings over time.

Unfortunately, there's _no_ git-p4.branchRegexp config option,
but it's fairly straightforward to implement -- patches welcome! ;)
(getBranchMapping() needs to apply a regex to branch names before doing anything serious with them)


The other option is to manually set git-p4.branchList for all your branch pairs like
git config --add git-p4.branchList staging:master
git config --add git-p4.branchList master:branchA
git config --add git-p4.branchList master:branchB
...
(or by manually editing .git/config)
Note, that you can't have master:staging together with staging:master,
otherwise you'll likely run into the same problem as before.

This might be simple or quite tedious depending on the history and branching strategies of your repositories.
It may be as easy as just dropping some p4 branch mappings.
However, one of the repositories I had to deal with had almost random branching strategy (with most integrations done without predefined branch mappings),
so I had to spend quite some time to trace the history and figure out which branches make the most sense in git.
(Revision Graph in p4v was very helpful for figuring branching history out)


As I said, git-p4 doesn't create merge commits in git (or I can't see how to make them),
so for repositories with simple/short history,
I recreated those merge commits manually (well, in a bash script) using `git replace --graft <commit> <parent1> <parent2>` followed by `git filter-branch --tag-name-filter cat -- --all` to make grafts permanent.
(`git filter-branch` is only needed once in the very end after all `git replace` manipulations are done)
It's perhaps better to teach git-p4 to produce merge commits, but a bash script was a low-tech low-risk option for me.

Also, beware that git-p4 doesn't handle branch-into-non-empty directory properly.
If I remember correctly, something like
`p4 copy //depot/branchA/... //depot/branchB/... ; p4 submit; p4 copy //depot/branchC/... //depot/branchB/...; p4 submit`
will result in branchB having _both_ branchA and branchC contents in git.
`git filter-branch` or `git rebase` are your friends to workaround this.
(or better fix git-p4, of course)

> I'm using Git 2.22.1.
>
> Thanks,
> Aaron

Hope this help,
Andrey.

Re: git-p4: Clone p4 path with bidirectional integrations

[Aaron Miller]

Hi Luke,

> It's possible that running with "-v" might give a bit more information.

Here's the output from that. I've set git-p4.branchUser in this test
to avoid needlessly cluttering the output since I have a huge amount
of branches in my Perforce repo, but otherwise I used the exact script
which I've included later in this email:

Importing from //depot/testing/...@all into .
Reinitialized existing Git repository in
/home/amiller/Code/git-migration/repos/testing/.git/
Reading pipe: ['git', 'config', '--bool', 'git-p4.useclientspec']
Reading pipe: ['git', 'config', 'git-p4.branchUser']
Reading pipe: ['git', 'config', 'git-p4.user']
Reading pipe: ['git', 'config', 'git-p4.password']
Reading pipe: ['git', 'config', 'git-p4.port']
Reading pipe: ['git', 'config', 'git-p4.host']
Reading pipe: ['git', 'config', 'git-p4.client']
Reading pipe: ['git', 'config', '--int', 'git-p4.retries']
Reading pipe: ['git', 'config', '--int', 'git-p4.retries']
Opening pipe: ['p4', '-r', '3', '-G', 'login', '-s']
Opening pipe: p4 -r 3 -G branches -u amiller
Reading pipe: ['git', 'config', '--get-all', 'git-p4.branchList']
p4-git branches: []
initial parents: {}
Getting p4 changes for //depot/testing/...
Opening pipe: ['p4', '-r', '3', '-G', 'changes', '-m', '1']
Opening pipe: ['p4', '-r', '3', '-G', 'changes',
'//depot/testing/...@1,1048577']
Opening pipe: ['p4', '-r', '3', '-G', 'changes',
'//depot/testing/...@1048578,1206544']
Opening pipe: ['p4', '-r', '3', '-G', 'describe', '-s', '1206099']
Importing revision 1206099 (25%)Reading pipe: ['git', 'config',
'--bool', 'core.ignorecase']
branch is master

    Importing new branch testing/master
Opening pipe: ['p4', '-r', '3', '-G', 'changes',
'//depot/testing/master/...@1,1048577']
Opening pipe: ['p4', '-r', '3', '-G', 'changes',
'//depot/testing/master/...@1048578,1206098']

    Resuming with change 1206099
parent determined through known branches: staging
looking for initial parent for refs/remotes/p4/testing/master; current
parent is refs/remotes/p4/testing/staging
Creating temporary branch: refs/git-p4-tmp/1206099
commit into refs/git-p4-tmp/1206099
Reading pipe: ['git', 'config', '--bool', 'git-p4.keepEmptyCommits']
Opening pipe: ['p4', '-r', '3', '-G', '-x', '-', 'print']
//depot/testing/master/test1 --> test1 (0 MB)
checkpoint finished: progress checkpoint

Reading pipe: ['git', 'rev-list', '--reverse', '--no-merges',
'refs/remotes/p4/testing/staging']
fatal: ambiguous argument 'refs/remotes/p4/testing/staging': unknown
revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
  File "/home/amiller/.bin/git-p4.py", line 4173, in <module>
    main()
  File "/home/amiller/.bin/git-p4.py", line 4167, in main
    if not cmd.run(args):
  File "/home/amiller/.bin/git-p4.py", line 3923, in run
    if not P4Sync.run(self, depotPaths):
  File "/home/amiller/.bin/git-p4.py", line 3790, in run
    self.importChanges(changes)
  File "/home/amiller/.bin/git-p4.py", line 3451, in importChanges
    blob = self.searchParent(parent, branch, tempBranch)
  File "/home/amiller/.bin/git-p4.py", line 3374, in searchParent
    "--no-merges", parent]):
  File "/home/amiller/.bin/git-p4.py", line 237, in read_pipe_lines
    die('Command failed: %s' % str(c))
  File "/home/amiller/.bin/git-p4.py", line 165, in die
    raise Exception(msg)
Exception: Command failed: ['git', 'rev-list', '--reverse',
'--no-merges', 'refs/remotes/p4/testing/staging']


> Can you write a test case, or even just a shell script, that might
> help figure out what's going on.

No problem:

#!/bin/bash

# perforce setup - assumes PWD is mapped to //depot/...
mkdir -p testing/master
touch testing/master/test1
p4 add testing/master/test1
p4 submit -d 'test changelist 1'

p4 integrate //depot/testing/master/... //depot/testing/staging/...
p4 submit -d 'test changelist 2'

touch testing/staging/test2
p4 add testing/staging/test2
p4 submit -d 'test changelist 3'

p4 integrate //depot/testing/staging/... //depot/testing/master/...
p4 submit -d 'test changelist 4'

# clone with git-p4:
git init p4_git_test
cd p4_git_test
git config git-p4.branchList master:staging
git config --add git-p4.branchList staging:master
git p4 clone //depot/testing/...@all --detect-branches --verbose .


Thanks,
Aaron

Re: git-p4: Clone p4 path with bidirectional integrations

[Aaron Miller]

Hi Andrey,

Thanks so much for this detailed response, I really appreciate it.

> First of all, git-p4 should normally take only one direction from bidirectional integrations on its own.
> Do you see "p4 branch <branchABC> defines a mapping from <path1> to <path2>, but there exists another mapping from <path2> to <path1> already!"?
> If you do, it means that git-p4 will ignore <branchABC> mapping.

I was afraid that was the case!

> Also, just FYI, as far as I know, git-p4 doesn't create "merge" commits,
> so bidirectional integrations won't look different from ordinary commits in git commit graph.

Ah, I didn't realize that, thank you. Perhaps I should just sync each
branch separately then, ignoring branch mappings entirely and be done
with it.

I was hoping to generate a commit graph that properly represents
integrations as merge commits because our Perforce branches are quite
large. Storing diffs of integrations rather than discrete commits
would result in a much smaller Git repository for us. There are
*quite* a lot of integration commits.

I wonder how hard it would be to modify git-p4 to use merge commits? I
will take a look at the source. I'm somewhat surprised that's not the
case to begin with though? Maybe someone else can chime in on why
merge commits aren't used.

> Uh-oh, 5M commits!
> But given that it fails at 25% instead of 1%,
> you've got some luck. :)

Hehe :) Fortunately I don't have to migrate *all* of those commits,
but still a fair number.

> This might not be just because of bidirectional integrations per se.

I should have mentioned - there were no P4 branch mappings defined for
the depot paths in the test case I shared (maybe that's obvious).

> This error may happen if, say, there's a P4 branch mapping from staging to master,
> but master was actually created before staging.
> git-p4 tries to find a "parent branch" for master, but it doesn't exist yet,
> so git-p4 fails in an ugly way.

Yeah this is exactly correct, I've tested precisely this scenario.
Actually that's what two of the real branches I am trying to migrate
look like.

> One way to filter out troublesome P4 branch mappings is to set git-p4.branchUser to a particular user.
> But most likely, this won't help you because different people created different branch mappings over time.

I was thinking I could set git-p4.branchUser to a user that hasn't
created any branches at all and then define branch mappings I care
about in git-p4.branchList.

But pretty much all of our branches either have bidirectional
integrations or a situation like you described above where branchA was
created before branchB, and branchB integrates into branchA.

> Unfortunately, there's _no_ git-p4.branchRegexp config option,
> but it's fairly straightforward to implement -- patches welcome! ;)
> (getBranchMapping() needs to apply a regex to branch names before doing anything serious with them)

That would be a neat option! It's not really filtering the branch
mappings which is my issue though. The number of branches is small
enough that I can manage the mappings manually.

> Note, that you can't have master:staging together with staging:master,
> otherwise you'll likely run into the same problem as before.

Yeah this is really the crux of my issue.

> This might be simple or quite tedious depending on the history and branching strategies of your repositories.
> It may be as easy as just dropping some p4 branch mappings.

I'm thinking if I'm not getting merge commits anyway I may as well
just forget about the branch mappings entirely.

> However, one of the repositories I had to deal with had almost random branching strategy (with most integrations done without predefined branch mappings), so I had to spend quite some time to trace the history and figure out which branches make the most sense in git.

Fortunately our branching strategy was *mostly* well-defined and we do
have Perforce mappings defined between our branches. However most
integrations didn't actually use the branch mappings, just manually
specified depot paths. But they did fall under the branch mapping
scopes.

> so for repositories with simple/short history,

Unfortunately our repo history is anything but short or simple :(

> I recreated those merge commits manually (well, in a bash script) using `git replace --graft <commit> <parent1> <parent2>` followed by `git filter-branch --tag-name-filter cat -- --all` to make grafts permanent.
>
> (`git filter-branch` is only needed once in the very end after all `git replace` manipulations are done)
> It's perhaps better to teach git-p4 to produce merge commits, but a bash script was a low-tech low-risk option for me.
>
> Also, beware that git-p4 doesn't handle branch-into-non-empty directory properly.
> If I remember correctly, something like
> `p4 copy //depot/branchA/... //depot/branchB/... ; p4 submit; p4 copy //depot/branchC/... //depot/branchB/...; p4 submit`
> will result in branchB having _both_ branchA and branchC contents in git.
> `git filter-branch` or `git rebase` are your friends to workaround this.
> (or better fix git-p4, of course)

These are great tips, thanks! Maybe what I will do is sync each branch
separately with no branch mappings, then use this technique to create
merge commits for the initial branch creation commits only and not
worry about any other integrations.

-Aaron

Re: git-p4: Clone p4 path with bidirectional integrations

[Andrey]

Aaron,

20.08.2019, 19:46, "Aaron Miller" <aaronkmiller@gmail.com>:
>>  Also, just FYI, as far as I know, git-p4 doesn't create "merge" commits,
>>  so bidirectional integrations won't look different from ordinary commits in git commit graph.
>
> Ah, I didn't realize that, thank you. Perhaps I should just sync each
> branch separately then, ignoring branch mappings entirely and be done
> with it.

You could do it this way too.
But this way your branches will be completely independent.
However, with --detect-branches, you'll preserve the information on which branch was branched off at which point.
(it's useful for release branches or feature branches; but not really for main and testing branching model)

> I was hoping to generate a commit graph that properly represents
> integrations as merge commits because our Perforce branches are quite
> large. Storing diffs of integrations rather than discrete commits
> would result in a much smaller Git repository for us. There are
> *quite* a lot of integration commits.
Not sure if there'll be a big difference in terms of Git repository size.
Git is quite efficient with its delta compression and pack files.

> I wonder how hard it would be to modify git-p4 to use merge commits? I
> will take a look at the source.
It's somewhere around importChanges() and commit(), I suppose.
It seems simple on git side (just specify additional parents when creating a commit using "merge" keyword for git fast-import),
but may be more complicated on Perforce side (to find the parent(s)).

> I'm somewhat surprised that's not the
> case to begin with though? Maybe someone else can chime in on why
> merge commits aren't used.
Yeah, it comes as a bad surprise.
One reason I can think of is bi-directional interaction between Perforce and Git -- perhaps some information is lost during submit to P4 and then reimport back to Git.
It might also get tricky during import of changelists that span several branches (shouldn't probably be considered a merge though).

>>  This might not be just because of bidirectional integrations per se.
>
> I should have mentioned - there were no P4 branch mappings defined for
> the depot paths in the test case I shared (maybe that's obvious).
Well, you had git-p4.branchList defined, which is enough for --detect-branches logic to kick in.

>>  One way to filter out troublesome P4 branch mappings is to set git-p4.branchUser to a particular user.
>>  But most likely, this won't help you because different people created different branch mappings over time.
>
> I was thinking I could set git-p4.branchUser to a user that hasn't
> created any branches at all and then define branch mappings I care
> about in git-p4.branchList.
Yeah, it's a good idea. (I did the same)
As far as I remember, you can even use a non-existing user for git-p4.branchUser.

>>  I recreated those merge commits manually (well, in a bash script) using `git replace --graft <commit> <parent1> <parent2>` followed by `git filter-branch --tag-name-filter cat -- --all` to make grafts permanent.
>>
>>  (`git filter-branch` is only needed once in the very end after all `git replace` manipulations are done)
>>  It's perhaps better to teach git-p4 to produce merge commits, but a bash script was a low-tech low-risk option for me.
>>
>>  Also, beware that git-p4 doesn't handle branch-into-non-empty directory properly.
>>  If I remember correctly, something like
>>  `p4 copy //depot/branchA/... //depot/branchB/... ; p4 submit; p4 copy //depot/branchC/... //depot/branchB/...; p4 submit`
>>  will result in branchB having _both_ branchA and branchC contents in git.
>>  `git filter-branch` or `git rebase` are your friends to workaround this.
>>  (or better fix git-p4, of course)
Another suggestion: after import into git is done, it might be useful to cross-check tips of git branches against tips of corresponding Perforce branches.
(some files may exist in git when they are deleted from Perforce due to the aforementioned branch-into-non-empty directory issue)

> These are great tips, thanks! Maybe what I will do is sync each branch
> separately with no branch mappings, then use this technique to create
> merge commits for the initial branch creation commits only and not
> worry about any other integrations.
If commits for integrations have consistent commit messages,
you might be able to get merge commits using `git replace` cheaper than by modifying git-p4 itself (which can't rely on commit messages).


-- 
Andrey.