Pushing and pulling the result of `git replace` and objects/info/alternates

Pushing and pulling the result of `git replace` and objects/info/alternates

[Stephen Kelly]

Hello,

I have an 'integration repo' which contains other git repos as submodules.

One of the submodules is to be split in two to extract a library.

A common way of doing that is to use git-filter-branch. A disadvantage
of that is that it results in duplicated partial-history in the
extracted repo. So, git log shows the entire history, but there is not
one canonical sha which represents the history at that point. The
split repo will contain 'false history', and checking it out will not
be useful.

So, I want to avoid git-filter-branch.

I have tried out using `git replace --graft` and
.git/objects/info/alternates to 'refer to' the history in the origin
repo instead of 'duplicating' it. This is similar to how Qt5 repos
refer to Qt 4 history in a different repo.

Question 1) Is this a reasonable thing to do for this scenario?

Question 2) Is there a way to push the `git replace` result and the
`info/alternates` content so that clients cloning the 'integration
repo' do not have to do that 'manually' or with a 'setup-repo.sh'
script?

The sequence of commands below can be pasted into a tmp directory to
see the scenario in action.

Thanks!


mkdir calculator
cd calculator
mkdir mainui libcalc
echo "print \"hello\"" > mainui/app.py
echo "print \"hello\"" > libcalc/adder.py
echo "print \"hello\"" > libcalc/subtracter.py
git init
git add .
git commit -am "Initial commit"
git checkout `git rev-parse HEAD`

cd ..
mkdir appsuite
cd appsuite
git init
git submodule add ../calculator
git commit -m "Add calculator submodule"

# Add other submodules in the suite...

cd calculator

echo "print \"goodbye\"" > libcalc/subtracter.py
git add libcalc/subtracter.py
git commit -am "Fix bug in subtracter"

echo "print \"Hi\"" > libcalc/adder.py
git add libcalc/adder.py
git commit -am "Make adder more efficient"

echo "print \"Hello, world!\"" > mainui/app.py
git add mainui/app.py
git commit -am "Improve app"

echo "print \"hello, hello\"" > libcalc/multiplier.py
git add libcalc/multiplier.py
git commit -am "Add multiplier"

cd ..
git add calculator
git commit -m "Update calculator submodule"

mkdir compute
cd calculator
mv libcalc ../compute

extraction_sha=`git rev-parse HEAD`
git commit -am "Remove libcalc from calculator repo" -m "It is moved
to a new compute repo"
removal_sha=`git rev-parse HEAD`
git push

cd ../compute
git init
git add .
git commit -m "Create the compute repo." -m "This commit will not be
normally visible after the replace --graft below."

echo "This is the compute framework. It contains the libcalc library." > README
git add README
git commit -m "Initialize the compute repo." -m "This has been
extracted from calculator.git at $removal_sha"
git checkout `git rev-parse HEAD`

cd ..
mv compute ..
git submodule add ../compute

git add calculator compute
git commit -m "Split compute framework out of calculator repo."

cd compute
git log --oneline
# We don't see older history from the calculator repo

# Let's add alternates
echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates

# ... and graft onto the extraction commit
git replace --graft HEAD $extraction_sha

git log --oneline
# Great, now we see history from the calculator repo.

cd ../..
git clone appsuite appsuite-clone
cd appsuite-clone
git submodule update --init
cd compute
ls ../.git/modules/compute/objects/info
git log --oneline
# The replacement and alternatives did not get cloned ... :(

echo "../../calculator/objects" >
../.git/modules/compute/objects/info/alternates
git replace --graft HEAD $extraction_sha

# And now we see the history from the calculator repo. Great. But, it
required user action after the clone.

Re: Pushing and pulling the result of `git replace` and objects/info/alternates

[Christian Couder]

Hi,

On Fri, May 22, 2015 at 4:38 PM, Stephen Kelly <steveire@gmail.com> wrote:
> Hello,
>
> I have an 'integration repo' which contains other git repos as submodules.
>
> One of the submodules is to be split in two to extract a library.
>
> A common way of doing that is to use git-filter-branch. A disadvantage
> of that is that it results in duplicated partial-history in the
> extracted repo. So, git log shows the entire history, but there is not
> one canonical sha which represents the history at that point. The
> split repo will contain 'false history', and checking it out will not
> be useful.
>
> So, I want to avoid git-filter-branch.
>
> I have tried out using `git replace --graft` and
> .git/objects/info/alternates to 'refer to' the history in the origin
> repo instead of 'duplicating' it. This is similar to how Qt5 repos
> refer to Qt 4 history in a different repo.
>
> Question 1) Is this a reasonable thing to do for this scenario?

I think it should work without too much work, but see the answer to
the next question.

> Question 2) Is there a way to push the `git replace` result and the
> `info/alternates` content so that clients cloning the 'integration
> repo' do not have to do that 'manually' or with a 'setup-repo.sh'
> script?

In short no.

"git replace" creates replace refs in "refs/replace/". To fetch these
refs, people need to use either "git fetch
'refs/replace/*:refs/replace/*'" or add a "fetch =
refs/replace/*:refs/replace/*" line in their config.
For simplicity and security reasons fetching replace refs is not on by default.

Also changing the objects/info/alternates to make it point to another
repo cannot be done automatically when cloning.

> The sequence of commands below can be pasted into a tmp directory to
> see the scenario in action.
>
> Thanks!
>
>
> mkdir calculator
> cd calculator
> mkdir mainui libcalc
> echo "print \"hello\"" > mainui/app.py
> echo "print \"hello\"" > libcalc/adder.py
> echo "print \"hello\"" > libcalc/subtracter.py
> git init
> git add .
> git commit -am "Initial commit"
> git checkout `git rev-parse HEAD`
>
> cd ..
> mkdir appsuite
> cd appsuite
> git init
> git submodule add ../calculator
> git commit -m "Add calculator submodule"
>
> # Add other submodules in the suite...
>
> cd calculator
>
> echo "print \"goodbye\"" > libcalc/subtracter.py
> git add libcalc/subtracter.py
> git commit -am "Fix bug in subtracter"
>
> echo "print \"Hi\"" > libcalc/adder.py
> git add libcalc/adder.py
> git commit -am "Make adder more efficient"
>
> echo "print \"Hello, world!\"" > mainui/app.py
> git add mainui/app.py
> git commit -am "Improve app"
>
> echo "print \"hello, hello\"" > libcalc/multiplier.py
> git add libcalc/multiplier.py
> git commit -am "Add multiplier"
>
> cd ..
> git add calculator
> git commit -m "Update calculator submodule"
>
> mkdir compute
> cd calculator
> mv libcalc ../compute
>
> extraction_sha=`git rev-parse HEAD`
> git commit -am "Remove libcalc from calculator repo" -m "It is moved
> to a new compute repo"
> removal_sha=`git rev-parse HEAD`
> git push
>
> cd ../compute
> git init
> git add .
> git commit -m "Create the compute repo." -m "This commit will not be
> normally visible after the replace --graft below."
>
> echo "This is the compute framework. It contains the libcalc library." > README
> git add README
> git commit -m "Initialize the compute repo." -m "This has been
> extracted from calculator.git at $removal_sha"
> git checkout `git rev-parse HEAD`
>
> cd ..
> mv compute ..
> git submodule add ../compute
>
> git add calculator compute
> git commit -m "Split compute framework out of calculator repo."
>
> cd compute
> git log --oneline
> # We don't see older history from the calculator repo
>
> # Let's add alternates
> echo "../../calculator/objects" >
> ../.git/modules/compute/objects/info/alternates
>
> # ... and graft onto the extraction commit
> git replace --graft HEAD $extraction_sha
>
> git log --oneline
> # Great, now we see history from the calculator repo.
>
> cd ../..
> git clone appsuite appsuite-clone
> cd appsuite-clone
> git submodule update --init
> cd compute
> ls ../.git/modules/compute/objects/info
> git log --oneline
> # The replacement and alternatives did not get cloned ... :(
>
> echo "../../calculator/objects" >
> ../.git/modules/compute/objects/info/alternates
> git replace --graft HEAD $extraction_sha

Maybe use the following instead of the above line:

git fetch 'refs/replace/*:refs/replace/*'

> # And now we see the history from the calculator repo. Great. But, it
> required user action after the clone.

Yeah, but if the 2 above commands are in a script maybe it's
reasonable to ask the user to launch the script once after cloning.

Best,
Christian.

Re: Pushing and pulling the result of `git replace` and objects/info/alternates

[Stephen Kelly]

On 05/24/2015 07:28 AM, Christian Couder wrote:
> Hi,
>
> On Fri, May 22, 2015 at 4:38 PM, Stephen Kelly <steveire@gmail.com> wrote:
>> I have tried out using `git replace --graft` and
>> .git/objects/info/alternates to 'refer to' the history in the origin
>> repo instead of 'duplicating' it. This is similar to how Qt5 repos
>> refer to Qt 4 history in a different repo.
>>
>> Question 1) Is this a reasonable thing to do for this scenario?
> I think it should work without too much work, but see the answer to
> the next question.

Ok, thanks. The concern is that there is plenty of documentation for
git-filter-branch, but no documentation or porcelain for info/alternates
and little out on the internet about it or git replace and using them
together.

However, it seems to be a reasonable thing to do.

>> echo "../../calculator/objects" >
>> ../.git/modules/compute/objects/info/alternates
>> git replace --graft HEAD $extraction_sha
> Maybe use the following instead of the above line:
>
> git fetch 'refs/replace/*:refs/replace/*'

Thanks.

>> # And now we see the history from the calculator repo. Great. But, it
>> required user action after the clone.
> Yeah, but if the 2 above commands are in a script maybe it's
> reasonable to ask the user to launch the script once after cloning.

Would it be possible to do this in a hook in the 'integration repo'
which contains both submodules in the example I posted? Like a fetch
hook or something?

Thanks,

Steve.

Re: Pushing and pulling the result of `git replace` and objects/info/alternates

[Christian Couder]

On Mon, May 25, 2015 at 11:49 AM, Stephen Kelly <steveire@gmail.com> wrote:
> On 05/24/2015 07:28 AM, Christian Couder wrote:
>> Hi,
>>
>> On Fri, May 22, 2015 at 4:38 PM, Stephen Kelly <steveire@gmail.com> wrote:
>>> I have tried out using `git replace --graft` and
>>> .git/objects/info/alternates to 'refer to' the history in the origin
>>> repo instead of 'duplicating' it. This is similar to how Qt5 repos
>>> refer to Qt 4 history in a different repo.
>>>
>>> Question 1) Is this a reasonable thing to do for this scenario?
>> I think it should work without too much work, but see the answer to
>> the next question.
>
> Ok, thanks. The concern is that there is plenty of documentation for
> git-filter-branch, but no documentation or porcelain for info/alternates
> and little out on the internet about it or git replace and using them
> together.
>
> However, it seems to be a reasonable thing to do.

Yeah.

By the way it looks like I was wrong in my answer to your second question.

You might want to clone using the --reference option to automatically
set up .git/objects/info/alternates properly.

(There is an ongoing related thread on the list:

http://thread.gmane.org/gmane.comp.version-control.git/269537)

You might also be able to clone using an option like "--config
remote.origin.fetch = 'refs/replace/*:refs/replace/*'" to fetch the
replace ref when cloning.

So it looks like you might just need to clone with a few more options
than usual.

I haven't tested it so please tell me if it works :-)

>>> echo "../../calculator/objects" >
>>> ../.git/modules/compute/objects/info/alternates
>>> git replace --graft HEAD $extraction_sha
>> Maybe use the following instead of the above line:
>>
>> git fetch 'refs/replace/*:refs/replace/*'
>
> Thanks.
>
>>> # And now we see the history from the calculator repo. Great. But, it
>>> required user action after the clone.
>> Yeah, but if the 2 above commands are in a script maybe it's
>> reasonable to ask the user to launch the script once after cloning.
>
> Would it be possible to do this in a hook in the 'integration repo'
> which contains both submodules in the example I posted? Like a fetch
> hook or something?

It is possible to do whatever you want in a hook, but the question is
why would you do it in a hook as it looks like it needs to be done
only once?

Or maybe I am missing something?

Best,
Christian.

Re: Pushing and pulling the result of `git replace` and objects/info/alternates

[Stephen Kelly]

On Tue, May 26, 2015 at 12:39 PM, Christian Couder
<christian.couder@gmail.com> wrote:
> First it looks like you sent the email to me only, so I am replying to you only.
> If this was a mistake, feel free to post this email to the Git mailing list.

Thanks, sorry for the mis-post.

>> 1) How would Alice push the content to a remote host so that Bob would
>> get that automatically?
>
> I am not sure what you want exactly, but let me try to answer anyway.
>
> Let's suppose that the content is in another submodule, let's call it
> subA, and let's call subB the submodule that should reference content
> in subA.

Yes, that's the scenario in my script.

> If subA has been pushed on the remote host, then Bob can clone subA
> first, and then clone subB using the --reference option to point to
> the content of subA.

Ah. Here's some confusion maybe.

Alice pushes subA and subB *and* the supermodule. In my script, these
were named calculator, compute and appsuite. The supermodule is the
entry point that everyone uses.

Bob clones the supermodule, appsuite, and expects that to 'just work'
regarding history.

So, I want to somehow specify the --reference in the .gitmodules of
the appsuite supermodule. Then when Bob runs git submodule update
--init, the right thing will be done.

>
> Please note that I don't know much about git submodules, as I try not
> to use them myself,

Me too :), but needs must.

> so I am not sure there is a way to make them do
> exactly what you want. Maybe you should look at the threads about
> submodules on the Git mailing list, see who are the people involved
> and send an email on the list with those people in CC and a subject
> related to submodules and with your specific questions about
> submodules in the content.

Ok, thanks. I think the solution of running a script after initial
clone/update is probably the most suitable for now anyway without
getting deeper into git.

> For example I don't know if there is a way to tell that subA should be
> cloned before subB or something like that.

Right. A step of performing actions like this would need to be done
after all fetches are done I guess.

>> 2) Can git submodules be configured to use particular options when
>> cloning particular repos? I see no relevant options in the
>>
>>  https://www.kernel.org/pub/software/scm/git/docs/gitmodules.html
>>
>> page.
>
> I don't know. Maybe it's possible to add a
> "submodule.<name>.cloneOptions" option to specify them. Or maybe it's
> possible to use the "submodule.<name>.update" config option with a
> specific command (see "custom command" in
> http://git-scm.com/docs/git-submodule) to do it.

Yes, actually the 'custom command' section there might be useful... I
might try it.

>>> You might also be able to clone using an option like "--config
>>> remote.origin.fetch = 'refs/replace/*:refs/replace/*'" to fetch the
>>> replace ref when cloning.
>>
>> Cool, but I guess the same second question applies here about whether
>> a submodule can be configured to fetch like that when the user does a
>> update --init ?
>
> Yeah, the same question applies.
>
>> Otherwise, I'm not sure what you are suggesting.
>
> I am not suggesting anything else.
>
>>>>>> echo "../../calculator/objects" >
>>>>>> ../.git/modules/compute/objects/info/alternates
>>>>>> git replace --graft HEAD $extraction_sha
>>>>> Maybe use the following instead of the above line:
>>>>>
>>>>> git fetch 'refs/replace/*:refs/replace/*'
>>>>
>>>> Thanks.
>>>>
>>>>>> # And now we see the history from the calculator repo. Great. But, it
>>>>>> required user action after the clone.
>>>>> Yeah, but if the 2 above commands are in a script maybe it's
>>>>> reasonable to ask the user to launch the script once after cloning.
>>>>
>>>> Would it be possible to do this in a hook in the 'integration repo'
>>>> which contains both submodules in the example I posted? Like a fetch
>>>> hook or something?
>>>
>>> It is possible to do whatever you want in a hook, but the question is
>>> why would you do it in a hook as it looks like it needs to be done
>>> only once?
>>>
>>> Or maybe I am missing something?
>>
>> The goal is to make it transparent to users, so that no one needs to
>> remember to 'do something once', but just gets a working checkout by
>> cloning the submodule in the plain, normal, 'what you learn on day
>> one' way. That is,
>>
>> git clone git://some/remote/appsuite appsuite-clone
>> cd appsuite-clone
>> git submodule update --init
>> cd compute
>> ls ../.git/modules/compute/objects/info
>> git log --oneline
>>
>> should show the history despite the split.
>
> Yeah, it would be nice if that would work, but, I am not sure it can
> work like that right now.
>
> And using hooks doesn't change anything as you have to setup those hooks anyway.
>
>>> So it looks like you might just need to clone with a few more options
>>> than usual.
>>>
>>> I haven't tested it so please tell me if it works :-)
>>
>> I changed the last 20 or so lines with one of your suggestions. I put
>> the initial revision and the update on a gist:
>>
>>  https://gist.github.com/steveire/a57bc48a460e11284d81/revisions
>>
>> The script I posted is easy to modify if you want to try something
>> out. I would be happy if you would try it out and see if you can make
>> your suggestion work.
>
> I tried it and it looks like it works for me as it works for you.
>
> There is:
>
>> git fetch origin 'refs/replace/*:refs/replace/*'
>> # Don't seem to need this? Why?
>> # Does the push of the replace refs copy them to the remote repo?
>> # How do I find out?
>> # echo "../../calculator/objects" > ../.git/modules/compute/objects/info/alternates
>
> The above comments probably mean that you didn't expect that fetching
> replace refs would also fetch the git objects (commits, trees, blobs,
> ...) pointed to by the replace refs. But that's what fetching does
> with any king of ref (branches, tags, notes and replace refs).

Actually, what I didn't expect is that

 # Push the replacement to the remote submodule clone
 git push origin 'refs/replace/*'

would push a copy of the content reachable by the 'refs/replace/*',
totally bypassing what I did with info/objects/alternates.

I updated the gist again with some output which I think shows that
that is happening.

 https://gist.github.com/steveire/a57bc48a460e11284d81/revisions

> Yes, this means that you need the commented "echo ..." line above,
> only if you don't want objects to be duplicated. But what you should
> really do in this case, is first use the "echo ..." line, and then
> fetch the replace refs.

Because the issue occurs with the push, I don't think this will help.

> If I have time, I will have a look at using the
> submodule.<name>.update option to see if it can help.

Thanks for your help so far!

Steve.

Re: Pushing and pulling the result of `git replace` and objects/info/alternates

[Christian Couder]

On Tue, May 26, 2015 at 4:10 PM, Stephen Kelly <steveire@gmail.com> wrote:
> On Tue, May 26, 2015 at 12:39 PM, Christian Couder
> <christian.couder@gmail.com> wrote:
>
>>> 1) How would Alice push the content to a remote host so that Bob would
>>> get that automatically?
>>
>> I am not sure what you want exactly, but let me try to answer anyway.
>>
>> Let's suppose that the content is in another submodule, let's call it
>> subA, and let's call subB the submodule that should reference content
>> in subA.
>
> Yes, that's the scenario in my script.
>
>> If subA has been pushed on the remote host, then Bob can clone subA
>> first, and then clone subB using the --reference option to point to
>> the content of subA.
>
> Ah. Here's some confusion maybe.
>
> Alice pushes subA and subB *and* the supermodule. In my script, these
> were named calculator, compute and appsuite. The supermodule is the
> entry point that everyone uses.
>
> Bob clones the supermodule, appsuite, and expects that to 'just work'
> regarding history.
>
> So, I want to somehow specify the --reference in the .gitmodules of
> the appsuite supermodule. Then when Bob runs git submodule update
> --init, the right thing will be done.

Yeah I understand and I am trying to help you do something like that,
though I can only talk about some of the steps involved, and this may
or may not help you find a complete solution that is good enough for
your needs.

>> Please note that I don't know much about git submodules, as I try not
>> to use them myself,
>
> Me too :), but needs must.
>
>> so I am not sure there is a way to make them do
>> exactly what you want. Maybe you should look at the threads about
>> submodules on the Git mailing list, see who are the people involved
>> and send an email on the list with those people in CC and a subject
>> related to submodules and with your specific questions about
>> submodules in the content.
>
> Ok, thanks. I think the solution of running a script after initial
> clone/update is probably the most suitable for now anyway without
> getting deeper into git.

Yeah, the user might just run a script instead of "git submodule update --init".
This way it doesn't increase the number of steps that have to be performed.

>> For example I don't know if there is a way to tell that subA should be
>> cloned before subB or something like that.
>
> Right. A step of performing actions like this would need to be done
> after all fetches are done I guess.
>
>>> 2) Can git submodules be configured to use particular options when
>>> cloning particular repos? I see no relevant options in the
>>>
>>>  https://www.kernel.org/pub/software/scm/git/docs/gitmodules.html
>>>
>>> page.
>>
>> I don't know. Maybe it's possible to add a
>> "submodule.<name>.cloneOptions" option to specify them. Or maybe it's
>> possible to use the "submodule.<name>.update" config option with a
>> specific command (see "custom command" in
>> http://git-scm.com/docs/git-submodule) to do it.
>
> Yes, actually the 'custom command' section there might be useful... I
> might try it.

Great, tell us what you come up with.

>>>> So it looks like you might just need to clone with a few more options
>>>> than usual.
>>>>
>>>> I haven't tested it so please tell me if it works :-)
>>>
>>> I changed the last 20 or so lines with one of your suggestions. I put
>>> the initial revision and the update on a gist:
>>>
>>>  https://gist.github.com/steveire/a57bc48a460e11284d81/revisions
>>>
>>> The script I posted is easy to modify if you want to try something
>>> out. I would be happy if you would try it out and see if you can make
>>> your suggestion work.
>>
>> I tried it and it looks like it works for me as it works for you.
>>
>> There is:
>>
>>> git fetch origin 'refs/replace/*:refs/replace/*'
>>> # Don't seem to need this? Why?
>>> # Does the push of the replace refs copy them to the remote repo?
>>> # How do I find out?
>>> # echo "../../calculator/objects" > ../.git/modules/compute/objects/info/alternates
>>
>> The above comments probably mean that you didn't expect that fetching
>> replace refs would also fetch the git objects (commits, trees, blobs,
>> ...) pointed to by the replace refs. But that's what fetching does
>> with any king of ref (branches, tags, notes and replace refs).
>
> Actually, what I didn't expect is that
>
>  # Push the replacement to the remote submodule clone
>  git push origin 'refs/replace/*'
>
> would push a copy of the content reachable by the 'refs/replace/*',
> totally bypassing what I did with info/objects/alternates.

Well if you had also setup info/objects/alternates on the server, it
would have been used there.

When pushing refs, as well as when fetching refs, Git sends the
objects pointed to by the refs that are transfered, if those objects
are not already available, so that the result is in a consistent
state. So if you setup info/objects/alternates on the server before
pushing, the server will see the objects as already available in the
alternates repo, and they will not be sent to the server.

> I updated the gist again with some output which I think shows that
> that is happening.
>
>  https://gist.github.com/steveire/a57bc48a460e11284d81/revisions
>
>> Yes, this means that you need the commented "echo ..." line above,
>> only if you don't want objects to be duplicated. But what you should
>> really do in this case, is first use the "echo ..." line, and then
>> fetch the replace refs.
>
> Because the issue occurs with the push, I don't think this will help.

This will help on the client, as the client will already have the
objects when fetching.

Best,
Christian.