share object storage for multiple clones of different repositories

share object storage for multiple clones of different repositories

[Gelonida N]

Hi,

(Thunderbird frozewhile sending my previous message so here a resend)


I wondered whether it is possible, that all of my git repository clones
share the same object storage whether they are cloned from they same
remote repository or not.

In my case this might save a lot of diskspace and accelerate cloning
as some huge files are part of several repositories' history and as I'd
like to clone the same repository multiple times in order to have
parallel working directories for parallel tests on different versions /
branches / tags

Further this might reduce clone times.



My goal would be to:
- reduce disk space
- reduce clone time, as objects would be taken from the existing shared
  object storage

If possible it would be great if also all new created shared objects
would end up in the new object storage.


If git doesn't support this natively, then would following approach work
for reducing the disk space (clone time would not be reduced though)



SHARED_STORAGE=$HOME/shared_storage
mkdir $SHARED_STORAGE

git clone remotehost1:repo1
cd repo1
rsync -av .git/objects $SHARED_REPO
rm -rf .git/objects
ln -s $SHARED_REPO/objects .git/


git clone remotehost2:repo2
cd repo2
rsync -av .git/objects $SHARED_REPO
rm -rf .git/objects
ln -s $SHARED_REPO/objects .git/

Thanks for any feedback or other suggestions.

Re: share object storage for multiple clones of different repositories

[Junio C Hamano]

Gelonida N <gelonida@gmail.com> writes:

> SHARED_STORAGE=$HOME/shared_storage
> mkdir $SHARED_STORAGE
>
> git clone remotehost1:repo1
> cd repo1
> rsync -av .git/objects $SHARED_REPO

Up to this part it is probably OK.  Repeat that for all your local
repositories to collect all objects in $HOME/shared_storage.

After doing that, do this in all of your local repositories:

	rm -rf .git/objects
        mkdir -p .git/objects/info
        echo $HOME/shared/storage >.git/objects/info/alternates

The reason why nobody should follow your original recipe is because any
"git gc"/"git repack" in any of your local repositories would break others
with that approach.

Re: share object storage for multiple clones of different repositories

[Frans Klaver]

On Sat, 05 Nov 2011 03:26:11 +0100, Junio C Hamano <gitster@pobox.com>  
wrote:

> Gelonida N <gelonida@gmail.com> writes:
>
>> SHARED_STORAGE=$HOME/shared_storage
>> mkdir $SHARED_STORAGE
>>
>> git clone remotehost1:repo1
>> cd repo1
>> rsync -av .git/objects $SHARED_REPO
>
> Up to this part it is probably OK.  Repeat that for all your local
> repositories to collect all objects in $HOME/shared_storage.
>
> After doing that, do this in all of your local repositories:
>
> 	rm -rf .git/objects
>         mkdir -p .git/objects/info
>         echo $HOME/shared/storage >.git/objects/info/alternates
>
> The reason why nobody should follow your original recipe is because any
> "git gc"/"git repack" in any of your local repositories would break  
> others
> with that approach.


Alternatively there's the git-new-workdir script in contrib/workdir in  
git.git. Haven't tested it, but it seems like it does what you want.

Frans

Re: share object storage for multiple clones of different repositories

[Gelonida N]

Thanks for both of your replies,


I'll probably mix both approaches and try out how it works.
I didn't know about git.git
What is the best url for an intrudoction into git.git?




On 11/05/2011 08:37 AM, Frans Klaver wrote:
> On Sat, 05 Nov 2011 03:26:11 +0100, Junio C Hamano <gitster@pobox.com>
> wrote:
> 
>> Gelonida N <gelonida@gmail.com> writes:
>>
>>> SHARED_STORAGE=$HOME/shared_storage
>>> mkdir $SHARED_STORAGE
>>>
>>> git clone remotehost1:repo1
>>> cd repo1
>>> rsync -av .git/objects $SHARED_REPO
>>
>> Up to this part it is probably OK.  Repeat that for all your local
>> repositories to collect all objects in $HOME/shared_storage.
>>
>> After doing that, do this in all of your local repositories:
>>
>>     rm -rf .git/objects
>>         mkdir -p .git/objects/info
>>         echo $HOME/shared/storage >.git/objects/info/alternates
>>
>> The reason why nobody should follow your original recipe is because any
>> "git gc"/"git repack" in any of your local repositories would break
>> others
>> with that approach.
> 
> 
> Alternatively there's the git-new-workdir script in contrib/workdir in
> git.git. Haven't tested it, but it seems like it does what you want.
> 
> Frans

Re: share object storage for multiple clones of different repositories

[Gelonida N]

On 11/05/2011 05:06 PM, Gelonida N wrote:
> Thanks for both of your replies,
> 
> 
> I'll probably mix both approaches and try out how it works.
> I didn't know about git.git
> What is the best url for an intrudoction into git.git?
> 
> 
>>
>> Alternatively there's the git-new-workdir script in contrib/workdir in
>> git.git. Haven't tested it, but it seems like it does what you want.
>>
>> Frans
> 
> 

In an installed git (at least on Ubuntu) the contrib directory can be
found at /usr/share/doc/git/contrib