git with large files...

git with large files...

[Darek Bridges]

I use git for many things, but I am trying to work out the workflow to use git for deployment. I have a wordpress site that i need to track multiple branches of and it has many many resource files, images, video, etc... how can i streamline the performance. ive read about git-annex and also something about git-bigfiles being merged in but i dont know how any of that has progressed, and I am just trying to work out the best way to handle this large site.

Happy Cake Oven -->  -[__]-

Re: git with large files...

[Randal L. Schwartz]

>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:

Darek> I use git for many things, but I am trying to work out the
Darek> workflow to use git for deployment.

Don't.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.posterous.com/ for Smalltalk discussion

Re: git with large files...

[Martin Langhoff]

On Fri, Jul 20, 2012 at 6:54 PM, Randal L. Schwartz
<merlyn@stonehenge.com> wrote:
>>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:
>
> Darek> I use git for many things, but I am trying to work out the
> Darek> workflow to use git for deployment.
>
> Don't.

Heh. Best to keep in mind that it just doesn't work very well.
git-bigfiles, git-annex might help you, but look at the docs and
caveats carefully.

Perhaps use rsync, unison work better for you.



m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: git with large files...

[David Aguilar]

On Fri, Jul 20, 2012 at 4:28 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> On Fri, Jul 20, 2012 at 6:54 PM, Randal L. Schwartz
> <merlyn@stonehenge.com> wrote:
>>>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:
>>
>> Darek> I use git for many things, but I am trying to work out the
>> Darek> workflow to use git for deployment.
>>
>> Don't.
>
> Heh. Best to keep in mind that it just doesn't work very well.
> git-bigfiles, git-annex might help you, but look at the docs and
> caveats carefully.
>
> Perhaps use rsync, unison work better for you.

I'm not sure if it was the "big files" part that Randal was responding
to.  IIUC it was the "using git for deployment" part.

Packaging tools (Makefiles, .rpm, .deb, etc) are a better suited for
deploying software.
-- 
David

Re: git with large files...

[Elia Pinto]

Well, many folks use puppet in serverless configuration pushing the
manifest from a central git server via cron and applying locally the
configuration fetched. In this sense git IS used for deployement. And,
for a configuration management system as puppet this could be a
sensible thing to do - reduce load, scalability ecc.

Best regards

2012/7/21, David Aguilar <davvid@gmail.com>:
> On Fri, Jul 20, 2012 at 4:28 PM, Martin Langhoff
> <martin.langhoff@gmail.com> wrote:
>> On Fri, Jul 20, 2012 at 6:54 PM, Randal L. Schwartz
>> <merlyn@stonehenge.com> wrote:
>>>>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:
>>>
>>> Darek> I use git for many things, but I am trying to work out the
>>> Darek> workflow to use git for deployment.
>>>
>>> Don't.
>>
>> Heh. Best to keep in mind that it just doesn't work very well.
>> git-bigfiles, git-annex might help you, but look at the docs and
>> caveats carefully.
>>
>> Perhaps use rsync, unison work better for you.
>
> I'm not sure if it was the "big files" part that Randal was responding
> to.  IIUC it was the "using git for deployment" part.
>
> Packaging tools (Makefiles, .rpm, .deb, etc) are a better suited for
> deploying software.
> --
> David
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
Inviato dal mio dispositivo mobile

Re: git with large files...

[Martin Langhoff]

On Fri, Jul 20, 2012 at 11:47 PM, David Aguilar <davvid@gmail.com> wrote:
> I'm not sure if it was the "big files" part that Randal was responding
> to.  IIUC it was the "using git for deployment" part.
>
> Packaging tools (Makefiles, .rpm, .deb, etc) are a better suited for
> deploying software.

Fair enough. On that topic, I have to say that git, with suitable
scripts controlling it, has been good to me for deployment to
webservers (ie: synchronized deployment on server clusters). Limited
to interpreted languages (py, php, etc).

I am not afraid of Makefiles, rpms or debs but sometimes they are a
bit too much overhead.

cheers,



m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: git with large files...

[Martin Langhoff]

On Sat, Jul 21, 2012 at 3:11 AM, Elia Pinto <gitter.spiros@gmail.com> wrote:
> Well, many folks use puppet in serverless configuration pushing the
> manifest from a central git server via cron and applying locally the
> configuration fetched. In this sense git IS used for deployement. And,
> for a configuration management system as puppet this could be a
> sensible thing to do - reduce load, scalability ecc.

That's a great thing to know, and timely too. I am looking at Puppet,
and worried a bit about reported memory load on the server side (and
some rumours of memory footprint issues on the client side too).

Can you point me to more information & discussion?

thanks!



m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: git with large files...

[Elia Pinto]

Sure . There is a complete paragraph " creating decentralized puppet
architecture" in "puppet 2.7 cookbook". I am also sure to have read
the same topic on web on the site bitfieldconsulting.com  , not
casually : the book's author have his blog there. Sorry i have not the
complete url now - old smartphone, no wifi etc. But i hope this can
help you the  same.

Best regards

2012/7/21, Martin Langhoff <martin.langhoff@gmail.com>:
> On Sat, Jul 21, 2012 at 3:11 AM, Elia Pinto <gitter.spiros@gmail.com>
> wrote:
>> Well, many folks use puppet in serverless configuration pushing the
>> manifest from a central git server via cron and applying locally the
>> configuration fetched. In this sense git IS used for deployement. And,
>> for a configuration management system as puppet this could be a
>> sensible thing to do - reduce load, scalability ecc.
>
> That's a great thing to know, and timely too. I am looking at Puppet,
> and worried a bit about reported memory load on the server side (and
> some rumours of memory footprint issues on the client side too).
>
> Can you point me to more information & discussion?
>
> thanks!
>
>
>
> m
> --
>  martin.langhoff@gmail.com
>  martin@laptop.org -- Software Architect - OLPC
>  - ask interesting questions
>  - don't get distracted with shiny stuff  - working code first
>  - http://wiki.laptop.org/go/User:Martinlanghoff
>

-- 
Inviato dal mio dispositivo mobile

Re: git with large files...

[Nick]

On 21/07/12 15:42, Martin Langhoff wrote:
> On Sat, Jul 21, 2012 at 3:11 AM, Elia Pinto <gitter.spiros@gmail.com> wrote:
>> Well, many folks use puppet in serverless configuration pushing the
>> manifest from a central git server via cron and applying locally the
>> configuration fetched. In this sense git IS used for deployement. And,
>> for a configuration management system as puppet this could be a
>> sensible thing to do - reduce load, scalability ecc.
> 
> That's a great thing to know, and timely too. I am looking at Puppet,
> and worried a bit about reported memory load on the server side (and
> some rumours of memory footprint issues on the client side too).
> 
> Can you point me to more information & discussion?

We use "masterless" puppet, deployed using gitolite post-receive hooks.  The
most useful clues I found are here.

 http://groups.google.com/group/puppet-users/browse_thread/thread/ed9a4032b31bd8d4/e1a68aa8ea91305d

 http://semicomplete.com/presentations/puppet-at-loggly/puppet-at-loggly.pdf.html

 http://current.workingdirectory.net/posts/2011/puppet-without-masters/

 http://bitfieldconsulting.com/scaling-puppet-with-distributed-version-control

We had to join the dots ourselves. Works for us so far, but it's only about six
months old.  We don't have lots of servers or very exacting requirements, just a
inclination against the pull orthodoxy and the freedom to experiment. Can't
comment about memory footprint, except it's not been a problem for us.  (On the
other hand I am not that enamoured with Puppet's DSL design, I might prefer to
avoid it if I could.)


N

Re: git with large files...

[Elia Pinto]

I forgot to tell that using git for deploying puppet have also some
disadvantages : you cant'use some puppet feature as external node
classifiers or stored configurations. As always, there are tradeoff.
For more

http: //bitfieldconsulting.com/scaling-puppet-with-distributed-version-control
(the missing link)

cheers


2012/7/21, Elia Pinto <gitter.spiros@gmail.com>:
> Sure . There is a complete paragraph " creating decentralized puppet
> architecture" in "puppet 2.7 cookbook". I am also sure to have read
> the same topic on web on the site bitfieldconsulting.com  , not
> casually : the book's author have his blog there. Sorry i have not the
> complete url now - old smartphone, no wifi etc. But i hope this can
> help you the  same.
>
> Best regards
>
> 2012/7/21, Martin Langhoff <martin.langhoff@gmail.com>:
>> On Sat, Jul 21, 2012 at 3:11 AM, Elia Pinto <gitter.spiros@gmail.com>
>> wrote:
>>> Well, many folks use puppet in serverless configuration pushing the
>>> manifest from a central git server via cron and applying locally the
>>> configuration fetched. In this sense git IS used for deployement. And,
>>> for a configuration management system as puppet this could be a
>>> sensible thing to do - reduce load, scalability ecc.
>>
>> That's a great thing to know, and timely too. I am looking at Puppet,
>> and worried a bit about reported memory load on the server side (and
>> some rumours of memory footprint issues on the client side too).
>>
>> Can you point me to more information & discussion?
>>
>> thanks!
>>
>>
>>
>> m
>> --
>>  martin.langhoff@gmail.com
>>  martin@laptop.org -- Software Architect - OLPC
>>  - ask interesting questions
>>  - don't get distracted with shiny stuff  - working code first
>>  - http://wiki.laptop.org/go/User:Martinlanghoff
>>
>
> --
> Inviato dal mio dispositivo mobile
>

-- 
Inviato dal mio dispositivo mobile

Re: git with large files...

[Kalle Launiala]

While my example isn't about puppet, it's about another templating
technology - that is by technical design hence also completely open
source. We have a completely modulirized and completely distributed
deployment scenario that is based on git.

The fundamental core how git addresses the solution is explained here
- especially on the visualization.
http://abstractiondev.wordpress.com/git-based-distribution/

I'd like to add a point of view that will underline git behaviour as a
core requirement on distributed depoloyment - if there is any issue
with big files, that's a good concern to address.

While this can be well justified and argued to benefit for possibly
platform specific deployment tools such as rpm or msi modules, the
audit trail of distributed software (the core requirement for source
code andyway) still remains. If there is a "big file" that is part of
the audit process, it's consistency needs to be guaranteed. And there
is no way around it but the git way of doing complete secure hashed
history trail.

This is not "nice to have" feature, but critical requirement to deploy
embedded software for automated machinery to suffice safety
regulations. Hospital, aviation, nuclear device software all apply
here. Distributed software processes and projects all benefit from
this - and trying to "cut short" some part of the complete audit trail
of development will cause larger pain points elsewhere.

So is there real problem with big files performance or storage files?
Anything else than "uncommon" slowness on calculating and comparing
large file sha1 hashes?


Kalle

Re: git with large files...

[Junio C Hamano]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

>>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:
>
> Darek> I use git for many things, but I am trying to work out the
> Darek> workflow to use git for deployment.
>
> Don't.

Yeah, "don't think 'git checkout' is a way to 'deploy'".  Using Git
as a transport measure is probably fine.

Re: git with large files...

[Sitaram Chamarty]

On Mon, Jul 23, 2012 at 2:24 AM, Junio C Hamano <gitster@pobox.com> wrote:
> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>
>>>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:
>>
>> Darek> I use git for many things, but I am trying to work out the
>> Darek> workflow to use git for deployment.
>>
>> Don't.
>
> Yeah, "don't think 'git checkout' is a way to 'deploy'".  Using Git
> as a transport measure is probably fine.

You can also try
http://sitaramc.github.com/the-list-and-irc/deploy.html.  Whether it's
saying you *can* use git for deploying something, or you *can* but
*should not*, or you *cannot*, will depend on your own thoughts on the
matter ;-)

Re: git with large files...

[Kalle Launiala]

2012/7/23 Sitaram Chamarty <sitaramc@gmail.com>:
> On Mon, Jul 23, 2012 at 2:24 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>>
>>>>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:
>>>
>>> Darek> I use git for many things, but I am trying to work out the
>>> Darek> workflow to use git for deployment.
>>>
>>> Don't.
>>
>> Yeah, "don't think 'git checkout' is a way to 'deploy'".  Using Git
>> as a transport measure is probably fine.
>
> You can also try
> http://sitaramc.github.com/the-list-and-irc/deploy.html.  Whether it's
> saying you *can* use git for deploying something, or you *can* but
> *should not*, or you *cannot*, will depend on your own thoughts on the
> matter ;-)

Nice summary list of options!

If you combine that with several key concepts:
1. You plan and design to deploy - hence you have separate deploy
repositories dedicated for that
2. You design for modularity and complete audit trail, hence you have this:
http://abstractiondev.files.wordpress.com/2011/12/git-distribution-overview.png

You can combine the staging with any approach, that *tries* to
maintain the local version copy. But if any problem arises in
pull/fetch, simply trash that part and get it from fresh (or just use
the git archive approach).

Now this model would introduce complete and as detailed security
enforcement - as the deployment can validate with certificates (from
the additional catalogue-metadata binding, whether there is authorized
party confirmed signature available for the wished deployable
version).

I don't see much overhead in any of the steps here - and the
deployment is as detailed and as securely controlled as desired. With
just Git (and well, possibly GnuPG or alike common tool for digital
certificate work).

Everything is also transparent - which is very important in having
that complete control and audit trail.

Re: git with large files...

[Drew Northup]

On Mon, Jul 23, 2012 at 12:23 AM, Sitaram Chamarty <sitaramc@gmail.com> wrote:
> On Mon, Jul 23, 2012 at 2:24 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>>
>>>>>>>> "Darek" == Darek Bridges <darek.bridges@me.com> writes:
>>>
>>> Darek> I use git for many things, but I am trying to work out the
>>> Darek> workflow to use git for deployment.
>>>
>>> Don't.
>>
>> Yeah, "don't think 'git checkout' is a way to 'deploy'".  Using Git
>> as a transport measure is probably fine.
>
> You can also try
> http://sitaramc.github.com/the-list-and-irc/deploy.html.  Whether it's
> saying you *can* use git for deploying something, or you *can* but
> *should not*, or you *cannot*, will depend on your own thoughts on the
> matter ;-)

After a couple of false starts, I think that Sitaram came closest to
what Darek was asking about.

Now, as somebody who is using Git currently to stage things to
"deployment" (I may change to SVN due to office politics--which will
double my workload on rollout of updates, but I'm not saying any more
than that in public) on production web servers, I have a few comments.

We have several WordPress instances @$work where we are using Git to
stage template changes out to our development server (where I've
contemplated putting the lessons in Sitaram's article to use) before
merging those changes back into the "Production" branch (after
suitable testing) and pulling them from a central gitolite into the
live server. It works and it respects the posix extended ACLs on the
destination host (which is what you actually want on a live web
server). Even better, it provides a safe way of tracking and merging
back in any "opportunistic" changes that were made directly in the
development or production servers so that they are not lost.

Thought must be applied to do this safely, but that's the way it
usually is on web servers. To those who say admins should be using
RPM, DEB, or any other "generic package management" software to deploy
non-system updates to in-house web servers may I kindly indicate that
it often doesn't make sense to do so unless each and every website has
its own server and IP address--and you are deploying tens of thousands
of them. Most of us can't afford that. (Yes, there is an overhead to
building packages. I've done it enough times to know about that quite
intimately.)

Packages and package management are great for system software but they
are not a good solution for installing client code into a webspace on
a shared server (yes, heresy, I know). For this common use case Git is
not a half-bad ADDITION to the toolkit of a website development and
maintenance team.

-- 
-Drew Northup
--------------------------------------------------------------
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59