Re: cannot fetch arm git tree

Re: cannot fetch arm git tree

[Uwe Kleine-König]

Hello Jello,

On Sun, Jan 16, 2011 at 09:23:15AM +0000, Russell King - ARM Linux wrote:
> On Sun, Jan 16, 2011 at 10:28:55AM +0800, Jello huang wrote:
> > Dear Russell,
> > 
> > when i use git fetch,there arm some wrong with it.
> > 
> > jello@jello-laptop:~/git/russell/linux-2.6-arm$git pull
> > error: Unable to find 89e4d4b145bb7e73b4c45671a84b401a5d8694c1 under
> > http://ftp.arm.linux.org.uk/pub/armlinux/kernel/git-cur/linux-2.6-arm.git
> > Cannot obtain needed blob 89e4d4b145bb7e73b4c45671a84b401a5d8694c1
> > while processing commit eda2e5dcc914b4d70f665443efc9780e89a5e5c1.
> > error: Fetch failed.
> > 
> > What is the wrong?
> 
> No idea.  The tree has the object file in one of its pack files:
> 
> | rmk@ZenIV:[linux-2.6-arm.git]:<1020> GIT_DIR=. git cat-file -p 89e4d4b145bb7e73b4c45671a84b401a5d8694c1|head
> | 
> |         List of maintainers and how to submit kernel changes
> | 
> | Please try to follow the guidelines below.  This will make things
> | easier on the maintainers.  Not all of these guidelines matter for every
> | trivial patch so apply some common sense.
> 
> and the pack info file lists all the pack files.  Maybe you have a hash
> collision on a pack file with your repository?
Probably you have a corrupted pack file.  Try renaming all packfiles
listed in
http://ftp.arm.linux.org.uk/pub/armlinux/kernel/git-cur/linux-2.6-arm.git/objects/info/packs
(i.e.
.git/objects/pack/pack-74405a23171b6debd894d4791e06956d6387022a.pack
etc.) and try refetching then.  This happend to me after Ctrl-Cing an
earlier git-fetch.  Git doesn't seem to handle that case.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: cannot fetch arm git tree

[Russell King - ARM Linux]

On Sun, Jan 16, 2011 at 09:10:17PM +0800, Jello huang wrote:
> yes,git doesn't  handle that case and i rename the pack name,but there is
> also the similar error.Now i just delet the git tree and  clone it again
> tonight .

_Always_ without fail fetch Linus' tree before pulling my tree.

My tree is a rsync clone of the objects and pack files in Linus' tree,
plus whatever git decided to build on top of that - for local commits
that's individual object files.  For remote pulls, that's probably a few
small pack files.

There is *no* repacking of my tree.  So the only times it gets 'repacked'
is when Linus repacks his tree.

Let's say you already have a copy of my tree from a month ago, and Linus
has pulled some work from me into his tree, and repacked his tree into one
single pack file.  At the moment, the largest pack file from Linus is
400MB plus a 50MB index.

You already have most of the contents of that 400MB pack file, but if
you're missing even _one_ object which is contained within it, git will
have to download the _entire_ 400MB pack file and index file to retrieve
it.

However, if you first fetch Linus' tree via the git protocol, it can just
request the objects it doesn't have from the git server.  That will mean
you'll have all the objects in the large pack files before you start trying
to pull my tree, and git won't have to download 400MB for the sake of
retrieving just maybe 10k that you didn't have.

This isn't something special with my tree - it's a side effect of the
http protocol git uses.  So, before you fetch _any_ http-based git tree,
first make sure you're up to date with Linus'.

(I update my tree from Linus' in rsync mode to make http-based stuff a
lot more friendly to people using it - some of whom are stuck behind
firewalls which can only do http.  Fetching a constantly repacked git
tree via http results in hundreds of megabytes needing to be fetched
every time.)

So please, whenever possible, always fetch Linus' latest tree _first_
and then mine.  Same goes for any other http based tree which doesn't
auto-repack.

Re: cannot fetch arm git tree

[Jello huang]

[-- Attachment #1.1: Type: text/plain, Size: 2353 bytes --]

Russell,thanks for your reply so elaborate and i have gotten the idea now.

On 16 January 2011 21:42, Russell King - ARM Linux
<linux@arm.linux.org.uk>wrote:

> On Sun, Jan 16, 2011 at 09:10:17PM +0800, Jello huang wrote:
> > yes,git doesn't  handle that case and i rename the pack name,but there is
> > also the similar error.Now i just delet the git tree and  clone it again
> > tonight .
>
> _Always_ without fail fetch Linus' tree before pulling my tree.
>
> My tree is a rsync clone of the objects and pack files in Linus' tree,
> plus whatever git decided to build on top of that - for local commits
> that's individual object files.  For remote pulls, that's probably a few
> small pack files.
>
> There is *no* repacking of my tree.  So the only times it gets 'repacked'
> is when Linus repacks his tree.
>
> Let's say you already have a copy of my tree from a month ago, and Linus
> has pulled some work from me into his tree, and repacked his tree into one
> single pack file.  At the moment, the largest pack file from Linus is
> 400MB plus a 50MB index.
>
> You already have most of the contents of that 400MB pack file, but if
> you're missing even _one_ object which is contained within it, git will
> have to download the _entire_ 400MB pack file and index file to retrieve
> it.
>
> However, if you first fetch Linus' tree via the git protocol, it can just
> request the objects it doesn't have from the git server.  That will mean
> you'll have all the objects in the large pack files before you start trying
> to pull my tree, and git won't have to download 400MB for the sake of
> retrieving just maybe 10k that you didn't have.
>
> This isn't something special with my tree - it's a side effect of the
> http protocol git uses.  So, before you fetch _any_ http-based git tree,
> first make sure you're up to date with Linus'.
>
> (I update my tree from Linus' in rsync mode to make http-based stuff a
> lot more friendly to people using it - some of whom are stuck behind
> firewalls which can only do http.  Fetching a constantly repacked git
> tree via http results in hundreds of megabytes needing to be fetched
> every time.)
>
> So please, whenever possible, always fetch Linus' latest tree _first_
> and then mine.  Same goes for any other http based tree which doesn't
> auto-repack.
>



-- 
JUST DO IT,NOTHING IS IMPOSSIBLE

[-- Attachment #1.2: Type: text/html, Size: 2867 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: cannot fetch arm git tree

[Detlef Vollmann]

On 01/16/11 14:42, Russell King - ARM Linux wrote:
> Let's say you already have a copy of my tree from a month ago, and Linus
> has pulled some work from me into his tree, and repacked his tree into one
> single pack file.  At the moment, the largest pack file from Linus is
> 400MB plus a 50MB index.
>
> You already have most of the contents of that 400MB pack file, but if
> you're missing even _one_ object which is contained within it, git will
> have to download the _entire_ 400MB pack file and index file to retrieve
> it.
I thought this has changed with "smart http" in git 1.6.6.
Am I missing something?

   Detlef

Re: cannot fetch arm git tree

[Uwe Kleine-König]

Hi Detlef,

On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
> On 01/16/11 14:42, Russell King - ARM Linux wrote:
> >Let's say you already have a copy of my tree from a month ago, and Linus
> >has pulled some work from me into his tree, and repacked his tree into one
> >single pack file.  At the moment, the largest pack file from Linus is
> >400MB plus a 50MB index.
> >
> >You already have most of the contents of that 400MB pack file, but if
> >you're missing even _one_ object which is contained within it, git will
> >have to download the _entire_ 400MB pack file and index file to retrieve
> >it.
> I thought this has changed with "smart http" in git 1.6.6.
> Am I missing something?
Well, not all http repos offer smart http.  E.g. Russell doesn't[1],
probably because the serving machine doesn't have the power to nice
serve a repo via git:// or smart http://.

Best regards
Uwe

[1] I didn't recheck though

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: cannot fetch arm git tree

[Russell King - ARM Linux]

On Fri, Jan 21, 2011 at 02:47:28PM +0100, Uwe Kleine-König wrote:
> Hi Detlef,
> 
> On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
> > On 01/16/11 14:42, Russell King - ARM Linux wrote:
> > >Let's say you already have a copy of my tree from a month ago, and Linus
> > >has pulled some work from me into his tree, and repacked his tree into one
> > >single pack file.  At the moment, the largest pack file from Linus is
> > >400MB plus a 50MB index.
> > >
> > >You already have most of the contents of that 400MB pack file, but if
> > >you're missing even _one_ object which is contained within it, git will
> > >have to download the _entire_ 400MB pack file and index file to retrieve
> > >it.
> > I thought this has changed with "smart http" in git 1.6.6.
> > Am I missing something?
> Well, not all http repos offer smart http.  E.g. Russell doesn't[1],
> probably because the serving machine doesn't have the power to nice
> serve a repo via git:// or smart http://.

What is smart http?  I don't particularly follow git developments.

Re: cannot fetch arm git tree

[Detlef Vollmann]

On 01/21/11 14:57, Russell King - ARM Linux wrote:
> On Fri, Jan 21, 2011 at 02:47:28PM +0100, Uwe Kleine-König wrote:
>> Hi Detlef,
>>
>> On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
>>> On 01/16/11 14:42, Russell King - ARM Linux wrote:
>>>> Let's say you already have a copy of my tree from a month ago, and Linus
>>>> has pulled some work from me into his tree, and repacked his tree into one
>>>> single pack file.  At the moment, the largest pack file from Linus is
>>>> 400MB plus a 50MB index.
>>>>
>>>> You already have most of the contents of that 400MB pack file, but if
>>>> you're missing even _one_ object which is contained within it, git will
>>>> have to download the _entire_ 400MB pack file and index file to retrieve
>>>> it.
>>> I thought this has changed with "smart http" in git 1.6.6.
>>> Am I missing something?
>> Well, not all http repos offer smart http.  E.g. Russell doesn't[1],
>> probably because the serving machine doesn't have the power to nice
>> serve a repo via git:// or smart http://.
>
> What is smart http?  I don't particularly follow git developments.
It seems to be an implementation of the git protocol using
HTTP as transport.
Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.

   Detlef

Re: cannot fetch arm git tree

[Jello huang]

[-- Attachment #1.1: Type: text/plain, Size: 1570 bytes --]

not a lucky dog. i used 1.6.0,there was a large pack.

On 21 January 2011 22:28, Detlef Vollmann <dv@vollmann.ch> wrote:

> On 01/21/11 14:57, Russell King - ARM Linux wrote:
>
>> On Fri, Jan 21, 2011 at 02:47:28PM +0100, Uwe Kleine-König wrote:
>>
>>> Hi Detlef,
>>>
>>> On Fri, Jan 21, 2011 at 02:38:11PM +0100, Detlef Vollmann wrote:
>>>
>>>> On 01/16/11 14:42, Russell King - ARM Linux wrote:
>>>>
>>>>> Let's say you already have a copy of my tree from a month ago, and
>>>>> Linus
>>>>> has pulled some work from me into his tree, and repacked his tree into
>>>>> one
>>>>> single pack file.  At the moment, the largest pack file from Linus is
>>>>> 400MB plus a 50MB index.
>>>>>
>>>>> You already have most of the contents of that 400MB pack file, but if
>>>>> you're missing even _one_ object which is contained within it, git will
>>>>> have to download the _entire_ 400MB pack file and index file to
>>>>> retrieve
>>>>> it.
>>>>>
>>>> I thought this has changed with "smart http" in git 1.6.6.
>>>> Am I missing something?
>>>>
>>> Well, not all http repos offer smart http.  E.g. Russell doesn't[1],
>>> probably because the serving machine doesn't have the power to nice
>>> serve a repo via git:// or smart http://.
>>>
>>
>> What is smart http?  I don't particularly follow git developments.
>>
> It seems to be an implementation of the git protocol using
> HTTP as transport.
> Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.
>
>  Detlef
>
>


-- 
JUST DO IT,NOTHING IS IMPOSSIBLE

[-- Attachment #1.2: Type: text/html, Size: 2490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 176 bytes --]

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: cannot fetch arm git tree

[Russell King - ARM Linux]

On Fri, Jan 21, 2011 at 03:28:14PM +0100, Detlef Vollmann wrote:
> It seems to be an implementation of the git protocol using
> HTTP as transport.
> Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.

Setting up Smart HTTP

...
   To set it up, it■s best to walk through the instructions on the
   `git-http-backend` documentation page. Basically, you have to install Git
   v1.6.6 or higher on a server with an Apache 2.x webserver (it has to be
   Apache, currently - other CGI servers don■t work, last I checked). Then
   you add something similar to this to your http.conf file:

 SetEnv GIT_PROJECT_ROOT /var/www/git
 SetEnv GIT_HTTP_EXPORT_ALL
 ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/

Great.  Deciding that it will be http://servername.example.com/git/ is
really damned annoying as that's traditionally where gitweb lives,
which requires a different script alias.

It seems that due to a lack of coordination between different git
developers, people running webservers have a choice between providing
gitweb or this http extension.

I'm really not interested in working out how to bodge this into working
along side the existing gitweb setup by adding lots of rewrite rules, so
as gitweb got there first I think it has priority, that's what we have
and we'll have to live without the smart http extensions.

It's really not that big a deal if you follow the advice I've given.

Re: cannot fetch arm git tree

[Detlef Vollmann]

On 01/21/11 15:50, Russell King - ARM Linux wrote:
> On Fri, Jan 21, 2011 at 03:28:14PM +0100, Detlef Vollmann wrote:
>> It seems to be an implementation of the git protocol using
>> HTTP as transport.
>> Some info on this is at<http://progit.org/2010/03/04/smart-http.html>.
>
> Setting up Smart HTTP
>
> ...
>     To set it up, it■s best to walk through the instructions on the
>     `git-http-backend` documentation page. Basically, you have to install Git
>     v1.6.6 or higher on a server with an Apache 2.x webserver (it has to be
>     Apache, currently - other CGI servers don■t work, last I checked). Then
>     you add something similar to this to your http.conf file:
>
>   SetEnv GIT_PROJECT_ROOT /var/www/git
>   SetEnv GIT_HTTP_EXPORT_ALL
>   ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
>
> Great.  Deciding that it will be http://servername.example.com/git/ is
> really damned annoying as that's traditionally where gitweb lives,
> which requires a different script alias.
>
> It seems that due to a lack of coordination between different git
> developers, people running webservers have a choice between providing
> gitweb or this http extension.
Huh?
/git/ is just the example here, you can use any name you want.
E.g. I use /auth/ for authenticated users, and others use /gitmob/
or /gitanon/ for non-authenticated users.
And you can use something like gitsmart, githttp, or whatever...

   Detlef


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: cannot fetch arm git tree

[Uwe Kleine-König]

On Fri, Jan 21, 2011 at 02:50:26PM +0000, Russell King - ARM Linux wrote:
> On Fri, Jan 21, 2011 at 03:28:14PM +0100, Detlef Vollmann wrote:
> > It seems to be an implementation of the git protocol using
> > HTTP as transport.
> > Some info on this is at <http://progit.org/2010/03/04/smart-http.html>.
> 
> Setting up Smart HTTP
> 
> ...
>    To set it up, it■s best to walk through the instructions on the
>    `git-http-backend` documentation page. Basically, you have to install Git
>    v1.6.6 or higher on a server with an Apache 2.x webserver (it has to be
>    Apache, currently - other CGI servers don■t work, last I checked). Then
>    you add something similar to this to your http.conf file:
> 
>  SetEnv GIT_PROJECT_ROOT /var/www/git
>  SetEnv GIT_HTTP_EXPORT_ALL
>  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
> 
> Great.  Deciding that it will be http://servername.example.com/git/ is
> really damned annoying as that's traditionally where gitweb lives,
> which requires a different script alias.
> 
> It seems that due to a lack of coordination between different git
> developers, people running webservers have a choice between providing
> gitweb or this http extension.
> 
> I'm really not interested in working out how to bodge this into working
> along side the existing gitweb setup by adding lots of rewrite rules, so
> as gitweb got there first I think it has priority, that's what we have
> and we'll have to live without the smart http extensions.
IIRC it's designed to live along side the http:// clone url.
git-http-backend can still serve dumb http clients including a web
browser.
 
But note that as git-http-backend less info it has to calculate much
more.  So the load it introduces should be comparable to running
git-daemon as should be the times to fetch from it.  So AFAIK the only
reason to run it is that more corporate users can access port 80.

Best regards
Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |

Re: cannot fetch arm git tree

[Miles Bader]

Russell King - ARM Linux <linux@arm.linux.org.uk> writes:
> I'm really not interested in working out how to bodge this into working
> along side the existing gitweb setup by adding lots of rewrite rules, so
> as gitweb got there first I think it has priority, that's what we have
> and we'll have to live without the smart http extensions.
...
> It's really not that big a deal if you follow the advice I've given.

Smart http is actually a very big deal -- the old git http protocol is
almost unusable in practice with big repos, at least over somewhat
latency-limited network connections.

If you don't intend to support people pulling over http, then maybe you
don't care.  But if you do care, it's very much worth a second look.

[My personal reason for caring is that I'm behind a corporate firewall
that's latency limited, although it seems to have pretty good bandwidth.
With some public repos, pulling via the old http protocol was a
multi-hour operation; the new http protocol is typically multiple orders
of magnitude faster in these cases.]

-Miles

-- 
Omochiroi!

Re: cannot fetch arm git tree

[Johannes Sixt]

Am 1/21/2011 15:50, schrieb Russell King - ARM Linux:
>  SetEnv GIT_PROJECT_ROOT /var/www/git
>  SetEnv GIT_HTTP_EXPORT_ALL
>  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
> 
> Great.  Deciding that it will be http://servername.example.com/git/ is
> really damned annoying as that's traditionally where gitweb lives,
> which requires a different script alias.
> ...
> I'm really not interested in working out how to bodge this into working
> along side the existing gitweb setup by adding lots of rewrite rules,...

It has been worked out for you already. It's just a single rule (although
a bit longish). Look for 'ScriptAliasMatch' in
http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

-- Hannes

Re: cannot fetch arm git tree

[J.H.]

On 01/23/2011 11:21 PM, Johannes Sixt wrote:
> Am 1/21/2011 15:50, schrieb Russell King - ARM Linux:
>>  SetEnv GIT_PROJECT_ROOT /var/www/git
>>  SetEnv GIT_HTTP_EXPORT_ALL
>>  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
>>
>> Great.  Deciding that it will be http://servername.example.com/git/ is
>> really damned annoying as that's traditionally where gitweb lives,
>> which requires a different script alias.
>> ...
>> I'm really not interested in working out how to bodge this into working
>> along side the existing gitweb setup by adding lots of rewrite rules,...
> 
> It has been worked out for you already. It's just a single rule (although
> a bit longish). Look for 'ScriptAliasMatch' in
> http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

I was going to say, I have both gitweb and smart http support running in
parallel, it's not terribly complicated.  The way I'm doing it is a bit
older (mainly that long ScriptAliasMatch broken out more), but it's
still only 11 lines to the apache config:

# Various environment setup, should be obvious
SetEnv GIT_PROJECT_ROOT /path
SetEnv GIT_HTTP_EXPORT_ALL 1
SetEnv GIT_EXEC_PATH /usr/libexec/git-core/

# Here so that we can do straight wget fetches for the objects files,
# pack files and refs heads.
# This isn't strictly neccisary, but if it's done we can use the inbuilt
# sendfile() of apache for the sending of those files vs. funneling them
# through the CGI
AliasMatch ^/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$		/path/$1
AliasMatch ^/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$	/path/$1
AliasMatch ^/(.*/refs/heads/.*)$				/path/$1

# Meat of how to make this all work
ScriptAliasMatch "^/(.*/HEAD)$"
/usr/libexec/git-core/git-http-backend/path/$1
ScriptAliasMatch "^/(.*/info/refs)$"
/usr/libexec/git-core/git-http-backend/path/$1
ScriptAliasMatch "^/(.*/objects/info/[^/]+)$"
/usr/libexec/git-core/git-http-backend/path/$1
ScriptAliasMatch "^/pub/scm/(.*/git-upload-pack)$"
/usr/libexec/git-core/git-http-backend/path/$1
ScriptAliasMatch "^/pub/scm/(.*/git-receive-pack)$"
/usr/libexec/git-core/git-http-backend/path/$1

Just an example anyway, the paths have been shortened to fit e-mail better.

- John 'Warthog9' Hawley

Re: cannot fetch arm git tree

[Russell King - ARM Linux]

On Mon, Jan 24, 2011 at 08:21:09AM +0100, Johannes Sixt wrote:
> Am 1/21/2011 15:50, schrieb Russell King - ARM Linux:
> >  SetEnv GIT_PROJECT_ROOT /var/www/git
> >  SetEnv GIT_HTTP_EXPORT_ALL
> >  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/
> > 
> > Great.  Deciding that it will be http://servername.example.com/git/ is
> > really damned annoying as that's traditionally where gitweb lives,
> > which requires a different script alias.
> > ...
> > I'm really not interested in working out how to bodge this into working
> > along side the existing gitweb setup by adding lots of rewrite rules,...
> 
> It has been worked out for you already. It's just a single rule (although
> a bit longish). Look for 'ScriptAliasMatch' in
> http://www.kernel.org/pub/software/scm/git/docs/git-http-backend.html

Thanks, added taking two server restarts (one of them took a few minutes
for apache to restart as it was soo busy...)

I assume this also means that I need to update the URLs people are using
(as the tree wasn't exported under http://hostname.example.com/git/)?