git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Nicolas Pitre <nico@cam.org>
To: Linus Torvalds <torvalds@osdl.org>
Cc: Jakub Narebski <jnareb@gmail.com>, git@vger.kernel.org
Subject: Re: Figured out how to get Mozilla into git
Date: Fri, 09 Jun 2006 13:38:13 -0400 (EDT)	[thread overview]
Message-ID: <Pine.LNX.4.64.0606091326550.2703@localhost.localdomain> (raw)
In-Reply-To: <Pine.LNX.4.64.0606090926550.5498@g5.osdl.org>

On Fri, 9 Jun 2006, Linus Torvalds wrote:

> 
> 
> On Fri, 9 Jun 2006, Nicolas Pitre wrote:
> > 
> > > So a single 2GB pack is already very much pushing it. It's really really 
> > > hard to map in a 2GB file on a 32-bit platform: your VM is usually 
> > > fragmented enough that it simply isn't practical. In fact, I think the 
> > > limit for _practical_ usage of single packs is probably somewhere in the 
> > > half-gig region, unless you just have 64-bit machines.
> > 
> > Sure, but have we already reached that size?
> 
> Not for the Linux repos.
> 
> But apparently the mozilla repo ends up being 2GB in git. From Martin:
> 
>   >> oh, I went back to a cvsimport that I started a couple days ago.
>   >> Completed with no problems...
>   >> 
>   >> Last commit:
>   >> commit 5ecb56b9c4566618fad602a8da656477e4c6447a
>   >> Author: wtchang%redhat.com <wtchang%redhat.com>
>   >> Date:   Fri Jun 2 17:20:37 2006 +0000
>   >> 
>   >>    Import NSPR 4.6.2 and NSS 3.11.1
>   >> 
>   >> mozilla.git$ du -sh .git/
>   >> 2.0G    .git/

He also sais:

| git-repack -a -d but it OOMs on my 2GB+2GBswap machine :(

> now that was done with _incremental_ repacking (ie his .git directory
> won't be just one large pack),

So given the nature of packs, incrementally packing an imported 
repository _might_ cause worse problems since each pack must be self 
referenced by definition.  That means you may end up with multiple 
revisions of the same file distributed amongst as many packs hence none 
of those revisions are ever deltified, and to repack that you currently 
have to mmap all those packs at once.

> but I bet that if you were to clone it
> (without using the "-l" flag or rsync/http), you'd end up with serious
> trouble because of the single-pack limit.

Maybe that single pack would instead be under the 512MB limit?  I'd be 
curious to know.

> So we're starting to see archives where single packs are problematic for
> a 32-bit architecture. 

Depending on the operation, the single pack might actually be better, 
especially for a full clone where everything gets mapped.  Multiple 
packs will always take more space, which is fine if you don't need 
access to all objects at once since individual packs are small, but the 
whole of them (when repacking or cloning) isn't.


Nicolas

  reply	other threads:[~2006-06-09 17:38 UTC|newest]

Thread overview: 67+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2006-06-09  2:17 Figured out how to get Mozilla into git Jon Smirl
2006-06-09  2:56 ` Nicolas Pitre
2006-06-09  3:06 ` Martin Langhoff
2006-06-09  3:28   ` Jon Smirl
2006-06-09  7:17     ` Jakub Narebski
2006-06-09 15:01       ` Linus Torvalds
2006-06-09 16:11         ` Nicolas Pitre
2006-06-09 16:30           ` Linus Torvalds
2006-06-09 17:38             ` Nicolas Pitre [this message]
2006-06-09 17:49               ` Linus Torvalds
2006-06-09 17:10           ` Jakub Narebski
2006-06-09 18:13   ` Jon Smirl
2006-06-09 19:00     ` Linus Torvalds
2006-06-09 20:17       ` Jon Smirl
2006-06-09 20:40         ` Linus Torvalds
2006-06-09 20:56           ` Jon Smirl
2006-06-09 21:57             ` Linus Torvalds
2006-06-09 22:17               ` Linus Torvalds
2006-06-09 23:16               ` Greg KH
2006-06-09 23:37               ` Martin Langhoff
2006-06-09 23:43                 ` Linus Torvalds
2006-06-10  0:00                   ` Jon Smirl
2006-06-10  0:11                     ` Linus Torvalds
2006-06-10  0:16                       ` Jon Smirl
2006-06-10  0:45                         ` Jon Smirl
2006-06-09 20:44         ` Jakub Narebski
2006-06-09 21:05         ` Nicolas Pitre
2006-06-09 21:46           ` Jon Smirl
2006-06-10  1:23         ` Martin Langhoff
2006-06-10  1:14   ` Martin Langhoff
2006-06-10  1:33     ` Linus Torvalds
2006-06-10  1:43       ` Linus Torvalds
2006-06-10  1:48         ` Jon Smirl
2006-06-10  1:59           ` Linus Torvalds
2006-06-10  2:21             ` Jon Smirl
2006-06-10  2:34               ` Carl Worth
2006-06-10  3:08                 ` Linus Torvalds
2006-06-10  8:21                   ` Jakub Narebski
2006-06-10  9:00                     ` Junio C Hamano
2006-06-10  8:36                   ` Rogan Dawes
2006-06-10  9:08                     ` Junio C Hamano
2006-06-10 14:47                       ` Rogan Dawes
2006-06-10 14:58                         ` Jakub Narebski
2006-06-10 15:14                         ` Nicolas Pitre
2006-06-10 17:53                     ` Linus Torvalds
2006-06-10 18:02                       ` Jon Smirl
2006-06-10 18:36                       ` Rogan Dawes
2006-06-10  3:01               ` Linus Torvalds
2006-06-10  2:30             ` Jon Smirl
2006-06-10  3:41             ` Martin Langhoff
2006-06-10  3:55               ` Junio C Hamano
2006-06-10  4:02               ` Linus Torvalds
2006-06-10  4:11                 ` Linus Torvalds
2006-06-10  6:02                   ` Jon Smirl
2006-06-10  6:15                     ` Junio C Hamano
2006-06-10 15:44                       ` Jon Smirl
2006-06-10 16:15                         ` Timo Hirvonen
2006-06-10 18:37                         ` Petr Baudis
2006-06-10 18:55                         ` Lars Johannsen
2006-06-11 22:00       ` Nicolas Pitre
2006-06-18 19:26         ` Linus Torvalds
2006-06-18 21:40           ` Martin Langhoff
2006-06-18 22:36             ` Linus Torvalds
2006-06-18 22:51               ` Broken PPC sha1.. (Re: Figured out how to get Mozilla into git) Linus Torvalds
2006-06-18 23:25                 ` [PATCH] Fix PPC SHA1 routine for large input buffers Paul Mackerras
2006-06-19  5:02                   ` Linus Torvalds
2006-06-09  3:12 ` Figured out how to get Mozilla into git Pavel Roskin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Pine.LNX.4.64.0606091326550.2703@localhost.localdomain \
    --to=nico@cam.org \
    --cc=git@vger.kernel.org \
    --cc=jnareb@gmail.com \
    --cc=torvalds@osdl.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).