[BUG] malloc error when using large file.

[BUG] malloc error when using large file.

[Emmanuel Puerto]

Hi All,

I use Git with some Virtual Machine and I have an issue with 2 of them.
When I use some VM I have created with a "pre-allocated" disk split into a 2Go file. For example a "ls" give:

-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f001.vmdk
-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f002.vmdk
-rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f003.vmdk
-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f004.vmdk
-rwx------ 1 epuerto epuerto 2147221504 juin 18 17:05 All-f005.vmdk
-rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f006.vmdk
-rwx------ 1 epuerto epuerto 2147221504 juin 18 12:07 All-f007.vmdk
-rwx------ 1 epuerto epuerto    1835008 mars 10 21:44 All-f008.vmdk
-rwx------ 1 epuerto epuerto       1007 juin 18 09:30 All.vmdk
-rw------- 1 epuerto epuerto        548 juin 18 09:40 devel
-rw------- 1 epuerto epuerto 1538850816 juin 18 17:05 devel-s001
-rw------- 1 epuerto epuerto 1125384192 juin 18 17:05 devel-s002
-rw------- 1 epuerto epuerto 1384382464 juin 18 17:05 devel-s003
-rw------- 1 epuerto epuerto 1632763904 juin 18 17:05 devel-s004
-rw------- 1 epuerto epuerto    1572864 juin 18 17:05 devel-s005
-rwx------ 1 epuerto epuerto       8684 juin 18 17:07 Ubuntu.nvram
-rw------- 1 epuerto epuerto  536870912 juin 18 17:06 Ubuntu.vmem
-rwx------ 1 epuerto epuerto          0 mars 10 21:31 Ubuntu.vmsd
-rw------- 1 epuerto epuerto  135732052 juin 18 17:07 Ubuntu.vmss
-rwx------ 1 epuerto epuerto       2908 juin 18 17:07 Ubuntu.vmx
-rwx------ 1 epuerto epuerto        261 mars 10 21:31 Ubuntu.vmxf

So if I just do a "git init" and after do a "git add -A" I have the following error:
fatal: Out of memory, malloc failed

no more information is given if I use the verbose option of git (git add -v -A).

In fact the error appear when I try to add file name "all-f00?.vmdk"

It appears that this size of file is not accepted by git.
If I do the same with other VM that they have a biggest size or a smallest size, all it is ok

For example:
-rw------- 1 epuerto epuerto       8684 avril  6 15:56 CentOs 5.2 Dev.nvram
-rw------- 1 epuerto epuerto       8684 juin 18 11:36 CentOs 5.3 Dev.nvram
-rw------- 1 epuerto epuerto  536870912 juin 18 11:36 CentOs 5.3 Dev.vmem
-rw------- 1 epuerto epuerto          0 avril  3 12:26 CentOs 5.3 Dev.vmsd
-rw------- 1 epuerto epuerto   18408288 juin 18 11:36 CentOs 5.3 Dev.vmss
-rwxr-xr-x 1 epuerto epuerto       2239 juin 18 11:36 CentOs 5.3 Dev.vmx
-rw------- 1 epuerto epuerto        269 avril  6 10:43 CentOs 5.3 Dev.vmxf
-rw------- 1 epuerto epuerto 1766981632 juin 18 11:36 sda-s001.vmdk
-rw------- 1 epuerto epuerto 2094596096 juin 18 11:36 sda-s002.vmdk
-rw------- 1 epuerto epuerto 1808138240 juin 18 11:36 sda-s003.vmdk
-rw------- 1 epuerto epuerto 1429667840 juin 18 11:36 sda-s004.vmdk
-rw------- 1 epuerto epuerto      65536 juin 18 11:36 sda-s005.vmdk
-rw------- 1 epuerto epuerto        907 juin 18 10:57 sda.vmdk

OR

-rwxrwxrwx 1 epuerto epuerto       8684 juin 17 21:08 FreeBSD 64-bit.nvram
-rwxrwxrwx 1 epuerto epuerto    1114112 mars 27  2008 FreeBSD 64-bit.vmdk
-rwxrwxrwx 1 epuerto epuerto          0 mars 27  2008 FreeBSD 64-bit.vmsd
-rwxrwxrwx 1 epuerto epuerto       1652 juin 17 19:48 FreeBSD 64-bit.vmx
-rwxrwxrwx 1 epuerto epuerto        269 mars 27  2008 FreeBSD 64-bit.vmxf
-rwxrwxrwx 1 epuerto epuerto 9118810112 juin 17 21:08 hdd.vmdk


My primary OS (where I use Git) is 
$uname -a
Linux epuerto-FC10.innova-card.internal 2.6.27.24-170.2.68.fc10.x86_64 #1 SMP Wed May 20 22:47:23 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

And my git version is:
$ git --version
git version 1.6.0.6

my config is:
ThinkPad T400 with a Intel(R) Core(TM)2 Duo CPU T9400 and 4Go

Thanks by advance for your help.

Best regards,
Emmanuel.

Before you print Think about 
ENVIRONMENTAL responsibility and commitment

Emmanuel PUERTO
Senior MTS, Field Applications (Microcontroller)
Linux Embedded Architect

MAXIM Integrated Products, Inc.
ZI Athelia IV - Le Forum, Bat A
Quartier Roumagoua
13600 La Ciotat - France
Phone : +33 44 298 1338
Fax   : +33 44 208 3319
Email : emmanuel.puerto@maxim-ic.com
Web   : www.maxim-ic.com

Re: [BUG] malloc error when using large file.

[Jeff King]

On Mon, Jun 22, 2009 at 02:54:18AM -0700, Emmanuel Puerto wrote:

> I use Git with some Virtual Machine and I have an issue with 2 of them.
> When I use some VM I have created with a "pre-allocated" disk split
> into a 2Go file. For example a "ls" give:
> [...]
> So if I just do a "git init" and after do a "git add -A" I have the
> following error:
> fatal: Out of memory, malloc failed

Many operations in git assume that a whole file can be held in memory at
once. Do you actually have 2G available on the machine? Are there any OS
limits that might be a problem?

-Peff

RE: [BUG] malloc error when using large file.

[Emmanuel Puerto]

> Many operations in git assume that a whole file can be held in memory at
> once. Do you actually have 2G available on the machine? Are there any OS
> limits that might be a problem?

Thanks for your answer Jeff, 

Yes I have more that 2 Go available when a do it, and after this allocation error if I try to do a 'git add' with a 9Go file, I did not have any issue.

Emmanuel.

Re: [BUG] malloc error when using large file.

[Jeff King]

On Tue, Jun 23, 2009 at 12:05:11AM -0700, Emmanuel Puerto wrote:

> Yes I have more that 2 Go available when a do it, and after this
> allocation error if I try to do a 'git add' with a 9Go file, I did not
> have any issue.

It may be that doing a whole bunch of files in one process is leading to
some memory fragmentation that doesn't let malloc get a contiguous 2G
slice. I'm not sure what else to suggest trying, except perhaps to
search the list archives for past discussions on memory fragmentation.

Maybe somebody else has suggestions.

-Peff

RE: [BUG] malloc error when using large file.

[Linus Torvalds]

On Tue, 23 Jun 2009, Emmanuel Puerto wrote:
> 
> 
> > Many operations in git assume that a whole file can be held in memory at
> > once. Do you actually have 2G available on the machine? Are there any OS
> > limits that might be a problem?
> 
> Thanks for your answer Jeff, 
> 
> Yes I have more that 2 Go available when a do it, and after this 
> allocation error if I try to do a 'git add' with a 9Go file, I did not 
> have any issue.

Can you add the 2GB files individually, using

	for i in All-f00?.vmdk; do git add $i; done

or similar?

It does sound like an unlucky malloc VM space fragmentation issue, 
although since the files you add are the same size I'd have not really 
expected that with any normal malloc library. But with some mixed small 
and large allocations, I could imagine that the fragmentation happens, and 
you end up having huge areas that aren't _quite_ useful for the next file 
due to having some small allocation taking part of it.

		Linus

RE: [BUG] malloc error when using large file.

[Emmanuel Puerto]

Hi Linus,

> > 
> > 
> > > Many operations in git assume that a whole file can be held in memory at
> > > once. Do you actually have 2G available on the machine? Are there any OS
> > > limits that might be a problem?
> > 
> > Thanks for your answer Jeff, 
> > 
> > Yes I have more that 2 Go available when a do it, and after this 
> > allocation error if I try to do a 'git add' with a 9Go file, I did not 
> > have any issue.
>
> Can you add the 2GB files individually, using
>
>	for i in All-f00?.vmdk; do git add $i; done
>
> or similar?
>
> It does sound like an unlucky malloc VM space fragmentation issue, 
> although since the files you add are the same size I'd have not really 
> expected that with any normal malloc library. But with some mixed small 
> and large allocations, I could imagine that the fragmentation happens, and 
> you end up having huge areas that aren't _quite_ useful for the next file 
> due to having some small allocation taking part of it.
>
>		Linus

Sorry, But I have already made this without any success. I really think that the issue is the size of this file. But we never know so I redo it again and here you have the result:

$ free
             total       used       free     shared    buffers     cached
Mem:       3954752     662264    3292488          0      17304     233092
-/+ buffers/cache:     411868    3542884
Swap:      8385920          0    8385920
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ for i in All-f00?.vmdk; do git add $i; done
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
fatal: Out of memory, malloc failed
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$ free
             total       used       free     shared    buffers     cached
Mem:       3954752    3928772      25980          0       6052    3510268
-/+ buffers/cache:     412452    3542300
Swap:      8385920         88    8385832
[epuerto@epuerto-FC10 My_Ubuntu_8.04.2]$

I do nothing else, only a top to know the memory use and I see that just before this 'malloc failed', I have something like this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
  4323 epuerto   20   0 2115m 2.0g 2.0g D 20.2 51.7   0:24.64 git                

And you can see that it appear some memory allocation is not free after this 'malloc failed' error.

If you have any suggestion, any test (or beta version with more debug message) I can do, you are welcome.

Best regards,
Emmanuel.