git status --> Out of memory, realloc failed

git status --> Out of memory, realloc failed

[Carsten Fuchs]

Dear Git group,

I use Git at a web hosting service, where my user account has a memory 
limit of 768 MB:

(uiserver):p7715773:~$ uname -a
Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian 
3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux

(uiserver):p7715773:~$ git --version
git version 2.1.4

The problem is that `git status` fails with an out of memory error:

(uiserver):p7715773:~/cafu$ git status
fatal: Out of memory, realloc failed
fatal: recursion detected in die handler

I talked to their support and their suggestion was to set a couple of 
memory constraints and to run `git gc`. This seemed to work well – but 
`git status` still fails:

(uiserver):p7715773:~/cafu$ cat ~/.gitconfig
[color]
     ui = auto
[user]
     name = Carsten Fuchs
     email = carsten.fuchs@cafu.de
[core]
     editor = nano
     pager = less -M -FRXS
     packedgitwindowsize = 30m
     packedgitlimit = 40m
[i18n]
     commitEncoding = ISO-8859-1
[pack]
     threads = 1
     windowMemory = 10m
     packSizeLimit = 20m
     deltaCacheSize = 30m

(uiserver):p7715773:~/cafu$ git gc
Zähle Objekte: 44293, Fertig.
Komprimiere Objekte: 100% (24534/24534), Fertig.
Schreibe Objekte: 100% (44293/44293), Fertig.
Total 44293 (delta 17560), reused 41828 (delta 16708)

(uiserver):p7715773:~/cafu$ git status
fatal: Out of memory, realloc failed
fatal: recursion detected in die handler

The repository is tracking about 19000 files which together take 260 MB.
The git server version is 2.7.4.1.g5468f9e (Bitbucket)

Well, their next response was that they have no solution for me – 
except, unsurprisingly, coaxing me into a more expensive hosting package.

I've read the Git man page about `git config`, but was not able to come 
up with anything to improve the situation.

Any ideas what I could do to reduce the memory consumption of `git status`?

Best regards,
Carsten

PS: Many thanks to Philip Oakley for initial advice at git-users, to 
which I should have properly subscribed in the first place, as the 
Google Groups interface seems to lose messages (mine at least, and 
inadvertently posts them as HTML) and gmane's NNTP interface reports 
that it is unidirectional/read-only.

Re: git status --> Out of memory, realloc failed

[René Scharfe]

Am 25.02.2017 um 11:13 schrieb Carsten Fuchs:
> Dear Git group,
>
> I use Git at a web hosting service, where my user account has a memory
> limit of 768 MB:
>
> (uiserver):p7715773:~$ uname -a
> Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian
> 3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux

What's the output of "ulimit -a"?

> (uiserver):p7715773:~$ git --version
> git version 2.1.4

That's quite old.  Can you try a more recent version easily (2.12.0 just 
came out)?  I don't remember a specific fix, and memory usage perhaps 
even increased with newer versions, but ruling out already fixed issues 
would be nice.

> (uiserver):p7715773:~/cafu$ git gc
> Zähle Objekte: 44293, Fertig.
> Komprimiere Objekte: 100% (24534/24534), Fertig.
> Schreibe Objekte: 100% (44293/44293), Fertig.
> Total 44293 (delta 17560), reused 41828 (delta 16708)
>
> (uiserver):p7715773:~/cafu$ git status
> fatal: Out of memory, realloc failed
> fatal: recursion detected in die handler
>
> The repository is tracking about 19000 files which together take 260 MB.
> The git server version is 2.7.4.1.g5468f9e (Bitbucket)

Is your repository publicly accessible?

René

Re: git status --> Out of memory, realloc failed

[Carsten Fuchs]

Hi René,

Am 01.03.2017 um 11:02 schrieb René Scharfe:
>> I use Git at a web hosting service, where my user account has a memory
>> limit of 768 MB:
>>
>> (uiserver):p7715773:~$ uname -a
>> Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian
>> 3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux
>
> What's the output of "ulimit -a"?

(uiserver):p7715773:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 1
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 512
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 1800
max user processes              (-u) 42
virtual memory          (kbytes, -v) 786432
file locks                      (-x) unlimited

>> (uiserver):p7715773:~$ git --version
>> git version 2.1.4
>
> That's quite old.  Can you try a more recent version easily (2.12.0 just came out)?

I don't think that they have any newer version available, having just upgraded from Git 
1.7 a couple of weeks ago... (1und1)

Git 1.7 used to work for me in the same environment, but iirc they also said they 
switched from 32-bit to 64-bit edition and blame the increased memory consumption on 
that change.

I'll ask for a newer version, but I'd be surprised if this happened.

>> The repository is tracking about 19000 files which together take 260 MB.
>> The git server version is 2.7.4.1.g5468f9e (Bitbucket)
>
> Is your repository publicly accessible?

Unfortunately, no. There are no big secrets in there, but just a couple of database 
details so that I cannot make it universally available. I can gladly give you access 
though. (E.g. by adding your public SSH key?)

Best regards,
Carsten

Re: git status --> Out of memory, realloc failed

[Duy Nguyen]

On Thu, Mar 2, 2017 at 3:12 AM, Carsten Fuchs <carsten.fuchs@cafu.de> wrote:
>>> The repository is tracking about 19000 files which together take 260 MB.
>>> The git server version is 2.7.4.1.g5468f9e (Bitbucket)
>>
>>
>> Is your repository publicly accessible?
>
>
> Unfortunately, no. There are no big secrets in there, but just a couple of
> database details so that I cannot make it universally available. I can
> gladly give you access though. (E.g. by adding your public SSH key?)

You could also try "git fast-export --anonymize" (read the doc first).
I'm not sure if it keeps blob size though, and delta chain might also
matter so an anonymized repo may or may not share the same problem,
I'm not sure.
-- 
Duy

Re: git status --> Out of memory, realloc failed

[René Scharfe]

Am 01.03.2017 um 21:12 schrieb Carsten Fuchs:
> Hi René,
>
> Am 01.03.2017 um 11:02 schrieb René Scharfe:
>>> I use Git at a web hosting service, where my user account has a memory
>>> limit of 768 MB:
>>>
>>> (uiserver):p7715773:~$ uname -a
>>> Linux infongp-de15 3.14.0-ui16322-uiabi1-infong-amd64 #1 SMP Debian
>>> 3.14.79-2~ui80+4 (2016-11-17) x86_64 GNU/Linux
>>
>> What's the output of "ulimit -a"?
>
> (uiserver):p7715773:~$ ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 1
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 16382
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 512
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) 1800
> max user processes              (-u) 42
> virtual memory          (kbytes, -v) 786432
> file locks                      (-x) unlimited

When I use ulimit -v with lower and lower numbers I can provoke mmap 
failures on bigger pack files, but not the realloc failures that you're 
seeing.  And your packs should be only up to 20MB anyway (you can check 
that with "ls -l .git/objects/pack/*.pack").

>>> The repository is tracking about 19000 files which together take 260 MB.
>>> The git server version is 2.7.4.1.g5468f9e (Bitbucket)
>>
>> Is your repository publicly accessible?
>
> Unfortunately, no. There are no big secrets in there, but just a couple
> of database details so that I cannot make it universally available. I
> can gladly give you access though. (E.g. by adding your public SSH key?)

I'd rather not look at semi-confidential data, and you probably 
shouldn't hand it to a stranger on the internet anyway. ;)

So a shot in the dark: Do you have a lot of untracked files?  You could 
check by cloning your repository locally (which copies only tracked 
contents).  Does "git status" work on the clone?

Another one, darker yet: Does "git config core.preloadIndex 0" help?

René

Re: git status --> Out of memory, realloc failed

[Carsten Fuchs]

Hi René,

Am 02.03.2017 um 21:04 schrieb René Scharfe:
> When I use ulimit -v with lower and lower numbers I can provoke mmap failures on
> bigger pack files, but not the realloc failures that you're seeing.  And your
> packs should be only up to 20MB anyway (you can check that with "ls -l
> .git/objects/pack/*.pack").

Yes, there are 9 of them, all about 20 MB in size.

> So a shot in the dark: Do you have a lot of untracked files?  You could check by
> cloning your repository locally (which copies only tracked contents).  Does "git
> status" work on the clone?

I had about 40 modified or untracked files with a combined file size of 
about 5 MB.

Before I got your latest mail, I tried something else: Instead of 
connecting to the system with the repository via SSH as usual, I 
accessed the repository by mounting its file system via gvfs-mount onto 
my local desktop machine. Then I used my local, normally working Git to 
commit all modified and some untracked files (and deleted those that 
were left). Running `git status` and other commands took a very long 
time over a network connection of only 6 MBit/s, but eventually I got 
everything committed and pushed.

Quite in the spirit of your above words, getting rid of untracked files 
helped: Logging normally via SSH into the remote machine again, `git 
status` ran normally.

Only then did I see your mail and got another clone, where `git status` 
worked equally well – but then `git diff` failed in a similar manner, 
continuing the problems of `git status`:

(uiserver):p7715773:~/__TEST__$ git clone 
https://CarstenF@bitbucket.org/CarstenF/website-cafu.git
Cloning into 'website-cafu'...
Password for 'https://CarstenF@bitbucket.org':
remote: Counting objects: 44359, done.
remote: Compressing objects: 100% (28777/28777), done.
remote: Total 44359 (delta 19201), reused 38661 (delta 14143)
Receiving objects: 100% (44359/44359), 168.28 MiB | 20.95 MiB/s, done.
Resolving deltas: 100% (19201/19201), done.
Checking connectivity... done.
Checking out files: 100% (18524/18524), done.

(uiserver):p7715773:~/__TEST__$ cd website-cafu/

(uiserver):p7715773:~/__TEST__/website-cafu$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

(uiserver):p7715773:~/__TEST__/website-cafu$ git diff
fatal: unable to create threaded lstat

(uiserver):p7715773:~/__TEST__/website-cafu$ git diff
fatal: Out of memory, realloc failed
fatal: Out of memory, realloc failed
fatal: Out of memory, realloc failed
fatal: recursion detected in die handler

> Another one, darker yet: Does "git config core.preloadIndex 0" help?

That's it!  :-)

Once that was set, both `git status` and `git diff` worked normally again!

I cannot test `git status` right now in the same situation as before, as 
I no longer have a working directory that is "as dirty" as before, but 
the above setting seems to fix the problem promptly and thoroughly!

Many thanks for your help!   :-)

Best regards,
Carsten

Re: git status --> Out of memory, realloc failed

[Carsten Fuchs]

Hi Duy,

Am 2017-03-02 um 10:31 schrieb Duy Nguyen:
> You could also try "git fast-export --anonymize" (read the doc first).

Eventually this was not needed, but thanks for letting me know about it! 
I've not been aware of this feature beforehand, learned something.

Best regards,
Carsten