From: "Dipl. Ing. Sergey Brester" <serg.brester@sebres.de>
To: "René Scharfe" <l.s.r@web.de>
Cc: Jeff King <peff@peff.net>,
git@vger.kernel.org,
"brian m. carlson" <sandals@crustytoothpaste.net>
Subject: Re: git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump
Date: Thu, 15 Oct 2020 18:19:34 +0200 [thread overview]
Message-ID: <181f35ca5dff05d8d2288f020354a860@sebres.de> (raw)
In-Reply-To: <79cbeb4c-5840-d5b7-a2b9-d72cf47968df@web.de>
15.10.2020 17:52, René Scharfe wrote:
> In case someone else is wondering about the meaning of those hashes,
> here are their reference strings:
> e83c516331 (Initial revision of "git", the information manager from
> hell, 2005-04-07)
> dc04167d37 (Fourth batch, 2020-08-04)
> 61addb841f (Merge branch 'jk/strvec' into next, 2020-08-04)
>
These are just revisions of master branch, and the process was splittet
into two parts to generate a large marks file with a small partial dump,
that can be imported over first marks.
So quasi to simulate normal situation (large repo, small partial export
dump), for which import/export with marks is basically used.
It doesn't matter here, just helpful to export/import single branch only
(to bypass signed tags, avoid installing gpg keys, etc).
> So you use the marks generated by the first export to import the second
> export.
It also doesn't matter, because you could import whole first dump
(several gigabytes, I guess) in order to generate new import marks...
Which will then be the same after all :) just because it is the same
repository used for export (and all the revisions are already there,
moreover having the same hash values, let alone internal marks index and
the export sequence).
> I wonder if that's relevant to trigger the memory allocation issue.
No, it is not relevant.
Also note my initial report where it is affected by real situation (over
import marks generated previosly with git fast-import).
> I can reproduce this on Debian -- but don't get a crash report, just a
> notice from the OOM killer. It bisects to ddddf8d7e2 (fast-import:
> permit reading multiple marks files, 2020-02-22).
Sure, OOM manager kills this process (before it is able to crash e. g.
create a dump on failed malloc)...
If you'll disable OOMK (or try to run it on windows), you'll see the
crash and dump is there. :)
Regards,
Sergey.
prev parent reply other threads:[~2020-10-15 16:19 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-14 9:22 git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump Dipl. Ing. Sergey Brester
2020-10-15 1:26 ` Jeff King
2020-10-15 11:50 ` Dipl. Ing. Sergey Brester
2020-10-15 15:38 ` [PATCH] fast-import: fix over-allocation of marks storage Jeff King
2020-10-15 17:29 ` Junio C Hamano
2020-10-15 17:34 ` Junio C Hamano
2020-10-15 18:09 ` Dipl. Ing. Sergey Brester
2020-10-15 18:35 ` Junio C Hamano
2020-10-15 18:58 ` Jeff King
2020-10-15 19:13 ` Junio C Hamano
2020-10-16 2:37 ` brian m. carlson
2020-10-15 19:05 ` Jeff King
2020-10-15 19:06 ` Jeff King
2020-10-16 3:18 ` brian m. carlson
2020-10-16 20:25 ` Jeff King
2020-10-15 19:17 ` Dipl. Ing. Sergey Brester
2020-10-15 20:15 ` Junio C Hamano
2020-10-15 17:57 ` René Scharfe
2020-10-15 15:52 ` git fast-import leaks memory drastically, so crashes with out of memory by attempt to import 22MB export dump René Scharfe
2020-10-15 16:19 ` Dipl. Ing. Sergey Brester [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=181f35ca5dff05d8d2288f020354a860@sebres.de \
--to=serg.brester@sebres.de \
--cc=git@vger.kernel.org \
--cc=l.s.r@web.de \
--cc=peff@peff.net \
--cc=sandals@crustytoothpaste.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).