From: Jeff King <peff@peff.net>
To: Jonathan Nieder <jrnieder@gmail.com>
Cc: Andreas Krey <a.krey@gmx.de>, Git Users <git@vger.kernel.org>
Subject: Re: git repack leaks disk space on ENOSPC
Date: Thu, 12 Oct 2017 09:27:30 -0400 [thread overview]
Message-ID: <20171012132730.bvglyiar4h6win4b@sigill.intra.peff.net> (raw)
In-Reply-To: <20171012031702.GB155740@aiede.mtv.corp.google.com>
On Wed, Oct 11, 2017 at 08:17:03PM -0700, Jonathan Nieder wrote:
> I can imagine this behavior of retaining tmp_pack being useful for
> debugging in some circumstances, but I agree with you that it is
> certainly not a good default.
>
> Chasing this down, I find:
>
> pack-write.c::create_tmp_packfile chooses the filename
> builtin/pack-objects.c::write_pack_file writes to it and the .bitmap, calling
> pack-write.c::finish_tmp_packfile to rename it into place
>
> Nothing tries to install an atexit handler to do anything special to it
> on exit.
>
> The natural thing, I'd expect, would be for pack-write to use the
> tempfile API (see tempfile.h) to create and finish the file. That way,
> we'd get such atexit handlers for free. If we want a way to keep temp
> files for debugging on abnormal exit, we could set that up separately as
> a generic feature of the tempfile API (e.g. an envvar
> GIT_KEEP_TEMPFILES_ON_FAILURE), making that an orthogonal topic.
Yes, I think this is the right direction. I've had a patch in GitHub's
fork for years that does so (since otherwise failures can fill up your
disk and need manual intervention).
The main reason that I hadn't submitted it upstream was because of the
"you can never free a struct tempfile" requirement. So my patch just
leaks the tempfile structs. That's OK for packs, of which we tend to
create only a few in a given process, but doesn't scale for loose
objects.
Now that 89563ec379 (Merge branch 'jk/incore-lockfile-removal',
2017-09-19) has landed, I think it makes sense to pursue that direction.
My patch roughly looks like:
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 4ff567db47..7f261e56c4 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -308,9 +348,11 @@ static const char *open_pack_file(const char *pack_name)
input_fd = 0;
if (!pack_name) {
struct strbuf tmp_file = STRBUF_INIT;
+ struct tempfile *t = xcalloc(1, sizeof(*t));
output_fd = odb_mkstemp(&tmp_file,
"pack/tmp_pack_XXXXXX");
pack_name = strbuf_detach(&tmp_file, NULL);
+ register_tempfile(t, pack_name);
} else {
output_fd = open(pack_name, O_CREAT|O_EXCL|O_RDWR, 0600);
if (output_fd < 0)
but note that's not quite what we'd want. It never closes the tempfile,
so:
1. Under the new regime, we'd still leak the struct!
2. Git will still try to unlink the tempfile on exit, even if we
successfully moved it into place.
So I think all the code around open_pack_file() needs to learn to pass
around the tempfile struct, and eventually use rename_tempfile() to
cement it in place. I also suspect that odb_mkstemp should just take a
"struct tempfile".
-Peff
prev parent reply other threads:[~2017-10-12 13:27 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-11 15:05 git repack leaks disk space on ENOSPC Andreas Krey
2017-10-12 3:17 ` Jonathan Nieder
2017-10-12 9:34 ` Andreas Krey
2017-10-12 11:01 ` brian m. carlson
2017-10-12 13:36 ` Jeff King
2017-10-12 13:27 ` Jeff King [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171012132730.bvglyiar4h6win4b@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=a.krey@gmx.de \
--cc=git@vger.kernel.org \
--cc=jrnieder@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).