From: "Ævar Arnfjörð Bjarmason" <firstname.lastname@example.org> To: Jeff King <email@example.com> Cc: Junio C Hamano <firstname.lastname@example.org>, Eric Wong <email@example.com>, firstname.lastname@example.org Subject: Re: [PATCH v3] repack: enable bitmaps by default on bare repos Date: Fri, 24 May 2019 09:55:05 +0200 Message-ID: <email@example.com> (raw) In-Reply-To: <20190524072724.GH25694@sigill.intra.peff.net> On Fri, May 24 2019, Jeff King wrote: > On Thu, May 23, 2019 at 09:26:34PM +0200, Ævar Arnfjörð Bjarmason wrote: > >> > I spent a while thinking and experimenting with this tonight. The result >> > is the patch below. Ævar, do you still have a copy of the repo that >> > misbehaved? I'd be curious to see how it fares. >> >> No, sorry. I think we could make an artificial test to emulate it, which >> would be something like: >> >> * ~1 million commits >> * local clone setup to fetch all branches/tags (default 'git clone') >> * local a bit ahead/behind >> * Have old "main" pack with *.bitmap, bunch of other packs/loose objects without that >> * push try to push the local change upstream (or to a topic branch) >> >> I tried briefly to emulate this with git.git with: >> [...] >> But didn't get anywhere, probably because here my topics are all stuff I >> have already, and I just have 2x packs. > > Yeah, I haven't been able to find a reproduction for this problem at > will. The bitmaps are _supposed_ to be sprinkled around through the > commit graph so that we don't have to walk far. But presumably in your > case that was not so. > > I'm not sure what tickles the bitmap-writer to fail so hard. Is it > having too many refs? Weird patterns in the graph? Just a ton of > commits? Ah, why did only this ancient (big) pack have a bitmap? The bitmap writer had never failed, this was just a repository where some automation (on a dev/staging box) cloned a repo, and someone had once run a manual "repack" to make make a pack with a bitmap. Then as time passed that pack stayed around, and re-looking at this that could have only happened because I had gc.bigPackThreshold turned on. I.e. without that we'd have eventually done a full repack, so the bitmap would have gone away. So getting the repo into that state was a series of unlikely events. I think to the extent that this is an issue we can reproduce in the future the proper fix for it in lieu of some easy fix in the bitmap code would be to just teach "gc" to unlink old *.bitmap files if we detect they're too stale. I.e. we don't need to deal gracefully with some case where the bitmaps just cover some tiny part of the graph, we can just teach "gc" to either update them, or (if we're not currently writing them) unlink them. That seems to me to be a good idea in general, not just with bitmaps but also the commit graph. If we're doing a GC and our current settings aren't such that we'd update those files, shouldn't we just unlink them?
next prev parent reply index Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-02-14 4:31 [PATCH 0/3] some prune optimizations Jeff King 2019-02-14 4:35 ` [PATCH 1/3] prune: lazily perform reachability traversal Jeff King 2019-02-14 10:54 ` Eric Sunshine 2019-02-14 11:07 ` Jeff King 2019-02-14 4:37 ` [PATCH 2/3] prune: use bitmaps for " Jeff King 2019-03-09 2:49 ` bitmaps by default? [was: prune: use bitmaps for reachability traversal] Eric Wong 2019-03-10 23:39 ` Jeff King 2019-03-12 3:13 ` [PATCH] repack: enable bitmaps by default on bare repos Eric Wong 2019-03-12 9:07 ` Ævar Arnfjörð Bjarmason 2019-03-12 10:49 ` Jeff King 2019-03-12 12:05 ` Jeff King 2019-03-13 1:51 ` Eric Wong 2019-03-13 14:54 ` Jeff King 2019-03-14 9:12 ` [PATCH v3] " Eric Wong 2019-03-14 16:02 ` Jeff King 2019-03-15 6:21 ` [PATCH 0/2] enable bitmap hash-cache by default Jeff King 2019-03-15 6:22 ` [PATCH 1/2] t5310: correctly remove bitmaps for jgit test Jeff King 2019-03-15 13:25 ` SZEDER Gábor 2019-03-15 18:36 ` Jeff King 2019-03-15 6:25 ` [PATCH 2/2] pack-objects: default to writing bitmap hash-cache Jeff King 2019-04-09 15:10 ` [PATCH v3] repack: enable bitmaps by default on bare repos Ævar Arnfjörð Bjarmason 2019-04-10 22:57 ` Jeff King 2019-04-25 7:16 ` Junio C Hamano 2019-05-04 1:37 ` Jeff King 2019-05-04 6:52 ` Ævar Arnfjörð Bjarmason 2019-05-04 13:23 ` SZEDER Gábor 2019-05-08 20:17 ` Ævar Arnfjörð Bjarmason 2019-05-09 4:24 ` Junio C Hamano 2019-05-07 7:45 ` Jeff King 2019-05-07 8:12 ` Ævar Arnfjörð Bjarmason 2019-05-08 7:11 ` Jeff King 2019-05-08 14:20 ` Derrick Stolee 2019-05-08 16:13 ` Ævar Arnfjörð Bjarmason 2019-05-08 22:25 ` Jeff King 2019-05-23 11:30 ` Jeff King 2019-05-23 12:53 ` Derrick Stolee 2019-05-24 7:24 ` Jeff King 2019-05-24 10:33 ` Derrick Stolee 2019-05-23 19:26 ` Ævar Arnfjörð Bjarmason 2019-05-24 7:27 ` Jeff King 2019-05-24 7:55 ` Ævar Arnfjörð Bjarmason [this message] 2019-05-24 8:26 ` Jeff King 2019-05-24 9:01 ` Ævar Arnfjörð Bjarmason 2019-05-24 9:29 ` SZEDER Gábor 2019-05-24 11:17 ` Ævar Arnfjörð Bjarmason 2019-05-24 11:41 ` SZEDER Gábor 2019-05-24 11:58 ` Ævar Arnfjörð Bjarmason 2019-05-24 12:34 ` SZEDER Gábor 2019-05-24 13:41 ` Ævar Arnfjörð Bjarmason 2019-05-24 11:31 ` [PATCH] pack-bitmap: look for an uninteresting bitmap Derrick Stolee 2019-04-15 15:00 ` [PATCH 2/3] prune: use bitmaps for reachability traversal Derrick Stolee 2019-04-18 19:49 ` Jeff King 2019-04-18 20:08 ` [PATCH] t5304: add a test for pruning with bitmaps Jeff King 2019-04-20 1:01 ` Derrick Stolee 2019-04-20 3:24 ` Jeff King 2019-04-20 21:01 ` Derrick Stolee 2019-02-14 4:38 ` [PATCH 3/3] prune: check SEEN flag for reachability Jeff King
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://vger.kernel.org/majordomo-info.html * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
firstname.lastname@example.org list mirror (unofficial, one of many) Archives are clonable: git clone --mirror https://public-inbox.org/git git clone --mirror http://ou63pmih66umazou.onion/git git clone --mirror http://czquwvybam4bgbro.onion/git git clone --mirror http://hjrcffqmbrq6wope.onion/git Example config snippet for mirrors Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.version-control.git nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git nntp://news.gmane.io/gmane.comp.version-control.git note: .onion URLs require Tor: https://www.torproject.org/ AGPL code for this site: git clone https://public-inbox.org/public-inbox.git