Re: 2.22.0 repack -a duplicating pack contents

[Janos Farkas <chexum@gmail.com>]

Thanks for looking at this, yes, it seems it's purely the .bitmap
writing.  With disabled repack.writeBitmaps, it all behaves as before
(and yes, these are bare repros for keeping an intermediate backup).

With the easy workaround in place, I'm back at 2.22 for now, thanks!

On Sun, Jun 23, 2019 at 3:54 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Sun, Jun 23 2019, Janos Farkas wrote:
>
> > I'm using .keep files to... well.. keep packs to avoid some CPU time
> > spent on repacking huge packs and make the process somewhat more
> > incremental.
> >
> > Something changed with 22.2.0.  Now .bitmap files are also created,
> > and no simple repacks re-create the pack data in a completely new
> > file, wasting quite some storage:
> >
> > 02d03::master> find objects/pack/pack* -type f|xargs ls -sht
> > 108K objects/pack/pack-879f2c28d15e57d353eb8e0ddbcb540655c844c9.bitmap
> > 524K objects/pack/pack-879f2c28d15e57d353eb8e0ddbcb540655c844c9.idx
> > 4.7M objects/pack/pack-879f2c28d15e57d353eb8e0ddbcb540655c844c9.pack
> > 108K objects/pack/pack-e7a7aebfc6dc6b1431f6f56bb8b2f7e730cc4a0c.bitmap
> > 524K objects/pack/pack-e7a7aebfc6dc6b1431f6f56bb8b2f7e730cc4a0c.idx
> > 4.6M objects/pack/pack-e7a7aebfc6dc6b1431f6f56bb8b2f7e730cc4a0c.pack
> > 116K objects/pack/pack-994c76cb1999e3b29552677d05e6364e6be2ae5e.bitmap
> > 524K objects/pack/pack-994c76cb1999e3b29552677d05e6364e6be2ae5e.idx
> > 4.6M objects/pack/pack-994c76cb1999e3b29552677d05e6364e6be2ae5e.pack
> >    0 objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.keep
> > 108K objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.bitmap
> > 524K objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.idx
> > 4.6M objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.pack
> > 02d03::master > git repack -af
> > Enumerating objects: 19001, done.
> > Counting objects: 100% (19001/19001), done.
> > Delta compression using up to 2 threads
> > Compressing objects: 100% (18952/18952), done.
> > Writing objects: 100% (19001/19001), done.
> > warning: ignoring extra bitmap file:
> > ./objects/pack/pack-e7a7aebfc6dc6b1431f6f56bb8b2f7e730cc4a0c.pack
> > warning: ignoring extra bitmap file:
> > ./objects/pack/pack-994c76cb1999e3b29552677d05e6364e6be2ae5e.pack
> > warning: ignoring extra bitmap file:
> > ./objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.pack
> > Reusing bitmaps: 104, done.
> > Selecting bitmap commits: 2550, done.
> > Building bitmaps: 100% (130/130), done.
> > Total 19001 (delta 14837), reused 4162 (delta 0)
> > 02d03::master > find objects/pack/pack* -type f|xargs ls -sht
> > 108K objects/pack/pack-8702a2550b7e29940af8bc62bc6fca011ccbd455.bitmap
> > 524K objects/pack/pack-8702a2550b7e29940af8bc62bc6fca011ccbd455.idx
> > 4.6M objects/pack/pack-8702a2550b7e29940af8bc62bc6fca011ccbd455.pack   <= ????
> > 108K objects/pack/pack-879f2c28d15e57d353eb8e0ddbcb540655c844c9.bitmap
> > 524K objects/pack/pack-879f2c28d15e57d353eb8e0ddbcb540655c844c9.idx
> > 4.7M objects/pack/pack-879f2c28d15e57d353eb8e0ddbcb540655c844c9.pack
> > 108K objects/pack/pack-e7a7aebfc6dc6b1431f6f56bb8b2f7e730cc4a0c.bitmap
> > 524K objects/pack/pack-e7a7aebfc6dc6b1431f6f56bb8b2f7e730cc4a0c.idx
> > 4.6M objects/pack/pack-e7a7aebfc6dc6b1431f6f56bb8b2f7e730cc4a0c.pack
> > 116K objects/pack/pack-994c76cb1999e3b29552677d05e6364e6be2ae5e.bitmap
> > 524K objects/pack/pack-994c76cb1999e3b29552677d05e6364e6be2ae5e.idx
> > 4.6M objects/pack/pack-994c76cb1999e3b29552677d05e6364e6be2ae5e.pack
> >    0 objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.keep
> > 108K objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.bitmap
> > 524K objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.idx
> > 4.6M objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.pack
> >
> > The ccbd455 pack and its metadata seem quite pointless to be
> > containing apparently all the data based on the size.
> >
> > If I use -ad, a new pack is still created,which, judging by the size,
> > is essentially everything again, (but at least the extra packs are
> > removed)
> >
> > 02d03::master> git repack -ad
> > Enumerating objects: 19001, done.
> > Counting objects: 100% (19001/19001), done.
> > Delta compression using up to 2 threads
> > Compressing objects: 100% (4114/4114), done.
> > Writing objects: 100% (19001/19001), done.
> > warning: ignoring extra bitmap file:
> > ./objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.pack
> > Reusing bitmaps: 104, done.
> > Selecting bitmap commits: 2550, done.
> > Building bitmaps: 100% (130/130), done.
> > Total 19001 (delta 14838), reused 19001 (delta 14838)
> > 02d03::master 9060> find objects/pack/pack* -type f|xargs ls -sht
> > 116K objects/pack/pack-46ab64716d4220aac8d53b380d90a264d5293d3d.bitmap
> > 524K objects/pack/pack-46ab64716d4220aac8d53b380d90a264d5293d3d.idx
> > 4.6M objects/pack/pack-46ab64716d4220aac8d53b380d90a264d5293d3d.pack   <= ????
> >    0 objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.keep
> > 108K objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.bitmap
> > 524K objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.idx
> > 4.6M objects/pack/pack-e5b8848e7c1096274dba2430323ccaf5320c6846.pack
> >
> > Previously, the kept pack would be kept, and no additional packs would
> > be created if no new objects were born in the repro.
> >
> > With the .keep placeholder removed, the duplication does not happen,
> > but all the repro is rewritten into a new pack, which does not look
> > correct.  Am I doing something unexpected?
>
> I haven't looked at this for more than a couple of minutes (and don't
> have more time now), but this is almost certainly due to 36eba0323d
> ("repack: enable bitmaps by default on bare repos", 2019-03-14). Can you
> confirm when you re-run with repack.writeBitmaps=false in the config?
>
> I.e. something in the "yes I want bitmaps" code implies "*.keep"
> semantics changing from "keep" to "replace", which is obvious in
> retrospect, since we can only have one *.bitmap per-repo.