Re: What's cooking in git.git (Aug 2018, #04; Fri, 17)

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Sat, Aug 18 2018, Christian Couder wrote:

> On Sat, Aug 18, 2018 at 12:44 AM Junio C Hamano <gitster@pobox.com> wrote:
>
>> * cc/delta-islands (2018-08-16) 7 commits
>>  - pack-objects: move 'layer' into 'struct packing_data'
>>  - pack-objects: move tree_depth into 'struct packing_data'
>>  - t5320: tests for delta islands
>>  - repack: add delta-islands support
>>  - pack-objects: add delta-islands support
>>  - pack-objects: refactor code into compute_layer_order()
>>  - Add delta-islands.{c,h}
>>
>>  Lift code from GitHub to restrict delta computation so that an
>>  object that exists in one fork is not made into a delta against
>>  another object that does not appear in the same forked repository.
>>
>>  What's the doneness of this topic?
>
> All the suggestions by Peff, you, Duy, Dscho, Ramsay and Szeder Gabor
> have been taken into account in the v5 that is in pu.
>
> Except the suggestion by Duy to move 2 fields from 'struct
> object_entry' to 'struct packing_data' (which is implemented in
> patches 6/7 and 7/7) the suggestions have all been about relatively
> small things (documentation, code modernization, regex check,
> translation strings, ...) So the code is very similar to the original
> code in https://github.com/peff/git/commits/jk/delta-islands which has
> been used for years by GitHub in production.
>
> FYI this has been requested from GitLab by Drupal (as well as others)
> see https://www.drupal.org/drupalorg/blog/developer-tools-initiative-part-5-gitlab-partnership
> which contains:
>
> "The timeline for Phase 2 is dependent on GitLab’s resolution of a
> diskspace deduplication issue, which they have committed to on our
> behalf: https://gitlab.com/gitlab-org/gitlab-ce/issues/23029"

This is not a critique of the delta islands feature, just something I'm
curious about.

Why is Drupal blocked on something like delta-islands? The blog post
mentions they have 45k projects, which can be browsed at
https://cgit.drupalcode.org

Almost all of those are completely independent projects, so they
wouldn't benefit from delta islands, and I'd guess >98% are of them are
in an obscure long tail and probably won't have even a single fork.

That leaves forks of say drupal.git, which is ~150MB, the mirror on
GitHub has 1500 forks: https://github.com/drupal Even if there were 5000
forks of that that would be 750G of disk space.

So accounting for backups, me being off by a lot etc. let's say that's
5TB. That's relatively cheap today. Are they really just holding up
their GitLab migration plans to save something on the order of that disk
space, or have I missed something here?

Again, not a critique of delta-islands, because it's most certainly
useful for the likes of github/gitlab, but I wonder if for this
particular problem it wouldn't be more straightforward of a solution for
GitLab to allow anyone to push to
refs/for-merge/<their-username>/<some-name-they-pick> on any
repository. Then they could open a MR for an existing branch in the repo
(which GitLab already supports).