[PATCH 0/1] builtin/pack-objects.c: avoid iterating all refs

[Jacob Vosmaer <jacob@gitlab.com>]

This is a small patch for git-pack-objects which will help server side
performance on repositories with lots of refs. I will post a related
but slightly larger patch for ls-refs.c in a separate thread.

The back story is in
https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/746 but I
will try to summarize it here.

We have a particular Gitaly (Git RPC) server at GitLab that has a very
homogenous workload, dominated by CI. While trying to reduce CPU
utilization on the server we configured CI to fetch with the
'--no-tags' option. This had an unexpectedly large impact so I started
looking closer at why that may be.

What I learned is that by default, a fetch ends up using the
'--include-tag' command line option of git-pack-objects. This causes
git-pack-objects to iterate through all the tags of the repository to
see if any should be included in the pack because they point to packed
objects. The problem is that this "iterate through all the tags" uses
for_each_ref which iterates through all references in the repository,
and in doing so loads each associated object into memory to check if
the ref is broken. But all we need for '--include-tag' is to iterate
through refs/tags/.

On the repo we were testing this on, there are about
500,000 refs but only 2,000 tags. So we had to load a lot of objects
just for the sake of '--include-tag'. It was common to see more than
half the CPU time in git-pack-objects being spent in do_for_each_ref,
and that in turn was dominated by ref_resolves_to_object.

So, I think it would be nice to just iterate over those 2,000 tags and
not load 500,000 objects outside refs/tags we already know we don't
care about.

Jacob Vosmaer (1):
  builtin/pack-objects.c: avoid iterating all refs

 builtin/pack-objects.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.30.0