Re: [PATCH 3/3] packfile: use oidset for bad objects

[Jeff King <peff@peff.net>]

On Sat, Sep 11, 2021 at 10:01:40AM +0200, René Scharfe wrote:

> Store the object ID of broken pack entries in an oidset instead of
> keeping only their hashes in an unsorted array.  The resulting code is
> shorter and easier to read.  It also handles the (hopefully) very rare
> case of having a high number of bad objects better.

Yay, I'm very happy to see this kind of cleanup replacing ad hoc data
structures with well-tested ones.

> @@ -303,15 +304,9 @@ static int nth_midxed_pack_entry(struct repository *r,
>  	if (!is_pack_valid(p))
>  		return 0;
> 
> -	if (p->num_bad_objects) {
> -		uint32_t i;
> -		struct object_id oid;
> -		nth_midxed_object_oid(&oid, m, pos);
> -		for (i = 0; i < p->num_bad_objects; i++)
> -			if (hasheq(oid.hash,
> -				   p->bad_object_sha1 + the_hash_algo->rawsz * i))
> -				return 0;
> -	}
> +	nth_midxed_object_oid(&oid, m, pos);
> +	if (oidset_contains(&p->bad_objects, &oid))
> +		return 0;

Calling nth_midxed_object_oid() implies a memcpy() under the hood. In
the old code, we'd skip that in the common case that we had no corrupt
objects, but now we'll pay the cost regardless. memcpy() isn't _that_
expensive, but I'd expect this to be a relatively hot code path.

Is it worth sticking all of this inside:

  if (oidset_size(&p->bad_objects))

?

> diff --git a/packfile.c b/packfile.c
> index 04080a558b..8f6d1d6328 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -1163,29 +1163,17 @@ int unpack_object_header(struct packed_git *p,
> 
>  void mark_bad_packed_object(struct packed_git *p, const struct object_id *oid)
>  {
> -	unsigned i;
> -	const unsigned hashsz = the_hash_algo->rawsz;
> -	for (i = 0; i < p->num_bad_objects; i++)
> -		if (hasheq(oid->hash, p->bad_object_sha1 + hashsz * i))
> -			return;

I cringed at the hasheq() and hashcpy() calls in the earlier patches.
Happy to see them go away now. :)

> -	p->bad_object_sha1 = xrealloc(p->bad_object_sha1,
> -				      st_mult(GIT_MAX_RAWSZ,
> -					      st_add(p->num_bad_objects, 1)));
> -	hashcpy(p->bad_object_sha1 + hashsz * p->num_bad_objects, oid->hash);
> -	p->num_bad_objects++;
> +	oidset_insert(&p->bad_objects, oid);
>  }

So now marking a bad object  is a one-liner. We _could_ just inline it
at the callers, but I like keeping the implementation abstract.

>  const struct packed_git *has_packed_and_bad(struct repository *r,
>  					    const struct object_id *oid)
>  {
>  	struct packed_git *p;
> -	unsigned i;
> 
>  	for (p = r->objects->packed_git; p; p = p->next)
> -		for (i = 0; i < p->num_bad_objects; i++)
> -			if (hasheq(oid->hash,
> -				   p->bad_object_sha1 + the_hash_algo->rawsz * i))
> -				return p;
> +		if (oidset_contains(&p->bad_objects, oid))
> +			return p;
>  	return NULL;
>  }

Not related to your patch, but I noticed how terribly inefficient this
function could be in a repo with a lot of packs. But we only call it
once in the error case right before we die(), so a linear scan is no
problem.

> @@ -2016,13 +2004,8 @@ static int fill_pack_entry(const struct object_id *oid,
>  {
>  	off_t offset;
> 
> -	if (p->num_bad_objects) {
> -		unsigned i;
> -		for (i = 0; i < p->num_bad_objects; i++)
> -			if (hasheq(oid->hash,
> -				   p->bad_object_sha1 + the_hash_algo->rawsz * i))
> -				return 0;
> -	}
> +	if (oidset_contains(&p->bad_objects, oid))
> +		return 0;

And this one (and the previous) have the oid already, so they don't have
to worry about optimizing the is-it-empty check first.

-Peff