Re: [PATCH 2/9] pack-objects: add `--print-filtered` to print omitted objects

[Taylor Blau <me@ttaylorr.com>]

On Wed, Jun 21, 2023 at 01:11:38PM +0200, Christian Couder wrote:
> On Wed, Jun 21, 2023 at 12:52 PM Taylor Blau <me@ttaylorr.com> wrote:
> >
> > On Thu, Jun 15, 2023 at 03:50:17PM -0700, Junio C Hamano wrote:
>
> > > Makes sense.  It is a bit sad that we have to accumulate everything
> > > until the end at which time we have to dump the accumulated in bulk,
> > > but that is a current limitation of list-objects-filter API and not
> > > within the scope of this change.  We may in the longer term want to
> > > see if we can make the collection of filtered-out objects streamable
> > > by replacing the .omits object array with a callback function, or do
> > > something along that line.
> >
> > Hmm. I think it is possible to use something like `git pack-objects`'s
> > `--stdin-packs` mode to accomplish this without needing to keep track of
> > the set of discarded objects (i.e. those which don't match the filter).
> >
> > IIUC, the set of objects which don't match the filter is the same as the
> > set of all objects in packs beforehand, differenced with the set of
> > objects that shows up in the pack containing objects which *do* match
> > the filter.
> >
> > If you mark all of the "before" packs with `-` in the input to
> > `--stdin-packs`, and then pass along the pack containing the filtered
> > set without `-` (to indicate that the resulting pack should not contain
> > any objects which appear in that pack), I think you would end up with
> > the set of non-matching objects.
>
> I agree that it can be done like this, but I am not sure it's very
> efficient to do it like this. When we create the pack with filtered
> out objects, we know the set of objects we filtered out, so it doesn't
> seem efficient to make `git pack-objects --stdin-packs` read more
> packfiles or their indexes than necessary and compute that set of
> objects again.

Discovering that an object appears in one of the packs whose objects
we're excluding from the resulting pack is extremely cheap, since we
have a cache of just those packs (see the call to
`has_object_kept_pack()` in `want_found_object()`).

I'd think that in small cases the performance is probably worse, though
the working set would be small enough for the differences to not matter
that much in absolute terms.

In cases with many objects, storing all of their OIDs in memory would be
a pain.

Thanks,
Taylor