Re: [PATCH] [RFC] list-objects-filter: introduce new filter sparse:buffer=<spec>

[Derrick Stolee <derrickstolee@github.com>]

On 8/8/2022 12:15 PM, Junio C Hamano wrote:
> "ZheNing Hu via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> From: ZheNing Hu <adlternative@gmail.com>
>>
>> Although we already had a `--filter=sparse:oid=<oid>` which
>> can used to clone a repository with limited objects which meet
>> filter rules in the file corresponding to the <oid> on the git
>> server. But it can only read filter rules which have been record
>> in the git server before.
> 
> Was the reason why we have "we limit to an object we already have"
> restriction because we didn't want to blindly use a piece of
> uncontrolled arbigrary end-user data here?  Just wondering.

One of the ideas here was to limit the opportunity of sending an
arbitrary set of data over the Git protocol and avoid exactly the
scenario you mention.

Another was that it is incredibly expensive to compute the set of
reachable objects within an arbitrary sparse-checkout definition,
since it requires walking trees (bitmaps do not help here). This
is why (to my knowledge) no Git hosting service currently supports
this mechanism at scale. At minimum, using the stored OID would
allow the host to keep track of these pre-defined sets and do some
precomputing of reachable data using bitmaps to keep clones and
fetches reasonable at all.

The other side of the issue is that we do not have a good solution
for resolving how to change this filter in the future, in case the
user wants to expand their sparse-checkout definition and update
their partial clone filter.

There used to be a significant issue where a 'git checkout'
would fault in a lot of missing trees because the index needed to
reference the files outside of the sparse-checkout definition. Now
that the sparse index exists, this is less of an impediment, but
it can still cause some pain.

At this moment, I think path-scoped filters have a lot of problems
that need solving before they can be used effectively in the wild.
I would prefer that we solve those problems before making the
feature more complicated. That's a tall ask, since these problems
do not have simple solutions.

Thanks,
-Stolee