[TOPIC 6/8] State of sparsity work

[Taylor Blau <me@ttaylorr.com>]

# State of sparsity developments and future plans (Victoria)

- (Victoria) Integrating commands with sparse index, making them
	compatible with sparse indexes, not un-sparsifying the index before
	executing themselves
- Have worked on some more recently
- GSoC student has worked on a handful as well
- Near-term future is about finding commands that need to touch the
	index, don't support sparse index, and then make them compatible
- In some cases, that is going to require expanding the index to be
	non-sparse, especially if you are touching something outside of the
	sparsity cone
	 - That is somewhat straightforward
- More interesting questions: what is the future of sparsity? Recently,
	Elijah pushed a change to make sparse-checkout's cone mode the default
	(nice, since it is required by sparse index)
- As we move forward, what should we change the defaults of?
	 - Sparse index for sparse checkouts in cone mode?
- Scalar as a testing ground for larger features, including sparse index
	 - Could make sparse index the default in Scalar for cone-mode sparse
		 checkouts, and then see how it goes
	 - Or, could just go for it sooner (after we have integrated sparse
		 index with enough commands)
- A handful of internal, logistical things that would have to happen for
	sparse index to become the default. Currently, commands are assumed to
	not work with the sparse index.
- Question for everybody: what is a good balance between pushing sparse
	index, and waiting to introduce it to more users by holding off on
	changing the default.
- (Stolee): sparse checkout and submodules became a difficulty when
	mentoring their GSoC student.
- (JTan): possible to decouple sparse index and cone-mode sparse
	checkouts from each other? This would be easy to test - turn it on,
	all of the test suite automatically uses it. Jrnieder: This sounds ok
	for the filesystem, but I don't know how this would work for this
		"VFS-backed Git" idea on the spreadsheet. (other things…)
- (Stolee): We need cone mode today because they're the only way to
	definitively say that we've reached the boundary. But we can also
	expand the idea of "cone" to allow more paths (files instead of
	directories) in the cone.
- (Taylor): What do we need to tell subcommands to assume that sparse
	index is supported? (Victoria) Gut feeling for the most part.
	(Jrnieder): I'd prefer this to happen sooner rather than later. This
	is easier for maintainability since we don't have to worry about
	commands being in two possible modes of operation. We can break these
	incompatible APIs by renaming them to prevent them from being misused
	by new commands.
- (Victoria): So just break things that always use the full index?
	Sounds ok.  (Stolee) This sounds similar to the_index macros, which
	we've tried to remove for the most part but we've stopped. Doing this
	conversion everywhere sounds extremely difficult - we've done an audit
	on this. (Jrnieder): Oh, I just meant renaming the API without
	changing semantics. Intentionally break everything.
- (Victoria): We'd need to write new tests for lots of commands because
	the existing tests don't actually interact with the "sparse" parts of
	the index.
- (Ævar): Is this just a matter of telling `git init` to initialize a
	sparse index? (Stolee): No, we need to force the tests to work on
	sparse directory entries.
- (Jrnieder): This sounds like a good fit for feature.experimental
- (VIctoria): Is sparse index a good git default instead of just "for
	large repos"? I'd think yes. (Jrnieder) Yes I think any sparse
	checkout user would want this. (Stolee): LIterally every command that
	touches the index has been converted (used for Microsoft Office
	monorepo), so it's just a matter of doing this for the whole project.
- (Elijah): I would like partial filters from sparse patterns.
	--filter=blob:none doesn't let you disconnect from the server.
	 - (Jrnieder) The DX of sparse checkout + blob:none has been pretty
		 good.  (Elijah): but you need to stay connected to the server.
		 (Jrnieder) Ah, thanks for explaining, sorry for the confusion
		 (Elijah) It would be great to have "sparse clone"s and have
		 commands that work just inside of that cone when disconnected. Make
		 "grep", "log", etc respect the sparse pattern
	 - (Stolee): We've thought about this, but it is very expensive on the
		 server side and makes bitmaps unusable. Alternatively, we could
		 start with blob:none and then backfill. That sounds more promising,
		 but that's not just a plain partial clone.
	 - (Jonathantanmy): FYI there's a protocol feature that already allows
		 clones to specify a sparse filter (referencing a blob with sparse
		 patterns that's present on the server), but I don't know of any
		 implementation that has this enabled.
	 - (Jrnieder): Can we delete this? (Github folks): We don't like it,
		 we invented the uploadpackfilter config to disable it :) Is this
		 just cleanup?  (Jrnieder): Yes, and this dead end will stop being a
		 distraction.
	 - (Peff) We could already implement most of the backfilling using
		 current commands, but that might skip over some delta-ing
		 optimizations. We could have a protocol change to provide the path
		 as a hint to the server.