[TOPIC 4/8] Commit `--filter`'s

[Taylor Blau <me@ttaylorr.com>]

# git clone --filter=commit:0 (jonathantanmy)

- Partial clone that can omit commits (we already support trees and
	blobs)
- Pros:
	 - Don't need all commits, save network and disk I/O. There's a repo
		 at Google that grows so quickly that having just commits is too
		 much
- Cons
	 - Git assumes that all of the commits are present locally; very
		 pervasive assumption.
			- Blobs don't have outlinks, not a problem. Tree depth is somewhat
				limited. Commits go all the back to the beginning of the repo
			-  `git bisect` without commits
			- Lose out on optimizations like `fetch` skipping negotiator,
				commit graph generation numbers
- Has everyone else thought about this?
- Peff: Compare to shallow clone (create a ‘graft' and pretend that the
	commit has no more parents). How do we handle the continuous N + 1
	fetching?  Jonathantanmy: Not a big issue, we can batch fetch. It's
	jumping around that's a problem
- Peff: What if the server sends the commit graph?
	 - Taylor: we could just send the generation number(s) of the parents
		 of the commits on the boundary of what you're sending.
	 - Emily: We can't verify it though, we'd have to just trust the
		 server
	 - Taylor: true, but that's the case even if you send the whole commit
		 graph, too
- Jrn: Partial clone - we know the server is there, so we still have a
	"full clone", but part of the "full clone" lives in the server. There
	are git services that don't need a full copy of the repo, e.g. for CI,
	we only need a view of the directories we're building.
- Two use cases for partial clone
	 - Shallow clone replacement: user only cares about a single commit
	 - Operation that involves history walking (e.g. git describe). We
		 might as well fetch all the commits (i.e., convert to tree or blob
		 filtering when we notice this operation). Are these operations
		 distinguishable?
- Rdamazio: What if all of the history walking happens only on the
	server? (e.g.  git blame). Jrnieder: For git blame specifically, that
	makes sense, but are you thinking of other things like git log?
	Rodrigo: Yes
- Johannes: That doesn't sound like it will scale. Stolee: At GitHub, we
	already do run blame on the server Rodrigo: At google, we precompute
	that
- Terry: More and more things want "stateless" operations (don't care
	about history) - that's probably the majority of use cases. There's
	also a popular use case of "1 week/month" of history. It would be
	great to not pay the penalty of fetching all commits. Today, we only
	have shallow clone, which pretends that history is different from what
	it actually is, and it's very difficult to maintain this on the server
	(sending not enough objects, sending too many objects). But filters
	are much easier to maintain.
- Victoria: Is this a replacement for shallow clones then? Terry: Yes.
- Stolee: 2 technical areas of apprehension
	 - VFS for git tried to do this by having only the initial commit and
		 fetching later objects one-by-one. Didn't work at all, was very
		 slow.
	 - Treeless clones - when traversing the tree, we keep refetching the
		 tree when we traverse it.
- We would have to drastically rework how GIt interacts with partial
	clones
- Taylor: Or we could teach the server to preempt the operations ("i'm
	going to run git log, send me the right things")
	 - Stolee: Or run it on the server
	 - Taylor: yes, that would be the other extreme approach to this
- Jrnieder: With treeless clones, we don't propagate the filter on the
	catch up fetch, and there are some code locations that assume that if
	we have a tree, we have all of its children. If anything, commit
	filters are even easier because we have nothing - so we can do "all or
	nothing"
	 - Stolee: I agree that it's simpler, but I still think it'll be
		 really slow.  So either, we need to do something much smarter than
		 object-by-object fetches, or to prevent users from running
		 problematic commands. We would eventually have to fix the problem
		 for treeless clones, so what if we start with full commit history,
			 but not all of the trees. We can fix that first before starting
			 on commit filters
	 - Jrnieder: I can see the need for all of the commits up until a
		 certain point in time, but I don't know if there's a need to solve
		 the general problem of omitting arbitrary commits e.g. jumping
		 around in bisect
- Rodrigo: We have some experience for doing this with Mercurial at
	Google - we hide the full history, users know they exist, but they can
	refetch if they wish. Stolee/Peff: That sounds like reimplementing
	shallow clone.
- Taylor: Is there any other kind of filter other than commit:0?
	Jonathantanmy: No plans yet.
- Peff: Wouldn't you need to implement the general case to do batching
	of commits in "git log"? Jonathantanmy: Maybe not, we could e.g. reuse
	the shallow clone protocol.  Managing ever growing pack index sizes on
	servers
- Some repositories have over 15 years of history with 1000 active
	developers, so pack indices can be between 1 and 2 GB. "GC pack"
	contains everything reachable from refs/heads/- and refs/tags/-
- Time-based slicing for repositories to allow smaller repositories.
	"Remove" history from before a certain point. Done by taking a shallow
	clone and using that as the new repository.
- What about folks who are only interested in the last week's history?
- Pack repositories based on time-based slicing.  Moving back to older
	history can fall back to older packs as necessary.
- Some people, like documentation folks, don't need the entire history
	and might be fine with a more limited environment.
- Chromium packs to three packs: one is a cruft/garbage pack, and the
	other are reachable objects.  refs/heads is packed into one pack, and
	refs/changes (PR-equivalent) are in the other.
- JGit doesn't have a reverse index yet
- Taylor: considering packing reverse index into main index.  The
	tension is that we need to make using multiple packs more flexible.
	Introduce bitmap chains when repacking to make things more stable and
	less expensive.
- Stolee: Consider older packs that are stable and only repack newer
	things.
- Peff: One reason not to have lots of packs on disk is missing out on
	deltas.  We could use thin packs on disk.
- Stolee: Future goal is to only include full delta chains in the stable
	packs.
- git gc –aggressive used to make really deep deltas and has been fixed
	to be less aggressive to avoid runtime performance costs.  Between 10
	and 50 shows real performance improvements but the old default was
	like 200.
- The original numbers were picked randomly without measurement.
- Patrick: GitLab maintenance architecture is evolving.  Each push is
	incremental (repack into one pack) or full repack (everything into one
	pack with deltas).
- Stable ordering for determining preferred objects (SHA ordering is not
	suitable).