[TOPIC 6/12] Clarifying backwards compatibility and when we break it

[Taylor Blau <me@ttaylorr.com>]

(Presenter: Emily Shaffer, Notetaker: Taylor Blau)

* (Emily) In the last year, there were a handful of scenarios where we had
  issues with backwards compatibility.
   * E.g. config based hooks. As a result of this change, text coloration was
     broken from user-provided hooks. Missing tests, but still viewed it as
     backwards-compatibility-breaking.
   * E.g. deleted internal "git submodule--helper" that looked like a plumbing
     command, which other projects depended on. Was being used in the wild, even
     though we didn't expect it.
   * E.g. bare repository embedding. Interested in fixing that as a security
     concern
     (https://offensi.com/2019/12/16/4-google-cloud-shell-bugs-explained-bug-3/,
     https://github.com/justinsteven/advisories/blob/main/2022_git_buried_bare_repos_and_fsmonitor_various_abuses.md).
     Weren't able to do so in a widespread fashion, since many projects are
     using it for testing setups (i.e. test fixtures).
* (Emily) When do we consider odd behavior as bugs, versus when do we consider
  them as something that's part of our backwards compatibility guarantee.
* (Emily) What do we want backwards compatibility to look like for libraries?
  How do we want to handle this in the future?
* (Minh) Is there documentation on how this should behave?
   * (Emily): Typically try to guarantee backwards compatibility via integration
     tests. Have changed documented behavior in the past when it seems "just
     wrong". Is the documentation the source of truth?
   * (Jonathan Nieder): In the case of browsers, using a specification for
     compatibility is a useful tool. What will work at the same time across
     different implementations? When there is a single implementation (e.g. git)
     it is easier to capture your intention with the implementation instead of a
     specification.
   * (Jonathan Nieder): Converting that documentation into a specification can
     hurt readability or inhibit its other uses.
   * (Junio): Tend to ensure that observable behavior is locked in via
     integration tests. Tests are the source of truth, along with
     implementation. Documentation is often lying. Unlike the browser example,
     we don't have an external specification to rely on. Intention from
     developers is captured in the log proposed message.
* (Minh) Should we be testing more, then?
   * (Emily) That's part of it, but some older behavior (e.g. from the original
     implementation) has less information in the commit message as a result of
     project culture at the time.
   * (Junio) Working-as-designed, but the design outlived its usefulness.
   * (Jonathan Nieder) E.g. ‘git-add' versus ‘git add'. Outcry after we changed
     behavior, so we rolled it back. Much later we got to a place where people
     weren't relying on this behavior as much.
* (Jonathan Nieder) There is another kind of documentation besides
  specification. E.g. the kernel has a documented guarantee about compatibility:
  "the kernel never breaks userspace". This doesn't mean that we can't have
  observable behavior changes. Only that they maintain "depended-upon" behavior
  that the kernel is reasonably responsible for providing. Can only determine
  this surface area by rolling out changes and seeing if folks complain.
* (Jonathan Nieder) It sometimes feels like we have adopted a similar
  philosophy, but the kernel has an easier job since POSIX, System V, etc have
  defined the overall shape of the syscall interface.
* (Elijah) Difficult to distinguish between bug fixes and breaking backwards
  compatibility. When we break existing test cases, document a rationale for
  doing so in the proposed patch message. Cases where documentation was just
  wrong. Often comes down to a judgment call.
* (Minh) Is the consensus to keep tests up-to-date, and add more tests when
  behavior is unclear?
* (Jonathan Nieder) Problem is that there can be differences of opinion on what
  are safe compatibility guarantees to break.
* (Emily) Also the case that there are tests that are in the integration suite
  that are enforcing things that weren't meant to be compatibility guarantees.
  E.g. enforcing error messages. How do we cope with legacy behavior and legacy
  tests when making a judgment call? There is some documentation in general
  about backwards compatibility, plumbing commands are frozen, porcelain
  commands are not. Should we expand that documentation to clarify how to
  decide?
* (brian) This would be helpful, but not sure what it would look like. Kernel's
  approach may be too rigid for Git. Sometimes useful to break backwards
  compatibility. E.g. "we have it, but it isn't a good choice." Users depend on
  those error messages. When we make a change that is overwhelmingly beneficial,
  can't please everybody all of the time.
* (Jonathan Nieder) Back to guarantees for library code. Kind of view the
  plumbing/porcelain decision as a failed experiment. Of course scripters are
  going to use the plumbing. Want a better backwards compatibility guarantee.
  People are going to want to add more functionality there and lock in
  additional behavior. When people write scripts, they write using the commands
  that they understand how to use. End-user commands gain for-scripting
  functionality.
* (Junio) Worse is when new features are added only to porcelain, and plumbing
  code is left behind.
* (Jonathan Nieder) In a way, we made it harder on ourselves. If porcelains are
  written as scripts, you need plumbing commands to expose the functionality
  they need. Now porcelains use function calls, so the well maintained interface
  is more on the (internal) library side
* Libification moves us in a good direction, since it provides an alternative to
  the CLI as a well-defined programmatic access method.
* (Jonathan Nieder) If we succeed at this, the command-line backwards
  compatibility guarantee for porcelain commands can break down a bit to the
  extent that users start to adopt the library code as their interface to Git.
* (Emily) If we have suitable replacements in the library, can we deprecate the
  plumbing variant of that functionality eventually? Freeze a particular
  plumbing command instead of adding to it
* (Taylor) Can't break existing behavior, shouldn't have to force users to
  upgrade to library code for existing behavior. Apologies if this is what you
  were saying.
* (Jakub) Auto-generated CLI shim, like cloud providers often provide for their
  APIs?
* (Jonathan Tan) Might be hard to create scriptable interfaces for library
  commands. Library allows us to pass pointers and function callbacks, neither
  of these we can accomplish via the shell.
* (Minh) Is there an understanding that the library has to implement 100% of the
  functionality of plumbing commands?
* (Emily) Not convinced that we need a one-to-one match between the library and
  command-line interface. Want to expose the same intent, not necessarily exact
  incantations.
* (Jonathan Nieder) Let me try and summarize. Question resonates with people, no
  one has a silver bullet. Maybe some agreement for using more tests, but the
  general approach to figuring out our compatibility guarantees remains an open
  discussion.
* (brian, via chat) One final thought: maybe we could look at what Semantic
  Versioning defines a breaking change as, since they've defined this in a very
  public way.
* (Phillip, via chat) Thinking back to yesterday there were people saying that
  they chose the cli over a library because of concerns about memory leaks and
  the library crashing/dying as well as licensing concerns. If we were to add
  new functionality only in libraries we'd need to make sure that they were
  robust.