Re: [RFC/WIP] Pluggable reference backends

[Shawn Pearce <spearce@spearce.org>]

On Mon, Mar 10, 2014 at 4:00 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:
> I have started working on pluggable ref backends.  In this email I
> would like to share my plans and solicit feedback.

Yay!

JGit already has pluggable ref backends, so it is good to see this
starting in git-core.

FWIW the Gerrit Code Review community is interested in this project.

> * Store references in a SQLite database, to get correct transaction
>   handling.

No to SQLLite in git-core. Using it from JGit requires building
SQLLite and a JNI wrapper, which makes JGit significantly less
portable. I know SQLLite is pretty amazing, but implementing
compatibility with it from JGit will be a big nightmare for us.

> * Reference transactions that can be used across multiple Git
>   commands.  Imagine,
>
>       export GIT_TRANSACTION=$(git transaction begin)
>       trap 'git transaction rollback' ERR
>       git foo ...
>       git bar ...
>       git baz ...
>       if ! git transaction commit
>       then
>           # Transaction failed; all references rolled back
>       else
>           # Transaction succeeded; all references updated atomically
>       fi
>       trap '' ERR
>       unset GIT_TRANSACTION
>
>   The "GIT_TRANSACTION" environment variable would tell git to read
>   from the usual references, overridden with any reference changes
>   that have occurred during the transaction, but write any changes
>   (including both old and new values) to the transaction.  The command
>   "git transaction commit" would verify that the old values listed in
>   the transaction still agree with the current values, and then make
>   all of the changes atomically.

Yay!

Gerrit Code Review really wants to get transactions implemented. So I
am very much in favor of trying to improve the situation in git-core.

We want not only a transaction over 2+ references in the same
repository, but we also want to perform transactions across
repositories. Consider a git submodule child and parent being updated
at the same time. We really want to update refs/heads/master in both
repositories atomically at the central server.

>   Such transactions could also be broadcast to mirrors when they are
>   committed to keep multiple Git repositories in sync.

Ooh, this would be very interesting.

> Git hosters [1] will be likely to take advantage of alternate
> reference backends pretty easily, because they know which tools touch
> their repositories and need only update those tools.  It is expected
> that alternate reference backends will be useful for hosters even if
> they don't become practical for end-users.

Alternate reference backends are absolutely useful to large hosters.
The loose reference format isn't very scalable. The packed-refs helps,
but you can do better. IIRC our android.googlesource.com reference
backend uses only 79 bytes per reference on average, including both
the name string and the value. This super compact format is easy to
hold in RAM for hundreds of busy repositories.

> For end-users it is important that their repository be readable by all
> of the tools that they use.  So if we want to make a new format a
> viable option for normal Git users (let alone make it the new default
> format), some coordination will be needed between all of the
> commonly-used Git implementations (git-core, libgit2, JGit, and maybe
> Dulwich, Grit, ...).  Whether or not this happens in real life depends
> on how advantageous the hypothetical new format is to Git users and is
> beyond the scope of this proposal.

It is sad we have this many implementations, but as one of the authors
(JGit) I am happy to at least see you are worrying about compatibility
with them.