git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: David Turner <dturner@twopensource.com>
To: git mailing list <git@vger.kernel.org>
Subject: RFC/Pull Request: Refs db backend
Date: Mon, 22 Jun 2015 20:50:56 -0400	[thread overview]
Message-ID: <1435020656.28466.8.camel@twopensource.com> (raw)

I've revived and modified Ronnie Sahlberg's work on the refs db
backend.  

The work is on top of be3c13e5564, Junio's "First batch for 2.5 cycle".
I recognize that there have been changes to the refs code since then,
and that there are some further changes in-flight from e.g. Michael
Haggerty.  If there is interest in this, I can rebase once Michael's
changes land.

The changes can be found here:
https://github.com/dturner-tw/git.git on the dturner/pluggable-backends
branch

The db backend code was added in the penultimate commit; the rest is
just code rearrangement and minor changes to make alternate backends
possible.  There ended up being a fair amount of this rearrangement, but
the end result is that almost the entire git test suite runs under the
db backend without error (see below for details).

The db backend runs git for-each-ref about 30% faster than the files
backend with fully-packed refs on a repo with ~120k refs.  It's also
about 4x faster than using fully-unpacked refs.  In addition, and
perhaps more importantly, it avoids case-conflict issues on OS X.

I chose to use LMDB for the database.  LMDB has a few features that make
it suitable for usage in git:

1. It is relatively lightweight; it requires only one header file, and
the library itself is under 300k (as opposed to 700k for
e.g. sqlite).

2. It is well-tested: it's been used in OpenLDAP for years.

3. It's very fast.  LMDB's benchmarks show that it is among
the fastest key-value stores.

4. It has a relatively simple concurrency story; readers don't
block writers and writers don't block readers.

Ronnie Sahlberg's original version of this patchset used tdb.  The
advantage of tdb is that it's smaller (~125k).  The disadvantages are
that tdb is hard to build on OS X.  It's also not in homebrew.  So lmdb
seemed simpler.

To test this backend's correctness, I hacked test-lib.sh and
test-lib-functions.sh to run all tests under the refs backend. Dozens
of tests use manual ref/reflog reading/writing, or create submodules
without passing --refs-backend-type to git init.  If those tests are
changed to use the update-ref machinery or test-refs-be-db (or, in the
case of packed-refs, corrupt refs, and dumb fetch tests, are skipped),
the only remaining failing tests are the git-new-workdir tests and the
gitweb tests.  

Please let me know how it would be best to proceed. 

             reply	other threads:[~2015-06-23  0:51 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-06-23  0:50 David Turner [this message]
2015-06-23  5:36 ` RFC/Pull Request: Refs db backend Junio C Hamano
2015-06-23 10:23   ` Duy Nguyen
2015-06-23 18:47     ` David Turner
2015-06-23 17:29   ` David Turner
2015-06-23 11:47 ` Jeff King
2015-06-23 13:10   ` Duy Nguyen
2015-06-24  8:51     ` Jeff King
2015-06-23 18:18   ` David Turner
2015-06-24  9:14     ` Jeff King
2015-06-24 17:29       ` David Turner
2015-06-24  6:09   ` Shawn Pearce
2015-06-24  9:49     ` Jeff King
2015-06-25  1:08       ` brian m. carlson
2015-06-24 10:18     ` Duy Nguyen
2015-06-23 15:51 ` Michael Haggerty
2015-06-23 19:53   ` David Turner
2015-06-23 21:27     ` Michael Haggerty
2015-06-24 17:31       ` David Turner
2015-06-23 21:35     ` David Turner
2015-06-23 21:41       ` Junio C Hamano
2015-06-23 17:16 ` Stefan Beller
2015-06-23 20:04   ` David Turner
2015-06-23 20:10     ` Randall S. Becker
2015-06-23 20:22       ` David Turner
2015-06-23 20:27         ` Randall S. Becker

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1435020656.28466.8.camel@twopensource.com \
    --to=dturner@twopensource.com \
    --cc=git@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).