* Experimental TDB support for GIT REFS
@ 2014-06-27 21:30 Ronnie Sahlberg
2014-06-28 0:37 ` Shawn Pearce
0 siblings, 1 reply; 4+ messages in thread
From: Ronnie Sahlberg @ 2014-06-27 21:30 UTC (permalink / raw)
To: git@vger.kernel.org; +Cc: Michael Haggerty
List,
One of my ref transaction aims is to make define a stable public API
for accessing refs.
Once this is done I want to make it possible to replace the current
.git/refs/* model with a
different type of backend.
In my case I want to add support to store all refs and reflogs in a TDB database
but once we have a pluggable backend framework for refs, if someone
wants to store the refs
in a SQL database, that should be fairly trivial too.
There are a few series queued before this becomes possible, but is
anyone wants to test or play with my "git can use TDB database" you
can find an implementation of this at
https://github.com/rsahlberg/git/tree/backend-struct-tdb
Most everything works (except submodules) but the code needs more work.
It is also VERY LIKELY that the database schema WILL CHANGE so this is
only for testing
and you will need to blow the repo away and start all over again at some stage.
I.e. no production use.
To activate TDB support a new flag is added to the init-db and clone commands:
git init --tdb-repo-name=RONNIE .
This creates a new git repo and sets core.tdb_repo_name in the config.
This config variable tells git to dlopen and use libtdb instead of
using the files under .git/refs
repo-name=RONNIE is a string that identifies the repository and all
keys in the database is prefixed
with repo-name + '\0'
By default this will create the two database refs.tdb and logs.tdb in
the repositories .git directory
but it is also possible to store the databases somehwere else by
adding --tdb-repo-dir=/var/lib/git/
when creating the repo.
Since the keys in the database are prefixed with the repo name it is
even possible to store all the
refs for multiple independent repos in the same database :
- First repository
$ cd <somewhere>
$ git init-db --tdb-repo-name=MyFirstRepo --tdb-repo-dir=/var/lib/git
- Second repository
$ cd <somewhere else>
$ git init-db --tdb-repo-name=MySecondRepo --tdb-repo-dir=/var/lib/git
This can also be a solutions for platforms lacking case sensitive
filesystems where today two refs that only differ in case can not be
used.
The current prototype will still apply the naming collision rules that
the existing files backend has.
For example if you have a ref 'm' you can not create another ref 'm/foo'.
I left those checks as is in order to not break compatibility between
the TDB backend and the current Files backend.
Please feel free to test this out. And comment.
regards
ronnie sahlberg
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Experimental TDB support for GIT REFS
2014-06-27 21:30 Experimental TDB support for GIT REFS Ronnie Sahlberg
@ 2014-06-28 0:37 ` Shawn Pearce
2014-07-02 17:11 ` Ronnie Sahlberg
0 siblings, 1 reply; 4+ messages in thread
From: Shawn Pearce @ 2014-06-28 0:37 UTC (permalink / raw)
To: Ronnie Sahlberg; +Cc: git@vger.kernel.org, Michael Haggerty
On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:
> List,
>
> One of my ref transaction aims is to make define a stable public API
> for accessing refs.
> Once this is done I want to make it possible to replace the current
> .git/refs/* model with a
> different type of backend.
> In my case I want to add support to store all refs and reflogs in a TDB database
> but once we have a pluggable backend framework for refs, if someone
> wants to store the refs
> in a SQL database, that should be fairly trivial too.
>
> There are a few series queued before this becomes possible, but is
> anyone wants to test or play with my "git can use TDB database" you
> can find an implementation of this at
>
> https://github.com/rsahlberg/git/tree/backend-struct-tdb
Interesting! But the link 404s :(
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Experimental TDB support for GIT REFS
2014-06-28 0:37 ` Shawn Pearce
@ 2014-07-02 17:11 ` Ronnie Sahlberg
2014-07-02 22:34 ` Ronnie Sahlberg
0 siblings, 1 reply; 4+ messages in thread
From: Ronnie Sahlberg @ 2014-07-02 17:11 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git@vger.kernel.org, Michael Haggerty
On Fri, Jun 27, 2014 at 5:37 PM, Shawn Pearce <spearce@spearce.org> wrote:
> On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:
>> List,
>>
>> One of my ref transaction aims is to make define a stable public API
>> for accessing refs.
>> Once this is done I want to make it possible to replace the current
>> .git/refs/* model with a
>> different type of backend.
>> In my case I want to add support to store all refs and reflogs in a TDB database
>> but once we have a pluggable backend framework for refs, if someone
>> wants to store the refs
>> in a SQL database, that should be fairly trivial too.
>>
>> There are a few series queued before this becomes possible, but is
>> anyone wants to test or play with my "git can use TDB database" you
>> can find an implementation of this at
>>
>> https://github.com/rsahlberg/git/tree/backend-struct-tdb
>
> Interesting! But the link 404s :(
Yeah, sorry but there were issues :-( Issues solved now though and
refactoring done.
Please see
https://github.com/rsahlberg/git/tree/backend-struct-db
This branch adds a refs backend that communicates via a domain socket
to a refs daemon.
The refs daemon in turn interfaces with the actual database.
My branch contains one hack refs daemon that uses a TDB database for
the refs storage.
This daemon is only about 600 lines of code, most of which is
marshalling and socket handling and only a small amount of
TDB specific code.
This small daemon should make it easy for folks to add arbitrary
database interfaces easily.
Taking refsd-tdb.c and modifying it to instead attach to a SQL
database should only take a few hundred lines of changes or so.
To build the daemon and start it :
$ gcc refsd-tdb.c -o refsd-tdb -ltdb
<create /var/lib/git and /var/log/git>
$ ./refsd-tdb /var/lib/git/refs.socket /var/lib/git /var/log/git/refsd.log
Then you can inspect the database with
tdbdump /var/lib/git/refs.tdb
You can then create a repository that uses this database :
$ git init --db-repo-name=ROCC --db-socket=/var/lib/git/refs.socket .
Then further commands will operate on the refs tdb.
See this as a prototype to experiment with and see the direction of
the refs transaction and pluggable backend support.
There is a lot additional work and cleanup that needs to be done
before this will become production code.
(yes I know we should not do hand marshalling and unmarshalling for
the PDUs on the domain socket. We should use some lightweight encoding
library like XDR and rpcgen or similar.)
Please test, comment and have fun.
regards
ronnie sahlberg
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Experimental TDB support for GIT REFS
2014-07-02 17:11 ` Ronnie Sahlberg
@ 2014-07-02 22:34 ` Ronnie Sahlberg
0 siblings, 0 replies; 4+ messages in thread
From: Ronnie Sahlberg @ 2014-07-02 22:34 UTC (permalink / raw)
To: Shawn Pearce; +Cc: git@vger.kernel.org, Michael Haggerty
On Wed, Jul 2, 2014 at 10:11 AM, Ronnie Sahlberg <sahlberg@google.com> wrote:
> On Fri, Jun 27, 2014 at 5:37 PM, Shawn Pearce <spearce@spearce.org> wrote:
>> On Fri, Jun 27, 2014 at 2:30 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:
>>> List,
>>>
>>> One of my ref transaction aims is to make define a stable public API
>>> for accessing refs.
>>> Once this is done I want to make it possible to replace the current
>>> .git/refs/* model with a
>>> different type of backend.
>>> In my case I want to add support to store all refs and reflogs in a TDB database
>>> but once we have a pluggable backend framework for refs, if someone
>>> wants to store the refs
>>> in a SQL database, that should be fairly trivial too.
>>>
>>> There are a few series queued before this becomes possible, but is
>>> anyone wants to test or play with my "git can use TDB database" you
>>> can find an implementation of this at
>>>
>>> https://github.com/rsahlberg/git/tree/backend-struct-tdb
>>
>> Interesting! But the link 404s :(
>
>
> Yeah, sorry but there were issues :-( Issues solved now though and
> refactoring done.
>
> Please see
> https://github.com/rsahlberg/git/tree/backend-struct-db
>
> This branch adds a refs backend that communicates via a domain socket
> to a refs daemon.
> The refs daemon in turn interfaces with the actual database.
>
> My branch contains one hack refs daemon that uses a TDB database for
> the refs storage.
> This daemon is only about 600 lines of code, most of which is
> marshalling and socket handling and only a small amount of
> TDB specific code.
> This small daemon should make it easy for folks to add arbitrary
> database interfaces easily.
> Taking refsd-tdb.c and modifying it to instead attach to a SQL
> database should only take a few hundred lines of changes or so.
>
>
> To build the daemon and start it :
>
> $ gcc refsd-tdb.c -o refsd-tdb -ltdb
> <create /var/lib/git and /var/log/git>
> $ ./refsd-tdb /var/lib/git/refs.socket /var/lib/git /var/log/git/refsd.log
>
> Then you can inspect the database with
> tdbdump /var/lib/git/refs.tdb
>
>
> You can then create a repository that uses this database :
> $ git init --db-repo-name=ROCC --db-socket=/var/lib/git/refs.socket .
>
> Then further commands will operate on the refs tdb.
>
> See this as a prototype to experiment with and see the direction of
> the refs transaction and pluggable backend support.
> There is a lot additional work and cleanup that needs to be done
> before this will become production code.
>
> (yes I know we should not do hand marshalling and unmarshalling for
> the PDUs on the domain socket. We should use some lightweight encoding
> library like XDR and rpcgen or similar.)
>
>
> Please test, comment and have fun.
One enhancement to this could be to make it easier to use for trivial
users that do not want to share one database across multiple repos.
We could have something like
$ git init --db=tdb .
core.db=tdb could then mean that instead of connecting to a domain
socket for a shared daemon instead it would just
popen("refsd-<core.db> ...", "rw")
and automaticallt set the asguments ro refsd-tdb to point to the local
.git directory.
This would make the setup simpler for the trivial usecase since it
would avoid having to manuallt initializing a dedicated daemon before
the git commands will start working.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-07-02 22:34 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-06-27 21:30 Experimental TDB support for GIT REFS Ronnie Sahlberg
2014-06-28 0:37 ` Shawn Pearce
2014-07-02 17:11 ` Ronnie Sahlberg
2014-07-02 22:34 ` Ronnie Sahlberg
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).