Modifying a bare repo directly

Modifying a bare repo directly

[Julian Goacher]

Hi -

Is it possible to modify a bare repo directly? e.g. is it possible to
insert a file into a bare repo without first cloning a non-bare copy?
I'm thinking along the lines of a command or sequence of commands that
modifies the file index and then copies the file blob into /objects,
but in a situation where the new file exists separately from the
target repo.

Thanks -

j/

Re: Modifying a bare repo directly

[Ævar Arnfjörð Bjarmason]

On Mon, Apr 10, 2017 at 12:09 PM, Julian Goacher
<julian.goacher@gmail.com> wrote:
> Hi -
>
> Is it possible to modify a bare repo directly? e.g. is it possible to
> insert a file into a bare repo without first cloning a non-bare copy?
> I'm thinking along the lines of a command or sequence of commands that
> modifies the file index and then copies the file blob into /objects,
> but in a situation where the new file exists separately from the
> target repo.

Yes you use the plumbing commands (see "man git", search for plumbing), e.g.:

git init --bare mybare.git
cd mybare.git
echo hello | git hash-object --stdin -w >obj
$ printf "100644 blob $(cat obj)\thello.txt\n" | git mktree
aaa96ced2d9a1c8e72c56b253a0e2fe78393feb7
$ git commit-tree -m "1st commit" aaa96ced2d9a1c8e72c56b253a0e2fe78393feb7
318448647ab7a2b1f78c87cb8a05ac0cf172fbb8
$ git show 318448647ab7a2b1f78c87cb8a05ac0cf172fbb8
commit 3184486
Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date:   Mon Apr 10 12:29:22 2017 +0200

    1st commit

diff --git a/hello.txt b/hello.txt
new file mode 100644
index 0000000..ce01362
--- /dev/null
+++ b/hello.txt
@@ -0,0 +1 @@
+hello
$ git update-ref refs/heads/master 318448647ab7a2b1f78c87cb8a05ac0cf172fbb8

Now when you clone this repo you'll have that one commit.

Re: Modifying a bare repo directly

[Jeff King]

On Mon, Apr 10, 2017 at 12:33:18PM +0200, Ævar Arnfjörð Bjarmason wrote:

> On Mon, Apr 10, 2017 at 12:09 PM, Julian Goacher
> <julian.goacher@gmail.com> wrote:
> > Hi -
> >
> > Is it possible to modify a bare repo directly? e.g. is it possible to
> > insert a file into a bare repo without first cloning a non-bare copy?
> > I'm thinking along the lines of a command or sequence of commands that
> > modifies the file index and then copies the file blob into /objects,
> > but in a situation where the new file exists separately from the
> > target repo.
> 
> Yes you use the plumbing commands (see "man git", search for plumbing), e.g.:
> 
> git init --bare mybare.git
> cd mybare.git
> echo hello | git hash-object --stdin -w >obj
> $ printf "100644 blob $(cat obj)\thello.txt\n" | git mktree
> aaa96ced2d9a1c8e72c56b253a0e2fe78393feb7
> $ git commit-tree -m "1st commit" aaa96ced2d9a1c8e72c56b253a0e2fe78393feb7
> 318448647ab7a2b1f78c87cb8a05ac0cf172fbb8

This is definitely the right track, but if you're going to be modifying
an existing tree (and not just creating a new one), I'd suggest using
using a temporary index:

  export GIT_INDEX_FILE=my-index
  parent=$(git rev-parse HEAD)
  git read-tree $parent
  obj=$(echo hello | git hash-object --stdin -w)
  git update-index --add --cacheinfo 100644,$obj,hello.txt
  [...and other index changes if you want...]
  tree=$(git write-tree)
  commit=$(git commit-tree -p $parent $tree)
  git update-ref HEAD $commit $parent
  rm -f $GIT_INDEX_FILE

That's a bit simpler because it handles modifications to existing trees
(rather than overwriting), and it handles changes to nested subtrees (to
edit "deep/subdir/hello.txt" using mktree, you have to issue three
separate mktree commands).

-Peff

Re: Modifying a bare repo directly

[Eric Wong]

Julian Goacher <julian.goacher@gmail.com> wrote:
> Is it possible to modify a bare repo directly? e.g. is it possible to
> insert a file into a bare repo without first cloning a non-bare copy?
> I'm thinking along the lines of a command or sequence of commands that
> modifies the file index and then copies the file blob into /objects,
> but in a situation where the new file exists separately from the
> target repo.

Yes.  You can use the commands Ævar and Jeff talked about as
(those are probably the easiest).  "git fast-import" is also
great to avoid using an index entirely; it can be much faster
for mass modifications.

Re: Modifying a bare repo directly

[Julian Goacher]

Thanks for the replies; is there anything that needs to be known about
concurrent updates on a repo when using plumbing commands?

On 10 April 2017 at 19:22, Eric Wong <e@80x24.org> wrote:
> Julian Goacher <julian.goacher@gmail.com> wrote:
>> Is it possible to modify a bare repo directly? e.g. is it possible to
>> insert a file into a bare repo without first cloning a non-bare copy?
>> I'm thinking along the lines of a command or sequence of commands that
>> modifies the file index and then copies the file blob into /objects,
>> but in a situation where the new file exists separately from the
>> target repo.
>
> Yes.  You can use the commands Ævar and Jeff talked about as
> (those are probably the easiest).  "git fast-import" is also
> great to avoid using an index entirely; it can be much faster
> for mass modifications.

Re: Modifying a bare repo directly

[Jeff King]

On Tue, Apr 11, 2017 at 10:35:17PM +0100, Julian Goacher wrote:

> Thanks for the replies; is there anything that needs to be known about
> concurrent updates on a repo when using plumbing commands?

Concurrent writes to the object database are always safe (so
hash-object, write-tree, commit-tree are fine). Updates to the index
will take a lock on the index; use a unique GIT_INDEX_FILE if you want
multiple independent updates to avoid conflicting with each other.

So all of that should be able to proceed independently. The place where
you might see conflicts is when you actually update the branch. The
locking there uses a "lease" system:

  1. At the start of the operation, you note that the branch is at some
     sha1 X.

  2. When you're ready to write, you feed update-ref the refname, the
     new sha1, and that original sha1.

  3. Update-ref takes a lock, checks that we are still at the old sha1,
     and then atomically writes the new sha1.

So if you have updates to multiple independent branches, they'll never
conflict. If you have multiple updates to the same branch, they may
race, but you'll never "lose" an update; each one will either succeed or
fail. Just make sure you pass the "old" sha1 to update-ref (the example
I posted did so).

-Peff