Do porcelain command require lock management?

Do porcelain command require lock management?

[Konstantin Ryabitsev]

Hello:

A script I'm writing performs a succession of porcelain commands to 
create a commit in a bare git repository:

git hash-object
git mktree
git commit-tree
git update-ref

Do I need to manage external locking around these commands to avoid any 
concurrency problems, or will git take care of that?

-K

Re: Do porcelain command require lock management?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

On 2020-10-24 at 14:46:37, Konstantin Ryabitsev wrote:
> Hello:
> 
> A script I'm writing performs a succession of porcelain commands to
> create a commit in a bare git repository:
> 
> git hash-object
> git mktree
> git commit-tree
> git update-ref
> 
> Do I need to manage external locking around these commands to avoid any
> concurrency problems, or will git take care of that?

I'm almost certain that Git will do the same locking and object creation
semantics that it does in porcelain commands as in the plumbing commands
you're using.  For example, I happen to know that all loose object
creation goes through one function, which should gracefully handle
concurrent accesses.  Git is in general safe against concurrent accesses
and is designed not to lose or corrupt data in this case.

However, I will point out that ref updates may conflict and if so, Git
will fail instead of waiting.  So while your repository will remain
consistent and won't experience corruption, that doesn't mean that all
operations will complete successfully.  Some sort of retry mechanism or
other error handling will probably be warranted.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Do porcelain command require lock management?

[Konstantin Ryabitsev]

On Sun, Oct 25, 2020 at 11:02:03PM +0000, brian m. carlson wrote:
> > A script I'm writing performs a succession of porcelain commands to
> > create a commit in a bare git repository:
> > 
> > git hash-object
> > git mktree
> > git commit-tree
> > git update-ref
> > 
> > Do I need to manage external locking around these commands to avoid any
> > concurrency problems, or will git take care of that?
> 
> I'm almost certain that Git will do the same locking and object creation
> semantics that it does in porcelain commands as in the plumbing commands
> you're using.  For example, I happen to know that all loose object
> creation goes through one function, which should gracefully handle
> concurrent accesses.  Git is in general safe against concurrent accesses
> and is designed not to lose or corrupt data in this case.
> 
> However, I will point out that ref updates may conflict and if so, Git
> will fail instead of waiting.  So while your repository will remain
> consistent and won't experience corruption, that doesn't mean that all
> operations will complete successfully.  Some sort of retry mechanism or
> other error handling will probably be warranted.

Thanks for your advice. I think in this case it's easier to just use 
simple locking to make sure that concurrent processes don't step on 
each-other's toes.

Best regards,
-K

Re: Do porcelain command require lock management?

[Junio C Hamano]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:

> A script I'm writing performs a succession of porcelain commands to 
> create a commit in a bare git repository:

Meaning 'plumbing' commands?

>
> git hash-object
> git mktree

These take input and create a new object or if the data fed to them
were somehow identical to an existing object, becomes a no-op.

We take the data, hash to compute its name while writing it out as a
loose object to a temporary file.  After finishing this
compute/write we can check if we already have the object and if so
we can just clean up and leave.  There is no race in this case.  If
there is no existing object, we rename the temporary file to the
filename derived from the object name.  If somebody else were
creating exactly the same object at the same time, the renaming to
the final name will fail for all but one of them, but in the end
everybody gets the object they wanted to see in the repository
unless you found a hash collision.  These other "failing" folks
would probably need to "redo" the operation, but when they do so,
their compute/writes would safely become no-op.

> git commit-tree

This may write more than one objects; compute/write for each object
competes with compute/write by other processes independently, but
the overall story is the same.

> git update-ref

You can use the "old value MUST BE this" variant to avoid a race
with another process that tries to update the same ref to a
different value, but if the other process updates the same ref to a
different value _after_ you are done, their update will overwrite
what you did, so they also need to be using the same "old value MUST
BE this" variant.