Re: [SCRIPT] cg-rpush & locking

[Dan Holmsand <holmsand@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1658 bytes --]

Tony Lindgren wrote:
> Anybody have any better ideas for locking that also works with
> rsync?

Since this seems to be Push Day on the git list, here's my feeble 
attempt at the same thing (attached cg-push script).

I do stuff in this order:

     1. read remote HEAD
     2. check that a merge from local HEAD to remote would be
        fast-forward, otherwise tell people to pull and merge.
     3. push objects using --ignore-existing (or equivalent)
     4. write lock file with --ignore-existing. The lock file
        contains, in particular, the HEAD to be written.
     5. read remote HEAD (again) and the lock file. Bail if HEAD
        changed since the first read, or if the lock file isn't
        the one we attempted to write.
     6. write remote HEAD and delete lock file in using rsync's
        --delete-after

This should always be safe, since rsync (as I understand the man page) 
always writes to temp files, and then renames into place. Checking for 
fast-forward-mergability assures that other peoples changes don't get lost.

cg-push determines uri and foreign branch name using the same rules as 
cg-pull, which is real nice as it allows you to do:

$ cg-clone me@myserver.example.com:git-repos/myrepo.git mystuff
$ cd mystuff
# write some stuff
$ cg-commit
$ cg-push

and later

$ cg-update
# write more stuff
$ cg-commit
$ cg-push

and even later

$ cg-branch-add pub me@myserver.example.com:public-repos/myrepo.git
$ cg-push pub

So, you can work pretty much exactly as you would do in CVS or svn, if 
you're so inclined, safely sharing a common repository among many users.

Which is kinda neat, if I may say so myself...

/dan

[-- Attachment #2: cg-push --]
[-- Type: text/plain, Size: 6219 bytes --]

#! /usr/bin/env bash
#
# Push changes to a remote git repository
#
# Copyright (c) Dan Holmsand, 2005.
#
# Based on cg-pull,
# Copyright (c) Petr Baudis, 2005.
#
# Takes the branch name as an argument, defaulting to "origin" (see
# `cg-branch-add` for some description of branch names).
#
# Takes one optional option: --force, that makes cg-push write objects
# regardless of lock-files and remote state. Use with care...
#
# cg-push supports two types of location specifiers:
#
# 1. local paths - simple directory names of git repositories
# 2. rsync - using the "[user@]machine:some/path" syntax
#
# Typical use would look something like this:
#
#	# clone a remote branch:
#	cg-clone me@myserver.example.com:repo.git myrepo
#	cd myrepo
#
#	# make some changes, and then do:
#	cg-commit
#	cg-push
#	
# cg-push is safe to use even if multiple users concurrently push
# to the same repository. Here's how this works:
#
# First, cg-push checks that the local repository is fully merged with
# the remote one (this will always be the case if there is only one
# user). Otherwise, you need to cg-update from the remote repository
# before you can cg-push to it.
#
# Then, cg-push writes all the object files that are missing in the
# remote repository.
#
# To finish, cg-push writes a lock file to the remote site (ignoring
# any preexisting lock file), checks that our lock file actually got 
# written, checks that the remote head is still the same (i.e. that 
# the remote site hasn't been updated while we were copying objects), 
# writes the new remote head and removes the lock.
#
# The head of the local repository is also updated in the process (as if
# the remote branch had been cg-pull'ed).
#
# cg-push requires that there already is a repository in place at the
# remote location, but it actually only checks that it has a 
# "refs/heads" subdirectory. So, creating a remote repo ready for
# cg-pushing is as easy "mkdir -p repo.git/heads/refs" at the remote
# location.

# TODO: Write tags as well.

. ${COGITO_LIB}cg-Xlib

force=
if [ "$1" = --force ]; then
	force=1; shift
fi

name=$1
[ "$name" ] || { [ -s $_git/refs/heads/origin ] && name=origin; }
[ "$name" ] || die "what to push to?"
uri=$(cat "$_git/branches/$name" 2>/dev/null) || die "unknown branch: $name"

rembranch=master
if echo "$uri" | grep -q '#'; then
	rembranch=$(echo $uri | cut -d '#' -f 2)
	uri=$(echo $uri | cut -d '#' -f 1)
fi

case $uri in
	*:*)
	readhead=rsync_readhead
	writeobjects=rsync_writeobjects
	writehead=rsync_writehead
	lock=rsync_lock
	;;
	*)
	if [ -d "$uri" ]; then
		[ -d "$uri/.git" ] && uri=$uri/.git
		readhead=local_readhead
		writeobjects=local_writeobjects
		writehead=local_writehead
		lock=local_lock
	else
		die "Don't know how to push to $uri"
	fi
	;;
esac

tmpd=$(mktemp -d -t cgpush.XXXXXX) || exit 1
trap "rm -rf $tmpd" SIGTERM EXIT

cid=$(commit-id) || exit 1
lock_msg="locked by $USER@$HOSTNAME on $(date) for writing $cid"
unset locked remhead

rsync_readhead() {
	rm -f "$tmpd/*" || return 1
	rsync $RSYNC_FLAGS --include="$rembranch" --include="$rembranch.lock" \
		--exclude='*' -r "$uri/refs/heads/" "$tmpd/" >&2 || 
		die "Fetching heads from $uri failed. Aborting."
	if [ "$locked" ]; then
		[ -s "$tmpd/$rembranch.lock" ] ||
		die "Couldn't acquire lock. Aborting."

		local rem_lock_msg=$(cat "$tmpd/$rembranch.lock")
		[ "$lock_msg" = "$rem_lock_msg" ] ||
		die "Remote is locked ($rem_lock_msg)."
	fi
	[ ! -e "$tmpd/$rembranch" ] || cat "$tmpd/$rembranch"
}

rsync_writeobjects() {
	[ -d "$_git/objects/" ] || die "no objects to copy"
	rsync $RSYNC_FLAGS -vr --ignore-existing --whole-file \
		"$_git/objects/" "$uri/objects/" 
}

rsync_lock() {
	echo "$lock_msg" > $tmpd/new_head_lock_file || return 1
	rsync $RSYNC_FLAGS --ignore-existing --whole-file \
		$tmpd/new_head_lock_file "$uri/refs/heads/$rembranch.lock"
}

rsync_writehead() {
	local heads=$tmpd/newhead
	mkdir $heads && echo "$1" > "$heads/$rembranch" || return 1
	rsync $RSYNC_FLAGS --include="$rembranch" --include="$rembranch.lock" \
		--exclude='*' --delete-after -r $heads/ "$uri/refs/heads/" 
}

local_readhead() {
	local lheads=$uri/refs/heads
	[ -d "$lheads" ] || die "no remote heads found at $uri"
	[ ! -e "$lheads/$rembranch" ] || cat "$lheads/$rembranch" 
}

local_writeobjects() {
	[ -d "$_git/objects/" ] || die "no objects to copy"
	[ -d "$uri/objects" ] || 
		GIT_DIR=$uri GIT_OBJECT_DIRECTORY=$uri/objects git-init-db ||
		die "git-init-db failed"
        # Note: We could use git-local-pull here, but this is safer
	# (git-*-pull don't react well to failures or kills), and
	# has the same semantics as rsync pushing.
	local dest=$(cd "$uri/objects" && pwd) || exit 1
	( cd "$_git/objects" && find -type f | while read f; do
		[ -f "$dest/$f" ] && continue
		ln "$f" "$dest/$f" 2>/dev/null || 
		cp "$f" "$dest/$f" || exit 1
	done ) 
}

local_lock() {
	([ "$force" ] || set -C 
	echo "$lockmsg" > "$uri/refs/heads/$rembranch.lock") 2>/dev/null
}

local_writehead() {
	local head=$uri/refs/heads/$rembranch
	echo "$1" > "$head.new" && mv "$head.new" "$head" &&
	rm "$head.lock"
}

echo "Checking remote repository"
remhead=$($readhead) || exit 1
[ "$remhead" ] || echo "Creating new branch"

if [ "$remhead" -a -z "$force" ]; then
	if [ "$remhead" = "$cid" ]; then
		echo "Remote branch \`$name' is already pushed" 
		exit 0
	fi
	git-cat-file commit "$remhead" &> /dev/null ||
	die "You need to pull from $name first. Aborting."

	base=$(git-merge-base "$remhead" "$cid") && [ "$base" ] ||
	die "You need to merge $name. Aborting."

	if [ "$base" = "$cid" ]; then
		echo "No changes to push"; exit 0
	fi

	[ "$base" = "$remhead" ] || 
	die "You need to merge $name first. Aborting." 
fi

echo "Writing objects"
$writeobjects || die "Failed to write objects. Aborting."

echo
echo "Writing new head"
$lock || die "Couldn't acquire lock on remote. Aborting."

if [ ! "$force" ]; then
	locked=1
	remhead2=$($readhead) || die "Aborting."
	[ "$remhead" = "$remhead2" ] || 
		die "Remote head changed during copy. Aborting."
fi

$writehead "$cid" || die "WARNING: Error writing remote head. Aborting."
echo "Push to $name succeeded"

echo "$cid" > "$_git/refs/heads/$name"
echo "Updated local head for $name to $cid"