[PATCH 0/2] update-ref: Allow creation of multiple transactions

[PATCH 0/2] update-ref: Allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1803 bytes --]

Hi,

A few months back, I've implemented a more interactive protocol for
`git-update-ref --stdin`, which converted it to read commands linewise
and allow for more control via a set of new controlling commands. One
thing that was noticeably missing was the ability to create multiple
transactions in a single git-update-ref invocation.

As Peff rightly pointed out a few months back, we're actually almost
there to have that. So I took a look today and finally decided to
implement it while I was busy implementing another series. And in fact,
it really didn't require a lof of changes.

So with this small series, it's now possible to start new transaction
after the current transaction has either been committed or aborted. E.g.
sessions like the following are now possible:

    > start
    start: ok
    > create refs/heads/test $A
    > commit
    commit: ok
    > start
    start: ok
    > update refs/heads/test $B $A
    > commit
    commit: ok
    > start
    start: ok
    > update refs/heads/test $A $B
    > abort
    abort: ok
    > start
    start: ok
    > delete refs/heads/test $B
    > commit
    commit: ok

I've also updated the performance test in p1400 to use this new
mechanism, which is a much more direct way to test what we want to test
in there compared to using not-atomic pushes.

Patrick

Patrick Steinhardt (2):
  update-ref: Allow creation of multiple transactions
  p1400: Use `git-update-ref --stdin` to test multiple transactions

 Documentation/git-update-ref.txt |  3 +-
 builtin/update-ref.c             | 13 ++++++++-
 t/perf/p1400-update-ref.sh       | 20 +++++--------
 t/t1400-update-ref.sh            | 50 ++++++++++++++++++++++++++++++++
 4 files changed, 71 insertions(+), 15 deletions(-)

-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH 1/2] update-ref: Allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3670 bytes --]

While git-update-ref has recently grown commands which allow interactive
control of transactions in e48cf33b61 (update-ref: implement interactive
transaction handling, 2020-04-02), it is not yet possible to create
multiple transactions in a single session. To do so, one currently still
needs to invoke the executable multiple times.

This commit addresses this shortcoming by allowing the "start" command
to create a new transaction if the current transaction has already been
either committed or aborted.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-update-ref.txt |  3 +-
 builtin/update-ref.c             | 13 ++++++++-
 t/t1400-update-ref.sh            | 50 ++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-update-ref.txt b/Documentation/git-update-ref.txt
index d401234b03..48b6683071 100644
--- a/Documentation/git-update-ref.txt
+++ b/Documentation/git-update-ref.txt
@@ -125,7 +125,8 @@ option::
 start::
 	Start a transaction. In contrast to a non-transactional session, a
 	transaction will automatically abort if the session ends without an
-	explicit commit.
+	explicit commit. This command may create a new empty transaction when
+	the current one has been committed or aborted already.
 
 prepare::
 	Prepare to commit the transaction. This will create lock files for all
diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 8a2df4459c..bb65129012 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -446,7 +446,18 @@ static void update_refs_stdin(void)
 			state = cmd->state;
 			break;
 		case UPDATE_REFS_CLOSED:
-			die("transaction is closed");
+			if (cmd->state != UPDATE_REFS_STARTED)
+				die("transaction is closed");
+
+			/*
+			 * Open a new transaction if we're currently closed and
+			 * get a "start".
+			 */
+			state = cmd->state;
+			transaction = ref_transaction_begin(&err);
+			if (!transaction)
+				die("%s", err.buf);
+
 			break;
 		}
 
diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 4c01e08551..72d995aece 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1526,4 +1526,54 @@ test_expect_success 'transaction with prepare aborts by default' '
 	test_path_is_missing .git/$b
 '
 
+test_expect_success 'transaction can commit multiple times' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/branch-1 $A
+	commit
+	start
+	create refs/heads/branch-2 $B
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_cmp expect actual &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/branch-1 >actual &&
+	test_cmp expect actual &&
+	echo "$B" >expect &&
+	git rev-parse refs/heads/branch-2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'transaction can create and delete' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/create-and-delete $A
+	commit
+	start
+	delete refs/heads/create-and-delete $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_path_is_missing .git/refs/heads/create-and-delete
+'
+
+test_expect_success 'transaction can commit after abort' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/abort $A
+	abort
+	start
+	create refs/heads/abort $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start abort start commit >expect &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/abort >actual &&
+	test_cmp expect actual
+'
+
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH 2/2] p1400: Use `git-update-ref --stdin` to test multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

In commit 0a0fbbe3ff (refs: remove lookup cache for
reference-transaction hook, 2020-08-25), a new benchmark was added to
p1400 which has the intention to exercise creation of multiple
transactions in a single process. As git-update-ref wasn't yet able to
create multiple transactions with a single run we instead used git-push.
As its non-atomic version creates a transaction per reference update,
this was the best approximation we could make at that point in time.

Now that `git-update-ref --stdin` supports creation of multiple
transactions, let's convert the benchmark to use that instead. It has
less overhead and it's also a lot clearer what the actual intention is.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/p1400-update-ref.sh | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/t/perf/p1400-update-ref.sh b/t/perf/p1400-update-ref.sh
index ce5ac3ed85..dda8a74866 100755
--- a/t/perf/p1400-update-ref.sh
+++ b/t/perf/p1400-update-ref.sh
@@ -7,13 +7,14 @@ test_description="Tests performance of update-ref"
 test_perf_fresh_repo
 
 test_expect_success "setup" '
-	git init --bare target-repo.git &&
 	test_commit PRE &&
 	test_commit POST &&
-	printf "create refs/heads/%d PRE\n" $(test_seq 1000) >create &&
-	printf "update refs/heads/%d POST PRE\n" $(test_seq 1000) >update &&
-	printf "delete refs/heads/%d POST\n" $(test_seq 1000) >delete &&
-	git update-ref --stdin <create
+	for i in $(test_seq 5000)
+	do
+		printf "start\ncreate refs/heads/%d PRE\ncommit\n" $i &&
+		printf "start\nupdate refs/heads/%d POST PRE\ncommit\n" $i &&
+		printf "start\ndelete refs/heads/%d POST\ncommit\n" $i
+	done >instructions
 '
 
 test_perf "update-ref" '
@@ -26,14 +27,7 @@ test_perf "update-ref" '
 '
 
 test_perf "update-ref --stdin" '
-	git update-ref --stdin <update &&
-	git update-ref --stdin <delete &&
-	git update-ref --stdin <create
-'
-
-test_perf "nonatomic push" '
-	git push ./target-repo.git $(test_seq 1000) &&
-	git push --delete ./target-repo.git $(test_seq 1000)
+	git update-ref --stdin <instructions >/dev/null
 '
 
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] update-ref: Allow creation of multiple transactions

[Jeff King]

On Wed, Nov 04, 2020 at 03:57:17PM +0100, Patrick Steinhardt wrote:

> While git-update-ref has recently grown commands which allow interactive
> control of transactions in e48cf33b61 (update-ref: implement interactive
> transaction handling, 2020-04-02), it is not yet possible to create
> multiple transactions in a single session. To do so, one currently still
> needs to invoke the executable multiple times.
> 
> This commit addresses this shortcoming by allowing the "start" command
> to create a new transaction if the current transaction has already been
> either committed or aborted.

Thanks for working on this. The amount of change needed is indeed quite
pleasant.

> diff --git a/Documentation/git-update-ref.txt b/Documentation/git-update-ref.txt
> index d401234b03..48b6683071 100644
> --- a/Documentation/git-update-ref.txt
> +++ b/Documentation/git-update-ref.txt
> @@ -125,7 +125,8 @@ option::
>  start::
>  	Start a transaction. In contrast to a non-transactional session, a
>  	transaction will automatically abort if the session ends without an
> -	explicit commit.
> +	explicit commit. This command may create a new empty transaction when
> +	the current one has been committed or aborted already.

Reading this made me wonder what would happen if we send a "start" when
the current one _hasn't_ been committed or aborted. I.e., what does:

  git update-ref --stdin <<EOF
  start
  create refs/heads/foo ...
  start
  commit
  EOF

do? It turns out that the second start is ignored totally (and the
commit does indeed update foo). I wonder if we ought to complain about
it. But that is completely orthogonal to your patch. The behavior is the
same before and after.

> --- a/builtin/update-ref.c
> +++ b/builtin/update-ref.c
> @@ -446,7 +446,18 @@ static void update_refs_stdin(void)
>  			state = cmd->state;
>  			break;
>  		case UPDATE_REFS_CLOSED:
> -			die("transaction is closed");
> +			if (cmd->state != UPDATE_REFS_STARTED)
> +				die("transaction is closed");
> +
> +			/*
> +			 * Open a new transaction if we're currently closed and
> +			 * get a "start".
> +			 */
> +			state = cmd->state;
> +			transaction = ref_transaction_begin(&err);
> +			if (!transaction)
> +				die("%s", err.buf);
> +

Very nice. This duplicates the state and transaction setup at the start
of the function, which made me wonder if we could do this:

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index bb65129012..140f0d30e9 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -385,14 +385,10 @@ static const struct parse_cmd {
 static void update_refs_stdin(void)
 {
 	struct strbuf input = STRBUF_INIT, err = STRBUF_INIT;
-	enum update_refs_state state = UPDATE_REFS_OPEN;
+	enum update_refs_state state = UPDATE_REFS_CLOSED;
 	struct ref_transaction *transaction;
 	int i, j;
 
-	transaction = ref_transaction_begin(&err);
-	if (!transaction)
-		die("%s", err.buf);
-
 	/* Read each line dispatch its command */
 	while (!strbuf_getwholeline(&input, stdin, line_termination)) {
 		const struct parse_cmd *cmd = NULL;

and just have it auto-open. But of course that doesn't work because we
might not see an "open" command at all. Traditional callers will start
with create/update/etc, and our "auto-open" would complain.

> +test_expect_success 'transaction can create and delete' '
> +	cat >stdin <<-EOF &&
> +	start
> +	create refs/heads/create-and-delete $A
> +	commit
> +	start
> +	delete refs/heads/create-and-delete $A
> +	commit
> +	EOF
> +	git update-ref --stdin <stdin >actual &&
> +	printf "%s: ok\n" start commit start commit >expect &&
> +	test_path_is_missing .git/refs/heads/create-and-delete
> +'

The tests all look quite reasonable to me. Touching .git/refs like this
is a bit gross (and something we may have to deal with if we introduce
reftables, etc). But it's pretty pervasive in this file, so matching
the existing style is the best option for now.

-Peff

Re: [PATCH 2/2] p1400: Use `git-update-ref --stdin` to test multiple transactions

[Jeff King]

On Wed, Nov 04, 2020 at 03:57:22PM +0100, Patrick Steinhardt wrote:

> In commit 0a0fbbe3ff (refs: remove lookup cache for
> reference-transaction hook, 2020-08-25), a new benchmark was added to
> p1400 which has the intention to exercise creation of multiple
> transactions in a single process. As git-update-ref wasn't yet able to
> create multiple transactions with a single run we instead used git-push.
> As its non-atomic version creates a transaction per reference update,
> this was the best approximation we could make at that point in time.
> 
> Now that `git-update-ref --stdin` supports creation of multiple
> transactions, let's convert the benchmark to use that instead. It has
> less overhead and it's also a lot clearer what the actual intention is.

Good direction. The diff confused me for a moment...

> @@ -26,14 +27,7 @@ test_perf "update-ref" '
>  '
>  
>  test_perf "update-ref --stdin" '
> -	git update-ref --stdin <update &&
> -	git update-ref --stdin <delete &&
> -	git update-ref --stdin <create
> -'
> -
> -test_perf "nonatomic push" '
> -	git push ./target-repo.git $(test_seq 1000) &&
> -	git push --delete ./target-repo.git $(test_seq 1000)
> +	git update-ref --stdin <instructions >/dev/null
>  '

...because we're dropping _two_ tests here. But I think they were
testing the same thing, just with varying degrees of quality.

It could possibly be useful to have perf results broken down by
operation type (create vs delete vs update), but the original certainly
didn't do that. And it's not clear to me it would actually produce
interesting results; certainly not related to the hook, but possibly
related to benchmarking ref updates in general. So I don't think it's
worth worrying about.

-Peff

Re: [PATCH 1/2] update-ref: Allow creation of multiple transactions

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>> +test_expect_success 'transaction can create and delete' '
>> +	cat >stdin <<-EOF &&
>> +	start
>> +	create refs/heads/create-and-delete $A
>> +	commit
>> +	start
>> +	delete refs/heads/create-and-delete $A
>> +	commit
>> +	EOF
>> +	git update-ref --stdin <stdin >actual &&
>> +	printf "%s: ok\n" start commit start commit >expect &&
>> +	test_path_is_missing .git/refs/heads/create-and-delete
>> +'
>
> The tests all look quite reasonable to me. Touching .git/refs like this
> is a bit gross (and something we may have to deal with if we introduce
> reftables, etc). But it's pretty pervasive in this file, so matching
> the existing style is the best option for now.


Shouldn't "git show-ref --verify" be usable portably across ref backends
to test if a well-formed ref was created (or was not created)?

On the ref-creation side, there are cases where we need to directly
futz with the filesystem entity.  For example, "git update-ref"
cannot be used to place a non-commit at "refs/heads/foo", so
something like

	git rev-parse HEAD^{tree} >.git/refs/heads/bad-branch

cannot be avoided (this is a tangent but we probably should add a
way to force setting _any_ value to any ref, that may not even point
at an existing object or an object of a wrong type, to help test
scripts).

But I do not think this is such a case.

Re: [PATCH 1/2] update-ref: Allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

On Thu, Nov 05, 2020 at 02:29:01PM -0500, Jeff King wrote:
> On Wed, Nov 04, 2020 at 03:57:17PM +0100, Patrick Steinhardt wrote:
> 
> > While git-update-ref has recently grown commands which allow interactive
> > control of transactions in e48cf33b61 (update-ref: implement interactive
> > transaction handling, 2020-04-02), it is not yet possible to create
> > multiple transactions in a single session. To do so, one currently still
> > needs to invoke the executable multiple times.
> > 
> > This commit addresses this shortcoming by allowing the "start" command
> > to create a new transaction if the current transaction has already been
> > either committed or aborted.
> 
> Thanks for working on this. The amount of change needed is indeed quite
> pleasant.
> 
> > diff --git a/Documentation/git-update-ref.txt b/Documentation/git-update-ref.txt
> > index d401234b03..48b6683071 100644
> > --- a/Documentation/git-update-ref.txt
> > +++ b/Documentation/git-update-ref.txt
> > @@ -125,7 +125,8 @@ option::
> >  start::
> >  	Start a transaction. In contrast to a non-transactional session, a
> >  	transaction will automatically abort if the session ends without an
> > -	explicit commit.
> > +	explicit commit. This command may create a new empty transaction when
> > +	the current one has been committed or aborted already.
> 
> Reading this made me wonder what would happen if we send a "start" when
> the current one _hasn't_ been committed or aborted. I.e., what does:
> 
>   git update-ref --stdin <<EOF
>   start
>   create refs/heads/foo ...
>   start
>   commit
>   EOF
> 
> do? It turns out that the second start is ignored totally (and the
> commit does indeed update foo). I wonder if we ought to complain about
> it. But that is completely orthogonal to your patch. The behavior is the
> same before and after.

Agreed, that's a case where we should raise an error. Doing nothing
without any indication is a bad way of handling it.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 1/2] update-ref: Allow creation of multiple transactions

[Jeff King]

On Thu, Nov 05, 2020 at 01:34:20PM -0800, Junio C Hamano wrote:

> > The tests all look quite reasonable to me. Touching .git/refs like this
> > is a bit gross (and something we may have to deal with if we introduce
> > reftables, etc). But it's pretty pervasive in this file, so matching
> > the existing style is the best option for now.
> 
> 
> Shouldn't "git show-ref --verify" be usable portably across ref backends
> to test if a well-formed ref was created (or was not created)?
> 
> On the ref-creation side, there are cases where we need to directly
> futz with the filesystem entity.  For example, "git update-ref"
> cannot be used to place a non-commit at "refs/heads/foo", so
> something like
> 
> 	git rev-parse HEAD^{tree} >.git/refs/heads/bad-branch
> 
> cannot be avoided (this is a tangent but we probably should add a
> way to force setting _any_ value to any ref, that may not even point
> at an existing object or an object of a wrong type, to help test
> scripts).
> 
> But I do not think this is such a case.

Yeah, I agree completely that we could be using rev-parse in this
instance. But it's definitely not alone there:

  $ git grep -c test_path_is.*.git/refs t/t1400-update-ref.sh
  t/t1400-update-ref.sh:13

So it is a question of "do an ugly thing that fits in with neighbors" or
"be inconsistent but set a good example". And I am OK with either. Of
course, "improve the neighbors on top" would be better still. :)

-Peff

PS And yeah, I agree in the long run we may need some mechanism to
   override internal safeties in order to test broken cases with
   reftable. We have sometimes resorted to manually munging on-disk
   files in similar tests (e.g., for broken packs, etc), but it gets
   rather tricky.

Re: [PATCH 1/2] update-ref: Allow creation of multiple transactions

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Thu, Nov 05, 2020 at 01:34:20PM -0800, Junio C Hamano wrote:
>
>> > The tests all look quite reasonable to me. Touching .git/refs like this
>> > is a bit gross (and something we may have to deal with if we introduce
>> > reftables, etc). But it's pretty pervasive in this file, so matching
>> > the existing style is the best option for now.
>>  ...
> Yeah, I agree completely that we could be using rev-parse in this
> instance. But it's definitely not alone there:
> ...

Yup, this morning I was reviewing what we said in the previous day's
exchanges and noticed that you weren't advocating but merely saying
it is not making things worse, and I agree with the assessment.

Perhaps two #leftoverbits are to 

 (1) clean up this test to create refs using "update-ref", and
     verify refs using "show-ref --verify".

 (2) If (1) had to leave some direct filesystem access due to the
     built-in safety that cannot be circumvented, decide which is
     more appropirate between a test-update-ref test helper only to
     be used in tests, or a "--force" option usable to corrupt
     repositories with "update-ref", implement it, and use it to
     finish cleaning up tests.

Thanks.

[PATCH v2 0/4] update-ref: Allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

Hi,

this is the second version of this patch series implementing support for
creation of multiple reference transactions in a single git-update-ref
process.

There's two changes compared to v1:

    - A small refactoring of t1400, which refactors many tests to not
      touch references via the filesystem but instead to use
      git-update-ref and git-show-ref. There's still tests in there
      which do, but converting them is harder as they create and read
      broken references.

    - I've added another commit on top which disallows restarting of
      transactions. E.g. writing "start\nstart\n" to git-update-ref is
      now going to fail.

Patrick

Patrick Steinhardt (4):
  t1400: Avoid touching refs on filesystem
  update-ref: Allow creation of multiple transactions
  p1400: Use `git-update-ref --stdin` to test multiple transactions
  update-ref: Disallow restart of ongoing transactions

 Documentation/git-update-ref.txt |   3 +-
 builtin/update-ref.c             |  15 +++-
 t/perf/p1400-update-ref.sh       |  20 ++---
 t/t1400-update-ref.sh            | 124 ++++++++++++++++++++++++-------
 4 files changed, 119 insertions(+), 43 deletions(-)

-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 1/4] t1400: Avoid touching refs on filesystem

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8560 bytes --]

The testcase t1400 exercises the git-update-ref(1) utility. To do so,
many tests directly read and write references via the filesystem,
assuming that we always use loose and/or packed references. While this
is true now, it'll change with the introduction of the reftable backend.

Convert those tests to use git-update-ref(1) and git-show-ref(1) where
possible. As some tests exercise behaviour with broken references and
neither of those tools actually allows writing or reading broken
references, this commit doesn't adjust all tests.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1400-update-ref.sh | 63 ++++++++++++++++++++++++-------------------
 1 file changed, 35 insertions(+), 28 deletions(-)

diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 4c01e08551..957bef272d 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -14,6 +14,12 @@ n=$n_dir/fixes
 outside=refs/foo
 bare=bare-repo
 
+# Some of the tests delete HEAD, which causes us to not treat the current
+# working directory as a Git repository anymore. To avoid using any potential
+# parent repository to be discovered, we need to set up the ceiling directories.
+GIT_CEILING_DIRECTORIES="$PWD/.."
+export GIT_CEILING_DIRECTORIES
+
 create_test_commits ()
 {
 	prfx="$1"
@@ -48,17 +54,17 @@ test_expect_success "fail to delete $m with stale ref" '
 	test $B = "$(git show-ref -s --verify $m)"
 '
 test_expect_success "delete $m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref -d $m $B &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "delete $m without oldvalue verification" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	test $A = $(git show-ref -s --verify $m) &&
 	git update-ref -d $m &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "fail to create $n" '
@@ -80,26 +86,26 @@ test_expect_success "fail to delete $m (by HEAD) with stale ref" '
 	test $B = $(git show-ref -s --verify $m)
 '
 test_expect_success "delete $m (by HEAD)" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref -d HEAD $B &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "deleting current branch adds message to HEAD's log" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	git symbolic-ref HEAD $m &&
 	git update-ref -m delete-$m -d $m &&
-	test_path_is_missing .git/$m &&
+	test_must_fail git show-ref --verify -q $m &&
 	grep "delete-$m$" .git/logs/HEAD
 '
 
 test_expect_success "deleting by HEAD adds message to HEAD's log" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	git symbolic-ref HEAD $m &&
 	git update-ref -m delete-by-head -d HEAD &&
-	test_path_is_missing .git/$m &&
+	test_must_fail git show-ref --verify -q $m &&
 	grep "delete-by-head$" .git/logs/HEAD
 '
 
@@ -188,17 +194,17 @@ test_expect_success "move $m (by HEAD)" '
 	test $B = $(git show-ref -s --verify $m)
 '
 test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "rm -f git update-ref -d $m" &&
 	git update-ref -d HEAD $B &&
 	! grep "$m" .git/packed-refs &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 cp -f .git/HEAD .git/HEAD.orig
 test_expect_success 'delete symref without dereference' '
 	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
 	git update-ref --no-deref -d HEAD &&
-	test_path_is_missing .git/HEAD
+	test_must_fail git show-ref --verify -q HEAD
 '
 
 test_expect_success 'delete symref without dereference when the referred ref is packed' '
@@ -208,7 +214,7 @@ test_expect_success 'delete symref without dereference when the referred ref is
 	git commit -m foo &&
 	git pack-refs --all &&
 	git update-ref --no-deref -d HEAD &&
-	test_path_is_missing .git/HEAD
+	test_must_fail git show-ref --verify -q HEAD
 '
 
 git update-ref -d $m
@@ -226,25 +232,25 @@ test_expect_success 'update-ref --no-deref -d can delete self-reference' '
 	test_when_finished "rm -f .git/refs/heads/self" &&
 	test_path_is_file .git/refs/heads/self &&
 	git update-ref --no-deref -d refs/heads/self &&
-	test_path_is_missing .git/refs/heads/self
+	test_must_fail git show-ref --verify -q refs/heads/self
 '
 
 test_expect_success 'update-ref --no-deref -d can delete reference to bad ref' '
 	>.git/refs/heads/bad &&
 	test_when_finished "rm -f .git/refs/heads/bad" &&
 	git symbolic-ref refs/heads/ref-to-bad refs/heads/bad &&
-	test_when_finished "rm -f .git/refs/heads/ref-to-bad" &&
+	test_when_finished "git update-ref -d refs/heads/ref-to-bad" &&
 	test_path_is_file .git/refs/heads/ref-to-bad &&
 	git update-ref --no-deref -d refs/heads/ref-to-bad &&
-	test_path_is_missing .git/refs/heads/ref-to-bad
+	test_must_fail git show-ref --verify -q refs/heads/ref-to-bad
 '
 
 test_expect_success '(not) create HEAD with old sha1' '
 	test_must_fail git update-ref HEAD $A $B
 '
 test_expect_success "(not) prior created .git/$m" '
-	test_when_finished "rm -f .git/$m" &&
-	test_path_is_missing .git/$m
+	test_when_finished "git update-ref -d $m" &&
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success 'create HEAD' '
@@ -254,7 +260,7 @@ test_expect_success '(not) change HEAD with wrong SHA1' '
 	test_must_fail git update-ref HEAD $B $Z
 '
 test_expect_success "(not) changed .git/$m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	! test $B = $(git show-ref -s --verify $m)
 '
 
@@ -284,8 +290,8 @@ test_expect_success 'empty directory removal' '
 	test_path_is_file .git/refs/heads/d1/d2/r1 &&
 	test_path_is_file .git/logs/refs/heads/d1/d2/r1 &&
 	git branch -d d1/d2/r1 &&
-	test_path_is_missing .git/refs/heads/d1/d2 &&
-	test_path_is_missing .git/logs/refs/heads/d1/d2 &&
+	test_must_fail git show-ref --verify -q refs/heads/d1/d2 &&
+	test_must_fail git show-ref --verify -q logs/refs/heads/d1/d2 &&
 	test_path_is_file .git/refs/heads/d1/r2 &&
 	test_path_is_file .git/logs/refs/heads/d1/r2
 '
@@ -298,8 +304,8 @@ test_expect_success 'symref empty directory removal' '
 	test_path_is_file .git/refs/heads/e1/e2/r1 &&
 	test_path_is_file .git/logs/refs/heads/e1/e2/r1 &&
 	git update-ref -d HEAD &&
-	test_path_is_missing .git/refs/heads/e1/e2 &&
-	test_path_is_missing .git/logs/refs/heads/e1/e2 &&
+	test_must_fail git show-ref --verify -q refs/heads/e1/e2 &&
+	test_must_fail git show-ref --verify -q logs/refs/heads/e1/e2 &&
 	test_path_is_file .git/refs/heads/e1/r2 &&
 	test_path_is_file .git/logs/refs/heads/e1/r2 &&
 	test_path_is_file .git/logs/HEAD
@@ -1388,7 +1394,8 @@ test_expect_success 'handle per-worktree refs in refs/bisect' '
 		git rev-parse refs/bisect/something >../worktree-head &&
 		git for-each-ref | grep refs/bisect/something
 	) &&
-	test_path_is_missing .git/refs/bisect &&
+	git show-ref >actual &&
+	! grep 'refs/bisect' actual &&
 	test_must_fail git rev-parse refs/bisect/something &&
 	git update-ref refs/bisect/something HEAD &&
 	git rev-parse refs/bisect/something >main-head &&
@@ -1500,7 +1507,7 @@ test_expect_success 'transaction can handle abort' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start abort >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_expect_success 'transaction aborts by default' '
@@ -1511,7 +1518,7 @@ test_expect_success 'transaction aborts by default' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_expect_success 'transaction with prepare aborts by default' '
@@ -1523,7 +1530,7 @@ test_expect_success 'transaction with prepare aborts by default' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start prepare >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 2/4] update-ref: Allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3695 bytes --]

While git-update-ref has recently grown commands which allow interactive
control of transactions in e48cf33b61 (update-ref: implement interactive
transaction handling, 2020-04-02), it is not yet possible to create
multiple transactions in a single session. To do so, one currently still
needs to invoke the executable multiple times.

This commit addresses this shortcoming by allowing the "start" command
to create a new transaction if the current transaction has already been
either committed or aborted.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-update-ref.txt |  3 +-
 builtin/update-ref.c             | 13 ++++++++-
 t/t1400-update-ref.sh            | 50 ++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-update-ref.txt b/Documentation/git-update-ref.txt
index d401234b03..48b6683071 100644
--- a/Documentation/git-update-ref.txt
+++ b/Documentation/git-update-ref.txt
@@ -125,7 +125,8 @@ option::
 start::
 	Start a transaction. In contrast to a non-transactional session, a
 	transaction will automatically abort if the session ends without an
-	explicit commit.
+	explicit commit. This command may create a new empty transaction when
+	the current one has been committed or aborted already.
 
 prepare::
 	Prepare to commit the transaction. This will create lock files for all
diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 8a2df4459c..bb65129012 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -446,7 +446,18 @@ static void update_refs_stdin(void)
 			state = cmd->state;
 			break;
 		case UPDATE_REFS_CLOSED:
-			die("transaction is closed");
+			if (cmd->state != UPDATE_REFS_STARTED)
+				die("transaction is closed");
+
+			/*
+			 * Open a new transaction if we're currently closed and
+			 * get a "start".
+			 */
+			state = cmd->state;
+			transaction = ref_transaction_begin(&err);
+			if (!transaction)
+				die("%s", err.buf);
+
 			break;
 		}
 
diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 957bef272d..7ed41bb328 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1533,4 +1533,54 @@ test_expect_success 'transaction with prepare aborts by default' '
 	test_must_fail git show-ref --verify -q $b
 '
 
+test_expect_success 'transaction can commit multiple times' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/branch-1 $A
+	commit
+	start
+	create refs/heads/branch-2 $B
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_cmp expect actual &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/branch-1 >actual &&
+	test_cmp expect actual &&
+	echo "$B" >expect &&
+	git rev-parse refs/heads/branch-2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'transaction can create and delete' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/create-and-delete $A
+	commit
+	start
+	delete refs/heads/create-and-delete $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_must_fail git show-ref --verify refs/heads/create-and-delete
+'
+
+test_expect_success 'transaction can commit after abort' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/abort $A
+	abort
+	start
+	create refs/heads/abort $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start abort start commit >expect &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/abort >actual &&
+	test_cmp expect actual
+'
+
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 3/4] p1400: Use `git-update-ref --stdin` to test multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

In commit 0a0fbbe3ff (refs: remove lookup cache for
reference-transaction hook, 2020-08-25), a new benchmark was added to
p1400 which has the intention to exercise creation of multiple
transactions in a single process. As git-update-ref wasn't yet able to
create multiple transactions with a single run we instead used git-push.
As its non-atomic version creates a transaction per reference update,
this was the best approximation we could make at that point in time.

Now that `git-update-ref --stdin` supports creation of multiple
transactions, let's convert the benchmark to use that instead. It has
less overhead and it's also a lot clearer what the actual intention is.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/p1400-update-ref.sh | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/t/perf/p1400-update-ref.sh b/t/perf/p1400-update-ref.sh
index ce5ac3ed85..dda8a74866 100755
--- a/t/perf/p1400-update-ref.sh
+++ b/t/perf/p1400-update-ref.sh
@@ -7,13 +7,14 @@ test_description="Tests performance of update-ref"
 test_perf_fresh_repo
 
 test_expect_success "setup" '
-	git init --bare target-repo.git &&
 	test_commit PRE &&
 	test_commit POST &&
-	printf "create refs/heads/%d PRE\n" $(test_seq 1000) >create &&
-	printf "update refs/heads/%d POST PRE\n" $(test_seq 1000) >update &&
-	printf "delete refs/heads/%d POST\n" $(test_seq 1000) >delete &&
-	git update-ref --stdin <create
+	for i in $(test_seq 5000)
+	do
+		printf "start\ncreate refs/heads/%d PRE\ncommit\n" $i &&
+		printf "start\nupdate refs/heads/%d POST PRE\ncommit\n" $i &&
+		printf "start\ndelete refs/heads/%d POST\ncommit\n" $i
+	done >instructions
 '
 
 test_perf "update-ref" '
@@ -26,14 +27,7 @@ test_perf "update-ref" '
 '
 
 test_perf "update-ref --stdin" '
-	git update-ref --stdin <update &&
-	git update-ref --stdin <delete &&
-	git update-ref --stdin <create
-'
-
-test_perf "nonatomic push" '
-	git push ./target-repo.git $(test_seq 1000) &&
-	git push --delete ./target-repo.git $(test_seq 1000)
+	git update-ref --stdin <instructions >/dev/null
 '
 
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2 4/4] update-ref: Disallow restart of ongoing transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1882 bytes --]

It is currently possible to write multiple "start" commands into
git-update-ref(1) for a single session, which doesn't make a lot of
sense to do in the first place. It's also not quite obvious what should
actually happen. Would this just go on with the current transaction as
if nothing was written or would it recreate a new session which doesn't
yet have any references? Silently ignoring this usage isn't helping
either as it may indicate erroneous use of the interface.

This commit catches this use and instead raises an error if the user is
trying to restart an ongoing transaction.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/update-ref.c  |  2 ++
 t/t1400-update-ref.sh | 11 +++++++++++
 2 files changed, 13 insertions(+)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index bb65129012..6029a80544 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -436,6 +436,8 @@ static void update_refs_stdin(void)
 		switch (state) {
 		case UPDATE_REFS_OPEN:
 		case UPDATE_REFS_STARTED:
+			if (state == UPDATE_REFS_STARTED && cmd->state == UPDATE_REFS_STARTED)
+				die("cannot restart ongoing transaction");
 			/* Do not downgrade a transaction to a non-transaction. */
 			if (cmd->state >= state)
 				state = cmd->state;
diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 7ed41bb328..e53d973d04 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1583,4 +1583,15 @@ test_expect_success 'transaction can commit after abort' '
 	test_cmp expect actual
 '
 
+test_expect_success 'transaction cannot restart ongoing transaction' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/restart $A
+	start
+	commit
+	EOF
+	test_must_fail git update-ref --stdin <stdin >actual &&
+	test_must_fail git show-ref --verify refs/heads/restart
+'
+
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/4] t1400: Avoid touching refs on filesystem

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> The testcase t1400 exercises the git-update-ref(1) utility. To do so,
> many tests directly read and write references via the filesystem,
> assuming that we always use loose and/or packed references. While this
> is true now, it'll change with the introduction of the reftable backend.
>
> Convert those tests to use git-update-ref(1) and git-show-ref(1) where
> possible. As some tests exercise behaviour with broken references and
> neither of those tools actually allows writing or reading broken
> references, this commit doesn't adjust all tests.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/t1400-update-ref.sh | 63 ++++++++++++++++++++++++-------------------
>  1 file changed, 35 insertions(+), 28 deletions(-)
>
> diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
> index 4c01e08551..957bef272d 100755
> --- a/t/t1400-update-ref.sh
> +++ b/t/t1400-update-ref.sh
> @@ -14,6 +14,12 @@ n=$n_dir/fixes
>  outside=refs/foo
>  bare=bare-repo
>  
> +# Some of the tests delete HEAD, which causes us to not treat the current
> +# working directory as a Git repository anymore. To avoid using any potential
> +# parent repository to be discovered, we need to set up the ceiling directories.
> +GIT_CEILING_DIRECTORIES="$PWD/.."
> +export GIT_CEILING_DIRECTORIES
> +

Interesting.  The current tests do not need to do this because the
HEAD-less broken state is transitory and is corrected without using
"git" at all (e.g. echoing a valid value into .git/HEAD), I presume?

>  create_test_commits ()
>  {
>  	prfx="$1"
> @@ -48,17 +54,17 @@ test_expect_success "fail to delete $m with stale ref" '
>  	test $B = "$(git show-ref -s --verify $m)"
>  '
>  test_expect_success "delete $m" '
> -	test_when_finished "rm -f .git/$m" &&
> +	test_when_finished "git update-ref -d $m" &&
>  	git update-ref -d $m $B &&
> -	test_path_is_missing .git/$m
> +	test_must_fail git show-ref --verify -q $m
>  '
>  
>  test_expect_success "delete $m without oldvalue verification" '
> -	test_when_finished "rm -f .git/$m" &&
> +	test_when_finished "git update-ref -d $m" &&
>  	git update-ref $m $A &&
>  	test $A = $(git show-ref -s --verify $m) &&
>  	git update-ref -d $m &&
> -	test_path_is_missing .git/$m
> +	test_must_fail git show-ref --verify -q $m
>  '
>  
>  test_expect_success "fail to create $n" '
> @@ -80,26 +86,26 @@ test_expect_success "fail to delete $m (by HEAD) with stale ref" '
>  	test $B = $(git show-ref -s --verify $m)
>  '
>  test_expect_success "delete $m (by HEAD)" '
> -	test_when_finished "rm -f .git/$m" &&
> +	test_when_finished "git update-ref -d $m" &&
>  	git update-ref -d HEAD $B &&
> -	test_path_is_missing .git/$m
> +	test_must_fail git show-ref --verify -q $m
>  '

During the above test, we are on the branch ${m#refs/heads/}, so
"update-ref -d HEAD" is removing the underlying branch ref, making
it an unborn branch, without destroying the repository, so this is
perfectly sensible.

This is a tangent, but what makes this test doubly interesting is
that "git update-ref -d HEAD" would have allowed us to make it a
non-repository if HEAD were detached, and it seems that we do not
require "--force" to do so.  We probably should forbid removing HEAD
that id detached without "--force", which is such a destructive
operation.

>  cp -f .git/HEAD .git/HEAD.orig
>  test_expect_success 'delete symref without dereference' '
>  	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
>  	git update-ref --no-deref -d HEAD &&
> -	test_path_is_missing .git/HEAD
> +	test_must_fail git show-ref --verify -q HEAD
>  '

This is an example of breaking the repository.  I am not sure if the
test_must_fail is a good replacement--it would fail even if you say
"git show-ref --verify -q refs/heads/$branch" where $branch is a
name of a branch that exists, no?

For now, I think this is fine, but we'd probably clean it up so that
without --force update-ref would not corrupt the repository like
this.  When used with --force, or before adding such a safety
measure, how we test if we successfully corrupted the repository is
an interesting matter.  I'd say

	git update-ref --force --no-deref -d HEAD &&
	test_must_fail git show-ref --verify -q HEAD &&
	cp -f .git/HEAD.orig .git/HEAD &&
	git show-ref --verify -q "$m"

to ensure that other than removing HEAD and corrupting the
repository, it did not cause permanent damage by removing the
underlying ref, perhaps.

We may want to add some NEEDSWORK comment around such tests.

>  test_expect_success 'delete symref without dereference when the referred ref is packed' '
> @@ -208,7 +214,7 @@ test_expect_success 'delete symref without dereference when the referred ref is
>  	git commit -m foo &&
>  	git pack-refs --all &&
>  	git update-ref --no-deref -d HEAD &&
> -	test_path_is_missing .git/HEAD
> +	test_must_fail git show-ref --verify -q HEAD
>  '

Does this share the same issue as the above?

Thanks.

Re: [PATCH v2 4/4] update-ref: Disallow restart of ongoing transactions

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> Subject: Re: [PATCH v2 4/4] update-ref: Disallow restart of ongoing transactions

Our convention (cf. "git shortlog --no-merges") is not to Capitalize
the first word after the "<area>:".  Shared across all four patches.

> It is currently possible to write multiple "start" commands into
> git-update-ref(1) for a single session, which doesn't make a lot of
> sense to do in the first place. It's also not quite obvious what should
> actually happen. Would this just go on with the current transaction as
> if nothing was written or would it recreate a new session which doesn't
> yet have any references? Silently ignoring this usage isn't helping
> either as it may indicate erroneous use of the interface.
>
> This commit catches this use and instead raises an error if the user is
> trying to restart an ongoing transaction.

Hmph, if you think of it as "restart", perhaps it would not make
sense, but in other contexts where the concept of "transactions" are
commonly used, it is not so unusual to support "nested" transactions
where inside an ongoing transaction, a subtransaction whose effect
can be rolled back as a whole, can be opened.  So wouldn't it be
more appropriate to explain this change like so?

    Since we do not support nested transactions, make sure a "start"
    command, seen after we saw "start" that is not closed by either
    "abort" or "commit", is flagged as an error.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  builtin/update-ref.c  |  2 ++
>  t/t1400-update-ref.sh | 11 +++++++++++
>  2 files changed, 13 insertions(+)
>
> diff --git a/builtin/update-ref.c b/builtin/update-ref.c
> index bb65129012..6029a80544 100644
> --- a/builtin/update-ref.c
> +++ b/builtin/update-ref.c
> @@ -436,6 +436,8 @@ static void update_refs_stdin(void)
>  		switch (state) {
>  		case UPDATE_REFS_OPEN:
>  		case UPDATE_REFS_STARTED:
> +			if (state == UPDATE_REFS_STARTED && cmd->state == UPDATE_REFS_STARTED)
> +				die("cannot restart ongoing transaction");
>  			/* Do not downgrade a transaction to a non-transaction. */
>  			if (cmd->state >= state)
>  				state = cmd->state;
> diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
> index 7ed41bb328..e53d973d04 100755
> --- a/t/t1400-update-ref.sh
> +++ b/t/t1400-update-ref.sh
> @@ -1583,4 +1583,15 @@ test_expect_success 'transaction can commit after abort' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 'transaction cannot restart ongoing transaction' '
> +	cat >stdin <<-EOF &&
> +	start
> +	create refs/heads/restart $A
> +	start
> +	commit
> +	EOF
> +	test_must_fail git update-ref --stdin <stdin >actual &&
> +	test_must_fail git show-ref --verify refs/heads/restart
> +'
> +
>  test_done

Re: [PATCH v2 1/4] t1400: Avoid touching refs on filesystem

[Jeff King]

On Mon, Nov 09, 2020 at 11:48:20AM -0800, Junio C Hamano wrote:

> This is a tangent, but what makes this test doubly interesting is
> that "git update-ref -d HEAD" would have allowed us to make it a
> non-repository if HEAD were detached, and it seems that we do not
> require "--force" to do so.  We probably should forbid removing HEAD
> that id detached without "--force", which is such a destructive
> operation.

Yeah, I'd agree that is a good direction (but it definitely is a tangent
that should come in a separate series).

> >  cp -f .git/HEAD .git/HEAD.orig
> >  test_expect_success 'delete symref without dereference' '
> >  	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
> >  	git update-ref --no-deref -d HEAD &&
> > -	test_path_is_missing .git/HEAD
> > +	test_must_fail git show-ref --verify -q HEAD
> >  '
> 
> This is an example of breaking the repository.  I am not sure if the
> test_must_fail is a good replacement--it would fail even if you say
> "git show-ref --verify -q refs/heads/$branch" where $branch is a
> name of a branch that exists, no?

Perhaps we could more directly check that the repository is broken.
Coupled with the ceiling-limit from earlier in the script, then:

  git rev-parse --git-dir

should fail. Maybe that reveals the intent of what we're expecting a
little more clearly (and more so than your "show-ref before and after
restoring HEAD" example does, in my opinion).

I do have to wonder if this test even cares about HEAD. Could it equally
well work on another symref, like refs/remotes/origin/HEAD?

-Peff

Re: [PATCH v2 0/4] update-ref: Allow creation of multiple transactions

[Jeff King]

On Mon, Nov 09, 2020 at 11:06:43AM +0100, Patrick Steinhardt wrote:

> this is the second version of this patch series implementing support for
> creation of multiple reference transactions in a single git-update-ref
> process.
> 
> There's two changes compared to v1:
> 
>     - A small refactoring of t1400, which refactors many tests to not
>       touch references via the filesystem but instead to use
>       git-update-ref and git-show-ref. There's still tests in there
>       which do, but converting them is harder as they create and read
>       broken references.
> 
>     - I've added another commit on top which disallows restarting of
>       transactions. E.g. writing "start\nstart\n" to git-update-ref is
>       now going to fail.

Thanks. Aside from the issues raised by Junio, this all looks good to me
(and I agree on the fourth one it is just a matter of the commit
message; what the code is doing is a definite improvement).

-Peff

Re: [PATCH v2 0/4] update-ref: Allow creation of multiple transactions

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Mon, Nov 09, 2020 at 11:06:43AM +0100, Patrick Steinhardt wrote:
>
>> this is the second version of this patch series implementing support for
>> creation of multiple reference transactions in a single git-update-ref
>> process.
>> 
>> There's two changes compared to v1:
>> 
>>     - A small refactoring of t1400, which refactors many tests to not
>>       touch references via the filesystem but instead to use
>>       git-update-ref and git-show-ref. There's still tests in there
>>       which do, but converting them is harder as they create and read
>>       broken references.
>> 
>>     - I've added another commit on top which disallows restarting of
>>       transactions. E.g. writing "start\nstart\n" to git-update-ref is
>>       now going to fail.
>
> Thanks. Aside from the issues raised by Junio, this all looks good to me
> (and I agree on the fourth one it is just a matter of the commit
> message; what the code is doing is a definite improvement).

Yeah, I agree that it is just terminology.  If we were to explain it
as lack of nested transaction, however, the error message would need
to be updated.  Other than that, I think this is quite good.

Oh, Patrick, please do not forget that it is our convention not to
Capitalize the word immediately after <area>: on the title of the
commit (cf. "git shortlog --no-merges -200").

Thanks.

Re: [PATCH v2 1/4] t1400: Avoid touching refs on filesystem

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 4465 bytes --]

On Mon, Nov 09, 2020 at 11:48:20AM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
> > index 4c01e08551..957bef272d 100755
> > --- a/t/t1400-update-ref.sh
> > +++ b/t/t1400-update-ref.sh
> > @@ -14,6 +14,12 @@ n=$n_dir/fixes
> >  outside=refs/foo
> >  bare=bare-repo
> >  
> > +# Some of the tests delete HEAD, which causes us to not treat the current
> > +# working directory as a Git repository anymore. To avoid using any potential
> > +# parent repository to be discovered, we need to set up the ceiling directories.
> > +GIT_CEILING_DIRECTORIES="$PWD/.."
> > +export GIT_CEILING_DIRECTORIES
> > +
> 
> Interesting.  The current tests do not need to do this because the
> HEAD-less broken state is transitory and is corrected without using
> "git" at all (e.g. echoing a valid value into .git/HEAD), I presume?

Correct.

> > @@ -80,26 +86,26 @@ test_expect_success "fail to delete $m (by HEAD) with stale ref" '
> >  	test $B = $(git show-ref -s --verify $m)
> >  '
> >  test_expect_success "delete $m (by HEAD)" '
> > -	test_when_finished "rm -f .git/$m" &&
> > +	test_when_finished "git update-ref -d $m" &&
> >  	git update-ref -d HEAD $B &&
> > -	test_path_is_missing .git/$m
> > +	test_must_fail git show-ref --verify -q $m
> >  '
> 
> During the above test, we are on the branch ${m#refs/heads/}, so
> "update-ref -d HEAD" is removing the underlying branch ref, making
> it an unborn branch, without destroying the repository, so this is
> perfectly sensible.
> 
> This is a tangent, but what makes this test doubly interesting is
> that "git update-ref -d HEAD" would have allowed us to make it a
> non-repository if HEAD were detached, and it seems that we do not
> require "--force" to do so.  We probably should forbid removing HEAD
> that id detached without "--force", which is such a destructive
> operation.

That'd make sense to me, yes. It also took me quite some time to
actually figure out why tests were misbehaving when I converted it.
Until I finally realized: this is not a Git repo anymore, and refs have
now been modified directly in the real git.git repository. Just this
morning I still found some invalid refs created by the test in git.git.

> >  cp -f .git/HEAD .git/HEAD.orig
> >  test_expect_success 'delete symref without dereference' '
> >  	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
> >  	git update-ref --no-deref -d HEAD &&
> > -	test_path_is_missing .git/HEAD
> > +	test_must_fail git show-ref --verify -q HEAD
> >  '
> 
> This is an example of breaking the repository.  I am not sure if the
> test_must_fail is a good replacement--it would fail even if you say
> "git show-ref --verify -q refs/heads/$branch" where $branch is a
> name of a branch that exists, no?

Right, it's not. We're not detecting HEAD deletion by means of checking
that it doesn't exist anymore, but only detect it because the repo is
not a repo anymore. Which would in fact cause most git commands to fail
now.

> For now, I think this is fine, but we'd probably clean it up so that
> without --force update-ref would not corrupt the repository like
> this.  When used with --force, or before adding such a safety
> measure, how we test if we successfully corrupted the repository is
> an interesting matter.  I'd say
> 
> 	git update-ref --force --no-deref -d HEAD &&
> 	test_must_fail git show-ref --verify -q HEAD &&
> 	cp -f .git/HEAD.orig .git/HEAD &&
> 	git show-ref --verify -q "$m"
> 
> to ensure that other than removing HEAD and corrupting the
> repository, it did not cause permanent damage by removing the
> underlying ref, perhaps.
> 
> We may want to add some NEEDSWORK comment around such tests.

I think the proper way to do the test would be to create a non-mandatory
symref and delete it. That'd also be a nice preparation for the
`--force` parameter already.

> >  test_expect_success 'delete symref without dereference when the referred ref is packed' '
> > @@ -208,7 +214,7 @@ test_expect_success 'delete symref without dereference when the referred ref is
> >  	git commit -m foo &&
> >  	git pack-refs --all &&
> >  	git update-ref --no-deref -d HEAD &&
> > -	test_path_is_missing .git/HEAD
> > +	test_must_fail git show-ref --verify -q HEAD
> >  '
> 
> Does this share the same issue as the above?

Yup, it does.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/4] t1400: Avoid touching refs on filesystem

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>> For now, I think this is fine, but we'd probably clean it up so that
>> without --force update-ref would not corrupt the repository like
>> this.  When used with --force, or before adding such a safety
>> measure, how we test if we successfully corrupted the repository is
>> an interesting matter.  I'd say
>> 
>> 	git update-ref --force --no-deref -d HEAD &&
>> 	test_must_fail git show-ref --verify -q HEAD &&
>> 	cp -f .git/HEAD.orig .git/HEAD &&
>> 	git show-ref --verify -q "$m"
>> 
>> to ensure that other than removing HEAD and corrupting the
>> repository, it did not cause permanent damage by removing the
>> underlying ref, perhaps.
>> 
>> We may want to add some NEEDSWORK comment around such tests.
>
> I think the proper way to do the test would be to create a non-mandatory
> symref and delete it. That'd also be a nice preparation for the
> `--force` parameter already.

Excellent.  Yes, the test is about creation and removal of a symref;
the symref used for testing does not have to be the HEAD.

Thanks.

[PATCH v3 0/4] update-ref: allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Hi,

this is the third version of this patch series implementing support for
creation of multiple reference transactions in a single git-update-ref
process.

There's three changes compared to v2:

    - Commit subjects have been changed to start with a lowercase.

    - The commit message in patch 4 has been improved.

    - t1400 has been changed to not delete HEAD anymore.

Patrick

Patrick Steinhardt (4):
  t1400: avoid touching refs on filesystem
  update-ref: allow creation of multiple transactions
  p1400: use `git-update-ref --stdin` to test multiple transactions
  update-ref: disallow "start" for ongoing transactions

 Documentation/git-update-ref.txt |   3 +-
 builtin/update-ref.c             |  15 +++-
 t/perf/p1400-update-ref.sh       |  20 ++---
 t/t1400-update-ref.sh            | 133 +++++++++++++++++++++++--------
 4 files changed, 123 insertions(+), 48 deletions(-)

-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 1/4] t1400: avoid touching refs on filesystem

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8713 bytes --]

The testcase t1400 exercises the git-update-ref(1) utility. To do so,
many tests directly read and write references via the filesystem,
assuming that we always use loose and/or packed references. While this
is true now, it'll change with the introduction of the reftable backend.

Convert those tests to use git-update-ref(1) and git-show-ref(1) where
possible. As some tests exercise behaviour with broken references and
neither of those tools actually allows writing or reading broken
references, this commit doesn't adjust all tests.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1400-update-ref.sh | 72 +++++++++++++++++++++++--------------------
 1 file changed, 39 insertions(+), 33 deletions(-)

diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 4c01e08551..d7a57488ed 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -14,6 +14,12 @@ n=$n_dir/fixes
 outside=refs/foo
 bare=bare-repo
 
+# Some of the tests delete HEAD, which causes us to not treat the current
+# working directory as a Git repository anymore. To avoid using any potential
+# parent repository to be discovered, we need to set up the ceiling directories.
+GIT_CEILING_DIRECTORIES="$PWD/.."
+export GIT_CEILING_DIRECTORIES
+
 create_test_commits ()
 {
 	prfx="$1"
@@ -48,17 +54,17 @@ test_expect_success "fail to delete $m with stale ref" '
 	test $B = "$(git show-ref -s --verify $m)"
 '
 test_expect_success "delete $m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref -d $m $B &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "delete $m without oldvalue verification" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	test $A = $(git show-ref -s --verify $m) &&
 	git update-ref -d $m &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "fail to create $n" '
@@ -80,26 +86,26 @@ test_expect_success "fail to delete $m (by HEAD) with stale ref" '
 	test $B = $(git show-ref -s --verify $m)
 '
 test_expect_success "delete $m (by HEAD)" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref -d HEAD $B &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "deleting current branch adds message to HEAD's log" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	git symbolic-ref HEAD $m &&
 	git update-ref -m delete-$m -d $m &&
-	test_path_is_missing .git/$m &&
+	test_must_fail git show-ref --verify -q $m &&
 	grep "delete-$m$" .git/logs/HEAD
 '
 
 test_expect_success "deleting by HEAD adds message to HEAD's log" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	git symbolic-ref HEAD $m &&
 	git update-ref -m delete-by-head -d HEAD &&
-	test_path_is_missing .git/$m &&
+	test_must_fail git show-ref --verify -q $m &&
 	grep "delete-by-head$" .git/logs/HEAD
 '
 
@@ -188,27 +194,26 @@ test_expect_success "move $m (by HEAD)" '
 	test $B = $(git show-ref -s --verify $m)
 '
 test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "rm -f git update-ref -d $m" &&
 	git update-ref -d HEAD $B &&
 	! grep "$m" .git/packed-refs &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
-cp -f .git/HEAD .git/HEAD.orig
 test_expect_success 'delete symref without dereference' '
-	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
-	git update-ref --no-deref -d HEAD &&
-	test_path_is_missing .git/HEAD
+	git symbolic-ref SYMREF $m &&
+	git update-ref --no-deref -d SYMREF &&
+	test_must_fail git show-ref --verify -q SYMREF
 '
 
 test_expect_success 'delete symref without dereference when the referred ref is packed' '
-	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
 	echo foo >foo.c &&
 	git add foo.c &&
 	git commit -m foo &&
+	git symbolic-ref SYMREF $m &&
 	git pack-refs --all &&
-	git update-ref --no-deref -d HEAD &&
-	test_path_is_missing .git/HEAD
+	git update-ref --no-deref -d SYMREF &&
+	test_must_fail git show-ref --verify -q SYMREF
 '
 
 git update-ref -d $m
@@ -226,25 +231,25 @@ test_expect_success 'update-ref --no-deref -d can delete self-reference' '
 	test_when_finished "rm -f .git/refs/heads/self" &&
 	test_path_is_file .git/refs/heads/self &&
 	git update-ref --no-deref -d refs/heads/self &&
-	test_path_is_missing .git/refs/heads/self
+	test_must_fail git show-ref --verify -q refs/heads/self
 '
 
 test_expect_success 'update-ref --no-deref -d can delete reference to bad ref' '
 	>.git/refs/heads/bad &&
 	test_when_finished "rm -f .git/refs/heads/bad" &&
 	git symbolic-ref refs/heads/ref-to-bad refs/heads/bad &&
-	test_when_finished "rm -f .git/refs/heads/ref-to-bad" &&
+	test_when_finished "git update-ref -d refs/heads/ref-to-bad" &&
 	test_path_is_file .git/refs/heads/ref-to-bad &&
 	git update-ref --no-deref -d refs/heads/ref-to-bad &&
-	test_path_is_missing .git/refs/heads/ref-to-bad
+	test_must_fail git show-ref --verify -q refs/heads/ref-to-bad
 '
 
 test_expect_success '(not) create HEAD with old sha1' '
 	test_must_fail git update-ref HEAD $A $B
 '
 test_expect_success "(not) prior created .git/$m" '
-	test_when_finished "rm -f .git/$m" &&
-	test_path_is_missing .git/$m
+	test_when_finished "git update-ref -d $m" &&
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success 'create HEAD' '
@@ -254,7 +259,7 @@ test_expect_success '(not) change HEAD with wrong SHA1' '
 	test_must_fail git update-ref HEAD $B $Z
 '
 test_expect_success "(not) changed .git/$m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	! test $B = $(git show-ref -s --verify $m)
 '
 
@@ -284,8 +289,8 @@ test_expect_success 'empty directory removal' '
 	test_path_is_file .git/refs/heads/d1/d2/r1 &&
 	test_path_is_file .git/logs/refs/heads/d1/d2/r1 &&
 	git branch -d d1/d2/r1 &&
-	test_path_is_missing .git/refs/heads/d1/d2 &&
-	test_path_is_missing .git/logs/refs/heads/d1/d2 &&
+	test_must_fail git show-ref --verify -q refs/heads/d1/d2 &&
+	test_must_fail git show-ref --verify -q logs/refs/heads/d1/d2 &&
 	test_path_is_file .git/refs/heads/d1/r2 &&
 	test_path_is_file .git/logs/refs/heads/d1/r2
 '
@@ -298,8 +303,8 @@ test_expect_success 'symref empty directory removal' '
 	test_path_is_file .git/refs/heads/e1/e2/r1 &&
 	test_path_is_file .git/logs/refs/heads/e1/e2/r1 &&
 	git update-ref -d HEAD &&
-	test_path_is_missing .git/refs/heads/e1/e2 &&
-	test_path_is_missing .git/logs/refs/heads/e1/e2 &&
+	test_must_fail git show-ref --verify -q refs/heads/e1/e2 &&
+	test_must_fail git show-ref --verify -q logs/refs/heads/e1/e2 &&
 	test_path_is_file .git/refs/heads/e1/r2 &&
 	test_path_is_file .git/logs/refs/heads/e1/r2 &&
 	test_path_is_file .git/logs/HEAD
@@ -1388,7 +1393,8 @@ test_expect_success 'handle per-worktree refs in refs/bisect' '
 		git rev-parse refs/bisect/something >../worktree-head &&
 		git for-each-ref | grep refs/bisect/something
 	) &&
-	test_path_is_missing .git/refs/bisect &&
+	git show-ref >actual &&
+	! grep 'refs/bisect' actual &&
 	test_must_fail git rev-parse refs/bisect/something &&
 	git update-ref refs/bisect/something HEAD &&
 	git rev-parse refs/bisect/something >main-head &&
@@ -1500,7 +1506,7 @@ test_expect_success 'transaction can handle abort' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start abort >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_expect_success 'transaction aborts by default' '
@@ -1511,7 +1517,7 @@ test_expect_success 'transaction aborts by default' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_expect_success 'transaction with prepare aborts by default' '
@@ -1523,7 +1529,7 @@ test_expect_success 'transaction with prepare aborts by default' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start prepare >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 2/4] update-ref: allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3695 bytes --]

While git-update-ref has recently grown commands which allow interactive
control of transactions in e48cf33b61 (update-ref: implement interactive
transaction handling, 2020-04-02), it is not yet possible to create
multiple transactions in a single session. To do so, one currently still
needs to invoke the executable multiple times.

This commit addresses this shortcoming by allowing the "start" command
to create a new transaction if the current transaction has already been
either committed or aborted.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-update-ref.txt |  3 +-
 builtin/update-ref.c             | 13 ++++++++-
 t/t1400-update-ref.sh            | 50 ++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-update-ref.txt b/Documentation/git-update-ref.txt
index d401234b03..48b6683071 100644
--- a/Documentation/git-update-ref.txt
+++ b/Documentation/git-update-ref.txt
@@ -125,7 +125,8 @@ option::
 start::
 	Start a transaction. In contrast to a non-transactional session, a
 	transaction will automatically abort if the session ends without an
-	explicit commit.
+	explicit commit. This command may create a new empty transaction when
+	the current one has been committed or aborted already.
 
 prepare::
 	Prepare to commit the transaction. This will create lock files for all
diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 8a2df4459c..bb65129012 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -446,7 +446,18 @@ static void update_refs_stdin(void)
 			state = cmd->state;
 			break;
 		case UPDATE_REFS_CLOSED:
-			die("transaction is closed");
+			if (cmd->state != UPDATE_REFS_STARTED)
+				die("transaction is closed");
+
+			/*
+			 * Open a new transaction if we're currently closed and
+			 * get a "start".
+			 */
+			state = cmd->state;
+			transaction = ref_transaction_begin(&err);
+			if (!transaction)
+				die("%s", err.buf);
+
 			break;
 		}
 
diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index d7a57488ed..2890504edd 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1532,4 +1532,54 @@ test_expect_success 'transaction with prepare aborts by default' '
 	test_must_fail git show-ref --verify -q $b
 '
 
+test_expect_success 'transaction can commit multiple times' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/branch-1 $A
+	commit
+	start
+	create refs/heads/branch-2 $B
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_cmp expect actual &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/branch-1 >actual &&
+	test_cmp expect actual &&
+	echo "$B" >expect &&
+	git rev-parse refs/heads/branch-2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'transaction can create and delete' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/create-and-delete $A
+	commit
+	start
+	delete refs/heads/create-and-delete $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_must_fail git show-ref --verify refs/heads/create-and-delete
+'
+
+test_expect_success 'transaction can commit after abort' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/abort $A
+	abort
+	start
+	create refs/heads/abort $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start abort start commit >expect &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/abort >actual &&
+	test_cmp expect actual
+'
+
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 3/4] p1400: use `git-update-ref --stdin` to test multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

In commit 0a0fbbe3ff (refs: remove lookup cache for
reference-transaction hook, 2020-08-25), a new benchmark was added to
p1400 which has the intention to exercise creation of multiple
transactions in a single process. As git-update-ref wasn't yet able to
create multiple transactions with a single run we instead used git-push.
As its non-atomic version creates a transaction per reference update,
this was the best approximation we could make at that point in time.

Now that `git-update-ref --stdin` supports creation of multiple
transactions, let's convert the benchmark to use that instead. It has
less overhead and it's also a lot clearer what the actual intention is.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/p1400-update-ref.sh | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/t/perf/p1400-update-ref.sh b/t/perf/p1400-update-ref.sh
index ce5ac3ed85..dda8a74866 100755
--- a/t/perf/p1400-update-ref.sh
+++ b/t/perf/p1400-update-ref.sh
@@ -7,13 +7,14 @@ test_description="Tests performance of update-ref"
 test_perf_fresh_repo
 
 test_expect_success "setup" '
-	git init --bare target-repo.git &&
 	test_commit PRE &&
 	test_commit POST &&
-	printf "create refs/heads/%d PRE\n" $(test_seq 1000) >create &&
-	printf "update refs/heads/%d POST PRE\n" $(test_seq 1000) >update &&
-	printf "delete refs/heads/%d POST\n" $(test_seq 1000) >delete &&
-	git update-ref --stdin <create
+	for i in $(test_seq 5000)
+	do
+		printf "start\ncreate refs/heads/%d PRE\ncommit\n" $i &&
+		printf "start\nupdate refs/heads/%d POST PRE\ncommit\n" $i &&
+		printf "start\ndelete refs/heads/%d POST\ncommit\n" $i
+	done >instructions
 '
 
 test_perf "update-ref" '
@@ -26,14 +27,7 @@ test_perf "update-ref" '
 '
 
 test_perf "update-ref --stdin" '
-	git update-ref --stdin <update &&
-	git update-ref --stdin <delete &&
-	git update-ref --stdin <create
-'
-
-test_perf "nonatomic push" '
-	git push ./target-repo.git $(test_seq 1000) &&
-	git push --delete ./target-repo.git $(test_seq 1000)
+	git update-ref --stdin <instructions >/dev/null
 '
 
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v3 4/4] update-ref: disallow "start" for ongoing transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]

It is currently possible to write multiple "start" commands into
git-update-ref(1) for a single session, but none of them except for the
first one actually have any effect.

Using such nested "start"s may eventually have a sensible effect. One
may imagine that it restarts the current transaction, effectively
emptying it and creating a new one. It may also allow for creation of
nested transactions. But currently, none of these are implemented.

Silently ignoring this misuse is making it hard to iterate in the future
if "start" is ever going to have meaningful semantics in such a context.
This commit thus makes sure to error out in case we see such use.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/update-ref.c  |  2 ++
 t/t1400-update-ref.sh | 11 +++++++++++
 2 files changed, 13 insertions(+)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index bb65129012..6029a80544 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -436,6 +436,8 @@ static void update_refs_stdin(void)
 		switch (state) {
 		case UPDATE_REFS_OPEN:
 		case UPDATE_REFS_STARTED:
+			if (state == UPDATE_REFS_STARTED && cmd->state == UPDATE_REFS_STARTED)
+				die("cannot restart ongoing transaction");
 			/* Do not downgrade a transaction to a non-transaction. */
 			if (cmd->state >= state)
 				state = cmd->state;
diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 2890504edd..e109097970 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1582,4 +1582,15 @@ test_expect_success 'transaction can commit after abort' '
 	test_cmp expect actual
 '
 
+test_expect_success 'transaction cannot restart ongoing transaction' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/restart $A
+	start
+	commit
+	EOF
+	test_must_fail git update-ref --stdin <stdin >actual &&
+	test_must_fail git show-ref --verify refs/heads/restart
+'
+
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 1/4] t1400: avoid touching refs on filesystem

[SZEDER Gábor]

On Wed, Nov 11, 2020 at 07:58:38AM +0100, Patrick Steinhardt wrote:
> The testcase t1400 exercises the git-update-ref(1) utility. To do so,
> many tests directly read and write references via the filesystem,
> assuming that we always use loose and/or packed references. While this
> is true now, it'll change with the introduction of the reftable backend.
> 
> Convert those tests to use git-update-ref(1) and git-show-ref(1) where
> possible. As some tests exercise behaviour with broken references and
> neither of those tools actually allows writing or reading broken
> references, this commit doesn't adjust all tests.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  t/t1400-update-ref.sh | 72 +++++++++++++++++++++++--------------------
>  1 file changed, 39 insertions(+), 33 deletions(-)
> 
> diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
> index 4c01e08551..d7a57488ed 100755
> --- a/t/t1400-update-ref.sh
> +++ b/t/t1400-update-ref.sh

> @@ -188,27 +194,26 @@ test_expect_success "move $m (by HEAD)" '
>  	test $B = $(git show-ref -s --verify $m)
>  '
>  test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" '
> -	test_when_finished "rm -f .git/$m" &&
> +	test_when_finished "rm -f git update-ref -d $m" &&

There is a leftover 'rm -f' here.

>  	git update-ref -d HEAD $B &&
>  	! grep "$m" .git/packed-refs &&
> -	test_path_is_missing .git/$m
> +	test_must_fail git show-ref --verify -q $m
>  '

Re: [PATCH v3 1/4] t1400: avoid touching refs on filesystem

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1529 bytes --]

On Wed, Nov 11, 2020 at 10:04:54AM +0100, SZEDER Gábor wrote:
> On Wed, Nov 11, 2020 at 07:58:38AM +0100, Patrick Steinhardt wrote:
> > The testcase t1400 exercises the git-update-ref(1) utility. To do so,
> > many tests directly read and write references via the filesystem,
> > assuming that we always use loose and/or packed references. While this
> > is true now, it'll change with the introduction of the reftable backend.
> > 
> > Convert those tests to use git-update-ref(1) and git-show-ref(1) where
> > possible. As some tests exercise behaviour with broken references and
> > neither of those tools actually allows writing or reading broken
> > references, this commit doesn't adjust all tests.
> > 
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  t/t1400-update-ref.sh | 72 +++++++++++++++++++++++--------------------
> >  1 file changed, 39 insertions(+), 33 deletions(-)
> > 
> > diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
> > index 4c01e08551..d7a57488ed 100755
> > --- a/t/t1400-update-ref.sh
> > +++ b/t/t1400-update-ref.sh
> 
> > @@ -188,27 +194,26 @@ test_expect_success "move $m (by HEAD)" '
> >  	test $B = $(git show-ref -s --verify $m)
> >  '
> >  test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" '
> > -	test_when_finished "rm -f .git/$m" &&
> > +	test_when_finished "rm -f git update-ref -d $m" &&
> 
> There is a leftover 'rm -f' here.

Oops, thanks a lot for catching. Funny that it still passed.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v3 1/4] t1400: avoid touching refs on filesystem

[SZEDER Gábor]

On Wed, Nov 11, 2020 at 11:00:15AM +0100, Patrick Steinhardt wrote:
> On Wed, Nov 11, 2020 at 10:04:54AM +0100, SZEDER Gábor wrote:
> > On Wed, Nov 11, 2020 at 07:58:38AM +0100, Patrick Steinhardt wrote:
> > > The testcase t1400 exercises the git-update-ref(1) utility. To do so,
> > > many tests directly read and write references via the filesystem,
> > > assuming that we always use loose and/or packed references. While this
> > > is true now, it'll change with the introduction of the reftable backend.
> > > 
> > > Convert those tests to use git-update-ref(1) and git-show-ref(1) where
> > > possible. As some tests exercise behaviour with broken references and
> > > neither of those tools actually allows writing or reading broken
> > > references, this commit doesn't adjust all tests.
> > > 
> > > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > > ---
> > >  t/t1400-update-ref.sh | 72 +++++++++++++++++++++++--------------------
> > >  1 file changed, 39 insertions(+), 33 deletions(-)
> > > 
> > > diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
> > > index 4c01e08551..d7a57488ed 100755
> > > --- a/t/t1400-update-ref.sh
> > > +++ b/t/t1400-update-ref.sh
> > 
> > > @@ -188,27 +194,26 @@ test_expect_success "move $m (by HEAD)" '
> > >  	test $B = $(git show-ref -s --verify $m)
> > >  '
> > >  test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" '
> > > -	test_when_finished "rm -f .git/$m" &&
> > > +	test_when_finished "rm -f git update-ref -d $m" &&
> > 
> > There is a leftover 'rm -f' here.
> 
> Oops, thanks a lot for catching. Funny that it still passed.

The '-f' option makes it ignore missing files like 'git', 'update-ref'
and whatever $m happens to expand to in this test, and '/bin/rm' from
coreutils (and apparently on macOS) understands the '-d' option
("remove empty directories"), so there was no error to fail this test
on common setups.  (I didn't investigate whether and how the not
deleted $m ref affects subsequent tests...)  But nowadays we have a CI
job running the tests with 'busybox sh' whose builtin 'rm' doesn't
have a '-d' option, and it errors out with:

  + rm -f git update-ref -d refs/heads/master
  rm: unrecognized option: d
  BusyBox v1.31.1 () multi-call binary.
  Usage: ......


https://travis-ci.org/github/git/git/jobs/742890453#L2853

Re: [PATCH v3 1/4] t1400: avoid touching refs on filesystem

[Jeff King]

On Wed, Nov 11, 2020 at 07:58:38AM +0100, Patrick Steinhardt wrote:

> The testcase t1400 exercises the git-update-ref(1) utility. To do so,
> many tests directly read and write references via the filesystem,
> assuming that we always use loose and/or packed references. While this
> is true now, it'll change with the introduction of the reftable backend.
> 
> Convert those tests to use git-update-ref(1) and git-show-ref(1) where
> possible. As some tests exercise behaviour with broken references and
> neither of those tools actually allows writing or reading broken
> references, this commit doesn't adjust all tests.

Do you want to mention the switch away from using HEAD in some tests
here?

> +# Some of the tests delete HEAD, which causes us to not treat the current
> +# working directory as a Git repository anymore. To avoid using any potential
> +# parent repository to be discovered, we need to set up the ceiling directories.
> +GIT_CEILING_DIRECTORIES="$PWD/.."
> +export GIT_CEILING_DIRECTORIES

Do we still need this, now that we're not deleting HEAD? I think we do
still delete a branch via HEAD, but that should leave an unborn branch,
which is still a valid repo.

>  test_expect_success "deleting current branch adds message to HEAD's log" '
> -	test_when_finished "rm -f .git/$m" &&
> +	test_when_finished "git update-ref -d $m" &&
>  	git update-ref $m $A &&
>  	git symbolic-ref HEAD $m &&
>  	git update-ref -m delete-$m -d $m &&
> -	test_path_is_missing .git/$m &&
> +	test_must_fail git show-ref --verify -q $m &&
>  	grep "delete-$m$" .git/logs/HEAD
>  '

E.g., these ones should leave a valid repo (and must remain HEAD,
because it's special for reflogging).

> -cp -f .git/HEAD .git/HEAD.orig
>  test_expect_success 'delete symref without dereference' '
> -	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
> -	git update-ref --no-deref -d HEAD &&
> -	test_path_is_missing .git/HEAD
> +	git symbolic-ref SYMREF $m &&
> +	git update-ref --no-deref -d SYMREF &&
> +	test_must_fail git show-ref --verify -q SYMREF
>  '

And now this one is safe. Good.

I wonder, though...is it still testing the same thing as the original?
This is not related to the use of SYMREF vs HEAD, but wouldn't show-ref
similarly fail if we had deleted $m, but left SYMREF in place (i.e., if
--no-deref didn't actually do anything)?

Perhaps this would be better:

  # confirm that the pointed-to ref is still there
  git show-ref --verify $m &&
  # but our symref is not
  test_must_fail git show-ref --verify SYMREF &&
  test_must_fail git symbolic-ref SYMREF

>  test_expect_success 'delete symref without dereference when the referred ref is packed' '
> -	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
>  	echo foo >foo.c &&
>  	git add foo.c &&
>  	git commit -m foo &&
> +	git symbolic-ref SYMREF $m &&
>  	git pack-refs --all &&
> -	git update-ref --no-deref -d HEAD &&
> -	test_path_is_missing .git/HEAD
> +	git update-ref --no-deref -d SYMREF &&
> +	test_must_fail git show-ref --verify -q SYMREF
>  '

Likewise here.

-Peff

Re: [PATCH v3 1/4] t1400: avoid touching refs on filesystem

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2859 bytes --]

On Wed, Nov 11, 2020 at 06:06:59PM -0500, Jeff King wrote:
> On Wed, Nov 11, 2020 at 07:58:38AM +0100, Patrick Steinhardt wrote:
> > +# Some of the tests delete HEAD, which causes us to not treat the current
> > +# working directory as a Git repository anymore. To avoid using any potential
> > +# parent repository to be discovered, we need to set up the ceiling directories.
> > +GIT_CEILING_DIRECTORIES="$PWD/.."
> > +export GIT_CEILING_DIRECTORIES
> 
> Do we still need this, now that we're not deleting HEAD? I think we do
> still delete a branch via HEAD, but that should leave an unborn branch,
> which is still a valid repo.

Good point, we don't.

> >  test_expect_success "deleting current branch adds message to HEAD's log" '
> > -	test_when_finished "rm -f .git/$m" &&
> > +	test_when_finished "git update-ref -d $m" &&
> >  	git update-ref $m $A &&
> >  	git symbolic-ref HEAD $m &&
> >  	git update-ref -m delete-$m -d $m &&
> > -	test_path_is_missing .git/$m &&
> > +	test_must_fail git show-ref --verify -q $m &&
> >  	grep "delete-$m$" .git/logs/HEAD
> >  '
> 
> E.g., these ones should leave a valid repo (and must remain HEAD,
> because it's special for reflogging).
> 
> > -cp -f .git/HEAD .git/HEAD.orig
> >  test_expect_success 'delete symref without dereference' '
> > -	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
> > -	git update-ref --no-deref -d HEAD &&
> > -	test_path_is_missing .git/HEAD
> > +	git symbolic-ref SYMREF $m &&
> > +	git update-ref --no-deref -d SYMREF &&
> > +	test_must_fail git show-ref --verify -q SYMREF
> >  '
> 
> And now this one is safe. Good.
> 
> I wonder, though...is it still testing the same thing as the original?
> This is not related to the use of SYMREF vs HEAD, but wouldn't show-ref
> similarly fail if we had deleted $m, but left SYMREF in place (i.e., if
> --no-deref didn't actually do anything)?
> 
> Perhaps this would be better:
> 
>   # confirm that the pointed-to ref is still there
>   git show-ref --verify $m &&
>   # but our symref is not
>   test_must_fail git show-ref --verify SYMREF &&
>   test_must_fail git symbolic-ref SYMREF

It would be, but I bailed at this point because we don't actually have
"$m" at this point. But agreed, i'll also include this into both tests.

Patrick

> >  test_expect_success 'delete symref without dereference when the referred ref is packed' '
> > -	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
> >  	echo foo >foo.c &&
> >  	git add foo.c &&
> >  	git commit -m foo &&
> > +	git symbolic-ref SYMREF $m &&
> >  	git pack-refs --all &&
> > -	git update-ref --no-deref -d HEAD &&
> > -	test_path_is_missing .git/HEAD
> > +	git update-ref --no-deref -d SYMREF &&
> > +	test_must_fail git show-ref --verify -q SYMREF
> >  '
> 
> Likewise here.
> 
> -Peff

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 0/4] update-ref: allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

Hi,

this is the fourth version of this patch series implementing support for
creation of multiple reference transactions in a single git-update-ref
process.

All changes are only in the first patch updating t1400:

    - A stale 'rm -f' was removed.

    - The workaround around broken repos was removed as we no longer
      delete HEAD.

    - Tests have been improved to verify that deleting a symref won't
      delete their target.

Patrick


Patrick Steinhardt (4):
  t1400: avoid touching refs on filesystem
  update-ref: allow creation of multiple transactions
  p1400: use `git-update-ref --stdin` to test multiple transactions
  update-ref: disallow "start" for ongoing transactions

 Documentation/git-update-ref.txt |   3 +-
 builtin/update-ref.c             |  15 +++-
 t/perf/p1400-update-ref.sh       |  20 ++---
 t/t1400-update-ref.sh            | 138 +++++++++++++++++++++++--------
 4 files changed, 126 insertions(+), 50 deletions(-)

-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 1/4] t1400: avoid touching refs on filesystem

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8956 bytes --]

The testcase t1400 exercises the git-update-ref(1) utility. To do so,
many tests directly read and write references via the filesystem,
assuming that we always use loose and/or packed references. While this
is true now, it'll change with the introduction of the reftable backend.

Convert those tests to use git-update-ref(1) and git-show-ref(1) where
possible. Furthermore, two tests are converted to not delete HEAD
anymore, as this results in a broken repository. They've instead been
updated to create a non-mandatory symybolic reference and delete that
one instead.

Some tests remain which exercise behaviour with broken references, which
cannot currently be converted to use regular git tooling.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/t1400-update-ref.sh | 77 +++++++++++++++++++++++--------------------
 1 file changed, 42 insertions(+), 35 deletions(-)

diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 4c01e08551..b782dafff5 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -48,17 +48,17 @@ test_expect_success "fail to delete $m with stale ref" '
 	test $B = "$(git show-ref -s --verify $m)"
 '
 test_expect_success "delete $m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref -d $m $B &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "delete $m without oldvalue verification" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	test $A = $(git show-ref -s --verify $m) &&
 	git update-ref -d $m &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "fail to create $n" '
@@ -80,26 +80,26 @@ test_expect_success "fail to delete $m (by HEAD) with stale ref" '
 	test $B = $(git show-ref -s --verify $m)
 '
 test_expect_success "delete $m (by HEAD)" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref -d HEAD $B &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success "deleting current branch adds message to HEAD's log" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	git symbolic-ref HEAD $m &&
 	git update-ref -m delete-$m -d $m &&
-	test_path_is_missing .git/$m &&
+	test_must_fail git show-ref --verify -q $m &&
 	grep "delete-$m$" .git/logs/HEAD
 '
 
 test_expect_success "deleting by HEAD adds message to HEAD's log" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref $m $A &&
 	git symbolic-ref HEAD $m &&
 	git update-ref -m delete-by-head -d HEAD &&
-	test_path_is_missing .git/$m &&
+	test_must_fail git show-ref --verify -q $m &&
 	grep "delete-by-head$" .git/logs/HEAD
 '
 
@@ -188,31 +188,37 @@ test_expect_success "move $m (by HEAD)" '
 	test $B = $(git show-ref -s --verify $m)
 '
 test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	git update-ref -d HEAD $B &&
 	! grep "$m" .git/packed-refs &&
-	test_path_is_missing .git/$m
+	test_must_fail git show-ref --verify -q $m
 '
 
-cp -f .git/HEAD .git/HEAD.orig
 test_expect_success 'delete symref without dereference' '
-	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
-	git update-ref --no-deref -d HEAD &&
-	test_path_is_missing .git/HEAD
+	test_when_finished "git update-ref -d $m" &&
+	echo foo >foo.c &&
+	git add foo.c &&
+	git commit -m foo &&
+	git symbolic-ref SYMREF $m &&
+	git update-ref --no-deref -d SYMREF &&
+	git show-ref --verify -q $m &&
+	test_must_fail git show-ref --verify -q SYMREF &&
+	test_must_fail git symbolic-ref SYMREF
 '
 
 test_expect_success 'delete symref without dereference when the referred ref is packed' '
-	test_when_finished "cp -f .git/HEAD.orig .git/HEAD" &&
+	test_when_finished "git update-ref -d $m" &&
 	echo foo >foo.c &&
 	git add foo.c &&
 	git commit -m foo &&
+	git symbolic-ref SYMREF $m &&
 	git pack-refs --all &&
-	git update-ref --no-deref -d HEAD &&
-	test_path_is_missing .git/HEAD
+	git update-ref --no-deref -d SYMREF &&
+	git show-ref --verify -q $m &&
+	test_must_fail git show-ref --verify -q SYMREF &&
+	test_must_fail git symbolic-ref SYMREF
 '
 
-git update-ref -d $m
-
 test_expect_success 'update-ref -d is not confused by self-reference' '
 	git symbolic-ref refs/heads/self refs/heads/self &&
 	test_when_finished "rm -f .git/refs/heads/self" &&
@@ -226,25 +232,25 @@ test_expect_success 'update-ref --no-deref -d can delete self-reference' '
 	test_when_finished "rm -f .git/refs/heads/self" &&
 	test_path_is_file .git/refs/heads/self &&
 	git update-ref --no-deref -d refs/heads/self &&
-	test_path_is_missing .git/refs/heads/self
+	test_must_fail git show-ref --verify -q refs/heads/self
 '
 
 test_expect_success 'update-ref --no-deref -d can delete reference to bad ref' '
 	>.git/refs/heads/bad &&
 	test_when_finished "rm -f .git/refs/heads/bad" &&
 	git symbolic-ref refs/heads/ref-to-bad refs/heads/bad &&
-	test_when_finished "rm -f .git/refs/heads/ref-to-bad" &&
+	test_when_finished "git update-ref -d refs/heads/ref-to-bad" &&
 	test_path_is_file .git/refs/heads/ref-to-bad &&
 	git update-ref --no-deref -d refs/heads/ref-to-bad &&
-	test_path_is_missing .git/refs/heads/ref-to-bad
+	test_must_fail git show-ref --verify -q refs/heads/ref-to-bad
 '
 
 test_expect_success '(not) create HEAD with old sha1' '
 	test_must_fail git update-ref HEAD $A $B
 '
 test_expect_success "(not) prior created .git/$m" '
-	test_when_finished "rm -f .git/$m" &&
-	test_path_is_missing .git/$m
+	test_when_finished "git update-ref -d $m" &&
+	test_must_fail git show-ref --verify -q $m
 '
 
 test_expect_success 'create HEAD' '
@@ -254,7 +260,7 @@ test_expect_success '(not) change HEAD with wrong SHA1' '
 	test_must_fail git update-ref HEAD $B $Z
 '
 test_expect_success "(not) changed .git/$m" '
-	test_when_finished "rm -f .git/$m" &&
+	test_when_finished "git update-ref -d $m" &&
 	! test $B = $(git show-ref -s --verify $m)
 '
 
@@ -284,8 +290,8 @@ test_expect_success 'empty directory removal' '
 	test_path_is_file .git/refs/heads/d1/d2/r1 &&
 	test_path_is_file .git/logs/refs/heads/d1/d2/r1 &&
 	git branch -d d1/d2/r1 &&
-	test_path_is_missing .git/refs/heads/d1/d2 &&
-	test_path_is_missing .git/logs/refs/heads/d1/d2 &&
+	test_must_fail git show-ref --verify -q refs/heads/d1/d2 &&
+	test_must_fail git show-ref --verify -q logs/refs/heads/d1/d2 &&
 	test_path_is_file .git/refs/heads/d1/r2 &&
 	test_path_is_file .git/logs/refs/heads/d1/r2
 '
@@ -298,8 +304,8 @@ test_expect_success 'symref empty directory removal' '
 	test_path_is_file .git/refs/heads/e1/e2/r1 &&
 	test_path_is_file .git/logs/refs/heads/e1/e2/r1 &&
 	git update-ref -d HEAD &&
-	test_path_is_missing .git/refs/heads/e1/e2 &&
-	test_path_is_missing .git/logs/refs/heads/e1/e2 &&
+	test_must_fail git show-ref --verify -q refs/heads/e1/e2 &&
+	test_must_fail git show-ref --verify -q logs/refs/heads/e1/e2 &&
 	test_path_is_file .git/refs/heads/e1/r2 &&
 	test_path_is_file .git/logs/refs/heads/e1/r2 &&
 	test_path_is_file .git/logs/HEAD
@@ -1388,7 +1394,8 @@ test_expect_success 'handle per-worktree refs in refs/bisect' '
 		git rev-parse refs/bisect/something >../worktree-head &&
 		git for-each-ref | grep refs/bisect/something
 	) &&
-	test_path_is_missing .git/refs/bisect &&
+	git show-ref >actual &&
+	! grep 'refs/bisect' actual &&
 	test_must_fail git rev-parse refs/bisect/something &&
 	git update-ref refs/bisect/something HEAD &&
 	git rev-parse refs/bisect/something >main-head &&
@@ -1500,7 +1507,7 @@ test_expect_success 'transaction can handle abort' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start abort >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_expect_success 'transaction aborts by default' '
@@ -1511,7 +1518,7 @@ test_expect_success 'transaction aborts by default' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_expect_success 'transaction with prepare aborts by default' '
@@ -1523,7 +1530,7 @@ test_expect_success 'transaction with prepare aborts by default' '
 	git update-ref --stdin <stdin >actual &&
 	printf "%s: ok\n" start prepare >expect &&
 	test_cmp expect actual &&
-	test_path_is_missing .git/$b
+	test_must_fail git show-ref --verify -q $b
 '
 
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 2/4] update-ref: allow creation of multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3695 bytes --]

While git-update-ref has recently grown commands which allow interactive
control of transactions in e48cf33b61 (update-ref: implement interactive
transaction handling, 2020-04-02), it is not yet possible to create
multiple transactions in a single session. To do so, one currently still
needs to invoke the executable multiple times.

This commit addresses this shortcoming by allowing the "start" command
to create a new transaction if the current transaction has already been
either committed or aborted.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Documentation/git-update-ref.txt |  3 +-
 builtin/update-ref.c             | 13 ++++++++-
 t/t1400-update-ref.sh            | 50 ++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-update-ref.txt b/Documentation/git-update-ref.txt
index d401234b03..48b6683071 100644
--- a/Documentation/git-update-ref.txt
+++ b/Documentation/git-update-ref.txt
@@ -125,7 +125,8 @@ option::
 start::
 	Start a transaction. In contrast to a non-transactional session, a
 	transaction will automatically abort if the session ends without an
-	explicit commit.
+	explicit commit. This command may create a new empty transaction when
+	the current one has been committed or aborted already.
 
 prepare::
 	Prepare to commit the transaction. This will create lock files for all
diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 8a2df4459c..bb65129012 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -446,7 +446,18 @@ static void update_refs_stdin(void)
 			state = cmd->state;
 			break;
 		case UPDATE_REFS_CLOSED:
-			die("transaction is closed");
+			if (cmd->state != UPDATE_REFS_STARTED)
+				die("transaction is closed");
+
+			/*
+			 * Open a new transaction if we're currently closed and
+			 * get a "start".
+			 */
+			state = cmd->state;
+			transaction = ref_transaction_begin(&err);
+			if (!transaction)
+				die("%s", err.buf);
+
 			break;
 		}
 
diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index b782dafff5..3144e98b31 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1533,4 +1533,54 @@ test_expect_success 'transaction with prepare aborts by default' '
 	test_must_fail git show-ref --verify -q $b
 '
 
+test_expect_success 'transaction can commit multiple times' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/branch-1 $A
+	commit
+	start
+	create refs/heads/branch-2 $B
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_cmp expect actual &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/branch-1 >actual &&
+	test_cmp expect actual &&
+	echo "$B" >expect &&
+	git rev-parse refs/heads/branch-2 >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'transaction can create and delete' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/create-and-delete $A
+	commit
+	start
+	delete refs/heads/create-and-delete $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start commit start commit >expect &&
+	test_must_fail git show-ref --verify refs/heads/create-and-delete
+'
+
+test_expect_success 'transaction can commit after abort' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/abort $A
+	abort
+	start
+	create refs/heads/abort $A
+	commit
+	EOF
+	git update-ref --stdin <stdin >actual &&
+	printf "%s: ok\n" start abort start commit >expect &&
+	echo "$A" >expect &&
+	git rev-parse refs/heads/abort >actual &&
+	test_cmp expect actual
+'
+
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 3/4] p1400: use `git-update-ref --stdin` to test multiple transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

In commit 0a0fbbe3ff (refs: remove lookup cache for
reference-transaction hook, 2020-08-25), a new benchmark was added to
p1400 which has the intention to exercise creation of multiple
transactions in a single process. As git-update-ref wasn't yet able to
create multiple transactions with a single run we instead used git-push.
As its non-atomic version creates a transaction per reference update,
this was the best approximation we could make at that point in time.

Now that `git-update-ref --stdin` supports creation of multiple
transactions, let's convert the benchmark to use that instead. It has
less overhead and it's also a lot clearer what the actual intention is.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 t/perf/p1400-update-ref.sh | 20 +++++++-------------
 1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/t/perf/p1400-update-ref.sh b/t/perf/p1400-update-ref.sh
index ce5ac3ed85..dda8a74866 100755
--- a/t/perf/p1400-update-ref.sh
+++ b/t/perf/p1400-update-ref.sh
@@ -7,13 +7,14 @@ test_description="Tests performance of update-ref"
 test_perf_fresh_repo
 
 test_expect_success "setup" '
-	git init --bare target-repo.git &&
 	test_commit PRE &&
 	test_commit POST &&
-	printf "create refs/heads/%d PRE\n" $(test_seq 1000) >create &&
-	printf "update refs/heads/%d POST PRE\n" $(test_seq 1000) >update &&
-	printf "delete refs/heads/%d POST\n" $(test_seq 1000) >delete &&
-	git update-ref --stdin <create
+	for i in $(test_seq 5000)
+	do
+		printf "start\ncreate refs/heads/%d PRE\ncommit\n" $i &&
+		printf "start\nupdate refs/heads/%d POST PRE\ncommit\n" $i &&
+		printf "start\ndelete refs/heads/%d POST\ncommit\n" $i
+	done >instructions
 '
 
 test_perf "update-ref" '
@@ -26,14 +27,7 @@ test_perf "update-ref" '
 '
 
 test_perf "update-ref --stdin" '
-	git update-ref --stdin <update &&
-	git update-ref --stdin <delete &&
-	git update-ref --stdin <create
-'
-
-test_perf "nonatomic push" '
-	git push ./target-repo.git $(test_seq 1000) &&
-	git push --delete ./target-repo.git $(test_seq 1000)
+	git update-ref --stdin <instructions >/dev/null
 '
 
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v4 4/4] update-ref: disallow "start" for ongoing transactions

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1955 bytes --]

It is currently possible to write multiple "start" commands into
git-update-ref(1) for a single session, but none of them except for the
first one actually have any effect.

Using such nested "start"s may eventually have a sensible effect. One
may imagine that it restarts the current transaction, effectively
emptying it and creating a new one. It may also allow for creation of
nested transactions. But currently, none of these are implemented.

Silently ignoring this misuse is making it hard to iterate in the future
if "start" is ever going to have meaningful semantics in such a context.
This commit thus makes sure to error out in case we see such use.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/update-ref.c  |  2 ++
 t/t1400-update-ref.sh | 11 +++++++++++
 2 files changed, 13 insertions(+)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index bb65129012..6029a80544 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -436,6 +436,8 @@ static void update_refs_stdin(void)
 		switch (state) {
 		case UPDATE_REFS_OPEN:
 		case UPDATE_REFS_STARTED:
+			if (state == UPDATE_REFS_STARTED && cmd->state == UPDATE_REFS_STARTED)
+				die("cannot restart ongoing transaction");
 			/* Do not downgrade a transaction to a non-transaction. */
 			if (cmd->state >= state)
 				state = cmd->state;
diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh
index 3144e98b31..31b64be521 100755
--- a/t/t1400-update-ref.sh
+++ b/t/t1400-update-ref.sh
@@ -1583,4 +1583,15 @@ test_expect_success 'transaction can commit after abort' '
 	test_cmp expect actual
 '
 
+test_expect_success 'transaction cannot restart ongoing transaction' '
+	cat >stdin <<-EOF &&
+	start
+	create refs/heads/restart $A
+	start
+	commit
+	EOF
+	test_must_fail git update-ref --stdin <stdin >actual &&
+	test_must_fail git show-ref --verify refs/heads/restart
+'
+
 test_done
-- 
2.29.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 1/4] t1400: avoid touching refs on filesystem

[Jeff King]

On Fri, Nov 13, 2020 at 09:12:31AM +0100, Patrick Steinhardt wrote:

> The testcase t1400 exercises the git-update-ref(1) utility. To do so,
> many tests directly read and write references via the filesystem,
> assuming that we always use loose and/or packed references. While this
> is true now, it'll change with the introduction of the reftable backend.
> 
> Convert those tests to use git-update-ref(1) and git-show-ref(1) where
> possible. Furthermore, two tests are converted to not delete HEAD
> anymore, as this results in a broken repository. They've instead been
> updated to create a non-mandatory symybolic reference and delete that
> one instead.

s/symybolic/symbolic/

Other than, this whole series looks good to me. Thanks for taking the
time to do the extra cleanup (which ended up being way more complicated
than the original goal :) ).

-Peff

Re: [PATCH v4 1/4] t1400: avoid touching refs on filesystem

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

On Fri, Nov 13, 2020 at 03:40:31PM -0500, Jeff King wrote:
> On Fri, Nov 13, 2020 at 09:12:31AM +0100, Patrick Steinhardt wrote:
> 
> > The testcase t1400 exercises the git-update-ref(1) utility. To do so,
> > many tests directly read and write references via the filesystem,
> > assuming that we always use loose and/or packed references. While this
> > is true now, it'll change with the introduction of the reftable backend.
> > 
> > Convert those tests to use git-update-ref(1) and git-show-ref(1) where
> > possible. Furthermore, two tests are converted to not delete HEAD
> > anymore, as this results in a broken repository. They've instead been
> > updated to create a non-mandatory symybolic reference and delete that
> > one instead.
> 
> s/symybolic/symbolic/
> 
> Other than, this whole series looks good to me. Thanks for taking the
> time to do the extra cleanup (which ended up being way more complicated
> than the original goal :) ).

Thanks!

Junio, shall I fix this typo with another version or will you fix this
up locally?

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 1/4] t1400: avoid touching refs on filesystem

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> Junio, shall I fix this typo with another version or will you fix this
> up locally?

Please make it a habit to check what is on 'seen' before asking.

Given we have timezone differences (and this late at local night I
am not in an environment in which I can check it myself easily),
doing so would be much faster.

Thanks.

Re: [PATCH v4 0/4] update-ref: allow creation of multiple transactions

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> this is the fourth version of this patch series implementing support for
> creation of multiple reference transactions in a single git-update-ref
> process.

It is my impression that the series is now in good enough shape that
we didn't see much discussion on this round.

So I'll mark this to be merged to 'next', but I ask reviewers to
please holler to stop me otherwise.

Thanks.

Re: [PATCH v4 0/4] update-ref: allow creation of multiple transactions

[Jeff King]

On Wed, Nov 25, 2020 at 02:37:54PM -0800, Junio C Hamano wrote:

> Patrick Steinhardt <ps@pks.im> writes:
> 
> > this is the fourth version of this patch series implementing support for
> > creation of multiple reference transactions in a single git-update-ref
> > process.
> 
> It is my impression that the series is now in good enough shape that
> we didn't see much discussion on this round.
> 
> So I'll mark this to be merged to 'next', but I ask reviewers to
> please holler to stop me otherwise.

Yeah, this looks good to me for advancing. Thanks.

-Peff