[PATCH] t3429: try to protect against a potential racy todo file problem

["SZEDER Gábor" <szeder.dev@gmail.com>]

On Sun, Nov 24, 2019 at 10:44:10AM +0000, Phillip Wood wrote:
> On 24/11/2019 04:49, Junio C Hamano wrote:
> 
> Thanks for working on this Gábor
> 
> >SZEDER Gábor <szeder.dev@gmail.com> writes:
> >
> >>When 'git revert' or 'git cherry-pick --edit' is invoked with multiple
> >>commits, then after editing the first commit message is finished both
> >
> >"commits, then after editing the first commit message, both of" I
> >would say.
> >
> >>these commands should continue with processing the second commit and
> >>launch another editor for its commit message, assuming there are
> >>no conflicts, of course.
> >>
> >>Alas, this inadvertently changed with commit a47ba3c777 (rebase -i:
> >>check for updated todo after squash and reword, 2019-08-19): after
> >>editing the first commit message is finished, both 'git revert' and
> >>'git cherry-pick --edit' exit with error, claiming that "nothing to
> >>commit, working tree clean".
> >>...
> >>   - When invoking 'git revert' or 'git cherry-pick --edit' with
> >>     multiple commits they don't read a todo list file but assemble the
> >>     todo list in memory, thus the associated stat data used to check
> >>     whether the file has been updated is all zeroed out initially.
> >>
> >>     Then the sequencer writes all instructions (including the very
> >>     first) to the todo file, executes the first 'revert/pick'
> >>     instruction, and after the user finished editing the commit
> >>     message the changes of a47ba3c777 kick in, and it checks whether
> >>     the todo file has been modified.  The initial all-zero stat data
> >>     obviously differs from the todo file's current stat data, so the
> >>     sequencer concludes that the file has been modified.  Technically
> >>     it is not wrong, of course, because the file just has been written
> >>     indeed by the sequencer itself, though the file's contents still
> >>     match what the sequencer was invoked with in the beginning.
> >>     Consequently, after re-reading the todo file the sequencer
> >>     executes the same first instruction _again_, thus ending up in
> >>     that "nothing to commit" situation.
> >
> >Hmph, that makes it sound as if the right fix is to re-read after
> >writing the first version of the todo file out, so that the stat
> >data matches reality and tells us that it has never been modified?
> 
> I think we should update the stat data after we write the todo list.

Well, yes and no.

No, because we are dealing with regression in v2.24.0 here, so the
simpler the fix the better it is for maint.  I don't think a fix can
get any simpler than my patch, with or without the suggestions from
Phillip.

Yes, we should definitely consider updating the stat data after the
sequencer writes the todo list, or any other options with which the
sequencer could notice a modified todo list file with less subtlety.
Alas, there is a big can of worms in that direction, see the patch
below, and we have to be very careful going that way, so I think it's
only viable in the long term, but less suitable as a regression fix
for maint.

(Hrm, perhaps I spent too many words on the all zeroed out stat data,
and managed to sidetrack you a bit...)

  ---  >8  ---

Subject: [PATCH] t3429: try to protect against a potential racy todo file
 problem

During an interactive rebase session the user can run basically any
commands with an 'exec' instruction, including commands that modify
the todo list of the ongoing rebase.  Similarly, the user might choose
to modify the todo list while editing a commit message for a 'reword'
or a 'squash' instruction.  The ongoing interactive rebase process
should be able to notice the modified todo file and re-read it to
follow the updated instructions.  This has been working well for the
'exec' instruction for some time, but for 'reword' and 'squash' it has
been fixed only recently in a47ba3c777 (rebase -i: check for updated
todo after squash and reword, 2019-08-19).

To notice a changed todo file the sequencer stores the file's stat
data in its 'struct todo_list' instance, and compares it with the
file's current stat data after 'reword', 'squash' and 'exec'
instructions.  If the two stat data doesn't match, it re-reads the
todo file.

Sounds simple, but there are some subtleties going on here:

  - The 'struct todo_list' holds the stat data from the time when the
    todo file was last read.

  - This stat data in 'struct todo_list' is not updated when the
    sequencer itself writes the todo file.

  - Before executing each instruction during an interactive rebase,
    the sequencer always updates the todo file by removing the
    just-about-to-be-executed instruction.  This changes the file's
    size and inode [1].

Consequently, when the sequencer looks at the stat data after a
'reword', 'squash' or 'exec' instruction, it most likely finds that
they differ, even when the user didn't modify the todo list at all!
This is not an issue in practice, it just wastes a few cycles on
re-reading the todo list that matches what the sequencer already has
in memory anyway.

However, an unsuspecting Git developer might try to "fix" it by simply
updating the stat data each time the sequencer writes the todo list
for an interactive rebase.  On first sight it looks quite sensible and
straightforward, but we have to be very careful when doing that,
because potential racy problems lie that way.

It is possible to overwrite the todo list file without modifying
either its inode or size, and if such an overwrite were to happen in
the same second when the file was last read (our stat data has one
second granularity by default), then the actual stat data on the file
system would match the stat data that the sequencer has in memory.
Consequently, such a modification to the todo list file would go
unnoticed.

Add a test to 't3429-rebase-edit-todo.sh' that modifies the todo list
during an interactive rebase in a way that the file's stat data very
likely doesn't change.  Since we have no control over ctime repeat the
test a few times, just like we do in the racy git and racy split index
tests (t0010 and t1701).  This test succeeds right now because of all
the above, but it does fail most of the time if the stat data is
naively updated when writing the todo file [2].  Hopefully this test
will help prevent us from introducing such racy problems if we ever
change how the sequencer looks out for modified todo list files.

This new test only checks what it's supposed to when it doesn't change
the size of the todo file while modifying it, thus it critically
depends on the todo file format.  And the todo file format did change
in the past: early on it stored abbreviated commit object ids, but
since 75c6976655 (rebase -i: fix short SHA-1 collision, 2013-08-23) it
stores full object ids.  Perhaps it will change in the future as well,
so be extra defensive about this, and check that the file size before
and after the edit actually match, and complain with a "bug in the
test script" if they don't.

[1] The todo file is written using the lockfile API, i.e. first to the
    lockfile, which is then moved to place, so the new file can't
    possibly have the same inode as the file it replaces.  Note,
    however, that the file system might reuse inodes, so it is
    possible that the new todo file ends up with the same inode as is
    recorded in the 'struct todo_list' from the last time the file was
    read.

[2] To trigger failure in the new test updating the stat data when
    writing the todo file like the diff below does.  No other test in
    our test suite seems to fail with this change applied.

    diff --git a/sequencer.c b/sequencer.c
    index 3b05d0277d..b2acb7d536 100644
    --- a/sequencer.c
    +++ b/sequencer.c
    @@ -2730,6 +2730,7 @@ static int save_todo(struct todo_list *todo_list, struct replay_opts *opts)
     	struct lock_file todo_lock = LOCK_INIT;
     	const char *todo_path = get_todo_path(opts);
     	int next = todo_list->current, offset, fd;
    +	struct stat st;

     	/*
     	 * rebase -i writes "git-rebase-todo" without the currently executing
    @@ -2748,6 +2749,10 @@ static int save_todo(struct todo_list *todo_list, struct replay_opts *opts)
     	if (commit_lock_file(&todo_lock) < 0)
     		return error(_("failed to finalize '%s'"), todo_path);

    +	if (stat(todo_path, &st) < 0)
    +		return error(_("could not stat '%s'"), todo_path);
    +	fill_stat_data(&todo_list->stat, &st);
    +
     	if (is_rebase_i(opts) && next > 0) {
     		const char *done = rebase_path_done();
     		int fd = open(done, O_CREAT | O_WRONLY | O_APPEND, 0666);

Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
---
 t/t3429-rebase-edit-todo.sh | 59 +++++++++++++++++++++++++++++++++++++
 1 file changed, 59 insertions(+)

diff --git a/t/t3429-rebase-edit-todo.sh b/t/t3429-rebase-edit-todo.sh
index 8739cb60a7..c80c139b47 100755
--- a/t/t3429-rebase-edit-todo.sh
+++ b/t/t3429-rebase-edit-todo.sh
@@ -52,4 +52,63 @@ test_expect_success 'todo is re-read after reword and squash' '
 	test_cmp expected actual
 '
 
+test_expect_success 'setup for racy tests' '
+	write_script sequence-editor <<-EOS &&
+		cat >.git/rebase-merge/git-rebase-todo <<-\EOF
+			r $(git rev-parse second^0) second
+			r $(git rev-parse third^0) third
+		EOF
+	EOS
+
+	write_script commit-editor <<-\EOS &&
+		read first_line <"$1" &&
+		echo "$first_line - edited" >"$1" &&
+
+		todo=.git/rebase-merge/git-rebase-todo &&
+
+		if test "$first_line" = second
+		then
+			old_size=$(wc -c <"$todo") &&
+			# Replace the "reword <full-oid> third" line with
+			# a different instruction of the same line length.
+			# Overwrite the file in-place, so the inode stays
+			# the same as well.
+			cat >"$todo" <<-EOF &&
+				exec echo 0123456789012345678901234 >exec-cmd-was-run
+			EOF
+			new_size=$(wc -c <"$todo") &&
+
+			if test $old_size -ne $new_size
+			then
+				echo >&2 "error: bug in the test script: the size of the todo list must not change"
+				exit 1
+			fi
+		fi
+	EOS
+
+	cat >expect <<-\EOF
+	second - edited
+	first
+	EOF
+'
+
+# This test can give false success if your machine is sufficiently
+# slow or all trials happened to happen on second boundaries.
+
+for trial in 0 1 2 3 4
+do
+	test_expect_success "racy todo re-read #$trial" '
+		git reset --hard third &&
+		rm -rf exec-cmd-was-run &&
+
+		GIT_SEQUENCE_EDITOR=./sequence-editor \
+		GIT_EDITOR=./commit-editor \
+		git rebase -i HEAD^^ &&
+
+		test_path_is_file exec-cmd-was-run &&
+		git log --format=%s >actual &&
+		test_cmp expect actual
+	'
+done
+
 test_done
-- 
2.24.0.532.ge18579ded8