[PATCH 0/3] gc: fix handling of crontab magic markers

[PATCH 0/3] gc: fix handling of crontab magic markers

[Martin Ågren]

This is a fix for a new bug in the v2.30 cycle:

`git maintenance [start|stop]` add and/or remove entries to/from the
user's cron table. When inserting entries, we add magic "BEGIN" and
"END" markers. When we later removes these entries, we remove everything
from "BEGIN" to ... the end of file. A realistic scenario for hitting
this bug is

  $ git maintenance start
  $ crontab -e # add stuff at the end of the file
  $ git maintenance stop
  $ crontab -l # stuff from above is gone

The second patch is the actual fix. The first patch is just a very minor
fix to the documentation. The third patch future-proofs the magic
markers, so that we can be reasonably sure that all future versions of
Git stick to these strings.

Martin Ågren (3):
  git-maintenance.txt: add missing word
  gc: fix handling of crontab magic markers
  t7900-maintenance: test for magic markers

 Documentation/git-maintenance.txt |  2 +-
 t/t7900-maintenance.sh            | 16 ++++++++++++++++
 builtin/gc.c                      |  7 +++----
 3 files changed, 20 insertions(+), 5 deletions(-)

-- 
2.30.0.rc1

[PATCH 1/3] git-maintenance.txt: add missing word

[Martin Ågren]

Add a missing "a" before "bunch".

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 Documentation/git-maintenance.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt
index 6fec1eb8dc..d1f9b5172d 100644
--- a/Documentation/git-maintenance.txt
+++ b/Documentation/git-maintenance.txt
@@ -101,7 +101,7 @@ This is done to avoid disrupting the remote-tracking branches. The end users
 expect these refs to stay unmoved unless they initiate a fetch.  With prefetch
 task, however, the objects necessary to complete a later real fetch would
 already be obtained, so the real fetch would go faster.  In the ideal case,
-it will just become an update to bunch of remote-tracking branches without
+it will just become an update to a bunch of remote-tracking branches without
 any object transfer.
 
 gc::
-- 
2.30.0.rc1

[PATCH 2/3] gc: fix handling of crontab magic markers

[Martin Ågren]

On `git maintenance start`, we add a few entries to the user's cron
table. We wrap our entries using two magic markers, "# BEGIN GIT
MAINTENANCE SCHEDULE" and "# END GIT MAINTENANCE SCHEDULE". At a later
`git maintenance stop`, we will go through the table and remove these
lines. Or rather, we will remove the "BEGIN" marker, the "END" marker
and everything between them.

Alas, we have a bug in how we detect the "END" marker: we don't. As we
loop through all the lines of the crontab, if we are in the "old
region", i.e., the region we're aiming to remove, we make an early
`continue` and don't get as far as checking for the "END" marker. Thus,
once we've seen our "BEGIN", we remove everything until the end of the
file.

Rewrite the logic for identifying these markers. There are four cases
that are mutually exclusive: The current line starts a region or it ends
it, or it's firmly within the region, or it's outside of it (and should
be printed).

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 t/t7900-maintenance.sh | 7 +++++++
 builtin/gc.c           | 7 +++----
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
index d1e0c8f830..4bbfce31e9 100755
--- a/t/t7900-maintenance.sh
+++ b/t/t7900-maintenance.sh
@@ -446,6 +446,13 @@ test_expect_success 'start preserves existing schedule' '
 	grep "Important information!" cron.txt
 '
 
+test_expect_success 'stop preserves surrounding schedule' '
+	echo "Crucial information!" >>cron.txt &&
+	GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop &&
+	grep "Important information!" cron.txt &&
+	grep "Crucial information!" cron.txt
+'
+
 test_expect_success 'register preserves existing strategy' '
 	git config maintenance.strategy none &&
 	git maintenance register &&
diff --git a/builtin/gc.c b/builtin/gc.c
index b57fda4924..4c24f41852 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -1554,11 +1554,10 @@ static int update_background_schedule(int run_maintenance)
 	while (!strbuf_getline_lf(&line, cron_list)) {
 		if (!in_old_region && !strcmp(line.buf, BEGIN_LINE))
 			in_old_region = 1;
-		if (in_old_region)
-			continue;
-		fprintf(cron_in, "%s\n", line.buf);
-		if (in_old_region && !strcmp(line.buf, END_LINE))
+		else if (in_old_region && !strcmp(line.buf, END_LINE))
 			in_old_region = 0;
+		else if (!in_old_region)
+			fprintf(cron_in, "%s\n", line.buf);
 	}
 
 	if (run_maintenance) {
-- 
2.30.0.rc1

[PATCH 3/3] t7900-maintenance: test for magic markers

[Martin Ågren]

When we insert our "BEGIN" and "END" markers into the cron table, it's
so that a Git version from many years into the future would be able to
identify this region in the cron table. Let's add a test to make sure
that these markers don't ever change.

Signed-off-by: Martin Ågren <martin.agren@gmail.com>
---
 t/t7900-maintenance.sh | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
index 4bbfce31e9..99bf0c7582 100755
--- a/t/t7900-maintenance.sh
+++ b/t/t7900-maintenance.sh
@@ -446,6 +446,15 @@ test_expect_success 'start preserves existing schedule' '
 	grep "Important information!" cron.txt
 '
 
+test_expect_success 'magic markers are correct' '
+	grep "GIT MAINTENANCE SCHEDULE" cron.txt >actual &&
+	cat >expect <<-\EOF &&
+	# BEGIN GIT MAINTENANCE SCHEDULE
+	# END GIT MAINTENANCE SCHEDULE
+	EOF
+	test_cmp actual expect
+'
+
 test_expect_success 'stop preserves surrounding schedule' '
 	echo "Crucial information!" >>cron.txt &&
 	GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop &&
-- 
2.30.0.rc1

Re: [PATCH 0/3] gc: fix handling of crontab magic markers

[Derrick Stolee]

On 12/21/2020 4:26 PM, Martin Ågren wrote:
> This is a fix for a new bug in the v2.30 cycle:
> 
> `git maintenance [start|stop]` add and/or remove entries to/from the
> user's cron table. When inserting entries, we add magic "BEGIN" and
> "END" markers. When we later removes these entries, we remove everything
> from "BEGIN" to ... the end of file. A realistic scenario for hitting
> this bug is
> 
>   $ git maintenance start
>   $ crontab -e # add stuff at the end of the file
>   $ git maintenance stop
>   $ crontab -l # stuff from above is gone
> 
> The second patch is the actual fix. The first patch is just a very minor
> fix to the documentation. The third patch future-proofs the magic
> markers, so that we can be reasonably sure that all future versions of
> Git stick to these strings.

Thank you for the attention to detail here. I think this series
is succinct and completely correct. Apologies for missing the
obvious "test after the region, too!"

Thanks,
-Stolee

Re: [PATCH 2/3] gc: fix handling of crontab magic markers

[Junio C Hamano]

Martin Ågren <martin.agren@gmail.com> writes:

> On `git maintenance start`, we add a few entries to the user's cron
> table. We wrap our entries using two magic markers, "# BEGIN GIT
> MAINTENANCE SCHEDULE" and "# END GIT MAINTENANCE SCHEDULE". At a later
> `git maintenance stop`, we will go through the table and remove these
> lines. Or rather, we will remove the "BEGIN" marker, the "END" marker
> and everything between them.
>
> Alas, we have a bug in how we detect the "END" marker: we don't. As we
> loop through all the lines of the crontab, if we are in the "old
> region", i.e., the region we're aiming to remove, we make an early
> `continue` and don't get as far as checking for the "END" marker. Thus,
> once we've seen our "BEGIN", we remove everything until the end of the
> file.
>
> Rewrite the logic for identifying these markers. There are four cases
> that are mutually exclusive: The current line starts a region or it ends
> it, or it's firmly within the region, or it's outside of it (and should
> be printed).
>
> Signed-off-by: Martin Ågren <martin.agren@gmail.com>
> ---
>  t/t7900-maintenance.sh | 7 +++++++
>  builtin/gc.c           | 7 +++----
>  2 files changed, 10 insertions(+), 4 deletions(-)
>
> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
> index d1e0c8f830..4bbfce31e9 100755
> --- a/t/t7900-maintenance.sh
> +++ b/t/t7900-maintenance.sh
> @@ -446,6 +446,13 @@ test_expect_success 'start preserves existing schedule' '
>  	grep "Important information!" cron.txt
>  '
>  
> +test_expect_success 'stop preserves surrounding schedule' '
> +	echo "Crucial information!" >>cron.txt &&
> +	GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop &&

31345d55 (maintenance: extract platform-specific scheduling,
2020-11-24) in ds/maintenance-part-4 needs to adjust this
exported variable for the tests to pass in 'seen'

Is it just the matter of replacing it with

	GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ..."

or is there more to it?

Thanks

Re: [PATCH 2/3] gc: fix handling of crontab magic markers

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>> +test_expect_success 'stop preserves surrounding schedule' '
>> +	echo "Crucial information!" >>cron.txt &&
>> +	GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop &&
>
> 31345d55 (maintenance: extract platform-specific scheduling,
> 2020-11-24) in ds/maintenance-part-4 needs to adjust this
> exported variable for the tests to pass in 'seen'
>
> Is it just the matter of replacing it with
>
> 	GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ..."
>
> or is there more to it?

We'll see soon enough; I noticed the above while viewing

    https://github.com/git/git/runs/1597371646#step:6:2093

and since then I pushed another round of 'seen' with a non-textual
conflict resolution while merging the ds/maintenance-part-4 topic
in.

Re: [PATCH 2/3] gc: fix handling of crontab magic markers

[Eric Sunshine]

On Tue, Dec 22, 2020 at 5:47 PM Junio C Hamano <gitster@pobox.com> wrote:
> Martin Ågren <martin.agren@gmail.com> writes:
> > +test_expect_success 'stop preserves surrounding schedule' '
> > +     echo "Crucial information!" >>cron.txt &&
> > +     GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop &&
>
> 31345d55 (maintenance: extract platform-specific scheduling,
> 2020-11-24) in ds/maintenance-part-4 needs to adjust this
> exported variable for the tests to pass in 'seen'
>
> Is it just the matter of replacing it with
>         GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ..."
> or is there more to it?

Yes, renaming GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER and
prepending "crontab:" to the value should be sufficient (per the
proposal by [1] and realization by [2]).

[1]: https://lore.kernel.org/git/X6+iJNYEbpQjHCb0@flurp.local/
[2]: https://lore.kernel.org/git/4807342b0019be29bb369ed3403a485f0ce9c15d.1605647598.git.gitgitgadget@gmail.com/

Re: [PATCH 2/3] gc: fix handling of crontab magic markers

[Martin Ågren]

On Wed, 23 Dec 2020 at 04:51, Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> On Tue, Dec 22, 2020 at 5:47 PM Junio C Hamano <gitster@pobox.com> wrote:
> > Martin Ågren <martin.agren@gmail.com> writes:
> > > +test_expect_success 'stop preserves surrounding schedule' '
> > > +     echo "Crucial information!" >>cron.txt &&
> > > +     GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop &&
> >
> > 31345d55 (maintenance: extract platform-specific scheduling,
> > 2020-11-24) in ds/maintenance-part-4 needs to adjust this
> > exported variable for the tests to pass in 'seen'
> >
> > Is it just the matter of replacing it with
> >         GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ..."
> > or is there more to it?

Oh, I never realized this could be a problem. My merge with seen had a
textual conflict, but nothing difficult, and the tests passed, so I
didn't even stop to think if there was more to it. I clearly didn't
notice the environment variable changed both name and value.

> Yes, renaming GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER and
> prepending "crontab:" to the value should be sufficient (per the
> proposal by [1] and realization by [2]).
>
> [1]: https://lore.kernel.org/git/X6+iJNYEbpQjHCb0@flurp.local/
> [2]: https://lore.kernel.org/git/4807342b0019be29bb369ed3403a485f0ce9c15d.1605647598.git.gitgitgadget@gmail.com/

Thanks Junio and Eric for helping out.

Martin

Re: [PATCH 2/3] gc: fix handling of crontab magic markers

[Junio C Hamano]

Martin Ågren <martin.agren@gmail.com> writes:

> On Wed, 23 Dec 2020 at 04:51, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>
>> On Tue, Dec 22, 2020 at 5:47 PM Junio C Hamano <gitster@pobox.com> wrote:
>> > Martin Ågren <martin.agren@gmail.com> writes:
>> > > +test_expect_success 'stop preserves surrounding schedule' '
>> > > +     echo "Crucial information!" >>cron.txt &&
>> > > +     GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop &&
>> >
>> > 31345d55 (maintenance: extract platform-specific scheduling,
>> > 2020-11-24) in ds/maintenance-part-4 needs to adjust this
>> > exported variable for the tests to pass in 'seen'
>> >
>> > Is it just the matter of replacing it with
>> >         GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ..."
>> > or is there more to it?
>
> Oh, I never realized this could be a problem. My merge with seen had a
> textual conflict, but nothing difficult, and the tests passed, so I
> didn't even stop to think if there was more to it. I clearly didn't
> notice the environment variable changed both name and value.
>
>> Yes, renaming GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER and
>> prepending "crontab:" to the value should be sufficient (per the
>> proposal by [1] and realization by [2]).
>>
>> [1]: https://lore.kernel.org/git/X6+iJNYEbpQjHCb0@flurp.local/
>> [2]: https://lore.kernel.org/git/4807342b0019be29bb369ed3403a485f0ce9c15d.1605647598.git.gitgitgadget@gmail.com/
>
> Thanks Junio and Eric for helping out.

Thanks for coming up with the fix.