[PATCH 0/2] diff_free(): free a bit more, fix leaks

[PATCH 0/2] diff_free(): free a bit more, fix leaks

[Ævar Arnfjörð Bjarmason]

A short and simple two-part series to fix some leaks in after
diff_flush() and friends are called by making in diff_free() more
useful.

This series along with other small leak fixes I have outstanding
on-list now is the last step I have before the big one of submitting a
series to fix the "big one" across the test suite: the common leaks in
the revisions API.

Ævar Arnfjörð Bjarmason (2):
  diff.[ch]: have diff_free() call clear_pathspec(opts.pathspec)
  diff.[ch]: have diff_free() free options->parseopts

 add-interactive.c | 6 +++---
 blame.c           | 3 ---
 builtin/reset.c   | 1 -
 diff.c            | 2 ++
 notes-merge.c     | 2 --
 5 files changed, 5 insertions(+), 9 deletions(-)

-- 
2.35.1.1028.g2d2d4be19de

[PATCH 1/2] diff.[ch]: have diff_free() call clear_pathspec(opts.pathspec)

[Ævar Arnfjörð Bjarmason]

Have the diff_free() function call clear_pathspec(). Since the
diff_flush() function calls this all its callers can be simplified to
rely on it instead.

When I added the diff_free() function in e900d494dcf (diff: add an API
for deferred freeing, 2021-02-11) I simply missed this, or wasn't
interested in it. Let's consolidate this now. This means that any
future callers (and I've got revision.c in mind) that embed a "struct
diff_options" can simply call diff_free() instead of needing know that
it has an embedded pathspec.

This does fix a bunch of leaks, but I can't mark any test here as
passing under the SANITIZE=leak testing mode because in
886e1084d78 (builtin/: add UNLEAKs, 2017-10-01) an UNLEAK(rev) was
added, which plasters over the memory
leak. E.g. "t4011-diff-symlink.sh" would report fewer leaks with this
fix, but because of the UNLEAK() reports none.

I'll eventually loop around to removing that UNLEAK(rev) annotation as
I'll fix deeper issues with the revisions API leaking. This is one
small step on the way there, a new freeing function in revisions.c
will want to call this diff_free().

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 add-interactive.c | 6 +++---
 blame.c           | 3 ---
 builtin/reset.c   | 1 -
 diff.c            | 1 +
 notes-merge.c     | 2 --
 5 files changed, 4 insertions(+), 9 deletions(-)

diff --git a/add-interactive.c b/add-interactive.c
index 6498ae196f1..e1ab39cce30 100644
--- a/add-interactive.c
+++ b/add-interactive.c
@@ -797,14 +797,14 @@ static int run_revert(struct add_i_state *s, const struct pathspec *ps,
 	diffopt.flags.override_submodule_config = 1;
 	diffopt.repo = s->r;
 
-	if (do_diff_cache(&oid, &diffopt))
+	if (do_diff_cache(&oid, &diffopt)) {
+		diff_free(&diffopt);
 		res = -1;
-	else {
+	} else {
 		diffcore_std(&diffopt);
 		diff_flush(&diffopt);
 	}
 	free(paths);
-	clear_pathspec(&diffopt.pathspec);
 
 	if (!res && write_locked_index(s->r->index, &index_lock,
 				       COMMIT_LOCK) < 0)
diff --git a/blame.c b/blame.c
index 206c295660f..401990726e7 100644
--- a/blame.c
+++ b/blame.c
@@ -1403,7 +1403,6 @@ static struct blame_origin *find_origin(struct repository *r,
 		}
 	}
 	diff_flush(&diff_opts);
-	clear_pathspec(&diff_opts.pathspec);
 	return porigin;
 }
 
@@ -1447,7 +1446,6 @@ static struct blame_origin *find_rename(struct repository *r,
 		}
 	}
 	diff_flush(&diff_opts);
-	clear_pathspec(&diff_opts.pathspec);
 	return porigin;
 }
 
@@ -2328,7 +2326,6 @@ static void find_copy_in_parent(struct blame_scoreboard *sb,
 	} while (unblamed);
 	target->suspects = reverse_blame(leftover, NULL);
 	diff_flush(&diff_opts);
-	clear_pathspec(&diff_opts.pathspec);
 }
 
 /*
diff --git a/builtin/reset.c b/builtin/reset.c
index b97745ee94e..24968dd6282 100644
--- a/builtin/reset.c
+++ b/builtin/reset.c
@@ -274,7 +274,6 @@ static int read_from_tree(const struct pathspec *pathspec,
 		return 1;
 	diffcore_std(&opt);
 	diff_flush(&opt);
-	clear_pathspec(&opt.pathspec);
 
 	return 0;
 }
diff --git a/diff.c b/diff.c
index c862771a589..0aef3db6e10 100644
--- a/diff.c
+++ b/diff.c
@@ -6345,6 +6345,7 @@ void diff_free(struct diff_options *options)
 
 	diff_free_file(options);
 	diff_free_ignore_regex(options);
+	clear_pathspec(&options->pathspec);
 }
 
 void diff_flush(struct diff_options *options)
diff --git a/notes-merge.c b/notes-merge.c
index b4a3a903e86..7ba40cfb080 100644
--- a/notes-merge.c
+++ b/notes-merge.c
@@ -175,7 +175,6 @@ static struct notes_merge_pair *diff_tree_remote(struct notes_merge_options *o,
 		       oid_to_hex(&mp->remote));
 	}
 	diff_flush(&opt);
-	clear_pathspec(&opt.pathspec);
 
 	*num_changes = len;
 	return changes;
@@ -261,7 +260,6 @@ static void diff_tree_local(struct notes_merge_options *o,
 		       oid_to_hex(&mp->local));
 	}
 	diff_flush(&opt);
-	clear_pathspec(&opt.pathspec);
 }
 
 static void check_notes_merge_worktree(struct notes_merge_options *o)
-- 
2.35.1.1028.g2d2d4be19de

Re: [PATCH 1/2] diff.[ch]: have diff_free() call clear_pathspec(opts.pathspec)

[Elijah Newren]

On Wed, Feb 16, 2022 at 7:34 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> Have the diff_free() function call clear_pathspec(). Since the
> diff_flush() function calls this all its callers can be simplified to
> rely on it instead.
>
> When I added the diff_free() function in e900d494dcf (diff: add an API
> for deferred freeing, 2021-02-11) I simply missed this, or wasn't
> interested in it. Let's consolidate this now. This means that any
> future callers (and I've got revision.c in mind) that embed a "struct
> diff_options" can simply call diff_free() instead of needing know that
> it has an embedded pathspec.
>
> This does fix a bunch of leaks, but I can't mark any test here as
> passing under the SANITIZE=leak testing mode because in
> 886e1084d78 (builtin/: add UNLEAKs, 2017-10-01) an UNLEAK(rev) was
> added, which plasters over the memory
> leak. E.g. "t4011-diff-symlink.sh" would report fewer leaks with this
> fix, but because of the UNLEAK() reports none.
>
> I'll eventually loop around to removing that UNLEAK(rev) annotation as
> I'll fix deeper issues with the revisions API leaking. This is one
> small step on the way there, a new freeing function in revisions.c
> will want to call this diff_free().
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  add-interactive.c | 6 +++---
>  blame.c           | 3 ---
>  builtin/reset.c   | 1 -
>  diff.c            | 1 +
>  notes-merge.c     | 2 --
>  5 files changed, 4 insertions(+), 9 deletions(-)
>
> diff --git a/add-interactive.c b/add-interactive.c
> index 6498ae196f1..e1ab39cce30 100644
> --- a/add-interactive.c
> +++ b/add-interactive.c
> @@ -797,14 +797,14 @@ static int run_revert(struct add_i_state *s, const struct pathspec *ps,
>         diffopt.flags.override_submodule_config = 1;
>         diffopt.repo = s->r;
>
> -       if (do_diff_cache(&oid, &diffopt))
> +       if (do_diff_cache(&oid, &diffopt)) {
> +               diff_free(&diffopt);
>                 res = -1;
> -       else {
> +       } else {
>                 diffcore_std(&diffopt);
>                 diff_flush(&diffopt);
>         }
>         free(paths);
> -       clear_pathspec(&diffopt.pathspec);
>
>         if (!res && write_locked_index(s->r->index, &index_lock,
>                                        COMMIT_LOCK) < 0)
> diff --git a/blame.c b/blame.c
> index 206c295660f..401990726e7 100644
> --- a/blame.c
> +++ b/blame.c
> @@ -1403,7 +1403,6 @@ static struct blame_origin *find_origin(struct repository *r,
>                 }
>         }
>         diff_flush(&diff_opts);
> -       clear_pathspec(&diff_opts.pathspec);
>         return porigin;
>  }
>
> @@ -1447,7 +1446,6 @@ static struct blame_origin *find_rename(struct repository *r,
>                 }
>         }
>         diff_flush(&diff_opts);
> -       clear_pathspec(&diff_opts.pathspec);
>         return porigin;
>  }
>
> @@ -2328,7 +2326,6 @@ static void find_copy_in_parent(struct blame_scoreboard *sb,
>         } while (unblamed);
>         target->suspects = reverse_blame(leftover, NULL);
>         diff_flush(&diff_opts);
> -       clear_pathspec(&diff_opts.pathspec);
>  }
>
>  /*
> diff --git a/builtin/reset.c b/builtin/reset.c
> index b97745ee94e..24968dd6282 100644
> --- a/builtin/reset.c
> +++ b/builtin/reset.c
> @@ -274,7 +274,6 @@ static int read_from_tree(const struct pathspec *pathspec,
>                 return 1;
>         diffcore_std(&opt);
>         diff_flush(&opt);
> -       clear_pathspec(&opt.pathspec);
>
>         return 0;
>  }
> diff --git a/diff.c b/diff.c
> index c862771a589..0aef3db6e10 100644
> --- a/diff.c
> +++ b/diff.c
> @@ -6345,6 +6345,7 @@ void diff_free(struct diff_options *options)
>
>         diff_free_file(options);
>         diff_free_ignore_regex(options);
> +       clear_pathspec(&options->pathspec);
>  }
>
>  void diff_flush(struct diff_options *options)
> diff --git a/notes-merge.c b/notes-merge.c
> index b4a3a903e86..7ba40cfb080 100644
> --- a/notes-merge.c
> +++ b/notes-merge.c
> @@ -175,7 +175,6 @@ static struct notes_merge_pair *diff_tree_remote(struct notes_merge_options *o,
>                        oid_to_hex(&mp->remote));
>         }
>         diff_flush(&opt);
> -       clear_pathspec(&opt.pathspec);
>
>         *num_changes = len;
>         return changes;
> @@ -261,7 +260,6 @@ static void diff_tree_local(struct notes_merge_options *o,
>                        oid_to_hex(&mp->local));
>         }
>         diff_flush(&opt);
> -       clear_pathspec(&opt.pathspec);
>  }
>
>  static void check_notes_merge_worktree(struct notes_merge_options *o)
> --
> 2.35.1.1028.g2d2d4be19de

I have occasionally found it surprising that we have places where
callers are responsible for free'ing a single piece of some struct
despite the fact that we have a function for free'ing memory
associated with that struct.  Thanks for getting rid of another one of
these.

Re: [PATCH 1/2] diff.[ch]: have diff_free() call clear_pathspec(opts.pathspec)

[Ævar Arnfjörð Bjarmason]

On Wed, Feb 16 2022, Elijah Newren wrote:

> On Wed, Feb 16, 2022 at 7:34 AM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>>
>> Have the diff_free() function call clear_pathspec(). Since the
>> diff_flush() function calls this all its callers can be simplified to
>> rely on it instead.
>>
>> When I added the diff_free() function in e900d494dcf (diff: add an API
>> for deferred freeing, 2021-02-11) I simply missed this, or wasn't
>> interested in it. Let's consolidate this now. This means that any
>> future callers (and I've got revision.c in mind) that embed a "struct
>> diff_options" can simply call diff_free() instead of needing know that
>> it has an embedded pathspec.
>>
>> This does fix a bunch of leaks, but I can't mark any test here as
>> passing under the SANITIZE=leak testing mode because in
>> 886e1084d78 (builtin/: add UNLEAKs, 2017-10-01) an UNLEAK(rev) was
>> added, which plasters over the memory
>> leak. E.g. "t4011-diff-symlink.sh" would report fewer leaks with this
>> fix, but because of the UNLEAK() reports none.
>>
>> I'll eventually loop around to removing that UNLEAK(rev) annotation as
>> I'll fix deeper issues with the revisions API leaking. This is one
>> small step on the way there, a new freeing function in revisions.c
>> will want to call this diff_free().
>>
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>>  add-interactive.c | 6 +++---
>>  blame.c           | 3 ---
>>  builtin/reset.c   | 1 -
>>  diff.c            | 1 +
>>  notes-merge.c     | 2 --
>>  5 files changed, 4 insertions(+), 9 deletions(-)
>>
>> diff --git a/add-interactive.c b/add-interactive.c
>> index 6498ae196f1..e1ab39cce30 100644
>> --- a/add-interactive.c
>> +++ b/add-interactive.c
>> @@ -797,14 +797,14 @@ static int run_revert(struct add_i_state *s, const struct pathspec *ps,
>>         diffopt.flags.override_submodule_config = 1;
>>         diffopt.repo = s->r;
>>
>> -       if (do_diff_cache(&oid, &diffopt))
>> +       if (do_diff_cache(&oid, &diffopt)) {
>> +               diff_free(&diffopt);
>>                 res = -1;
>> -       else {
>> +       } else {
>>                 diffcore_std(&diffopt);
>>                 diff_flush(&diffopt);
>>         }
>>         free(paths);
>> -       clear_pathspec(&diffopt.pathspec);
>>
>>         if (!res && write_locked_index(s->r->index, &index_lock,
>>                                        COMMIT_LOCK) < 0)
>> diff --git a/blame.c b/blame.c
>> index 206c295660f..401990726e7 100644
>> --- a/blame.c
>> +++ b/blame.c
>> @@ -1403,7 +1403,6 @@ static struct blame_origin *find_origin(struct repository *r,
>>                 }
>>         }
>>         diff_flush(&diff_opts);
>> -       clear_pathspec(&diff_opts.pathspec);
>>         return porigin;
>>  }
>>
>> @@ -1447,7 +1446,6 @@ static struct blame_origin *find_rename(struct repository *r,
>>                 }
>>         }
>>         diff_flush(&diff_opts);
>> -       clear_pathspec(&diff_opts.pathspec);
>>         return porigin;
>>  }
>>
>> @@ -2328,7 +2326,6 @@ static void find_copy_in_parent(struct blame_scoreboard *sb,
>>         } while (unblamed);
>>         target->suspects = reverse_blame(leftover, NULL);
>>         diff_flush(&diff_opts);
>> -       clear_pathspec(&diff_opts.pathspec);
>>  }
>>
>>  /*
>> diff --git a/builtin/reset.c b/builtin/reset.c
>> index b97745ee94e..24968dd6282 100644
>> --- a/builtin/reset.c
>> +++ b/builtin/reset.c
>> @@ -274,7 +274,6 @@ static int read_from_tree(const struct pathspec *pathspec,
>>                 return 1;
>>         diffcore_std(&opt);
>>         diff_flush(&opt);
>> -       clear_pathspec(&opt.pathspec);
>>
>>         return 0;
>>  }
>> diff --git a/diff.c b/diff.c
>> index c862771a589..0aef3db6e10 100644
>> --- a/diff.c
>> +++ b/diff.c
>> @@ -6345,6 +6345,7 @@ void diff_free(struct diff_options *options)
>>
>>         diff_free_file(options);
>>         diff_free_ignore_regex(options);
>> +       clear_pathspec(&options->pathspec);
>>  }
>>
>>  void diff_flush(struct diff_options *options)
>> diff --git a/notes-merge.c b/notes-merge.c
>> index b4a3a903e86..7ba40cfb080 100644
>> --- a/notes-merge.c
>> +++ b/notes-merge.c
>> @@ -175,7 +175,6 @@ static struct notes_merge_pair *diff_tree_remote(struct notes_merge_options *o,
>>                        oid_to_hex(&mp->remote));
>>         }
>>         diff_flush(&opt);
>> -       clear_pathspec(&opt.pathspec);
>>
>>         *num_changes = len;
>>         return changes;
>> @@ -261,7 +260,6 @@ static void diff_tree_local(struct notes_merge_options *o,
>>                        oid_to_hex(&mp->local));
>>         }
>>         diff_flush(&opt);
>> -       clear_pathspec(&opt.pathspec);
>>  }
>>
>>  static void check_notes_merge_worktree(struct notes_merge_options *o)
>> --
>> 2.35.1.1028.g2d2d4be19de
>
> I have occasionally found it surprising that we have places where
> callers are responsible for free'ing a single piece of some struct
> despite the fact that we have a function for free'ing memory
> associated with that struct.  Thanks for getting rid of another one of
> these.

Yes, much of the later memory leak effort is focused on consolidating
those, there's a lot of it around "struct rev_info", e.g. callers
reaching into it to free a pathspec, the "struct object_array *pending"
etc.

There's some gray area there, particularly when the caller originally
allocates the resource and "hands it off" to the struct, but even in
those cases they should probably be freeing their own original variable,
not referring to it by reaching into a struct they handed it to.

That would amount to the exact same thing for the machine (same
pointer), but the former is surely more readable to the human reader.

Re: [PATCH 1/2] diff.[ch]: have diff_free() call clear_pathspec(opts.pathspec)

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> diff --git a/diff.c b/diff.c
> index c862771a589..0aef3db6e10 100644
> --- a/diff.c
> +++ b/diff.c
> @@ -6345,6 +6345,7 @@ void diff_free(struct diff_options *options)
>  
>  	diff_free_file(options);
>  	diff_free_ignore_regex(options);
> +	clear_pathspec(&options->pathspec);
>  }
>  
>  void diff_flush(struct diff_options *options)

Interesting.  As diff_flush() is the way to conclude the diff
session whose state was kept in the diff_options structure, it
probably makes sense to allow pathspec to be also cleared from
there.  It is somewhat surprising that we didn't do this when we
introduced diff_free(), but better late than never ;-)

> diff --git a/notes-merge.c b/notes-merge.c
> index b4a3a903e86..7ba40cfb080 100644
> --- a/notes-merge.c
> +++ b/notes-merge.c
> @@ -175,7 +175,6 @@ static struct notes_merge_pair *diff_tree_remote(struct notes_merge_options *o,
>  		       oid_to_hex(&mp->remote));
>  	}
>  	diff_flush(&opt);
> -	clear_pathspec(&opt.pathspec);
>  
>  	*num_changes = len;
>  	return changes;
> @@ -261,7 +260,6 @@ static void diff_tree_local(struct notes_merge_options *o,
>  		       oid_to_hex(&mp->local));
>  	}
>  	diff_flush(&opt);
> -	clear_pathspec(&opt.pathspec);
>  }
>  
>  static void check_notes_merge_worktree(struct notes_merge_options *o)


Looks quite sensible.  Will queue.  Thanks.

[PATCH 2/2] diff.[ch]: have diff_free() free options->parseopts

[Ævar Arnfjörð Bjarmason]

The "struct option" added in 4a288478394 (diff.c: prepare to use
parse_options() for parsing, 2019-01-27) would be free'd in the case
of diff_setup_done() being called.

But not all codepaths that allocate it reach that,
e.g. "t6427-diff3-conflict-markers.sh" will now free memory that it
didn't free before. By using FREE_AND_NULL() here (which
diff_setup_done() also does) we ensure that we free the memory, and
that we won't have double-free's.

Before this running:

    ./t6427-diff3-conflict-markers.sh -vixd --run=7

Would report:

    SUMMARY: LeakSanitizer: 7823 byte(s) leaked in 6 allocation(s).

But now we'll report:

    SUMMARY: LeakSanitizer: 703 byte(s) leaked in 5 allocation(s).

I.e. the largest leak in that particular test has now been addressed.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 diff.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/diff.c b/diff.c
index 0aef3db6e10..fb8bc8aadbf 100644
--- a/diff.c
+++ b/diff.c
@@ -6346,6 +6346,7 @@ void diff_free(struct diff_options *options)
 	diff_free_file(options);
 	diff_free_ignore_regex(options);
 	clear_pathspec(&options->pathspec);
+	FREE_AND_NULL(options->parseopts);
 }
 
 void diff_flush(struct diff_options *options)
-- 
2.35.1.1028.g2d2d4be19de

Re: [PATCH 2/2] diff.[ch]: have diff_free() free options->parseopts

[Elijah Newren]

On Wed, Feb 16, 2022 at 7:59 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> The "struct option" added in 4a288478394 (diff.c: prepare to use
> parse_options() for parsing, 2019-01-27) would be free'd in the case
> of diff_setup_done() being called.
>
> But not all codepaths that allocate it reach that,
> e.g. "t6427-diff3-conflict-markers.sh" will now free memory that it
> didn't free before. By using FREE_AND_NULL() here (which
> diff_setup_done() also does) we ensure that we free the memory, and
> that we won't have double-free's.
>
> Before this running:
>
>     ./t6427-diff3-conflict-markers.sh -vixd --run=7
>
> Would report:
>
>     SUMMARY: LeakSanitizer: 7823 byte(s) leaked in 6 allocation(s).
>
> But now we'll report:
>
>     SUMMARY: LeakSanitizer: 703 byte(s) leaked in 5 allocation(s).
>
> I.e. the largest leak in that particular test has now been addressed.
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  diff.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/diff.c b/diff.c
> index 0aef3db6e10..fb8bc8aadbf 100644
> --- a/diff.c
> +++ b/diff.c
> @@ -6346,6 +6346,7 @@ void diff_free(struct diff_options *options)
>         diff_free_file(options);
>         diff_free_ignore_regex(options);
>         clear_pathspec(&options->pathspec);
> +       FREE_AND_NULL(options->parseopts);
>  }
>
>  void diff_flush(struct diff_options *options)
> --
> 2.35.1.1028.g2d2d4be19de

Makes sense.

Re: [PATCH 0/2] diff_free(): free a bit more, fix leaks

[Elijah Newren]

On Wed, Feb 16, 2022 at 7:56 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> A short and simple two-part series to fix some leaks in after
> diff_flush() and friends are called by making in diff_free() more
> useful.

Reviewed-by: Elijah Newren <newren@gmail.com>

> This series along with other small leak fixes I have outstanding
> on-list now is the last step I have before the big one of submitting a
> series to fix the "big one" across the test suite: the common leaks in
> the revisions API.

Wahoo!