[PATCH] gc: fix cast in compare_tasks_by_selection()

[PATCH] gc: fix cast in compare_tasks_by_selection()

[René Scharfe]

compare_tasks_by_selection() is used with QSORT and gets passed pointers
to the elements of "static struct maintenance_task tasks[]".  It casts
the *addresses* of these passed pointers to element pointers, though,
and thus effectively compares some unrelated values from the stack.  Fix
the casts to actually compare array elements.

Detected by USan (make SANITIZE=undefined test).

Signed-off-by: René Scharfe <l.s.r@web.de>
---
 builtin/gc.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/builtin/gc.c b/builtin/gc.c
index 5cd2a43f9f..986b760b30 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -1213,10 +1213,8 @@ static struct maintenance_task tasks[] = {

 static int compare_tasks_by_selection(const void *a_, const void *b_)
 {
-	const struct maintenance_task *a, *b;
-
-	a = (const struct maintenance_task *)&a_;
-	b = (const struct maintenance_task *)&b_;
+	const struct maintenance_task *a = a_;
+	const struct maintenance_task *b = b_;

 	return b->selected_order - a->selected_order;
 }
--
2.29.2

Re: [PATCH] gc: fix cast in compare_tasks_by_selection()

[Jeff King]

On Tue, Nov 17, 2020 at 10:59:49PM +0100, René Scharfe wrote:

> compare_tasks_by_selection() is used with QSORT and gets passed pointers
> to the elements of "static struct maintenance_task tasks[]".  It casts
> the *addresses* of these passed pointers to element pointers, though,
> and thus effectively compares some unrelated values from the stack.  Fix
> the casts to actually compare array elements.
> 
> Detected by USan (make SANITIZE=undefined test).

I checked the caller here, and indeed, it's passing an array-of-struct
so your patch is doing the right thing (not that I doubted it, but
that's what review is for).

This qsort void-pointer convention seems to create a lot of confusion
(not just the lack of type-safety, but the fact that it's getting a
pointer to the element). I felt like we had a discussion about this a
while ago, and indeed, I found:

  https://lore.kernel.org/git/7b95417a-c8fb-4f1e-cb09-c36804a3a4d0@web.de/

The whole sub-thread is worth reading, but the macro you arrived at in:

  https://lore.kernel.org/git/c141fb44-904f-e8b6-119f-7d2d6bcfd81a@web.de/

seems pretty reasonable.

-Peff