[RFC PATCH 08/10] linear-assignment.c: detect signed add/mul on GCC and Clang

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

Extend the cost_index() inline function added in the preceding commit
with signed overflow detection that'll work on GCC[1] and
Clang[2]. See 320d0b493a2 (add helpers for detecting size_t overflow,
2016-02-19) for the existing git-compat-util.h helpers to detect
signed overflow.

This fixes a segfault on that happens when "range-diff" is given a
very large range to work with, such as the difference between
git.git's "master" the git-for-windows fork:

    $ git -P range-diff --creation-factor=50 origin/master...git-for-windows/main
    fatal: integer overflow in cost[47395 + ((47396 * 45309) = -2147454537)] addition

There are known bugs with using these functions in some versions of
GCC, as we'll see in a subsequent commit we're better off detecting
those with the "intprops.h" library, but for now let's add a simpler
implementation that relies on the bare minimum of compiler checking.

1. https://gcc.gnu.org/onlinedocs/gcc/Integer-Overflow-Builtins.html#Integer-Overflow-Builtins
2. https://clang.llvm.org/docs/LanguageExtensions.html#checked-arithmetic-builtins

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 linear-assignment.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/linear-assignment.c b/linear-assignment.c
index e9cec16132a..b6597b622c8 100644
--- a/linear-assignment.c
+++ b/linear-assignment.c
@@ -10,7 +10,14 @@ static inline int cost_index(int *cost, int a, int b, int c)
 {
 	int r;
 
+#if defined(__GNUC__) || defined(__clang__)
+	if (__builtin_mul_overflow(a, c, &r))
+		die(_("integer overflow in cost[%d + %d * %c] multiplication"), b, a, c);
+	if (__builtin_add_overflow(b, r, &r))
+		die(_("integer overflow in cost[%d + ((%d * %d) = %d)] addition"), b, a, c, r);
+#else
 	r = b + a * c;
+#endif
 
 	return r;
 }
-- 
2.34.1.930.g0f9292b224d