[PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[Ramsay Jones]

Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
---
 cache.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/cache.h b/cache.h
index a916bc79e..a0e3e362c 100644
--- a/cache.h
+++ b/cache.h
@@ -1243,8 +1243,8 @@ static inline unsigned int hexval(unsigned char c)
  */
 static inline int hex2chr(const char *s)
 {
-	int val = hexval(s[0]);
-	return (val < 0) ? val : (val << 4) | hexval(s[1]);
+	unsigned int val = hexval(s[0]);
+	return (val & ~0xf) ? val : (val << 4) | hexval(s[1]);
 }
 
 /* Convert to/from hex/sha1 representation */
-- 
2.14.0

Re: [PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[Jeff King]

On Thu, Sep 21, 2017 at 05:48:38PM +0100, Ramsay Jones wrote:

> diff --git a/cache.h b/cache.h
> index a916bc79e..a0e3e362c 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1243,8 +1243,8 @@ static inline unsigned int hexval(unsigned char c)
>   */
>  static inline int hex2chr(const char *s)
>  {
> -	int val = hexval(s[0]);
> -	return (val < 0) ? val : (val << 4) | hexval(s[1]);
> +	unsigned int val = hexval(s[0]);
> +	return (val & ~0xf) ? val : (val << 4) | hexval(s[1]);
>  }

Ironically, the unsigned return from hexval() comes from internally
converting the signed char in hexval_table. And then we again return it
as a signed int from hex2chr().

Would it make sense to return a signed int from hexval()? That would
make hex2chr just work as it tries to above. I admit that shifting
signed values is a little funny, but it should be fine here since we
know they're no larger than 8 bits in the first place.

As an aside, I also see some uses of hexval() that don't appear to be
quite as rigorous in checking for invalid characters. A few
unconditionally shift the first nibble and assume that there will still
be high bits set. I think that's generally true for twos-complement
negative numbers, but isn't shifting off the left side of a signed
integer undefined behavior?

And mailinfo's decode_q_segment() does not seem to check for errors at
all.

Handling those is getting far off your original patch, but I'm having
trouble figuring out if it's saner for us to consistently stick to
all-signed or all-unsigned here.

-Peff

Re: [PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[Ramsay Jones]

On 22/09/17 06:47, Jeff King wrote:
> On Thu, Sep 21, 2017 at 05:48:38PM +0100, Ramsay Jones wrote:
> 
>> diff --git a/cache.h b/cache.h
>> index a916bc79e..a0e3e362c 100644
>> --- a/cache.h
>> +++ b/cache.h
>> @@ -1243,8 +1243,8 @@ static inline unsigned int hexval(unsigned char c)
>>   */
>>  static inline int hex2chr(const char *s)
>>  {
>> -	int val = hexval(s[0]);
>> -	return (val < 0) ? val : (val << 4) | hexval(s[1]);
>> +	unsigned int val = hexval(s[0]);
>> +	return (val & ~0xf) ? val : (val << 4) | hexval(s[1]);
>>  }
> 
> Ironically, the unsigned return from hexval() comes from internally
> converting the signed char in hexval_table. And then we again return it
> as a signed int from hex2chr().

Yep, my first inclination was to change the return type of
hexval(), but after looking at all its callers, I decided
against that (because it wouldn't help). ;-)

> Would it make sense to return a signed int from hexval()? That would
> make hex2chr just work as it tries to above. I admit that shifting
> signed values is a little funny, but it should be fine here since we
> know they're no larger than 8 bits in the first place.

Indeed, shifting signed values is a no-no, which is why I decided
to go this route.

> As an aside, I also see some uses of hexval() that don't appear to be
> quite as rigorous in checking for invalid characters. A few
> unconditionally shift the first nibble and assume that there will still
> be high bits set. I think that's generally true for twos-complement
> negative numbers, but isn't shifting off the left side of a signed
> integer undefined behavior?

All uses of hexval() that I can see are shifting an unsigned value.
Have I missed something?
> And mailinfo's decode_q_segment() does not seem to check for errors at
> all.

Yes, I noticed that. (I put it on my TODO list).

> Handling those is getting far off your original patch, but I'm having
> trouble figuring out if it's saner for us to consistently stick to
> all-signed or all-unsigned here.

Oh, unsigned, without a doubt. :D

ATB,
Ramsay Jones

Re: [PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[Jeff King]

On Fri, Sep 22, 2017 at 05:05:03PM +0100, Ramsay Jones wrote:

> > As an aside, I also see some uses of hexval() that don't appear to be
> > quite as rigorous in checking for invalid characters. A few
> > unconditionally shift the first nibble and assume that there will still
> > be high bits set. I think that's generally true for twos-complement
> > negative numbers, but isn't shifting off the left side of a signed
> > integer undefined behavior?
> 
> All uses of hexval() that I can see are shifting an unsigned value.
> Have I missed something?

Hmm. get_hex_color() does:

  unsigned int val;
  val = (hexval(in[0]) << 4) | hexval(in[1]));

Isn't that shifting the signed return value of hexval(), and then
converting it to unsigned afterwards?

I've been confused by C's integer conversion rules before, though, so
perhaps I'm wrong.

I think if this function is fed an empty string that it will also read
past the end of the buffer for in[1]. It shouldn't matter, since the NUL
in in[0] would cause us to return an error regardless, but it's still
undefined behavior.

In fact, this whole function is just hex2chr() implemented badly. Who is
responsible for this terrible code? ;)

-Peff

Re: [PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[Jeff King]

On Fri, Sep 22, 2017 at 12:11:59PM -0400, Jeff King wrote:

> On Fri, Sep 22, 2017 at 05:05:03PM +0100, Ramsay Jones wrote:
> 
> > > As an aside, I also see some uses of hexval() that don't appear to be
> > > quite as rigorous in checking for invalid characters. A few
> > > unconditionally shift the first nibble and assume that there will still
> > > be high bits set. I think that's generally true for twos-complement
> > > negative numbers, but isn't shifting off the left side of a signed
> > > integer undefined behavior?
> > 
> > All uses of hexval() that I can see are shifting an unsigned value.
> > Have I missed something?
> 
> Hmm. get_hex_color() does:
> 
>   unsigned int val;
>   val = (hexval(in[0]) << 4) | hexval(in[1]));
> 
> Isn't that shifting the signed return value of hexval(), and then
> converting it to unsigned afterwards?
> 
> I've been confused by C's integer conversion rules before, though, so
> perhaps I'm wrong.

Oh, nevermind. I managed to confuse myself again. The return value from
hexval() is already unsigned. It's hex2chr that has the funny signed
return. So the signedness is fine.

> I think if this function is fed an empty string that it will also read
> past the end of the buffer for in[1]. It shouldn't matter, since the NUL
> in in[0] would cause us to return an error regardless, but it's still
> undefined behavior.

This is still a bug, though.

-Peff

Re: [PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[Jeff King]

On Fri, Sep 22, 2017 at 12:18:17PM -0400, Jeff King wrote:

> > I think if this function is fed an empty string that it will also read
> > past the end of the buffer for in[1]. It shouldn't matter, since the NUL
> > in in[0] would cause us to return an error regardless, but it's still
> > undefined behavior.
> 
> This is still a bug, though.

Last message, I promise. ;)

I started on the minimal fix for this, but actually it's OK by virtue of
its sole caller first checking that we have enough length (because we're
not parsing a string, in fact, but a ptr/len buffer).

So all is well, though I think get_hex_color() does serve as a poor
example if somebody were to try to adapt it generally (hopefully they
wouldn't, since hex2chr() is already globally available).

Sorry for the all the noise.

-Peff

Re: [PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[Ramsay Jones]

On 22/09/17 17:11, Jeff King wrote:
> On Fri, Sep 22, 2017 at 05:05:03PM +0100, Ramsay Jones wrote:
> 
>>> As an aside, I also see some uses of hexval() that don't appear to be
>>> quite as rigorous in checking for invalid characters. A few
>>> unconditionally shift the first nibble and assume that there will still
>>> be high bits set. I think that's generally true for twos-complement
>>> negative numbers, but isn't shifting off the left side of a signed
>>> integer undefined behavior?
>>
>> All uses of hexval() that I can see are shifting an unsigned value.
>> Have I missed something?
> 
> Hmm. get_hex_color() does:
> 
>   unsigned int val;
>   val = (hexval(in[0]) << 4) | hexval(in[1]));
> 
> Isn't that shifting the signed return value of hexval(), and then
> converting it to unsigned afterwards?

Err ... no. the return value of hexval() is *unsigned int*.
(which is kinda the point!)

> I've been confused by C's integer conversion rules before, though, so
> perhaps I'm wrong.
> 
> I think if this function is fed an empty string that it will also read
> past the end of the buffer for in[1]. It shouldn't matter, since the NUL
> in in[0] would cause us to return an error regardless, but it's still
> undefined behavior.

Correct.

> In fact, this whole function is just hex2chr() implemented badly. Who is
> responsible for this terrible code? ;)

;-)

ATB,
Ramsay Jones

Re: [PATCH 3/4] cache.h: hex2chr() - avoid -Wsign-compare warnings

[René Scharfe]

Am 22.09.2017 um 07:47 schrieb Jeff King:
> On Thu, Sep 21, 2017 at 05:48:38PM +0100, Ramsay Jones wrote:
> 
>> diff --git a/cache.h b/cache.h
>> index a916bc79e..a0e3e362c 100644
>> --- a/cache.h
>> +++ b/cache.h
>> @@ -1243,8 +1243,8 @@ static inline unsigned int hexval(unsigned char c)
>>    */
>>   static inline int hex2chr(const char *s)
>>   {
>> -	int val = hexval(s[0]);
>> -	return (val < 0) ? val : (val << 4) | hexval(s[1]);
>> +	unsigned int val = hexval(s[0]);
>> +	return (val & ~0xf) ? val : (val << 4) | hexval(s[1]);
>>   }
> 
> Ironically, the unsigned return from hexval() comes from internally
> converting the signed char in hexval_table. And then we again return it
> as a signed int from hex2chr().
> 
> Would it make sense to return a signed int from hexval()? That would
> make hex2chr just work as it tries to above. I admit that shifting
> signed values is a little funny, but it should be fine here since we
> know they're no larger than 8 bits in the first place.

I'd say yes.  We can replace that shift with a multiplication --
compilers will nevertheless use a shift even with -O0 (if
beneficial).

> As an aside, I also see some uses of hexval() that don't appear to be
> quite as rigorous in checking for invalid characters. A few
> unconditionally shift the first nibble and assume that there will still
> be high bits set. I think that's generally true for twos-complement
> negative numbers, but isn't shifting off the left side of a signed
> integer undefined behavior?
> 
> And mailinfo's decode_q_segment() does not seem to check for errors at
> all.

It also allocates and returns a strbuf, which seems odd as well.

René