integer overflow in mpn/get_d.c from GMP 5.1.2

[Niels Möller]

Vincent Lefevre <vincent at vinc17.net> writes:

> In GMP 5.1.2, there's an integer overflow in mpn/get_d.c:
>
>   if (UNLIKELY ((unsigned long) (GMP_NUMB_BITS * size)
>                 > (unsigned long) (LONG_MAX - exp)))
>
> It should be replaced by:
>
>   if (UNLIKELY ((unsigned long) (GMP_NUMB_BITS * size)
>                 > (unsigned long) LONG_MAX - exp))
>
> This integer overflow triggers the following failure

Assuming your C compiler uses two's complement representation for signed
values, I'm not sure I understand why that change makes any difference.
Both the variable exp and the constant LONG_MAX are of type long, and
all that should matter is the interpretation of the resulting bit
pattern.

Casting the result makes it clear that the result should be intepreted
as unsigned long. If you change it to (unsigned long) LONG_MAX - exp,
the type depends on C conversion rules. Which are a bit obscure. But as
far as I understand, the result is still of type unsigned long, and it's
the value of exp which is gets converted from long to unsigned long.

And the result should be in the range 0 to LONG_MAX - LONG_MIN, which
usually is the same as ULONG_MAX, and hence it always fits (but with
no margin!) in an unsigned long. 

So, can you explain why it goes wrong?

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.