integer overflow in mpn/get_d.c from GMP 5.1.2

[Niels Möller]

Vincent Lefevre <vincent at vinc17.net> writes:

> I don't know what led to the "wrong code" here. But for instance,
> here's what a compiler would be allowed to do. The test is:

Thanks for explaining. But I'm still not quite following you.

>   if (UNLIKELY ((unsigned long) (GMP_NUMB_BITS * size)
>                 > (unsigned long) (LONG_MAX - exp)))
>
> [...] Since exp <= LONG_MAX, the compiler knows that LONG_MAX - exp is
> also nonnegative and fits in a long.

Here, for non-negativity, we're also making the critical assumption that
the compiler's global optimization infers that the function is never
called with exp < 0, right?

> Hence a wrong result if exp is negative.

And then the generated code breaks when, in fact, the function *is*
called with exp < 0. That means that either the compiler's inference is
buggy. Or that the GMP bug is elsewhere, with the negative value of exp
resulting from some *earlier* computation with undefined behaviour.

Hmm, or ar you saying that the compiler's assumption exp >= 0 comes from
*local* inference? "If the caller provides a negative exp, then we get
undefined behaviour in this expression. Hence, let's just pretend that
the caller would never do such a thing".

Makes a bit sense in it's own way, and I guess that's the only way
moving around the cast can make a difference.

Regards,
/Niels

-- 
Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
Internet email is subject to wholesale government surveillance.