integer overflow in mpn/get_d.c from GMP 5.1.2

[Vincent Lefevre]

On 2013-09-20 19:29:51 +0200, Niels Möller wrote:
> I understand that it's a portability problem if you interpret the
> standard strictly.

There isn't any other document that specifies what can happen with
an integer overflow. What you can see with most code is just a side
effect on how most processors work. But if a processor has separate
signed and unsigned instructions (or a different representation, as
allowed by the C standard), you would see a different behavior in
case of integer overflow.

> But I'm a bit suprised if gcc, with *any* optimization options,
> implements something different than plain old two's complement
> semantics.

I'm not surprised at all. The optimizations are based on the
specification of the C standard. And there's no such thing as
"plain old two's complement semantics" in the standard (GCC
has the -fwrapv option, but it isn't related at all to the C
standard).

I don't know what led to the "wrong code" here. But for instance,
here's what a compiler would be allowed to do. The test is:

  if (UNLIKELY ((unsigned long) (GMP_NUMB_BITS * size)
                > (unsigned long) (LONG_MAX - exp)))

With LTO and VRP, the compiler can detect that GMP_NUMB_BITS * size
is always nonnegative and fits in a long. Since exp <= LONG_MAX, the
compiler knows that LONG_MAX - exp is also nonnegative and fits in a
long. Then there could be some optimization rule that eliminates the
conversions to unsigned long in such a case, and generates a signed
comparison. Hence a wrong result if exp is negative.

With a stack of independent advanced optimization rules, you cannot
know what will happen in case of undefined behavior.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)