a few integer undefined behaviors in GMP

[John Regehr]

Hi,

I ran GMP's "make check" under a tool that looks for integer undefined
behaviors in C code, and it found a few problems, listed in the attached
file.  Hopefully the error messages are self-explanatory, but if not
please let me know and I can explain more.

Just to be clear: these are not static analysis results, and there are
no false positives!  These are real problems and the last part of each
error message lists example arguments to the operator that result in a
violation.  So, referring to the first error in mul_i.h:

  left (int32): 0 right (int32): -2147483648

is saying that this file at line 61 has tried to evaluate

  (0 - -2147483648)

Of course this is an error when sizeof(int)==4.

This is GMP's development head from 8/20/2010.

Configured and built like this:

  ./configure CC=clang CXX=clang++ ABI=32
  make CFLAGS='-fcatch-undefined-ansic-behavior -O2'
CXXFLAGS='-fcatch-undefined-ansic-behavior -O2'

The Clang being used here is from LLVM's development head as of a day or
two ago, and it has been patched by us to detect integer bugs.

More info:

regehr at john-home:~/z/gmp$ uname -a
Linux john-home 2.6.32-24-generic #39-Ubuntu SMP Wed Jul 28 06:07:29 UTC
2010 i686 GNU/Linux
regehr at john-home:~/z/gmp$ ./config.guess
core2-pc-linux-gnu
regehr at john-home:~/z/gmp$ ./configfsf.guess
i686-pc-linux-gnu

Thanks,

John Regehr
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: gmp-bugs.txt
URL: <http://gmplib.org/list-archives/gmp-bugs/attachments/20100820/ad690715/attachment.txt>