integer overflow in mpz/clear.c of GMP 5.1.3

[Vincent Lefevre]

Hi,

I've found an integer overflow in mpz/clear.c of GMP 5.1.3 under
Linux/x86_64. It is triggered by the following program placed in
mpfr/tests (the reason one has to do this is that MPFR tracks the
correctness of the arguments of GMP's allocate functions in its
test suite).

#include "mpfr-test.h"

int main (void)
{
  mpz_t z;

  tests_start_mpfr ();
  printf ("GMP .....  Library: %-12s  Header: %d.%d.%d\n",
          gmp_version, __GNU_MP_VERSION, __GNU_MP_VERSION_MINOR,
          __GNU_MP_VERSION_PATCHLEVEL);
  mpz_init (z);
  mpz_set_ui (z, 1);
  printf ("_mp_size = %d, _mp_alloc = %d\n", z->_mp_size, z->_mp_alloc);
  gmp_printf ("%Zd\n", z);
  mpz_realloc2 (z, (long) INT_MAX * 8 - 55);
  printf ("_mp_size = %d, _mp_alloc = %d\n", z->_mp_size, z->_mp_alloc);
  gmp_printf ("%Zd\n", z);
  printf ("size = %lu\n", (unsigned long) (z->_mp_alloc * BYTES_PER_MP_LIMB));
  mpz_clear (z);
  printf ("OK\n");
  tests_end_mpfr ();
  return 0;
}

I get:

GMP .....  Library: 5.1.3         Header: 5.1.3
_mp_size = 1, _mp_alloc = 1
1
_mp_size = 1, _mp_alloc = 268435456
1
size = 18446744071562067968
tests_free(): bad size 18446744071562067968, should be 2147483648

mpz/clear.c contains:

  (*__gmp_free_func) (PTR (m), ALLOC (m) * BYTES_PER_MP_LIMB);

but both terms of the multiplication have int type.
There's a missing cast to size_t.

Other clear.c files may be affected by the same problem.

BTW, I suggest that you test GMP with:

  ./configure CC=clang CFLAGS="-fsanitize=undefined -fno-sanitize-recover"

There seem to be lots of bugs. :)

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)