integer overflow in mpz/clear.c of GMP 5.1.3

[Torbjorn Granlund]

Vincent Lefevre <vincent at vinc17.net> writes:

  #include "mpfr-test.h"
  
  int main (void)
  {
    mpz_t z;
  
    tests_start_mpfr ();
    printf ("GMP .....  Library: %-12s  Header: %d.%d.%d\n",
            gmp_version, __GNU_MP_VERSION, __GNU_MP_VERSION_MINOR,
            __GNU_MP_VERSION_PATCHLEVEL);
    mpz_init (z);
    mpz_set_ui (z, 1);
    printf ("_mp_size = %d, _mp_alloc = %d\n", z->_mp_size, z->_mp_alloc);
    gmp_printf ("%Zd\n", z);
    mpz_realloc2 (z, (long) INT_MAX * 8 - 55);
    printf ("_mp_size = %d, _mp_alloc = %d\n", z->_mp_size, z->_mp_alloc);
    gmp_printf ("%Zd\n", z);
    printf ("size = %lu\n", (unsigned long) (z->_mp_alloc * BYTES_PER_MP_LIMB));

This is buggy-----------------------------^

    mpz_clear (z);
    printf ("OK\n");
    tests_end_mpfr ();
    return 0;
  }
  
  size = 18446744071562067968

This printout is bad because you cast to long too late.  The (internal)
_mp_alloc field is of type int, when you multiply it by the (internal)
BYTES_PER_MP_LIMB you will get overflow.

I checked mpz_realloc2 and think the code's limit checks are correct.

If mpfr barfs in its sanity checks, it might be due to the same type of
bugs as above.


Torbjörn