integer overflow in mpz/clear.c of GMP 5.1.3

[Vincent Lefevre]

On 2014-01-18 22:38:44 +0100, Torbjorn Granlund wrote:
> Vincent Lefevre <vincent at vinc17.net> writes:
> 
>   #include "mpfr-test.h"
>   
>   int main (void)
>   {
>     mpz_t z;
>   
>     tests_start_mpfr ();
>     printf ("GMP .....  Library: %-12s  Header: %d.%d.%d\n",
>             gmp_version, __GNU_MP_VERSION, __GNU_MP_VERSION_MINOR,
>             __GNU_MP_VERSION_PATCHLEVEL);
>     mpz_init (z);
>     mpz_set_ui (z, 1);
>     printf ("_mp_size = %d, _mp_alloc = %d\n", z->_mp_size, z->_mp_alloc);
>     gmp_printf ("%Zd\n", z);
>     mpz_realloc2 (z, (long) INT_MAX * 8 - 55);
>     printf ("_mp_size = %d, _mp_alloc = %d\n", z->_mp_size, z->_mp_alloc);
>     gmp_printf ("%Zd\n", z);
>     printf ("size = %lu\n", (unsigned long) (z->_mp_alloc * BYTES_PER_MP_LIMB));
> 
> This is buggy-----------------------------^
> 
>     mpz_clear (z);
>     printf ("OK\n");
>     tests_end_mpfr ();
>     return 0;
>   }
>   
>   size = 18446744071562067968
> 
> This printout is bad because you cast to long too late.

I know. That's what I've said in my bug report! The printf outputs
the value computed by GMP, to show the bug:

  (*__gmp_free_func) (PTR (m), ALLOC (m) * BYTES_PER_MP_LIMB);
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It was not clear, but I added the printf in the program above to make
sure that the bug wasn't in MPFR's tests_free() function.

> The (internal) _mp_alloc field is of type int, when you multiply it
> by the (internal) BYTES_PER_MP_LIMB you will get overflow.
> 
> I checked mpz_realloc2 and think the code's limit checks are correct.
> 
> If mpfr barfs in its sanity checks, it might be due to the same type of
> bugs as above.

No, MPFR just outputs the value from the __gmp_free_func, which has
been computed by mpz/clear.c.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)