Overflow in mpz_cmp

[Guillaume Melquiond]

Dear GMP developers,

When the operand sizes do not match, the mpz_cmp function function just
returns the difference of the signed sizes. Unfortunately, this
difference might not fit inside the "int" return type, when the numbers
are of opposite sign.

This is extremely unlikely to happen in practice, as this requires 8GB
numbers. The bug was noticed when formally verifying this function.

Using GMP 6.1.2 (as packaged by Debian) on an x86-64 architecture, the
testcase at the end of the email displays:

    x: +1 * 2^85899345857
    y: -1 * 2^68719476673
    x < y

Obviously, it should display "x >= y" on the last line. The code of
mpz_cmp is unchanged at the tip of the hg repository, so the bug is
likely present there too.

#include <stdio.h>
#include <gmp.h>

int main() {
  mpz_t x, y;
  mpz_init(x); mpz_realloc(x, 0x50000000);
  mpz_init(y); mpz_realloc(y, 0x40000000);
  // create numbers by hand to avoid trashing 19GB of memory
  x->_mp_size = x->_mp_alloc; x->_mp_d[x->_mp_size - 1] = 1;
  y->_mp_size = y->_mp_alloc; y->_mp_d[y->_mp_size - 1] = 1;
  mpz_neg(y, y);
  printf("x: %+d * 2^%ld\n", mpz_sgn(x), mpz_sizeinbase(x, 2));
  printf("y: %+d * 2^%ld\n", mpz_sgn(y), mpz_sizeinbase(y, 2));
  int res = mpz_cmp(x, y);
  if (res >= 0) printf("x >= y\n"); else printf("x < y\n");
}