Overflow in mpz_cmp

[Marco Bodrato]

Ciao,

Il 2020-02-10 18:25 Guillaume Melquiond ha scritto:
> When the operand sizes do not match, the mpz_cmp function function just
> returns the difference of the signed sizes. Unfortunately, this
> difference might not fit inside the "int" return type, when the numbers
> are of opposite sign.

In mini-gmp we defined a macro:
#define GMP_CMP(a,b) (((a) > (b)) - ((a) < (b)))

We may use the same idea here too. I mean something like the following:

diff -r f5601c2a8b11 mpz/cmp.c
--- a/mpz/cmp.c	Sun Feb 09 16:16:19 2020 +0100
+++ b/mpz/cmp.c	Tue Feb 11 14:20:39 2020 +0100
@@ -35,15 +35,15 @@
  int
  mpz_cmp (mpz_srcptr u, mpz_srcptr v) __GMP_NOTHROW
  {
-  mp_size_t  usize, vsize, dsize, asize;
+  mp_size_t  usize, vsize, asize;
    mp_srcptr  up, vp;
    int        cmp;

    usize = SIZ(u);
    vsize = SIZ(v);
-  dsize = usize - vsize;
-  if (dsize != 0)
-    return dsize;
+  cmp = (usize > vsize) - (usize < vsize);
+  if (cmp != 0)
+    return cmp;

    asize = ABS (usize);
    up = PTR(u);
diff -r f5601c2a8b11 mpz/cmpabs.c
--- a/mpz/cmpabs.c	Sun Feb 09 16:16:19 2020 +0100
+++ b/mpz/cmpabs.c	Tue Feb 11 14:20:39 2020 +0100
@@ -36,15 +36,15 @@
  int
  mpz_cmpabs (mpz_srcptr u, mpz_srcptr v) __GMP_NOTHROW
  {
-  mp_size_t  usize, vsize, dsize;
+  mp_size_t  usize, vsize;
    mp_srcptr  up, vp;
    int        cmp;

    usize = ABSIZ (u);
    vsize = ABSIZ (v);
-  dsize = usize - vsize;
-  if (dsize != 0)
-    return dsize;
+  cmp = (usize > vsize) - (usize < vsize);
+  if (cmp != 0)
+    return cmp;

    up = PTR(u);
    vp = PTR(v);


Ĝis,
m