Overflow in mpz_cmp

Marco Bodrato bodrato at mail.dm.unipi.it
Tue Feb 11 13:22:07 UTC 2020

```Ciao,

Il 2020-02-10 18:25 Guillaume Melquiond ha scritto:
> When the operand sizes do not match, the mpz_cmp function function just
> returns the difference of the signed sizes. Unfortunately, this
> difference might not fit inside the "int" return type, when the numbers
> are of opposite sign.

In mini-gmp we defined a macro:
#define GMP_CMP(a,b) (((a) > (b)) - ((a) < (b)))

We may use the same idea here too. I mean something like the following:

diff -r f5601c2a8b11 mpz/cmp.c
--- a/mpz/cmp.c	Sun Feb 09 16:16:19 2020 +0100
+++ b/mpz/cmp.c	Tue Feb 11 14:20:39 2020 +0100
@@ -35,15 +35,15 @@
int
mpz_cmp (mpz_srcptr u, mpz_srcptr v) __GMP_NOTHROW
{
-  mp_size_t  usize, vsize, dsize, asize;
+  mp_size_t  usize, vsize, asize;
mp_srcptr  up, vp;
int        cmp;

usize = SIZ(u);
vsize = SIZ(v);
-  dsize = usize - vsize;
-  if (dsize != 0)
-    return dsize;
+  cmp = (usize > vsize) - (usize < vsize);
+  if (cmp != 0)
+    return cmp;

asize = ABS (usize);
up = PTR(u);
diff -r f5601c2a8b11 mpz/cmpabs.c
--- a/mpz/cmpabs.c	Sun Feb 09 16:16:19 2020 +0100
+++ b/mpz/cmpabs.c	Tue Feb 11 14:20:39 2020 +0100
@@ -36,15 +36,15 @@
int
mpz_cmpabs (mpz_srcptr u, mpz_srcptr v) __GMP_NOTHROW
{
-  mp_size_t  usize, vsize, dsize;
+  mp_size_t  usize, vsize;
mp_srcptr  up, vp;
int        cmp;

usize = ABSIZ (u);
vsize = ABSIZ (v);
-  dsize = usize - vsize;
-  if (dsize != 0)
-    return dsize;
+  cmp = (usize > vsize) - (usize < vsize);
+  if (cmp != 0)
+    return cmp;

up = PTR(u);
vp = PTR(v);

Ĝis,
m
```