Overflow in mpz_cmp

[Guillaume Melquiond]

Le 11/02/2020 à 14:22, Marco Bodrato a écrit :
>> When the operand sizes do not match, the mpz_cmp function function just
>> returns the difference of the signed sizes. Unfortunately, this
>> difference might not fit inside the "int" return type, when the numbers
>> are of opposite sign.
> 
> In mini-gmp we defined a macro:
> #define GMP_CMP(a,b) (((a) > (b)) - ((a) < (b)))
> 
> We may use the same idea here too. I mean something like the following:

>  mpz_cmpabs (mpz_srcptr u, mpz_srcptr v) __GMP_NOTHROW
>  {
> -  dsize = usize - vsize;
> +  cmp = (usize > vsize) - (usize < vsize);

Note that mpz_cmpabs does not have any overflow issue, since absolute
sizes are subtracted there. So, except maybe for homogeneity with
mpz_cmp, it can keep using the optimized version.

Best regards,

Guillaume