integer overflow in mini-gmp due to integer promotion

[Niels Möller]

Vincent Lefevre <vincent at vinc17.net> writes:

> ./configure --with-mini-gmp=/home/vlefevre/software/gmp/mini-gmp CC=gcc-13 CFLAGS="-O2 -fsanitize=undefined -fno-sanitize-recover -DMINI_GMP_LIMB_TYPE=short"
>
> I get lots of failures in mini-gmp.c (I suspect that the errors were
> hidden by some optimization in GCC 12 and before).

Interesting. It seems rather undocumented what settings of
MINI_GMP_LIMB_TYPE are supported/tested. I can try to guess your
motivation for using such a small size, but can you give a bit more
context?

> For instance:
>
> cventin:...ftware/mpfr/tests> ./texceptions
> mini-gmp.c:993:7: runtime error: signed integer overflow: 46604 * 61440 cannot be represented in type 'int'

That looks like a problem, since we aim to stick to standard C.

> I suppose that the simplest solution would be to add a cast to
> unsigned long, so that the multiplication is done in this type (and
> the subtraction too as a consequence), as this should be the largest
> MINI_GMP_LIMB_TYPE size... hoping that the compiler will optimize.

I think it's desirable to support setting MINI_GMP_LIMB_TYPE to long
long (which would make sense, e.g., on 64-bit windows, where for some
historical reasons long is still only 32 bits). Unconditionally casting
to unsigned long would break that.
 
> Otherwise the sizes of the types could be checked like in
> gmp_umul_ppmm.

I think that's needed, to be able to support any size of
MINI_GMP_LIMB_TYPE. Something like

#define umullo_limb(u, v) \
  (sizeof(mp_limb_t) >= sizeof(int)) ? (u)*(v) : (unsigned int)(u) * (v))

If I understand you correctly, the two multiplies in
gmp_udiv_qrnnd_preinv and gmp_udiv_qr_3by2 are the only places where we
need a mullo operation, i.e., producing the low limb of a limb product?

Regards,
/Niels

-- 
Niels Möller. PGP key CB4962D070D77D7FCB8BA36271D8F1FF368C6677.
Internet email is subject to wholesale government surveillance.