integer overflow in mini-gmp due to integer promotion

[Vincent Lefevre]

On 2023-07-19 21:24:03 +0200, Niels Möller wrote:
> Vincent Lefevre <vincent at vinc17.net> writes:
> 
> > ./configure --with-mini-gmp=/home/vlefevre/software/gmp/mini-gmp CC=gcc-13 CFLAGS="-O2 -fsanitize=undefined -fno-sanitize-recover -DMINI_GMP_LIMB_TYPE=short"
> >
> > I get lots of failures in mini-gmp.c (I suspect that the errors were
> > hidden by some optimization in GCC 12 and before).
> 
> Interesting. It seems rather undocumented what settings of
> MINI_GMP_LIMB_TYPE are supported/tested. I can try to guess your
> motivation for using such a small size, but can you give a bit more
> context?

The goal is to try to find bugs in MPFR. With limbs of small size,
particular cases (such as some limbs being 0 or -1) could occur
more often.

> I think it's desirable to support setting MINI_GMP_LIMB_TYPE to long
> long (which would make sense, e.g., on 64-bit windows, where for some
> historical reasons long is still only 32 bits). Unconditionally casting
> to unsigned long would break that.

OK.

> > Otherwise the sizes of the types could be checked like in
> > gmp_umul_ppmm.
> 
> I think that's needed, to be able to support any size of
> MINI_GMP_LIMB_TYPE. Something like
> 
> #define umullo_limb(u, v) \
>   (sizeof(mp_limb_t) >= sizeof(int)) ? (u)*(v) : (unsigned int)(u) * (v))
> 
> If I understand you correctly, the two multiplies in
> gmp_udiv_qrnnd_preinv and gmp_udiv_qr_3by2 are the only places where we
> need a mullo operation, i.e., producing the low limb of a limb product?

These are the only ones that affect MPFR, but I haven't looked
at the whole code.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)