integer overflow in mini-gmp due to integer promotion

[Vincent Lefevre]

Hi,

When I test MPFR with

./configure --with-mini-gmp=/home/vlefevre/software/gmp/mini-gmp CC=gcc-13 CFLAGS="-O2 -fsanitize=undefined -fno-sanitize-recover -DMINI_GMP_LIMB_TYPE=short"

I get lots of failures in mini-gmp.c (I suspect that the errors were
hidden by some optimization in GCC 12 and before).

For instance:

cventin:...ftware/mpfr/tests> ./texceptions
mini-gmp.c:993:7: runtime error: signed integer overflow: 46604 * 61440 cannot be represented in type 'int'

This is a call to gmp_udiv_qrnnd_preinv, which has

#define gmp_udiv_qrnnd_preinv(q, r, nh, nl, d, di)                      \
  do {                                                                  \
    mp_limb_t _qh, _ql, _r, _mask;                                      \
    gmp_umul_ppmm (_qh, _ql, (nh), (di));                               \
    gmp_add_ssaaaa (_qh, _ql, _qh, _ql, (nh) + 1, (nl));                \
    _r = (nl) - _qh * (d);                                              \
[...]

Here, both _qh and d have type mp_limb_t, which is an unsigned short.
Since this is a 16-bit type and int is on 32 bits, unsigned short is
promoted to int, which is now a signed integer type, hence the error.

Ditto in gmp_udiv_qr_3by2.

I suppose that the simplest solution would be to add a cast to
unsigned long, so that the multiplication is done in this type (and
the subtraction too as a consequence), as this should be the largest
MINI_GMP_LIMB_TYPE size... hoping that the compiler will optimize.

I've attached a patch.

On the following testcase, GCC generates the same code for i1 and i2
and for s1 and s2 (tested on 32-bit and 64-bit x86, on aarch64, and
on ppc64le):

unsigned int i1 (unsigned int a, unsigned int b, unsigned int c)
{
  return a - b * c;
}

unsigned int i2 (unsigned int a, unsigned int b, unsigned int c)
{
  return a - (unsigned long) b * c;
}

unsigned short s1 (unsigned short a, unsigned short b, unsigned short c)
{
  return a - b * c;
}

unsigned short s2 (unsigned short a, unsigned short b, unsigned short c)
{
  return a - (unsigned long) b * c;
}

Otherwise the sizes of the types could be checked like in
gmp_umul_ppmm.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mini-gmp-intoverflow.patch
Type: text/x-diff
Size: 956 bytes
Desc: not available
URL: <https://gmplib.org/list-archives/gmp-bugs/attachments/20230719/80b36556/attachment.bin>