out-of-bound write

[Zimmermann Paul]

       Hi,

the following program with GMP 5.1.3 exhibits an out-of-bound write on a
32-bit computer (or on a 64-bit computer with ABI=32). Indeed the input
number 19^3134+14 has exactly 417 digits in base 2^32, thus rp[417] should
not be written to, but it is:

tarte% gcc -I/tmp/include -m32 /tmp/bug.c /tmp/lib/libgmp.a
tarte% ./a.out
before mpn_set_str, rp[417]=17
after mpn_set_str,  rp[417]=0
rn=417

Note: this was discovered while trying to compile GNU MPFR with the CompCert
verified compiler (http://compcert.inria.fr/).

Paul Zimmermann

#include <stdio.h>
#include <string.h>
#include "gmp.h"

main()
{
  unsigned char str[3135];
  mp_limb_t rp[418];
  int i;
  mp_size_t rn;

  memset (str, 0, 3135);
  str[3134] = 1;
  str[0] = 14;
  rp[417] = 17;
  printf ("before mpn_set_str, rp[417]=%lu\n", rp[417]);
  rn = mpn_set_str (rp, str, 3135, 19);
  printf ("after mpn_set_str,  rp[417]=%lu\n", rp[417]);
  printf ("rn=%lu\n", rn);
}