Segmentation fault with mpz_inp_raw on gcc45

[Torbjörn Granlund]

Vincent Lefevre <vincent at vinc17.net> writes:

  In mpz/inp_raw.c, I think that abs_csize*8 yields an integer overflow
  on large sizes.

Indeed.

Going from byte count to bit count to limb count is non-trivial without
risking overflow, even if the end result will typically be smaller than
the incoming byte count (assuming we're not using gmp/asl.h with tiny
limbs...).

Below is a fix which works unless GMP_NUMB_BITS mod 8 != 0.  I think we
have this problem in more places in the library.  I spotted
mpz/import.c, but there might be more.

Perhaps, instead of the change below which triggers an errir in the
allocation statement later in the program flow, we could make an
explicit check of the byte count vs the max supported sizes, and
generate the overflow error locally.


*** /tmp/extdiff.b8_0au15/gmp.09a7d62c4438/mpz/inp_raw.c	Tue Sep 14 19:05:51 2021
--- /home/tege/prec/gmp/mpz/inp_raw.c	Thu Sep 16 22:35:46 2021
***************
*** 90,94 ****
  
    /* round up to a multiple of limbs */
!   abs_xsize = BITS_TO_LIMBS (abs_csize*8);
  
    if (abs_xsize != 0)
--- 90,99 ----
  
    /* round up to a multiple of limbs */
!   if (GMP_NUMB_BITS % 8 != 0)
!     abs_xsize = BITS_TO_LIMBS (abs_csize*8); /* FIXME: may overflow */
!   else
!     abs_xsize = (abs_csize + (GMP_NUMB_BITS / 8) - 1) / (GMP_NUMB_BITS / 8);
! 
!   printf ("XXX %08lx %08lx\n", csize, abs_xsize);
  
    if (abs_xsize != 0)


-- 
Torbjörn
Please encrypt, key id 0xC8601622