Latest commit introduces undefined behavior in hgcd2.c

Guido Vranken guidovranken at gmail.com
Wed Sep 18 19:24:28 UTC 2019


I don't mind if you don't fix it, but technically undefined behavior
can have consequences beyond the value of the involved variable.

On Wed, Sep 18, 2019 at 9:20 PM Torbjörn Granlund <tg at gmplib.org> wrote:
>
> Guido Vranken <guidovranken at gmail.com> writes:
>
>   My bignum fuzzer running at OSS-Fuzz came up with this:
>
>   hgcd2.c:223:42: runtime error: shift exponent 64 is too large for
>   64-bit type 'mp_limb_t' (aka 'unsigned long')
>   #0 0x76a4db in div2 /src/libgmp/mpn/hgcd2.c:223:42
>   #1 0x769684 in __gmpn_hgcd2 /src/libgmp/mpn/hgcd2.c:372:18
>   #2 0x74ac55 in __gmpn_gcd /src/libgmp/mpn/gcd.c:200:11
>   #3 0x73c209 in __gmpz_gcd /src/libgmp/mpz/gcd.c
>
>   Introduced in commit https://gmplib.org/repo/gmp/rev/f044264e2fe9
>
> I think it is a false positive.  The result of the shifted value is
> masked when the shift count is not in range.
>
> (We got the same false positive from our nightly testing using gcc's
> sanitized-something command-line option.)
>
> --
> Torbjörn
> Please encrypt, key id 0xC8601622


More information about the gmp-bugs mailing list