Latest commit introduces undefined behavior in hgcd2.c

[Vincent Lefevre]

On 2019-09-18 21:20:41 +0200, Torbjorn Granlund wrote:
> Guido Vranken <guidovranken at gmail.com> writes:
> 
>   My bignum fuzzer running at OSS-Fuzz came up with this:
> 
>   hgcd2.c:223:42: runtime error: shift exponent 64 is too large for
>   64-bit type 'mp_limb_t' (aka 'unsigned long')
>   #0 0x76a4db in div2 /src/libgmp/mpn/hgcd2.c:223:42
>   #1 0x769684 in __gmpn_hgcd2 /src/libgmp/mpn/hgcd2.c:372:18
>   #2 0x74ac55 in __gmpn_gcd /src/libgmp/mpn/gcd.c:200:11
>   #3 0x73c209 in __gmpz_gcd /src/libgmp/mpz/gcd.c
> 
>   Introduced in commit https://gmplib.org/repo/gmp/rev/f044264e2fe9
> 
> I think it is a false positive.  The result of the shifted value is
> masked when the shift count is not in range.

If the shift count is not in range, the behavior is undefined,
whatever you do with the "result". Having such undefined behavior
is a bug, as the compiler can do some optimization, such as
considering this as dead code. This is different from the case
where you have implementation-defined behavior.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)