Undefined behavior in t-sqrlo test

[Andrew Wock]

The t-sqrlo test invokes undefined behavior.

The mpn_ptr scratch as well as the boundary values of the mpn_ptr pp are
uninitialized values.  In t-sqrlo.c their values are recorded (lines
92,93,100,101) before mpn_sqrlo and mpn_sqr are called.  Their values are
subsequently checked (lines 105,106) to see if the values at these
locations in memory have changed.

I understand the programmers logic here- the code appears as if it simply
checks that some uninitialized set of bits remains the same uninitialized
set of bits after a function call.  However, the C standard doesn't
represent the value at that pointer as being a set of bits- it represents
it as being something like a floating point NaN- something that ideally
shouldn't be operated upon, a placeholder waiting for a real value.

I came across this undefined behavior when compiling tests/mpn/t-sqrlo.c
with clang and linking it to the otherwise correctly compiled gmp library.
I am an LLVM/Clang developer, so I at first suspected that the issue was a
compiler bug.

Clang assumes that the memory at and around pp is initialized within the
function calls; however, there is nothing to suggest that the memory at
scratch is ever initialized.  At -O2 optimization, clang optimizes the
check for scratch[-1] != s_before into an undefined check, which is later
treated as being always false, causing the test case to fail.  Although
this may seem like an issue with clang, it is within the right of the
compiler to do this since controlling the flow of the program based on
uninitialized memory is undefined behavior.

I was able to get the tests to pass by simply memset()ting scratch to any
value, and it should be trivially fixable by doing basically this but the
proper way one might do it if they were a GMP developer.  I would also
suggest doing the same with pp[-1] and pp[n] in this case, with care that
whatever values those are set to are unlikely to be values that a buggy
mpn_sqrlo() call would set those to.

GMP version 6.1.2
Investigated on RHEL on PowerPC, but recreated on FreeBSD on x86-64.
uname -a output:
Linux -----.----.---.--- 4.14.0-115.el7a.ppc64le #1 SMP Tue Sep 25 12:28:39
EDT 2018 ppc64le ppc64le ppc64le GNU/Linux
config.guess:
power9-unknown-linux-gnu
configfsf.guess:
powerpc64le-unkown-linux-gnu
The bug was triggered with an experimental version of LLVM/Clang