fat_init violates host ABI on Win64
nh at nosebud.de
Tue Apr 4 13:28:11 UTC 2017
Dear GMP maintainers,
the Windows x64 ABI requires callers to allocate a 32 byte "parameter
area" before calling into a function, which the callee is allowed to use
as it pleases. fat_init does not do this before calling
__gmpn_cpuvec_init, thus violating the ABI.
This is not usually an issue, since neither gcc nor clang seem to use
the space. MSVC, however, does, which results in rax/rcx/r8/r9
incorrectly being restored to clobbered values, and, in turn, random
crashes as fat_init jumps into the real routines with incorrect arguments.
I'm not sure whether MSVC is a supported compiler (I assume it isn't,
because --enable-fat requires support for variable-length arrays, which
MSVC will not compile in C mode), but it's possible that other compilers
may also use the parameter area as a scratch space, or even that
gcc/clang start using it in future releases.
Attached patch solves this problem by just allocating 32 bytes of stack
space before calling __gmpn_cpuvec_init, and discarding them afterwards.
 "Even if the called function has fewer than 4 parameters, these 4
stack locations ... may be used by the called function for other
purposes ..." https://msdn.microsoft.com/en-us/library/ew5tede7.aspx
-------------- next part --------------
@@ -167,7 +167,13 @@
+ subq $32, %rsp
+ addq $32, %rsp
More information about the gmp-bugs