fat_init violates host ABI on Win64

[Nicolas Hake]

Dear GMP maintainers,

the Windows x64 ABI requires callers to allocate a 32 byte "parameter 
area" before calling into a function, which the callee is allowed to use 
as it pleases[1]. fat_init does not do this before calling 
__gmpn_cpuvec_init, thus violating the ABI.

This is not usually an issue, since neither gcc nor clang seem to use 
the space. MSVC, however, does, which results in rax/rcx/r8/r9 
incorrectly being restored to clobbered values, and, in turn, random 
crashes as fat_init jumps into the real routines with incorrect arguments.

I'm not sure whether MSVC is a supported compiler (I assume it isn't, 
because --enable-fat requires support for variable-length arrays, which 
MSVC will not compile in C mode), but it's possible that other compilers 
may also use the parameter area as a scratch space, or even that 
gcc/clang start using it in future releases.

Attached patch solves this problem by just allocating 32 bytes of stack 
space before calling __gmpn_cpuvec_init, and discarding them afterwards.

Thanks,
Nicolas


[1] "Even if the called function has fewer than 4 parameters, these 4 
stack locations ... may be used by the called function for other 
purposes ..." https://msdn.microsoft.com/en-us/library/ew5tede7.aspx
-------------- next part --------------
--- a/mpn/x86_64/fat/fat_entry.asm
+++ b/mpn/x86_64/fat/fat_entry.asm
@@ -167,7 +167,13 @@
 	push	%r8
 	push	%r9
 	push	%rax
+ifdef(`HOST_DOS64',`dnl
+	subq	$32, %rsp
+')
 	CALL(	__gmpn_cpuvec_init)
+ifdef(`HOST_DOS64',`dnl
+	addq	$32, %rsp
+')
 	pop	%rax
 	pop	%r9
 	pop	%r8