fat_init violates host ABI on Win64

[Torbjörn Granlund]

Nicolas Hake <nh at nosebud.de> writes:

  the Windows x64 ABI requires callers to allocate a 32 byte "parameter
  area" before calling into a function, which the callee is allowed to
  use as it pleases[1]. fat_init does not do this before calling
  __gmpn_cpuvec_init, thus violating the ABI.
  
Right; apparently we're aware of this as such allocations are performed
in most cases in GMP.  There seem to be 5 forgotten places, the one you
found and 4 more.

  I'm not sure whether MSVC is a supported compiler (I assume it isn't,
  because --enable-fat requires support for variable-length arrays,
  which MSVC will not compile in C mode), but it's possible that other
  compilers may also use the parameter area as a scratch space, or even
  that gcc/clang start using it in future releases.
  
We should follow the ABI, of course.

Where do we (unconditionally) rely on variable-length arrays?

While such arrays are in the C standard since 18 years, it is missing
from C++.

  Attached patch solves this problem by just allocating 32 bytes of
  stack space before calling __gmpn_cpuvec_init, and discarding them
  afterwards.
  
Thanks.  Here is a slightly different patch:

diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/divrem_2.asm gmp-main/mpn/x86_64/divrem_2.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/divrem_2.asm	Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/divrem_2.asm	Tue Apr 25 20:52:18 2017
***************
*** 101,106 ****
--- 101,108 ----
  IFSTD(`	mov	%r11, %rdi	')
  IFDOS(`	mov	%r11, %rcx	')
+ IFDOS(`	sub	$32, %rsp	')
  	ASSERT(nz, `test $15, %rsp')
  	CALL(	mpn_invert_limb)
+ IFDOS(`	add	$32, %rsp	')
  	pop	%r11
  	pop	%r10
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/fat/fat_entry.asm gmp-main/mpn/x86_64/fat/fat_entry.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/fat/fat_entry.asm	Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/fat/fat_entry.asm	Tue Apr 25 20:52:18 2017
***************
*** 168,172 ****
--- 168,174 ----
  	push	%r9
  	push	%rax
+ IFDOS(`	sub	$32, %rsp	')
  	CALL(	__gmpn_cpuvec_init)
+ IFDOS(`	add	$32, %rsp	')
  	pop	%rax
  	pop	%r9
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/mod_1_1.asm gmp-main/mpn/x86_64/mod_1_1.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/mod_1_1.asm	Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/mod_1_1.asm	Tue Apr 25 20:52:18 2017
***************
*** 199,204 ****
--- 199,206 ----
  IFSTD(`	mov	%r12, %rdi	')	C pass parameter
  IFDOS(`	mov	%r12, %rcx	')	C pass parameter
+ IFDOS(`	sub	$32, %rsp	')
  	ASSERT(nz, `test $15, %rsp')
  	CALL(	mpn_invert_limb)
+ IFDOS(`	add	$32, %rsp	')
  	neg	%r12
  	mov	%r12, %r8
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/mod_1_2.asm gmp-main/mpn/x86_64/mod_1_2.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/mod_1_2.asm	Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/mod_1_2.asm	Tue Apr 25 20:52:18 2017
***************
*** 184,189 ****
--- 184,191 ----
  IFSTD(`	mov	%r12, %rdi	')	C pass parameter
  IFDOS(`	mov	%r12, %rcx	')	C pass parameter
+ IFDOS(`	sub	$32, %rsp	')
  	ASSERT(nz, `test $15, %rsp')
  	CALL(	mpn_invert_limb)
+ IFDOS(`	add	$32, %rsp	')
  	mov	%r12, %r8
  	mov	%rax, %r11
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/mod_1_4.asm gmp-main/mpn/x86_64/mod_1_4.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/mod_1_4.asm	Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/mod_1_4.asm	Tue Apr 25 20:52:18 2017
***************
*** 191,196 ****
--- 191,198 ----
  IFSTD(`	mov	%r12, %rdi	')	C pass parameter
  IFDOS(`	mov	%r12, %rcx	')	C pass parameter
+ IFDOS(`	sub	$32, %rsp	')
  	ASSERT(nz, `test $15, %rsp')
  	CALL(	mpn_invert_limb)
+ IFDOS(`	add	$32, %rsp	')
  	mov	%r12, %r8
  	mov	%rax, %r11


-- 
Torbjörn
Please encrypt, key id 0xC8601622