fat_init violates host ABI on Win64
Torbjörn Granlund
tg at gmplib.org
Tue Apr 25 18:54:01 UTC 2017
Nicolas Hake <nh at nosebud.de> writes:
the Windows x64 ABI requires callers to allocate a 32 byte "parameter
area" before calling into a function, which the callee is allowed to
use as it pleases[1]. fat_init does not do this before calling
__gmpn_cpuvec_init, thus violating the ABI.
Right; apparently we're aware of this as such allocations are performed
in most cases in GMP. There seem to be 5 forgotten places, the one you
found and 4 more.
I'm not sure whether MSVC is a supported compiler (I assume it isn't,
because --enable-fat requires support for variable-length arrays,
which MSVC will not compile in C mode), but it's possible that other
compilers may also use the parameter area as a scratch space, or even
that gcc/clang start using it in future releases.
We should follow the ABI, of course.
Where do we (unconditionally) rely on variable-length arrays?
While such arrays are in the C standard since 18 years, it is missing
from C++.
Attached patch solves this problem by just allocating 32 bytes of
stack space before calling __gmpn_cpuvec_init, and discarding them
afterwards.
Thanks. Here is a slightly different patch:
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/divrem_2.asm gmp-main/mpn/x86_64/divrem_2.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/divrem_2.asm Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/divrem_2.asm Tue Apr 25 20:52:18 2017
***************
*** 101,106 ****
--- 101,108 ----
IFSTD(` mov %r11, %rdi ')
IFDOS(` mov %r11, %rcx ')
+ IFDOS(` sub $32, %rsp ')
ASSERT(nz, `test $15, %rsp')
CALL( mpn_invert_limb)
+ IFDOS(` add $32, %rsp ')
pop %r11
pop %r10
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/fat/fat_entry.asm gmp-main/mpn/x86_64/fat/fat_entry.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/fat/fat_entry.asm Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/fat/fat_entry.asm Tue Apr 25 20:52:18 2017
***************
*** 168,172 ****
--- 168,174 ----
push %r9
push %rax
+ IFDOS(` sub $32, %rsp ')
CALL( __gmpn_cpuvec_init)
+ IFDOS(` add $32, %rsp ')
pop %rax
pop %r9
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/mod_1_1.asm gmp-main/mpn/x86_64/mod_1_1.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/mod_1_1.asm Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/mod_1_1.asm Tue Apr 25 20:52:18 2017
***************
*** 199,204 ****
--- 199,206 ----
IFSTD(` mov %r12, %rdi ') C pass parameter
IFDOS(` mov %r12, %rcx ') C pass parameter
+ IFDOS(` sub $32, %rsp ')
ASSERT(nz, `test $15, %rsp')
CALL( mpn_invert_limb)
+ IFDOS(` add $32, %rsp ')
neg %r12
mov %r12, %r8
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/mod_1_2.asm gmp-main/mpn/x86_64/mod_1_2.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/mod_1_2.asm Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/mod_1_2.asm Tue Apr 25 20:52:18 2017
***************
*** 184,189 ****
--- 184,191 ----
IFSTD(` mov %r12, %rdi ') C pass parameter
IFDOS(` mov %r12, %rcx ') C pass parameter
+ IFDOS(` sub $32, %rsp ')
ASSERT(nz, `test $15, %rsp')
CALL( mpn_invert_limb)
+ IFDOS(` add $32, %rsp ')
mov %r12, %r8
mov %rax, %r11
diff -Nrc2 gmp-main.253deadf9fc8/mpn/x86_64/mod_1_4.asm gmp-main/mpn/x86_64/mod_1_4.asm
*** gmp-main.253deadf9fc8/mpn/x86_64/mod_1_4.asm Tue Apr 25 20:52:18 2017
--- gmp-main/mpn/x86_64/mod_1_4.asm Tue Apr 25 20:52:18 2017
***************
*** 191,196 ****
--- 191,198 ----
IFSTD(` mov %r12, %rdi ') C pass parameter
IFDOS(` mov %r12, %rcx ') C pass parameter
+ IFDOS(` sub $32, %rsp ')
ASSERT(nz, `test $15, %rsp')
CALL( mpn_invert_limb)
+ IFDOS(` add $32, %rsp ')
mov %r12, %r8
mov %rax, %r11
--
Torbjörn
Please encrypt, key id 0xC8601622
More information about the gmp-bugs
mailing list