Invalid operand size in inline assembler instruction on x86-64 with LTO

[Oleg Oshmyan]

Hi,

GMP 6.1.1 (and probably earlier versions) uses a wrong constraint in the x86 inline assembler code for the MPN_IORD_U macro that blows up (fails to assemble) on x86-64 when compiled using GCC 6 with link-time optimization enabled. The problem occurs when a compile-time constant does not fit in a 32-bit immediate operand, so my guess is that it is specifically triggered by constant propagation across source files.

The code currently uses a constraint that allows any compile-time constant as an immediate operand, but x86-64 supports only 32-bit immediate values (even in 64-bit instructions), whereas the value MPN_IORD_U uses with this constraint may be longer than that. GCC, at least since version 3.1.1, actually provides a constraint just for such an occasion, which allows a compile-time constant as an immediate operand if and only if its value fits in 32 bits. In fact, GCC provides two such constraints: one for signed values (for sign-extending instructions) and one for unsigned values (for zero-extending instructions). MPN_IORD_U is used only with sign-extending instructions (ADD and SUB), so the fix is easy: simply replace the currently used constraint with the signed 32-bit immediate constraint.

I have not investigated whether the same bug appears in other places. GMP currently builds successfully for me with the below patch applied.

Here is a patch:

diff --git a/gmp-impl.h b/gmp-impl.h
index 6987581..dcc233b 100644
--- a/gmp-impl.h
+++ b/gmp-impl.h
@@ -2687,7 +2687,7 @@ __GMP_DECLSPEC mp_bitcnt_t mpn_remove (mp_ptr, mp_size_t *, mp_srcptr, mp_size_t
 	   ASM_L(done) ":\n"						\
 	   : "=r" (__ptr_dummy)						\
 	   : "0"  (ptr),						\
-	     "ri" ((mp_limb_t) (incr)), "n" (sizeof(mp_limb_t))		\
+	     "re" ((mp_limb_t) (incr)), "n" (sizeof(mp_limb_t))		\
 	   : "memory");							\
       }									\
   } while (0)