parameter size mismatch in mpn_pi1_bdiv_q_1

Robert Yates rob at
Thu Dec 14 22:14:48 CET 2023


I found an obscure bug in gmp 6.3.0.

#define mpn_pi1_bdiv_q_1 __MPN(pi1_bdiv_q_1)
__GMP_DECLSPEC mp_limb_t mpn_pi1_bdiv_q_1 (mp_ptr, mp_srcptr, mp_size_t, 
mp_limb_t, mp_limb_t, int);

define(`cnt', `x5')
	sub	n, n, #1
	subs	x6, x6, x6		C clear r6 and C flag
	ldr	x9, [up],#8
	cbz	cnt, L(norm)

the cbz instruction acts upon parameter 6(32bit int) with the x5(64bit 

i work on an obfuscation compiler at quarkslab, and its possible that 
optimisations or our transformation can leave random bits in the upper 
data of x5 which will make the implementation of mpn_pi1_bdiv_q_1 fail, 
since clangs codegen will not emit trunc instructions for the 64bit 
register as the function prototype clearly states `i32` type although 
the internal function uses a 64bit register.

i noticed this issue on apple-arm64.
i believe the implemetation should be changed to use the `w5` register, 
or the function prototype should use `long`, either of these solves the 
issues ive seen.

hope this helps :)

More information about the gmp-bugs mailing list