GMP failure on Mac OS X 10.5

[Torbjorn Granlund]

Matt Fago <fago at earthlink.net> writes:

  I installed gcc 4.2.2 via ../gcc/configure; make; make install and  
  then tried to install GMP 4.2.2 using this new compiler via ./ 
  configure; make; make check

  This results in the message:

  make  check-TESTS
  /bin/sh: line 1: 28320 Segmentation fault      ${dir}$tst
  FAIL: t-bswap
  PASS: t-constants
  PASS: t-count_zeros
  PASS: t-gmpmax
  PASS: t-hightomask
  PASS: t-modlinv
  /bin/sh: line 1: 28431 Segmentation fault      ${dir}$tst
  FAIL: t-popc
  PASS: t-parity
  PASS: t-sub
  ==================================
  2 of 9 tests failed
  Please report to gmp-bugs at swox.com
  ==================================
    
I got gcc 4.2.2 built on the Mac system, and could look reproduce the
problem.

The problem is that the FSF gcc (as opposed to the Apple hacked gcc)
does not align the stack at 16-byte boundaries.  The problematic
function is __gmp_randget_mt (from randmt.c), the stack is aligned 8
mod 16 at entry (as it should), then six 8-byte pushes are made, still
leaving the stack 8 mod 16 aligned.  For the calls later in the
function (to the dynamic loader and to __gmp_mt_recalc_buffer) the
stack will be aligned 0 mod 16 at function entry, which is wrong.

The dynamic loader executes a bunch of movdqa instructions with an
offset that is 8 mod 16 relative to the stack, which causes the
SIGSEGV signals sicne the stack is now incorrectly 0 mod 16 aligned.
(These instructions require 16-bit alignment.)

I haven't seen any ABI document for the x86_64 Mac, but the ELF ABI
(used for *BSD, GNU/Linux, Solaris) mandates 8 mod 16 alignment at
function entry.

I looked at some code generated on FreebSD, and it seems the FSF gcc
deviates from the ELF ABI stack alignment requirements there, but
luckily currently this does not trigger any problems.

No GMP bug, workaround within GMP impossible.

-- 
Torbjörn