register corruption under MS Windows / x86-64

[Torbjörn Granlund]

Vincent Lefevre <vincent at vinc17.net> writes:

  There appears to be a bug in mpn/x86_64/fastsse/com-palignr.asm,
  which is now used by the GMP trunk. If I understand correctly,
  the optimized loop uses xmm6 and xmm7 without restoring their
  values. This is correct under Linux, but not under MS Windows,

You are right, and we seem to have been aware of this at some point:

  2013-09-16  Torbjorn Granlund  <tege at gmplib.org>
    * mpn/x86_64/fastsse/copyi-palignr.asm: Preserve xmm6-xmm8 under DOS.

It is a flaw in our testing setup that this calling convention breach is
not caught by the automated testing.  I will fix both bugs.  :-)

Unfortunately the same issue might affect released versions of GMP.  It
is not immediately clear, as all problem code lives in x86_64/fastsse,
code which is then explicitly included from CPU specific subdirs.

-- 
Torbjörn
Please encrypt, key id 0xC8601622