register corruption under MS Windows / x86-64

[Vincent Lefevre]

There appears to be a bug in mpn/x86_64/fastsse/com-palignr.asm,
which is now used by the GMP trunk. If I understand correctly,
the optimized loop uses xmm6 and xmm7 without restoring their
values. This is correct under Linux, but not under MS Windows,
according to:

  https://en.wikipedia.org/wiki/X86_calling_conventions#x86-64_calling_conventions

which says:

  In x86-64, Visual Studio 2008 stores floating point numbers in XMM6
  and XMM7 (as well as XMM8 through XMM15); consequently, for x86-64,
  user-written assembly language routines must preserve XMM6 and XMM7
  (as compared to x86 wherein user-written assembly language routines
  did not need to preserve XMM6 and XMM7). In other words,
  user-written assembly language routines must be updated to
  save/restore XMM6 and XMM7 before/after the function when being
  ported from x86 to x86-64.

Since the commit https://gmplib.org/repo/gmp/rev/069ff6facc35 [*]
(which enables this asm file), MS Windows users get obscure failures
in MPFR 4.0.0-rc1 (debugging shows a floating-point value becoming
NaN) under some conditions (depending on whether optimizations are
used, on output of debug information, and on whether the library is
shared or static...).

[*] https://sympa.inria.fr/sympa/arc/mpfr/2017-12/msg00045.html

More information here:

  https://sympa.inria.fr/sympa/arc/mpfr/2017-12/msg00051.html

and in earlier messages of the thread.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)