GMP 6.3.0: Failure building on MinGW (defaulting to ABI=64)

[Simon Sobisch]

Problem when building from 6.3.0 release tarball:

libtool: compile:  ../mpn/m4-ccas --m4=m4 gcc -c -DHAVE_CONFIG_H -I. 
-I.. -D__GMP_WITHIN_GMP -I.. -DOPERATION_x86_64_add_n -Wno-attribute 
x86_64_add_n.asm  -DDLL_EXPORT -DPIC -o .libs/x86_64_add_n.o
m4  -DHAVE_CONFIG_H -D__GMP_WITHIN_GMP -DOPERATION_x86_64_add_n 
-DDLL_EXPORT -DPIC x86_64_add_n.asm >tmp-x86_64_add_n.s
  gcc -c -DHAVE_CONFIG_H -I. -I.. -D__GMP_WITHIN_GMP -I.. 
-DOPERATION_x86_64_add_n -Wno-attribute tmp-x86_64_add_n.s -DDLL_EXPORT 
-DPIC -o .libs/x86_64_add_n.o
tmp-x86_64_add_n.s: Assembler messages:
tmp-x86_64_add_n.s:112: Error: bad register name `%rdi'
tmp-x86_64_add_n.s:113: Error: bad register name `%rsi'
tmp-x86_64_add_n.s:114: Error: bad register name `%rcx'
tmp-x86_64_add_n.s:115: Error: bad register name `%rdx'
tmp-x86_64_add_n.s:116: Error: bad register name `%r8'
tmp-x86_64_add_n.s:117: Error: bad register name `%r9'
tmp-x86_64_add_n.s:119: Error: bad register name `%rsp)'
tmp-x86_64_add_n.s:121: Error: bad register name `%rcx'
tmp-x86_64_add_n.s:123: Error: bad register name `%r8'
tmp-x86_64_add_n.s:124: Error: `jrcxz' is only supported in 64-bit mode
[...]
make[2]: *** [x86_64_add_n.lo] Error 1


Environment:

$ ./config.guess && gcc --version && as --version
haswell-pc-mingw32

gcc.exe (MinGW.org GCC Build-2) 9.2.0

GNU assembler (GNU Binutils) 2.32
This assembler was configured for a target of `mingw32'.


configure options:

./configure --prefix=/mingw --enable-fat \
    --enable-shared --disable-static


configure outputs:

using ABI="64"
       CC="gcc"
       CFLAGS="-Wno-attribute"
       CPPFLAGS=""
       MPN_PATH=" x86_64/coreihwl x86_64/coreisbr x86_64/coreinhm 
x86_64/core2 x86_64 generic"

checking if the assembler knows about the mulx instruction... no
configure: WARNING: 
+----------------------------------------------------------
configure: WARNING: | WARNING WARNING WARNING
configure: WARNING: | Host CPU has the mulx instruction, but it can't be
configure: WARNING: | assembled by
configure: WARNING: |     gcc -c
configure: WARNING: | Older x86 instructions will be used.
configure: WARNING: | This will be an inferior build.
configure: WARNING: 
+----------------------------------------------------------

configure: summary of build options:

   Version:           GNU MP 6.3.0
   Host type:         haswell-pc-mingw32
   ABI:               64
   Install prefix:    /mingw
   Compiler:          gcc
   Static libraries:  no
   Shared libraries:  yes


As the issue "obviously" is the ABI that doesn't match I've tried with 
ABI=32, which works a bit better, to then error with:


libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -D__GMP_WITHIN_GMP -I.. 
-DOPERATION_fat_com -c fat_com.c  -DDLL_EXPORT -DPIC -o .libs/fat_com.o
In file included from ../gmp-impl.h:147,
                  from ../mpn/generic/com.c:31,
                  from .././mpn/x86/fat/com.c:32,
                  from fat_com.c:10:
../fib_table.h:4:1: warning: data definition has no type or storage class
     4 | Error, error, this data is for 64 bits
       | ^~~~~
../fib_table.h:4:1: warning: type defaults to 'int' in declaration of 
'Error' [-Wimplicit-int]
../fib_table.h:4:8: warning: type defaults to 'int' in declaration of 
'error' [-Wimplicit-int]
     4 | Error, error, this data is for 64 bits
       |        ^~~~~
../fib_table.h:4:20: error: expected '=', ',', ';', 'asm' or 
'__attribute__' before 'data'
     4 | Error, error, this data is for 64 bits
       |                    ^~~~
make[2]: *** [fat_com.lo] Error 1


This error seems to be related to changing the ABI after the initial 
configure. Using "make distclean" fixes this.

I therefore think that something like `AC_ARG_VAR([ABI], [Application 
Binary Interface to use])` (maybe also reference 
https://gmplib.org/manual/ABI-and-ISA in its description) should be 
added to configure.ac to make that error on an ABI change and also 
adding this missing piece to configure's help output.


Using the right ABI all tests pass.


Rechecked with GMP 6.2.1 - the ABI needs to be specified there manually, 
too, the same warning is seen in the output.


Just to let you know: make check resulted in multiple occurrences of

   libtool: warning: '-no-install' is ignored for haswell-pc-mingw32
   libtool: warning: assuming '-no-fast-install' instead


Side note, just FYI: to build without warnings on MinGW (also with 
recent MSYS2) the following additional option needs to be specified
   CFLAGS="-Wno-attributes -Wno-ignored-attributes"

Simon