FreeBSD links wrong library for tests if one is installed in $prefix

[Emmanuel Thomé]

Hi,

Here's a nasty bug that affects GMP.

Assume there's already a libgmp.so in $prefix/lib . Say, for the purpose
of this report, that it's a bastardized one, with the following patch for
example:

--- mpn/generic/mul.c	2017-06-27 10:12:07.151400000 +0200
+++ mpn/generic/mul.c.bastard	2017-06-27 10:11:50.132594000 +0200
@@ -115,11 +115,17 @@
 
 #define ITCH (16*vn + 100)
 
+#include <stdio.h>
+
 mp_limb_t
 mpn_mul (mp_ptr prodp,
 	 mp_srcptr up, mp_size_t un,
 	 mp_srcptr vp, mp_size_t vn)
 {
+
+	fprintf(stderr, "FLYING SAUCERS!\n");
+	abort();
+
   ASSERT (un >= vn);
   ASSERT (vn >= 1);
   ASSERT (! MPN_OVERLAP_P (prodp, un+vn, up, un));

The following assume that "./configure --prefix=$prefix && make && make
install" have already been run based on that bastardized source tree.

Now do a fresh untar of gmp (without that buggy change!), and:

    ./configure --prefix=$prefix
    make
    make check

And you get:

    [...]
    Abort trap (core dumped)
    FAIL: t-mul
    [...]

When I run t-mul directly:
    $ ./tests/mpn/t-mul
    FLYING SAUCERS!
    Abort trap (core dumped)

So in effect, GMP's make check links ./tests/mpn/t-mul with
$prefix/lib/libgmp.so instead of ./libgmp.la.

When we get fireworks like this, that's fine. But it needs not always be
so. One could imagine use cases where one is misled into thinking that
some change to the source tree is harmless while in fact it isn't
---that's already more worrisome.

I've witnessed this with gmp-6.1.2 on FreeBSD 11.0 (amd64). Not on any
other system. I came across this issue with the gf2x package I maintain,
and it happens similarly with gmp. I wouldn't be surprised if the
behaviour were the same with mpfr (Vincent ?).

I wonder whether it's a libtool bug, or a collective misuse of it (after
all, most of the configury here and there tends to be inspired from other
projects).

I'd be glad if you could share opinions on this.

Best,

E.