[PATCH] to fix use of out-of-scope stack memory and consequent coredump during 5.0.5 'make check' 'reuse' test on Solaris when compiling with Solaris Studio 'cc'

[Jason Vas Dias]

In mpz/powm.c 's mpz_powm() of gmp-5.0.5 , when compiled with Solaris
Studio 'cc', (NOT gcc) ,
a bad 'bp' pointer is generated having value 0x7, causing a coredump
when accessed by mpn_powm:

mpz/powm.c @ line 75:
  if (UNLIKELY (es <= 0))
    {
      mpz_t new_b;
      ...
#if HANDLE_NEGATIVE_EXPONENT
      MPZ_TMP_INIT (new_b, n + 1);
...
      b = new_b;
#endif
 }
mpz/powm.c @ line 188:
  bp = PTR(b);
  mpn_powm (rp, bp, bn, ep, en, mp, nodd, tp);


 I think alloca() is being used, in which case the memory pointed to
by 'new_b' and hence 'b'
 will be out-of-scope and re-usable (potentially trashed) when
accessed via bp . This suspicion
 was reinforced by the fact that simply moving the declaration of
'new_b' fixes the problem -
 all tests now pass with this patch:
 $ diff -u mpz/powm.c /tmp/powm.c
--- mpz/powm.c  2012-05-06 12:19:50.000000000 +0100
+++ /tmp/powm.c 2012-11-27 18:49:21.250229446 +0000
@@ -61,8 +61,9 @@
   mp_size_t n, nodd, ncnt;
   int cnt;
   mp_ptr rp, tp;
-  mp_srcptr bp, ep, mp;
+  mp_srcptr bp=0, ep, mp;
   mp_size_t rn, bn, es, en, itch;
+  mpz_t new_b;
   TMP_DECL;

   n = ABSIZ(m);
@@ -76,7 +77,6 @@
   es = SIZ(e);
   if (UNLIKELY (es <= 0))
     {
-      mpz_t new_b;
       if (es == 0)
        {
          /* b^0 mod m,  b is anything and m is non-zero.