unavoidable abort in mpz/realloc.c

[Theo Schlossnagle]

This affects all platforms.

In mpz/realloc.c, the following code runs before the user-provided
allocation function:

if (sizeof (mp_size_t) == sizeof (int))
    {
      if (UNLIKELY (new_alloc > ULONG_MAX / GMP_NUMB_BITS))
  {
    fprintf (stderr, "gmp: overflow in mpz type\n");
    abort ();
  }
    }
  else
    {
      if (UNLIKELY (new_alloc > INT_MAX))
  {
    fprintf (stderr, "gmp: overflow in mpz type\n");
    abort ();
  }
    }

This means that there is no way to subvert the abort if any piece of
code calls mpz_realloc with a large allocation request (in my case
mpz_n_pow_ui can do so with large user inputs).

If libgmp can't do what was requested (due to an overflow), it should
not result in the unconditional abortion of the host application.

I *suggest* replacing all aborts in libgmp (there are very few) with a
user-replaceable function.  This would have zero performance impact,
and no change on current function.  In other words, it has absolutely
no positive or negative impact on any current use of libgmp and only
benefits integrators (like me) that are attempting to embed/extend
other languages with libgmp's wonderful and performant capabilities.

I'll note that I sent a working patch to gmp-devel, however the
conversation was unproductive there.

http://gmplib.org/list-archives/gmp-devel/2011-March/001806.html

-- 
Theo Schlossnagle

http://omniti.com/is/theo-schlossnagle