Additional memory handler features.

[Victor Shoup]

A couple of comments...

First, it seems to me that Marc's example of custom allocators
can perhaps best be viewed as a work-around of GMP's lack of
"proper" error handling.  So one should be careful not to
conflate error handling with memory management.

Thus, I would suggest one consider custom memory management
and error handling as two separate issues.  If one does this,
then assuming a proper error handling interface is defined,
one can ask if there are any other useful examples of
custom allocators.  I assume there are, but perhaps they 
are a bit more obscure.

So let's ignore custom memory allocators for a minute,
and just think about error handling.
There are just a few viable approaches:
  1) abort with error message (current approach)
  2) return codes (the traditional C approach)
  3) C++ exceptions
  4) longjump

Approach (2) would require changes to the current interface.
For functions that currently return void, one could just make them
return int, and so this would be a fairly minor change.
But there are presumably some functions that already return
a non-void value, and so this would not work for such functions.
Another issue is that clients would now have to constantly
check these return codes -- failure to do so is even worse
than approach (1).
Internal GMP routines must also check these codes, which requires
writing a lot of code.
Anyway, there are lots of variations on this approach:
maybe define a whole set of functions which mirror the current
functions but with a return-code interface.

Approach (3) only work for C++ clients, and probably would require
compiling GMP under C++ and introducing RAII to reclaim resources
within GMP.  I know Marc is currently working on this.
While this approach may have some disadvantages, it has the
huge advantage that no changes need to be made to the
existing interfaces.  Moreover, clients could choose between
approach (1) and (3), either at compile time or possibly at 
runtime (with a global (thread local?) flag).
This is probably the easiest thing to implement in GMP,
especially assuming that GMP is compiled under C++.
With judicious use of macros and RAII classes, it should be possible 
to support a dual code base with approach (1) under pure C
and approach (3) under C++.
I've just completed retrofitting a large C++ library (NTL) in this
way.  It was a pain, but not as painful as I initially thought
it would be.

[NOTE: with this approach, one might also contemplate slowly
migrating GMP to a pure C++ code base, as has been done
with GCC...of course, this would have to be done *very*
slowly and with much care.]

Approach (4), at least to me, does not seem very workable.
As has already been commented elsewhere (I think by Marc),
this would just amount to reimplementing C++ exceptions
in C, which would never be as clean as actual C++ exceptions.

There are many articles and online discussions on error handling,
so most of what I said above is just repeating that...

Now, one can consider the custom allocator interface separately,
assuming it is actually important.  It seems to me there
are not too many approaches:
  1) global function pointers (current approach)
  2) thread-local function pointers 
  3) function pointers passed as parameters

In some sense, (3) is the cleanest solution, but it would probably
break too many interfaces to seriously considered.
So (1) or (2).
But I see mention of itching and scratching: could somebody
describe what that is or provide a link? Sorry for my ignorance.
....and sorry for the length of this post....

[FULL DISCLOSURE: my own, somewhat narrow and selfish goal
is to see GMP's mpn-level routines throw exceptions, rather
than abort, with no interface changes.  This is what would work
best for my own NTL library]

[And yet another issue: my understanding is that some OS's actually
have rather weird ways of dealing with out-of-memory errors:
malloc always succeeds and returns a non-null pointer, but
indirecting through that pointer may abort the program.
In such a setting, all of this memory-related error handling
stuff is pointless]

Sorry...my post just got longer :-(


-Victor


On Jan 4, 2015, at 10:50 AM, Niels Möller wrote:

> Marc Glisse <marc.glisse at inria.fr> writes:
> 
>> Er, people have been using the custom allocators for (possibly
>> imperfect) recovery on allocation failure for years. PPL's configure
>> script complains if gmp was compiled without -fexceptions for that
>> reason. Apparently SWI-Prolog uses longjmp
>> (http://stackoverflow.com/a/14245611/1918193).
> 
> Ok, good to hear about some examples. We ought to investigate these
> before deciding what to do to improve error recovery hooks.
> 
>> Although itch/scratch may force a compromise between overestimating
>> the space required or spending too long estimating it, which I hope
>> won't penalize small numbers too much.
> 
> I think a typical itch function will look like
> 
>  mp_size_t
>  foo_itch (mp_size_t n)
>  {
>    if (LIKELY (BELOW_THRESHOLD (n, FOO_THRESHOLD)))
>      return n; /* Or possibly zero or some other constant */
>    else
>      ... something possibly more complicated, perhaps
>          a full "synthetic computation" ...
>  }
> 
> So it's a function call (I think we usually want to avoid macros here),
> but for small operands it's just the function call overhead and a
> well-predicted branch.
> 
> Regards,
> /Niels
> 
> -- 
> Niels Möller. PGP-encrypted email is preferred. Keyid C0B98E26.
> Internet email is subject to wholesale government surveillance.
> _______________________________________________
> gmp-devel mailing list
> gmp-devel at gmplib.org
> https://gmplib.org/mailman/listinfo/gmp-devel