GCL, GMP linkage

[Camm Maguire]

Greetings!

[cc'ed to the gmp list for the discussion of sse accelerations in the
gmp library as potentially attainable in a lisp system doing large
integer calculations (toward the end).]

Richard Fateman <fateman at cs.berkeley.edu> writes:

> Camm Maguire wrote:
> 
> >Greetings!
> >
> >Well, the way it works in C, at least with gcc, is that only
> >needed/used symbols are taken from a library and placed in the symbol
> >table of the executable.  One should be able to override this with
> >-Wl,-E to the linker command line, but this would not be useful in a
> >lisp without a lisp interface to the functions in any case, except
> >arguably that the user could then write a lisp interface without
> > recompiling.
> This is exactly what I find useful. A  gmp.dll with a symbol table.  I
> use it from Allegro CL
> by doing this...
> 

Thanks for this example!  Something like this is definitely doable
(i.e. I think I know how to do it already :-).  I also think a generic
interface of this sort would be quite useful.  We'd probably need to
discuss which of a variety of similar approaches would be best,
e.g. uffi, before working up a generic interface -- I'm therefore
cc'ing to the list for discussion.  Also have to come to a consensus
on priority.  In the meantime, of course, we can bring forward any
particular gmp functions you or any other user might find helpful much
more quickly.  Is the set below comprehensive?

> :ld gmp41.dll
> 
> (eval-when (compile load eval)
>   (ff:def-foreign-type mpz (* :int))
> 
>   (ff:def-foreign-call
>    (mpz_init "__gmpz_init")
>    ((x (* :int)))
>    :returning :int
>    :arg-checking nil
>    :call-direct t)
>  (ff:def-foreign-call
>    (mpz_set_str "__gmpz_set_str")    ;changes value of mpz x.
>    ((x mpz)
>     (s (* :char))
>     (base :int))
>    :strings-convert t  :returning :int
>    :arg-checking nil)
>    ;;   :call-direct t
>     (ff:def-foreign-call
>    (mpz_get_str "__gmpz_get_str")
>    ((s :int);;let it allocate the space
>     (base :int)
>     (op mpz))
>    :returning  ((* :char) )
>    :arg-checking nil :call-direct t)
> 
>   (ff:def-foreign-call
>    (mpz_get_d  "__gmpz_get_d") ((x mpz))
>    :returning :double
>    :arg-checking nil :call-direct t)
>  (ff:def-foreign-call;; remove gmp number from memory
>    (mpz_clear  "__gmpz_clear") ((x mpz))
>    :returning :int
>    :arg-checking nil :call-direct t)
>  (ff:def-foreign-call
>    (mpz_mul  "__gmpz_mul")
>    ((target mpz)(op1 mpz)(op2 mpz))
>    :returning :void                           :arg-checking nil
> :call-direct t)
> 
>   (ff:def-foreign-call
>    (mpz_add  "__gmpz_add")
>    ((target mpz)(op1 mpz)(op2 mpz))
>    :returning :void
>    :arg-checking nil :call-direct t)
>  (ff:def-foreign-call
>    (mpz_addmul  "__gmpz_addmul");;rop <-rop+op1*op2
>    ((rop mpz)(op1 mpz)(op2 mpz))
>    :returning :void
>    :arg-checking nil :call-direct t)
>    (ff:def-foreign-call;; convert fixnum y to mpz and store in x
>    (mpz_set_si  "__gmpz_set_si") ((x mpz) (y :fixnum) )
>    :returning :void
>    :arg-checking nil
>    :call-direct t
>    )
> 
>   (ff:def-foreign-call
>    (mpz_mul_2exp  "__gmpz_mul_2exp")   ;; set target to op1*2^op2
>    ((target mpz)(op1 mpz)(op2 :long))  ;****
>    :returning :void
>    :arg-checking nil :call-direct t)
>  (ff:def-foreign-call
>    (mpz_sign  "__gmpz_sign")   ;; return -1, 0, 1
>    ((op1 mpz))  ;****
>    :returning :fixnum
>    :arg-checking nil :call-direct t)
>  (ff:def-foreign-call
>    (mpz_size  "__gmpz_size")   ;; number of limbs
>    ((op1 mpz))  ;****
>    :returning :fixnum
>    :arg-checking nil :call-direct t)
> 
> (ff:def-foreign-call
>    (mpz_limbn  "__gmpz_getlimbn")   ;; unsigned byte-32
>    ((op1 mpz) (op2 :long))  :returning :long ;**** might be negative??
> must make unsigned
>    :arg-checking nil :call-direct t)
>  ;;This operation can also be defined as a left shift by op2 bits.
>  ;; add more interface program declarations here
> )
> 
> > It should not be too hard to do this (including the lisp
> >interface) on a semi-automated basis for any given external shared
> >library, but I'm not really sure if this isn't a really low priority.
> >
> 
> As you can see, automating this is possible, but actually I just add
> what I need
> when it occurs to me that I need something.  I do not ever recompile
> gmp41.dll
> 

Just wondering, I assume this happens via dlopen.  What happens in
Allegro when you "save the system" and restart?  Is gmp4l.dll
automatically reloaded with your bindings?  We have a gcl developer
(Michael Koehne) working on something similar.

> 
> >GCL actually has hooks already, currently disabled for Linux, for
> >linking in libraries at runtime without loss of session.  If anyone
> >needs a particular gmp function in the short term, please let me know
> >and I'll make it available.
> >
> I think this is a very useful facility if you want to build a system
> that grows and grows ...
> 
> >One thing I've been mulling over regarding GCL built binaries is an
> >automatic selection of the fastest gmp routines possible for the
> >general architecture at runtime, i.e. sse1, sse2, etc.  I haven't
> >pursued this as I haven't been able to see any noticeable performance
> >increase when using sse2 routines even on heavy bignum lisp
> >benchmarks.  I think you might have made this observation in the
> >past.  But then again perhaps I'm using the wrong tests.
> >
> 
> What do you mean by heavy bignum benchmarks?  Really long numbers or lots of
> modest sized ones?   I think that huge numbers are not of much real
> interest. (or
> should I say rational interest...)
> but small ones,  1 - 100 decimal digits
> and especiall 1-6 decimal digits
> matter a lot.
> 

I agree that the case of many modest sized numbers is all that really
matters.   I basically just looked at Vadim's latest maxima benchmark,
ratsimp((x+y+z)^500)$, using a gmp with generic x86 code and one with
tuned SSE2 extensions.  And I could not see a difference, even when
the GBC was setup so as not to be the bottleneck.  I'd be most
interested if anyone else has encountered a real case where SSE or
SSE2 gmp speeds up bignum processing.  I've heard that clisp has a
good gmp setup, but I've not looked into this aspect myself.

Take care,

> 
> >
> >Take care,
> >
> >Richard Fateman <fateman at cs.berkeley.edu> writes:
> >
> >
> >>I would have expected that, if gcl is linked with gmp.dll,
> >>that all the entry points in gmp would be accessible via
> >>a foreign function linkage.  I believe all of the entry
> >>points are available via FFI in Allegro CL, which has its
> >>own bignum facility, but for bignums longer that a few words
> >>is not as fast as the GMP routines.
> >>
> >
> >
> 
> 
> 
> 

-- 
Camm Maguire			     			camm at enhanced.com
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah