FW: Is GMP safe for using on multi core processors with openmp or pthreads?

Thu Jul 30 22:40:36 CEST 2009

Torbjörn,

You are right that you can trigger the problem with a simple openmp (multithreaded) malloc loop, but don't you think an application library that has a strong dependency on malloc should have a caution in the documentation for the developer using OpenMP and multiple threads? 

The memory leak with malloc for the multithreaded GMP program was a difficult problem to track down and debug.  I was glad/lucky that replacing the standard malloc library routine with Hoard solved our problem and that we did not have to attempt to rewriting the application to use mpf_t with a pre-determined allocation or restrict all the mpz_t references to a single thread.  

In a perfect world the standard malloc library routine would have been replace when multi core processors became available and when compilers started enabling multithreaded applications.  Until that time we can only document the multi thread problem with malloc and warn other developers.  By the way, there are other multithreaded aware malloc routines, besides Hoard, available from vendors and on the internet.
http://developers.sun.com/solaris/articles/multiproc/multiproc.html

Here is that simple non-GMP GCC program that Valgrind reports a memory leak.

#include <stdlib.h>

// Just a big number to amplify the leak
#define ONEMEG 1048576

#define CORES 4 
long *array[CORES];

int main(int argc, char *argv[])
{
   int i;

//   for (;;)
#ifdef _OPENMP
#pragma omp parallel for default(shared) private(i) num_threads(CORES)
#endif
     for (i = 0 ; i < CORES ; i++)
     {
        array[i] = malloc(ONEMEG*sizeof(long));
        free(array[i]);
     }
//   }

   exit(0);
}

Standard malloc with GCC OpenMP

login3$ gcc -Wall -fopenmp -O0 -g -o memory-leak2 memory-leak2.c  -lm
login3$  valgrind --leak-check=full --show-reachable=yes --track-origins=yes  ./memory-leak2
==26509== Memcheck, a memory error detector.
==26509== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al.
==26509== Using LibVEX rev 1884, a library for dynamic binary translation.
==26509== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP.
==26509== Using valgrind-3.4.1, a dynamic binary instrumentation framework.
==26509== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al.
==26509== For more details, rerun with: -v
==26509== 
==26509== 
==26509== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 10 from 5)
==26509== malloc/free: in use at exit: 2,888 bytes in 6 blocks.
==26509== malloc/free: 14 allocs, 8 frees, 6,110 bytes allocated.
==26509== For counts of detected errors, rerun with: -v
==26509== searching for pointers to 6 not-freed blocks.
==26509== checked 6,466,152 bytes.
==26509== 
==26509== 40 bytes in 1 blocks are still reachable in loss record 1 of 3
==26509==    at 0x4A075EE: malloc (vg_replace_malloc.c:207)
==26509==    by 0x4A07777: realloc (vg_replace_malloc.c:429)
==26509==    by 0x4C233C8: gomp_realloc (alloc.c:54)
==26509==    by 0x4C28632: gomp_team_start (team.c:358)
==26509==    by 0x40075E: main (memory-leak2.c:13)
==26509== 
==26509== 
==26509== 864 bytes in 3 blocks are possibly lost in loss record 2 of 3
==26509==    at 0x4A053C4: calloc (vg_replace_malloc.c:397)
==26509==    by 0x375F50D742: _dl_allocate_tls (in /lib64/ld-2.3.4.so)
==26509==    by 0x3760206792: pthread_create@@GLIBC_2.2.5 (in /lib64/tls/libpthread-2.3.4.so)
==26509==    by 0x4C283C9: gomp_team_start (team.c:420)
==26509==    by 0x40075E: main (memory-leak2.c:13)
==26509== 
==26509== 
==26509== 1,984 bytes in 2 blocks are still reachable in loss record 3 of 3
==26509==    at 0x4A075EE: malloc (vg_replace_malloc.c:207)
==26509==    by 0x4C23418: gomp_malloc (alloc.c:36)
==26509==    by 0x4C28794: gomp_new_team (team.c:143)
==26509==    by 0x4C2714B: GOMP_parallel_start (parallel.c:108)
==26509==    by 0x40075E: main (memory-leak2.c:13)
==26509== 
==26509== LEAK SUMMARY:
==26509==    definitely lost: 0 bytes in 0 blocks.
==26509==      possibly lost: 864 bytes in 3 blocks.
==26509==    still reachable: 2,024 bytes in 3 blocks.
==26509==         suppressed: 0 bytes in 0 blocks.

Standard malloc with GCC and without openmp

login3$ gcc -Wall  -O0 -g -o memory-leak2 memory-leak2.c  -lm
login3$  valgrind --leak-check=full --show-reachable=yes --track-origins=yes  ./memory-leak2
==26546== Memcheck, a memory error detector.
==26546== Copyright (C) 2002-2008, and GNU GPL'd, by Julian Seward et al.
==26546== Using LibVEX rev 1884, a library for dynamic binary translation.
==26546== Copyright (C) 2004-2008, and GNU GPL'd, by OpenWorks LLP.
==26546== Using valgrind-3.4.1, a dynamic binary instrumentation framework.
==26546== Copyright (C) 2000-2008, and GNU GPL'd, by Julian Seward et al.
==26546== For more details, rerun with: -v
==26546== 
==26546== 
==26546== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 10 from 5)
==26546== malloc/free: in use at exit: 0 bytes in 0 blocks.
==26546== malloc/free: 4 allocs, 4 frees, 32 bytes allocated.
==26546== For counts of detected errors, rerun with: -v
==26546== All heap blocks were freed -- no leaks are possible.

Thanks,
David

-----Original Message-----
From: tg at gmplib.org [mailto:tg at gmplib.org] 
Sent: Thursday, July 30, 2009 1:36 PM
To: David Carver
Cc: gmp-discuss at gmplib.org
Subject: Re: FW: Is GMP safe for using on multi core processors with openmp or pthreads?

David Carver <dcarver at tacc.utexas.edu> writes:

  Finally, using the information from the previous two articles I was
  able to link the example program with the Hoards malloc library and
  have Valgrind not report any memory leaks.

So Hoards is a reentrant malloc unlike the malloc your gmp application
was linked to by default.

  I suggest maybe adding another section in the GMP documentation to
  address multi threaded issues with GMP.

What makes you think this is an issue with GMP?

If the memory allocation functions set by a call to
mp_set_memory_functions (or malloc and friends by default) are not
reentrant, then GMP will not be reentrant either.

I would be *very* surprised if you could not trigger the problem with
just a simple malloc loop, not involving GMP.

--
Torbjörn
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 474 bytes
Desc: not available
URL: <http://gmplib.org/list-archives/gmp-discuss/attachments/20090730/fdc32f80/attachment.bin>