Compression that can be used with mpz_*_raw

[David McKen]

Essentially it boiled down to using the import and export functions
and writing it out using the various reading/writing functions
provided by the compression library, using a temporary buffer.

I have essentially reverse enginered the file format that is used by
mpz_*_raw and found one dicrepancy in the manual concerning the
format. It seems there is an extra byte written after every number,
this byte has the value of '0A' in hex. Other than this byte the
manual is lays out the file format correctly.

If these compression libraries had a C++ stream interface then it
might be possible to implement a way to detect the ios::binary (or
something similar) flag and then have the stream operator write the
number in the binary format instead of text. As of yet I am yet to
find a way to detect this flag, to the best of my knowledge the flag
is disposed of once stream_name.open() returns.

A simpler interface without going to C++ streams is not possible due
to the fact that you cannot stick an intermediate function or layer
between FILE*'s and the implementation. Sadly I don't know how to
create my own specialized C++ stream (a new stream class) which would
have the buffer and call the compression libraries' functions.

Hope this Helps
David McKen

--- Dirk Osterkamp <osterkamp at gmx.de> wrote:
> Hello David,
> 
> I have read you question about "Compression that can be used...."
> in the 
> gmp-discuss mailing list. May I ask you if you have found a
> solution? I 
> have more or less the same problem.
> 
> I would be glad if you can report me (and the gmp-discuss) a little
> bit 
> about your experience with the suggestions (zlib and bzip2).
> 
> Thank you very much!
> 
> Dirk Osterkamp
> 
> > Ok, I have a program that writes a bunch of numbers to a file
> (the
> > numbers are aprox 300 digits long). 
> > 
> > Putting 1,000,000 of them in a file with mpz_out_raw gives me a
> 129Mb
> > file, which needless to say is kinda big.
> > 
> > Just on a hunch I decided to try compressing the file. The
> results
> > were quite impressive, .zip (Microsoft's built in compressor,
> Yikes)
> > - 1.5Mb, .bz2 (win32 command line bin util I found online) -
> 553Kb, I
> > didn't try .gz experience has shown me that it usually falls
> right
> > between bz2 and zip. This is great! I can store and transport the
> > numbers easily, one problem though. When I want to do something
> with
> > said numbers do I have to de-compress the entire file and open it
> > normally?
> > 
> > Essentially it boils down to, Does anyone know a compression
> library
> > that can work well with GMP? I know of a C++ zip stream library
> but
> > the mpz_*_raw functions use a FILE * which kinda makes it hard to
> put
> > a man in the middle. This brings up another question, are the
> > mpz_*_raw functions accessible through the C++ interface? From
> what I
> > can tell they are not.
> > 
> > I have considered spliting the file so each individual file is
> > smaller and just have more of them but that brings a problem with
> > both .gz and .bz2, I would have to use a tar archive as well
> which
> > would require that I still de-compress the entire tar archive and
> > then access the files I want. Luckily this does not knock out zip
> but
> > I would prefer the better compression.
> > 
> > Thanks in Advance
> > David McKen
> > 
> > 
> > 	
> > 		
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! Movies - Buy advance tickets for 'Shrek 2'
> > http://movies.yahoo.com/showtimes/movie?mid=1808405861 
> > _______________________________________________
> > gmp-discuss mailing list
> > gmp-discuss at swox.com
> > https://gmplib.org/mailman/listinfo/gmp-discuss
> > 
> > 
> 



	
		
__________________________________
Do you Yahoo!?
Friends.  Fun.  Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/