Bug in mpz_out_str...

[Torbjorn Granlund]

David Cleaver <wraithx at morpheus.net> writes:

  I have been looking at mpz_out_str recently, and found what appears to
  be a bug (or logical flaw) in the code.  I am looking at out_str.c in
  the mpz directory of GMP 5.0.1.  When checking on the base, bases >= 0
  are handled correctly, but bases < 0 have no check to make sure they
  are "in bounds".  If you pass a base that is < -36, you can get
  undefined behavior since num_to_text is only 36 characters long, and a
  call to num_to_text[str[i]] can run off the end of this character
  array.  Also, there is no check to handle the case when base = 1 or
  -1.  When I ran the following program with base = 1 (or -1), the
  program never completed, it just hung.
  
  Here is an example program that on my system output a null character
  into the output file:
  
  #include <stdio.h>
  #include <gmp.h>
  
  int main(int argc, char* argv[])
  {
    mpz_t num;
    char filename[] = "output.txt";
    FILE *output;
  
    if ((output = fopen(filename, "w")) == NULL)
    {
      printf(" ***ERROR***: Unable to open %s\n", filename);
      return 1;
    }/* end if */
  
    mpz_init_set_ui(num, 43112609);
  
    mpz_out_str(output, -60, num);
  
    if (fclose(output) != 0)
      printf(" ***ERROR***: Error closing %s\n", filename);
  
    mpz_clear(num);
  
    return 0;
  }
  
It is a philosophical question if a library should always behave in a
nice way if asked to do things that is undefined according to the
documentation.

I agree that it is a little strange that the current code checks for
undefined bases > 62, when it does not check for other undefined bases.

Undefined bases:

  (1) -oo ... -37
  (2) -1
  (3) +1
  (4) +63 ... +oo

Of these, we only recognise case 4.  What is right to do?  GMP doesn't
do a whole lot of checks of consistency of arguments, and I am not sure
we should change that.

Opinions?

-- 
Torbjörn