major formatted output function bug with %c and the value 0

[Vincent Lefevre]

With GMP 6.3.0, the formatted output functions do not handle %c
with the value 0 correctly. For gmp_sprintf, the return value is
incorrect. For gmp_asprintf and gmp_vasprintf, this is either a
buffer overflow (according to the GMP manual: "The block will be
the size of the string and null-terminator.") or, in case this
is an error in the GMP manual, possible memory corruption when
freeing the allocated memory, if the custom memory allocation
function cares about the size parameter.

Testcase for gmp_sprintf:

------------------------------------------------------------
#include <stdio.h>
#include <gmp.h>

static void test (int flag)
{
  char s[3] = { 1, 1, 1 };
  int r;

  r = (flag ? sprintf : gmp_sprintf) (s, "%c", 0);
  printf ("%4s: r = %d, s = { %d %d %d }\n",
          flag ? "libc" : "gmp", r, s[0], s[1], s[2]);
}

int main (void)
{
  test (0);
  test (1);
  return 0;
}
------------------------------------------------------------

which currently gives:

 gmp: r = 0, s = { 0 0 1 }
libc: r = 1, s = { 0 0 1 }

MPFR has various issues concerning %c with the value 0, but an
attempt to fix them fails due to

  length = gmp_vasprintf (...);
[...]
  mpfr_free_str (s);

which is similar to GMP's tests/misc/t-printf.c file, which contains

  got_len = gmp_vasprintf (&got, fmt, ap);
[...]
  (*__gmp_free_func) (got, strlen(got)+1);

But replacing

  mpfr_free_str (s);

by

  mpfr_free_func (s, length + 1);

i.e. using the return value length instead of strlen(s), also fails.
I suppose that this is related to the incorrect return value.

-- 
Vincent Lefèvre <vincent at vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)