0.1387778780781445675529539585113525390625e31

[Paul Zimmermann]

       John,

> With version 4.2.4 (and also older versions) the subject number is
> converted to this string of hex digits:
> 
>  00000011 8427b3b4 a05bc8a8 a4de8459 86800000 00000000 00000000
> 
> The number is then rounded to a 112-bit number (a L floating constant
> in IBM parlance).  The truncation point is between the 6 and 8.  IBM's
> High Level Assembler truncates the last digit to 6 (for reasons it
> does not divulge), while with gmp the last digit becomes 7, because of
> rounding.

this is strange, since the above hex number has 29 hex digits, but only 110
bits since the leading 1 and the trailing 8 need only one bit. Thus if you
convert to a 112-bit number *no rounding* should occur (I assume you consider
floating-point numbers, not fixed-point numbers).

My guess is that what you mean by "112-bit number" is in fact a number with
28 hex digits, which is quite different. In that case 68 can be either rounded
to 60 or to 70 depending on the tie breaking rule. If you follow IEEE 754
which rounds to even it would give 60.

> IBM's HLASM uses an IBM-proprietary infinite precision facility.
> 
> One of the two has to be wrong.  To nail this down, I'd like to follow
> the various stages of the conversion to see what carries run how far.
> Howto?
>
> I tried doing the conversion to a variable that had ten million digits
> precision, but the result was the same (as it should be).
> 
> Thanks,
> 
>    j.

note that if you want to avoid the double rounding (decimal -> hex and
hex -> binary) you can use MPFR (mpfr.org), which guarantees correct
rounding. For example with the following program you get:

$ ./hartmann
1.18427b3b4a05bc8a8a4de8459868 at 25
1.000110000100001001111011001110110100101000000101101111001000101010001010010011011110100001000101100110000110100e100

Paul Zimmermann

#include <stdio.h>
#include "mpfr.h"

int
main()
{
  mpfr_t x;

  mpfr_init2 (x, 112);
  mpfr_set_str (x, "0.1387778780781445675529539585113525390625e31", 10, GMP_RNDN);
  mpfr_out_str (stdout, 16, 29, x, GMP_RNDN); printf ("\n");
  mpfr_out_str (stdout, 2, 112, x, GMP_RNDN); printf ("\n");
  mpfr_clear (x);
  return 0;
}