In my article “Fifteen Digits Don’t Round-Trip Through SQLite Reals” I showed examples of decimal floating-point numbers — 15 significant digits or less — that don’t round-trip through double-precision binary floating-point variables stored in SQLite. The round-trip failures occur because SQLite’s floating-point to decimal conversion routine uses limited-precision floating-point arithmetic.
My quick and dirty floating-point to decimal conversion routine, which I wrote to demonstrate conversion inaccuracies caused by limited-precision, also fails to round-trip some decimal numbers of 15 digits or less. Since I hadn’t demonstrated this failure previously, I will do so here.
Incorrect Quick and Dirty 15-Digit Conversions
To find examples of incorrect 15-digit round-trips, I modified my quick and dirty floating-point to decimal conversion routine to make it a function, accepting as a parameter the floating-point number to be converted. I called this function on randomly generated decimal inputs, each of which were converted correctly beforehand to double-precision floating-point by David Gay’s strtod() function (this ruled out incorrect decimal to floating-point conversion as a cause for a bad round-trip). Using additional code, I normalized the output of my conversion routine and rounded it to 15 significant decimal digits. I compared this output to the input decimal string, seeing if the significant digits matched; here are three examples that didn’t match:
| Input | Floating-Point Approximation | Q&D to 15 Digits | ULPs in error |
|---|---|---|---|
| 3.409452297963e-288 | 0x1.09cf3929ffbc3p-955 | 3.40945229796301e-288 | 1 |
| 9.08344e+307 | 0x1.02b4782a6c378p+1023 | 9.08343999999998e+307 | 2 |
| 9.88819e+221 | 0x1.5e258a3929ee5p+737 | 9.88819000000003e+221 | 3 |
These examples show a one, two, and three decimal unit-in-the-last-place (ULP) error (three ULPs was the maximum error I found).
I traced each conversion as in my previous article, comparing — step by step — the quick and dirty conversion with a GMP arbitrary-precision floating-point implementation of the same algorithm. The traces are too long to list here, but here are the highlights:
- 0x1.09cf3929ffbc3p-955 (from 3.409452297963e-288)
The GMP conversion takes 1007 iterations, and requires 721 bits of precision (at steps 288 and 289). The quick and dirty conversion takes 338 iterations, and is hobbled by its limit of 53 bits of precision.
The unrounded GMP conversion, shown to 18 digits, is 3.40945229796299974…e-288; the unrounded quick and dirty conversion, shown to 18 digits, is 3.40945229796300619…e-288. This shows why the GMP conversion rounds correctly to 15 digits, and why the quick and dirty conversion does not.
- 0x1.02b4782a6c378p+1023 (from 9.08344e+307)
The GMP conversion takes 308 iterations, and requires 1018 bits of precision (at step 1). The quick and dirty conversion takes 308 iterations as well, although its limited precision means the generated digits are inaccurate after a point.
The unrounded GMP conversion, shown to 18 digits, is 9.08344000000000015…e+307; the unrounded quick and dirty conversion, shown to 18 digits, is 9.08343999999998406…e+307.
- 0x1.5e258a3929ee5p+737 (from 9.88819e+221)
The GMP conversion takes 222 iterations, and requires 735 bits of precision (at step 1). The quick and dirty conversion takes 222 iterations as well, although its limited precision means the generated digits are inaccurate after a point.
The unrounded GMP conversion, shown to 18 digits, is 9.88818999999999996…e+221; the unrounded quick and dirty conversion, shown to 18 digits, is 9.88819000000002828…e+221.
Discussion
My previous article showed that quick and dirty floating-point to decimal conversion has two flaws: it sometimes fails to generate the full decimal equivalent of a binary floating-point number, and in some of those cases, it fails to round correctly to 17 digits. An incorrectly rounded 17-digit decimal number can result in floating-point numbers that don’t round-trip.
In this article, I showed a third flaw in quick and dirty floating-point to decimal conversion: it can fail to round correctly to 15 digits, meaning that some decimal numbers won’t round-trip.
