CODEWITHSUNDEEP
c++jsonqtdoubleqjson
mrg95
mrg95

Reputation: 2418

QJsonDocument::toJson() incorrect double precision

In my project I read from a json file with QJsonDocument::fromJson(). This works great, however when I try to write the QJsonDocument back to file with toJson() some of the doubles have messed up precision.

For example, calling toJson() on a document with a QJsonValue with a double value of 0.15 will save to file as 0.14999999999999999. I do not want this.

This is because the Qt source file qjsonwriter.cpp at line 126 (Qt 5.6.2) reads:

json += QByteArray::number(d, 'g', std::numeric_limits<double>::digits10 + 2); // ::digits10 is 15

That +2 at the end there is messing me up. If this same call to QByteArray::number() instead has a precision of 15 (instead of 17), the result is exactly as I need... 0.15.

I understand how the format of floating point precision causes the double to be limited in what it can represent. But if I limit the precision to 15 instead of 17, this has the effect of matching the input double precision, which I want.

How can I get around this?

Obviously... I could write my own Json parser, but that's last resort. And obviously I could edit the Qt source code, however my software is already deployed with the Qt5Core.dll included in everyone's install directory, and my updater is not designed to update any dll's. So I cannot edit the Qt source code.

Fingers crossed someone has a magic fix for this :)

Upvotes: 1

Views: 2147

Answers (2)

BuvinJ
BuvinJ

Reputation: 11056

I just encountered this as well. Rather than replace an entire Qt JSON implementation with a third party library (or roll my own!), however, I kludged a solution...

My full code base related to this is too extensive and elaborate to post and explain here. But the gist of the solution to this point is simple enough.

First, I use a QVariantMap (or QVariantHash) to collect my data, and then convert that to json via the built-in QJsonObject::fromVariantMap or QJsonDocument::fromVariant functions. To control the serialization, I define a class called DataFormatOptions which has a decimalPrecision member (and sets up easy expansion to other such formatting options..) and then I call a function called toMagicVar to create "magic variants" for my data structure to be converted to json bytes. To control for the number format / precision toMagicVar converts doubles and floats to strings that are in the desired format, and surrounds the string value with some "magic bytes". The way my actual code is written, one can easily do this on any "level" of the map/hash I'm building / formatting via recursive processing, but I've omitted those details...

const QString NO_QUOTE( "__NO_QUOT__" );

QVariant toMagicVar( const QVariant &var, const DataFormatOptions &opt )
{
...
    const QVariant::Type type( var.type() );
    const QMetaType::Type metaType( (QMetaType::Type)type );
... 
    if( opt.decimalPrecision != DataFormatOptions::DEFAULT_PRECISION
        && (type == QVariant::Type::Double || metaType == QMetaType::Float) )
    {
            static const char FORMAT( 'f' );
            static const QRegExp trailingPointAndZeros( "\\.?0+$" );
            QString formatted( QString::number(
                var.toDouble(), FORMAT, opt.decimalPrecision ) );
            formatted.remove( trailingPointAndZeros );
            return QVariant( QString( NO_QUOTE + formatted + NO_QUOTE ) );
    }
... 
}

Note that I trim off any extraneous digits via formatted.remove. If you want the data to always include exactly X digits after the decimal point, you may opt to skip that step. (Or you might want to control that via DataFormatOptions?)

Once I have the json bytes I'm going to send across the network as a QByteArray, I remove the magic bytes so my numbers represented as quoted strings become numbers again in the json.

// This is where any "magic residue" is removed, or otherwise manipulated,
// to produce the desired final json bytes...
void scrubMagicBytes( QByteArray &bytes )
{
    static const QByteArray EMPTY, QUOTE( "\"" ),
        NO_QUOTE_PREFIX( QUOTE + NO_QUOTE.toLocal8Bit() ),
        NO_QUOTE_SUFFIX( NO_QUOTE.toLocal8Bit() + QUOTE );
    bytes.replace( NO_QUOTE_PREFIX, EMPTY );
    bytes.replace( NO_QUOTE_SUFFIX, EMPTY );
}

Upvotes: 0

Matteo Italia
Matteo Italia

Reputation: 126857

this has the effect of matching the input double precision, which I want.

This request doesn't make much sense. A double doesn't carry any information about its precision - it only carries a value. 0.15, 0.1500 and 0.14999999999999999 are the exact same double value, and the JSON writer has no way to know how it was read from the file in first place (if it was read from a file at all).

In general you cannot ask for maximum 15 digits of precision as you propose, as, depending from the particular value, up to 17 are required for a precise double->text->double roundtrip, so you would write incorrectly rounded values. What some JSON writers do however is to write numbers with the minimum number of decimals required to read the same double back. This is far from trivial to do numerically correctly unless you do - as many do - a loop from 15 to 17, write the number with such precision, parse it back and see if it comes back as the exact same double value. While this generates "nicer" (and smaller) output, it's more work and slows down the JSON write, so that's why probably Qt doesn't do this.

Still, you can write your own JSON write code and have this feature, for a simple recursive implementation I expect ~15 lines of code.

That being said, again, if you want to precisely match your input this won't save you - as it's simply impossible.

Upvotes: 1

Related Questions