CODEWITHSUNDEEP
qtencodingcharacter-encodingdisqusqstring
air-dex
air-dex

Reputation: 4180

Encoding issues with Disqus API results (2 Unicode characters which should be interpreted as one)

I have got a Qt application which receive strings in JSON objects from the Disqus API:

{ "title": "Swiftkey pr\u0102\u0160dit votre choix d\u2019emoji gr\u0102\u02d8ce au clavier Swiftmoji" }

(there's more but I only write what matters here)

Then I put the title string in a QString:

// Assuming that "reply" is the QNetworkReply * containing the Disqus API response.
QByteArray disqusReply = reply->readAll();
// disqusReply == "{ \"title\": \"Swiftkey pr\u0102\u0160dit votre choix d\u2019emoji\ gr\u0102\u02d8ce au clavier Swiftmoji" }"
QJsonDocument doc = QJsonDocument::fromJson(disqusReply);
QJsonObject obj = doc.object();
QString title = obj["title"].toString();

Later I write it in a QML Text. It should display "Swiftkey prédit votre choix d'emoji grâce au clavier Swiftmoji" but it displays "Swiftkey prĂŠdit votre choix d'emoji grâce au clavier Swiftmoji" instead.

As you can see there are some encoding issues, with two successive Unicode characters which should be interpreted as one character only ("ĂŠ" instead of "é" and "â" instead of "à" here). How can I do (with Qt or QML) to display the right characters? Which encoding conversions I have to perform (with Qt or QML) in order to solve those encoding issues?

Additional informations: the bug occurs on Windows 10 64-bit.

EDIT : you can find the bug here: https://disqus.com/api/3.0/threads/list.json?since=2016-05-18T14%3A08%3A27%2B00%3A00&forum=frandroid&api_key=7o0xSBOEzN2AG6yxcJgeJbeEbACBfGhgnoIRHu7umbifKAvXQpisYKT3KSXF9nPN

Upvotes: 3

Views: 206

Answers (3)

user122933
user122933

Reputation:

QJsonDocument requires UTF-8 encoded strings. Is your document encoded in UTF-8 when you load it?

Upvotes: 0

Shtol Krakov
Shtol Krakov

Reputation: 1280

Well, I think the problem in double encoding or something else from server side, not from client's. Because your code must be pr\u00e9dit instead of pr\u0102\u0160dit. If you'll use clean_title instead of title of your json-answer you'll get the right string cause it's encode is correct.

UPD:

As I said in comment - there are two replies with same news. Id 4836688567 has wrong encode string. And 4836587900 correct. There are many double variants of news with different encoding.

First:

{
"feed":"https://frandroid.disqus.com/httpwwwfrandroidcomandroidapplications358721_swiftkey_predire_choix_demoji_grace_clavier_swiftmoji/latest.rss",
"identifiers":[],
"dislikes":0,
"likes":0,
"message":"",
"id":"4836688567",
"createdAt":"2016-05-18T09:08:43",
"category":"448171",
"author":"3938134",
"userScore":0,
"isSpam":false,
"signedLink":"http://disq.us/?url=http%3A%2F%2Fwww.frandroid.com%2Fandroid%2Fapplications%2F358721_swiftkey-predire-choix-demoji-grace-clavier-swiftmoji&key=nqCbe6jgfwM-skLyqTf3lg",
"isDeleted":false,
"raw_message":"",
"isClosed":false,
"link":"http://www.frandroid.com/android/applications/358721_swiftkey-predire-choix-demoji-grace-clavier-swiftmoji",
"slug":"httpwwwfrandroidcomandroidapplications358721_swiftkey_predire_choix_demoji_grace_clavier_swiftmoji",
"forum":"frandroid",
"clean_title":"Swiftkey pr\u0102\u0160dit votre choix d\u2019emoji gr\u0102\u02d8ce au clavier Swiftmoji",
"posts":0,
"userSubscription":false,
"title":"Swiftkey pr\u0102\u0160dit votre choix d\u2019emoji gr\u0102\u02d8ce au clavier Swiftmoji",
"highlightedPost":null
}

Second:

{
"feed":"https://frandroid.disqus.com/swiftkey_predit_votre_choix_d8217emoji_grace_au_clavier_swiftmoji/latest.rss",
"identifiers":["358721 http://www.frandroid.com/?p=358721"],
"dislikes":0,
"likes":1,
"message":"",
"id":"4836587900",
"createdAt":"2016-05-18T08:16:30",
"category":"448171",
"author":"3938134",
"userScore":0,
"isSpam":false,
"signedLink":"http://disq.us/?url=http%3A%2F%2Fwww.frandroid.com%2Fandroid%2Fapplications%2Fgoogle-apps%2F358721_swiftkey-predire-choix-demoji-grace-clavier-swiftmoji&key=UU8IrLN_UDXEggF6wHjAYg",
"isDeleted":false,
"raw_message":"",
"isClosed":false,
"link":"http://www.frandroid.com/android/applications/google-apps/358721_swiftkey-predire-choix-demoji-grace-clavier-swiftmoji",
"slug":"swiftkey_predit_votre_choix_d8217emoji_grace_au_clavier_swiftmoji",
"forum":"frandroid",
"clean_title":"Swiftkey pr\u00e9dit votre choix d\u2019emoji gr\u00e2ce au clavier Swiftmoji",
"posts":13,
"userSubscription":false,
"title":"Swiftkey pr\u00e9dit votre choix d’emoji gr\u00e2ce au clavier Swiftmoji",
"highlightedPost":null
}

As you see, difference in url from where news comes. But why some of them encoded wrong - this is a question.

UPD 2:

Or maybe it's RSS bug. Let's take prédit word. In second variant RSS returns xml with content which has been already encoded to é and feed link looks like normal. In first variant RSS returns ĂŠ and its feed link looks like anormal - feel like it takes whole url and do another encode.

Upvotes: 1

Aconcagua
Aconcagua

Reputation: 25526

Have you tried

QByteArray disqusReply
    = QString::fromUtf8(reply->readAll().data()).toLocal8Bit();

? You might prefer delaying this conversion until actually writing the string into the QML text.

Upvotes: 0

Related Questions