CODEWITHSUNDEEP
utf-8binarydart
darkzangel
darkzangel

Reputation: 949

How to convert binary data to utf8 string after dart2js

I'm storing my data in the localStorage. Because I store a lot of data, I need to compress it. I do it with GZip. Everything works, but I found out that my convertion from List which the result of GZip to Utf8 is extremely slow. On 500k of data, it can take up to 5 min on a fast computer.

The problem appear mainly during the loading function.

Load:

import 'dart:convert';
import 'package:archive/archive.dart';
import 'package:crypto/crypto.dart';

var base64 = CryptoUtils.base64StringToBytes(window.localStorage[storageName]);
var decompress = new GZipDecoder().decodeBytes(base64);
storage = JSON.decode(new Utf8Decoder().convert(decompress));

Save:

var g = JSON.encode(storage);
List<int> bz2 = new GZipEncoder().encode(new Utf8Encoder().convert(g));
window.localStorage[storageName] = CryptoUtils.bytesToBase64(bz2);

Am I doing something wrong? Is there other way of converting a List to a String in Dart? Also, I do both save and load, so there is no need of utf8.

I tried Ascii, but it throws error for unsupported character. I got kind of working latin1, for which a much higher performance, but I still get some unsupported character.

Upvotes: 2

Views: 405

Answers (2)

darkzangel
darkzangel

Reputation: 949

This is just a hack and is not the best, but it works and it's giving me good performance for now:

String convertListIntToUtf8(List<int> codeUnits) {
  var endIndex = codeUnits.length;
  StringBuffer buffer = new StringBuffer();
  int i = 0;
  while (i < endIndex) {
    var c = codeUnits[i];
    if (c < 0x80) {
      buffer.write(new String.fromCharCode(c));
      i += 1;
    } else if (c < 0xE0) {
      buffer.write(new String.fromCharCode((c << 6) + codeUnits[i+1] - 0x3080));
      i += 2;
    } else if (c < 0xF0) {
      buffer.write(new String.fromCharCode((c << 12) + (codeUnits[i+1] << 6) + codeUnits[i+2] - 0xE2080));
      i += 3;
    } else {
      buffer.write(new String.fromCharCode((c << 18) + (codeUnits[i+1] << 12) + (codeUnits[i+2] << 6) + codeUnits[i+3] - 0x3C82080));
      i += 4;
    }
    if (i >= endIndex) break;
  }
  return buffer.toString();
}

Upvotes: 0

Pixel Elephant
Pixel Elephant

Reputation: 21383

If UTF-8 is really the bottleneck, and you don't need it, you can just use Dart's native String encoding which is UTF-16.

var compressed = new GZipEncoder().encode(str.codeUnits);
var decompressed = new GZipDecoder().decodeBytes(compressed);
var finalStr = new String.fromCharCodes(decompressed);

Upvotes: 4

Related Questions