CODEWITHSUNDEEP
jsondartflutter
Doc
Doc

Reputation: 11661

Optimal parsing of large JSON in background

I am parsing large JSON data similar to one like this.

There are around 3000+ chapter objects here. I require only those chapter objects with "lang_code":"gb", which will be around 1300 with some basic objects like title, description. So basically 55% of JSON is not for my use.

I am generating the classes for JSON parsing using https://app.quicktype.io/ which gives me correct classes but this method is too slow.

Any suggestion to speed it up.

Upvotes: 3

Views: 2021

Answers (1)

Richard Heap
Richard Heap

Reputation: 51750

If most of the information that you need is sparse, it's probably best to pick it out in a targeted fashion, rather than create objects for everything.

You can't get around decoding the whole json string, which takes about 60ms on my laptop. Pruning the non-gb chapters takes just a few ms, and mapping what's left to some usable objects takes another few ms. Total time to something usable: <70ms.

import 'dart:convert';
import 'dart:io';

main() {
  String manga = new File('manga.json').readAsStringSync();
  int t1 = DateTime.now().millisecondsSinceEpoch;

  Map<String, dynamic> data = json.decode(manga);
  Map<String, dynamic> jChapters = data['chapter'];
  jChapters.removeWhere((_, m) => m['lang_code'] != 'gb');

  Map<String, Chapter> chapters = jChapters.map((_, m) {
    String number = m['chapter'];
    return MapEntry(number, Chapter(number, m['title']));
  });

  int t2 = DateTime.now().millisecondsSinceEpoch;
  print(t2 - t1);

  print(chapters);
}

class Chapter {
  String number;
  String title;

  Chapter(this.number, this.title);

  @override
  String toString() => 'Chapter #$number:$title';
}

Upvotes: 3

Related Questions