CODEWITHSUNDEEP
androiddartflutterdart-html
dheeraj reddy
dheeraj reddy

Reputation: 1197

Http.get loads and parse website partially before getting HTML in Flutter

I am trying to perform web parsing in flutter. I want to grab all episode links and numbers from a certain website https://www2.9anime.to/watch/black-clover-dub.2y44/0wql03

This is my code to parse the html:

    var url = 'https://www2.9anime.to/watch/black-clover-dub.2y44/0wql03';

          http.Response response = await http.get((url));
          dom.Document document = parse(response.body);

List<dom.Element> rapidvideoepisodelinks = document.getElementsByTagName('#servers-container');
List<Map<String, dynamic>> rapidvideoepisodelinkMap = [];
for (var link in rapidvideoepisodelinks) {
        rapidvideoepisodelinkMap.add(
        {
        /////////////////////some logic////////////////////
      });
    }
var rapidvideoepisodejson = json.encode(rapidvideoepisodelinkMap);
rapidvideoepisodelist = (json.decode(rapidvideoepisodejson) as List)
          .map((data) => new Rapidvideoepisodelist.fromJson(data))
          .toList();
      setState(() {
        isLoading = false;
      });

But the thing is, the episodes content area takes a few seconds to load. And the http.get is loading the website too early before this part is even loaded. Because of this, I am unable to parse it completely. This area containing the episode is not even loaded, so its HTML isn't parsed. Everything else seems to be working fine except for the areas like this that take additional time to load.

enter image description here

enter image description here

Is there a way to solve this issue? Like parsing the website after it is completely loaded or something like that.

Any help really appreciated.

Upvotes: 1

Views: 1408

Answers (1)

Sintaloo
Sintaloo

Reputation: 75

Your thinking is not really correct. The reason why you can not parse it is NOT because of partial load. http.get is getting the HTML file. That's all. You are just getting the HTML file and you got it. What you see in your browser is not that HTML file. Your browser first gets HTML file and then find what else it should load from the HTML file and then load JPG files, CSS files, JS scripts etc...

The contents you are trying to parse is manipulated by executing JS script inside the Browser. You can not achieve this with http.get. I am not sure how to achieve what you want in flutter. You may need some kind of pseudo browser in dart if any to load the URL and then parse the resulted html. You will never be able to do it with http.get because you do get the HTML file, but you are actually not looking for that HTML file. I am not sure if you can understand what I mean or not.

Upvotes: 1

Related Questions