CODEWITHSUNDEEP
javahtmlweb-crawler
user3757805
user3757805

Reputation: 31

Get HTML code from dynamic page

I am trying to get the HTML code of the following website. http://fortune.com/fortune500/list/

But the problem is when we visit this website in browser, it only shows the first 20 companies and when we go to the bottom part of website it loads the next 50 companies.

How do i get the first 700 companies in HTML code from this website? I tried the code from this website https://www.mkyong.com/java/how-to-get-url-content-in-java/ to get the HTML content but as expected it gives only the top 20 companies

Any help is much appreciated Thanks

Upvotes: 0

Views: 148

Answers (2)

Julien Nioche
Julien Nioche

Reputation: 4864

You should use Selenium for this. Here is a tutorial on how to use it with StormCrawler. You could also use it directly if you wanted to.

Upvotes: 0

Angelo C
Angelo C

Reputation: 66

CURL: http://fortune.com/api/v2/list/2013055/expand/item/ranking/asc/{{start_from}}/{{num_limit}}

Example: http://fortune.com/api/v2/list/2013055/expand/item/ranking/asc/1/100

The site "fortune.com" return max 100 elements form CURL.

The CURL return a JSON.

Upvotes: 1

Related Questions