CODEWITHSUNDEEP
ajaxhtml-parsingnokogiri
Mohit Jain
Mohit Jain

Reputation: 43939

How do I fetch AJAX-loaded content from an another site using Nokogiri?

I was trying to parse some HTML content from a site. Nokogiri works perfectly for content loaded the first time.

Now the issue is how to fetch that content which is loaded using AJAX. For example, there is a "see more" link and more items are fetched using AJAX, or consider a case for AJAX-based tabs.

How can I fetch that content?

Upvotes: 1

Views: 1671

Answers (2)

the Tin Man
the Tin Man

Reputation: 160551

It isn't completely clear what you want to do, but if you are trying to get access to additional HTML that is loaded by AJAX, then you will need to study the code, figure out what URL is being used for the AJAX request, whether any session IDs or cookies have been set, then create a new URL that reproduces what AJAX is using. Request that, and you should get the new content back.

That can be difficult to do though. As @Nuby said, Mechanize could be good help, as it is designed to manage cookies and sessions for you in the background. Mechanize uses Nokogiri internally so if you request a page from Mechanize, you can use Nokogiri searches against it to drill down and extract any particular JavaScript strings. They'll be present as text, so then you can use regex or substring matches to get at the particular parameters you need, then construct the new URL and ask Mechanize to get it.

Upvotes: 2

d11wtq
d11wtq

Reputation: 35308

You won't be able to parse anything that requires a JavaScript runtime to produce that content using Nokogiri. Nokogiri is a HTML/XML parser, not a web browser.

PhantomJS on the other hand is a web browser, albeit a special kind of browser ;) Take a look at that and have a play.

Upvotes: 3

Related Questions