CODEWITHSUNDEEP
pythonweb-scrapingscrapy
Luis Ramon Ramirez Rodriguez
Luis Ramon Ramirez Rodriguez

Reputation: 10383

Deal with XMLHttpRequest in scrapy

I'm scraping the table of this kind of pages:

http://www.trulia.com/school-district/CA-San_Francisco_County/San_Francisco_Unified/

There is a table with several pages whose change is through an XMLHttpRequest, inspecting the page I can get te url for this request

http://www.trulia.com/q_schools_in_district.php?did=600116051&grade=elementary&page=2&sortby=testRating&sortdir=desc

But I can't scrape this url with scrapy

When I try:

scrapy shell http://www.trulia.com/q_schools_in_district.php?did=600116051&grade=elementary&page=2&sortby=testRating&sortdir=desc

I don't get response, how can this data be scraped?

Upvotes: 2

Views: 289

Answers (1)

Nelewout
Nelewout

Reputation: 6554

[originally posted by eLRuLL in the comments]

Surround your url with "", because the second url contains & which is also recognized by the shell to run another process.

Upvotes: 1

Related Questions