CODEWITHSUNDEEP
pythonjsonweb-crawlerpython-requests
qanknku
qanknku

Reputation: 147

python crawl with requests to get json

When I do the crawl, I usually utilize scripts before parsing with python. Since this allows to get JSON which can be easily structured and parsed. 

 >>> import requests
 >>> r = requests.get('~.json')
 >>> r.json()

However, encountering this page, https://www.eiganetflix.jp/%E3%82%BF%E3%82%A4%E3%83%97/tv-%E3%82%B7%E3%83%AA%E3%83%BC%E3%82%BA

It seems there's no interaction to call JSON to show materials on the page. And it is hard to find pagination javascript functions. (Actually, there is, but I mean it seems hard to execute. )

In this case, how can I utilize existing requests and json method? Or is there any easy way to crawl this?

Upvotes: 0

Views: 481

Answers (1)

tayfun
tayfun

Reputation: 3135

If I understand correctly, you want to scrape a webpage which does not have a JSON response. Check to be sure that the website does not have an API that allows you to get JSON data. Or even any other structured data such as XML would also be helpful. If there is no way, you would have to screen scrape, which is not the easiest method to do. Check scrapy which is a framework for doing this, or you can use a library like beautifulsoup for a custom solution.

If the page uses Javascript, you would somehow need to run it on the page to get content and browse pages. You can spynner or Selenium to do that.

Upvotes: 1

Related Questions