CODEWITHSUNDEEP
pythonweb-scrapingdryscrape
Sayhan
Sayhan

Reputation: 165

Dryscrape response always returns with "Product unavailable" when using proxy

I was trying to scrap javascript based website using dryscrape and was getting the right content when not using proxy. But it returns "Product unavailable" when using proxy. Here is the code snippet.

session = dryscrape.Session()
session.set_proxy('host=<ipaddress>', port=80, user='<username>', password='<password>')
session.visit(url)

Please let me know how we can solve it.

Upvotes: -1

Views: 135

Answers (1)

Sayhan
Sayhan

Reputation: 165

Found the issue. Actually the problem was not with the proxy. Everything was fine, the problem was with headers. The problem can be fixed by setting request headers(User-Agent, Accept-Language etc.).

set_header(key, value)

This method is available in Dryscrape session object.

Upvotes: 0

Related Questions