CODEWITHSUNDEEP
web-scrapingyql
aateeque
aateeque

Reputation: 2231

HTML scraping using YQL

I am trying to use YQL to scrape some websites. When I test various queries in the YQL console I get an results node. So for example when I run:

select * from html where url="http://www.reverbnation.com/" and xpath='/html/body'

I get an empty <results /> node (permalink). Thanks in advance!

Upvotes: 0

Views: 2159

Answers (1)

opyate
opyate

Reputation: 5428

http://www.reverbnation.com may be blocking the request coming from Yahoo! based on certain criteria, like headers. I had a look at reverbnation's robots.txt, and they aren't blocking Yahoo! based on the "Yahoo Pipes 2.0" user agent, so it must be something else.

To re-create the issue, make a YQL query against your own site, then look at the full access logs to see the full request and all headers that came from Yahoo! Then make a similar request using a tool like cURL.

You can also try and run netcat on a port and query with http://yoursite.com:PORT to see the full request.

Related issue discussed here.

Upvotes: 1

Related Questions