Tips for Scraping data from sites with latest technologies

Question

I am trying to scrape data from asp, jsp sites which uses ajax and session and POST variables for data navigation and display. I have gone through various articles and SO for data scraping but not helped much I parsed some sites with modifying headers but most of sites shows header redirection for my custom header. What is the proper way of parsing data from site which are javascript enabled and sites in asp having viewstate variable.

Answer 1

if you want to learn web scrapers i recomment you read that book

Webbots, Spiders, and Screen Scrapers: A Guide to Developing Internet Agents with PHP/CURL

really this is the only book dedicated for web scrapers it's written for php developer but i think the basic that book teaches will help any developer to understand how web bots work

I also emailed the author about a some questions and he got back to me in few minutes really i highly recommend reading that book for any one wants to learn about web scraping