CODEWITHSUNDEEP
c#web-crawlerweb-scraping
Umer
Umer

Reputation: 1921

crawling / scraping a search form based webpages

I want to crawl/scrape a webpage which has a form to be precise following is the URL

http://lafayetteassessor.com/propertysearch.cfm

The problem is, i want to make a search and save the result in a webpage.

  1. my search string will always give a unique page, so result count won't be a problem.
  2. the search over there doesn't search on URL (e.g. google searching url contains parameters to search). How can i search from starting page (as above) and get the result page ?

please give me some idea. I am using C#/.NET.

Upvotes: 0

Views: 654

Answers (1)

Mike Caron
Mike Caron

Reputation: 14561

If you look at the forms on that page, you will notice that they use the POST method, rather than the GET method. As I'm sure you know, GET forms pass their parameters as part of the URL, eg mypage?arg1=value&arg2=value

However, for POST requests, you need to pass the parameters as the request body. It takes the same format, it's just passed in differently. To do this, use code similar to this:

HttpRequest myRequest = (HttpRequest)WebRequest.Create(theURL);
myRequest.Method = "post";

using(TextWriter body = new StreamWriter(myRequest.GetRequestStream())) {
    body.Write("arg1=value1&arg2=value2");
}

WebResponse theResponse = myRequest.GetResponse();

//do stuff with the response

Don't forget that you still need to escape the arguments, etc.

Upvotes: 1

Related Questions