CODEWITHSUNDEEP
c#httpwebrequestscreen-scraping
G Berdal
G Berdal

Reputation: 1164

How to make C# HttpWebRequest wait for query results

I'm trying to scrape a quote engine but HttpWebResponse returns only the "please wait" screen, not the final result.

Is there a way to make the request wait for the response? Status codes won't work because the "please wait" screen returns 200. - I guess it redirects after it retrieved the results from the database but that redirect comes after the initial 200.

AllowAutoRedirect is true but it doesn't make any difference in this case.

Has any of you seen this before?

Upvotes: 0

Views: 1313

Answers (2)

Joel Mueller
Joel Mueller

Reputation: 28764

The site is probably using either Javascript or a meta refresh tag to load the ultimate destination page. Neither of those are directly supported by HttpWebRequest. You've got the source to the "please wait" screen - read it to see exactly how they are redirecting your browser to the end result. There may be a URL you can parse out of the content for the "please wait" screen. Once you have that URL, you can just load it.

Upvotes: 4

Hogan
Hogan

Reputation: 70538

My guess is the site can tell you are scraping and is telling you to bug off.

Upvotes: 1

Related Questions