CODEWITHSUNDEEP
c#httpwebrequest
user1455112
user1455112

Reputation: 183

Going from web browser to web request

I am working on a program that automates tasks in a browser like entering text, clicking, etc and right now everything is working fine when using the Web Browser tool in Visual Studio 2010.

What I'd like to know is how should I approach converting all of this so I can use send requests instead of the browser? I heard its a lot more efficient and a lot better if you are going to be using multi threading but I have so much code that already works now and am not sure how I should do this without scraping quite a bit of it.

Upvotes: 0

Views: 249

Answers (1)

Dai
Dai

Reputation: 155035

You'd use HttpWebRequest / HttpWebResponse to craft HTTP requests to servers, then retrieve the document body using HttpWebResponse.GetResponseStream(). Then use a HTML framework (such as HtmlAgilityPack) to parse the HTML and get a DOM graph of the document, then you can traverse the DOM to extract data and craft request bodies for any consequent requests back to the server.

If the response document is valid XML (e.g. XHTML1.1) then you can save some time by loading it into System.Xml.XmlDocument, but in practice HTML documents frequently contain errors and frameworks like HtmlAgilityPack are more forgiving of parser errors than System.Xml.

With respect to multi-threading, you can fire-off multiple HttpWebRequests at the same each (each request is associated with its own thread, or use the Async methods). Note you will need to fiddle with some settings to override the "two concurrent HTTP requests per host" policy present in .NET networking libraries.

Upvotes: 1

Related Questions