CODEWITHSUNDEEP
c#
NikRED
NikRED

Reputation: 1195

accessing multiple web pages at a time using c#

my project need to read multiple web pages at a time(eg: for a particular keyword google will display 100 pages result maximum) I tried to read that 100 pages using 'for' loop in URL.But it is showing http request exception in c#.How can i read that pages within sort period of time???

Upvotes: 2

Views: 521

Answers (4)

Poma
Poma

Reputation: 8484

Here is simple solution. But doing so many requests in parallel is very bad, so you have to do some mechanism to reduce number off simultaneous threads.

void QueueWorkItems()
{
    for (int i = 0; i < 10; i++)
        new Action<string>(Request).BeginInvoke("http://www.google.com/search?q=test&start=" + (i * 10), null, null);
}

void Request(string url)
{
    //Process your request
}

Upvotes: 0

Lucero
Lucero

Reputation: 60236

My guess is that you either try to access a disposed object or you don't dispose it, thus leaving too much connections open.

Post some code for more help.

Upvotes: 0

Henk Holterman
Henk Holterman

Reputation: 273464

To do it in parallel, push the code onto the ThreadPool. It will not run 100 threads at once (but you don't want that).

Upvotes: 1

Pablo Castilla
Pablo Castilla

Reputation: 2741

this is really nice

http://blogs.msdn.com/somasegar/archive/2009/11/18/reactive-extensions-for-net-rx.aspx http://www.minddriven.de/?p=550

But maybe you can not use RX .

Upvotes: 0

Related Questions