CODEWITHSUNDEEP
vb.nethttpwebrequestweb-scraping
John Cliven
John Cliven

Reputation: 1231

Reading lines of GetResponseStream

I'm using the following to GET web pages:

Dim request As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
request.CookieContainer = logincookie
Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
Dim reader As New StreamReader(response.GetResponseStream())
Dim thesource As String = reader.ReadToEnd

I'm trying to extract data from the source. thesource holds the source code of the page successfully, however, I can't loop through each line using a For Each loop, as it merely responds with each character instead of each line.

Can somebody suggest how to fix this? Thanks guys, Stan.

Upvotes: 0

Views: 6105

Answers (1)

Joel Coehoorn
Joel Coehoorn

Reputation: 415880

I see three options.

Option 1. Read line by line in a While loop instead of For Each. Use this if everything was going to be in the same function anyway.

Dim request As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
request.CookieContainer = logincookie
Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
Dim reader As New StreamReader(response.GetResponseStream())
Dim line As String = reader.ReadLine()

While line IsNot Nothing

    'Contents of your For Each loop go here

    line = reader.ReadLine()
End While

Option 2: You have at least Visual Studio 2012 and want to return the result from this function so you can use the For Each loop in the code that called this code.

Public Iterator Function GetUrl(ByVal url As String) As IEnumerable(Of String)

    Dim request As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
    request.CookieContainer = logincookie
    Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
    Dim reader As New StreamReader(response.GetResponseStream())
    Dim line As String = reader.ReadLine()

    While line IsNot Nothing

        Yield line
        line = reader.ReadLine()
    End While
End Function

Call it like this:

For Each line As String In GetUrl("http://example.com")
    '...
Next line

Option 3: You want to return from a function, like with the previous option, but you can't use the new iterators language feature.

Public Sub GetUrl(ByVal url As String, ByVal lineAction As Action(Of String))

    Dim request As HttpWebRequest = DirectCast(WebRequest.Create(url), HttpWebRequest)
    request.CookieContainer = logincookie
    Dim response As HttpWebResponse = DirectCast(request.GetResponse(), HttpWebResponse)
    Dim reader As New StreamReader(response.GetResponseStream())
    Dim line As String = reader.ReadLine()

    While line IsNot Nothing
        lineAction(line)
        line = reader.ReadLine()
    End While
End Sub

Call it like this:

GetUrl("http://example.com", _
   Sub(l)
       'Do stuff here
   End Sub)

Upvotes: 1

Related Questions