CODEWITHSUNDEEP
c#
Cheung Brian
Cheung Brian

Reputation: 755

C# is slow in downloading website using StreamReader

I have a desktop app for downloading the content of website.

Originally this is written in Python with the following code:

import requests
from bs4 import BeautifulSoup

def getSoup(url):
    try:
        r = requests.get(url)
    except requests.exceptions.ConnectionError:
        print "wrong"
    soup = BeautifulSoup(r.text, "html.parser")
    return soup

Then I change to using C#, the code is as follows:

public string getHtml(string theUrl) {
    try {
        string result = null;
        WebRequest req = HttpWebRequest.Create(theUrl);
        req.Method = "GET";
        req.Proxy = null;
        using(StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream(), Encoding.UTF8) {
            result = reader.ReadToEnd();
        }
        return result;
    } catch (Exception e) {
        //handle exception
    }
}

But the C# code is much slower than the Python code. Often, C# code just doesn't have any response. Can you suggest ways to improve the C# code to make it faster?

Thank you.

Upvotes: 0

Views: 196

Answers (2)

MSD88
MSD88

Reputation: 31

Try Restsharp

public string GetHtml(string theUrl)
    {
        var client = new RestClient(theUrl);
        var request = new RestRequest(Method.GET);
        request.AddHeader("postman-token", "c9d3ea79-0e8e-b377-a6c7-0042b1f82d51");
        request.AddHeader("cache-control", "no-cache");
        request.AddHeader("content-type", "application/x-www-form-urlencoded");
        request.AddParameter("application/x-www-form-urlencoded", "tokenDate=10%2F08%2F2017&enrollId=446", ParameterType.RequestBody);
        IRestResponse response = client.Execute(request);
        return response.Content;
    }

Upvotes: 0

Martin
Martin

Reputation: 11

1) You could use Wireshark to verify if you actually have tcp communication. 2) You should verify if it catches an exception. 3) Use HttpWebRequest rather than WebRequest

Whatvworks for me is:

        try
        {

            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            using (StreamWriter streamOut = new StreamWriter(request.GetRequestStream()))
            {
                streamOut.Write(yourMessage);
            }
            using (StreamReader streamIn = new StreamReader(request.GetResponse().GetResponseStream()))
            {
                Console.WriteLine(streamIn.ReadToEnd());
            }
        }
        catch (SystemException se)
        {
            Console.WriteLine(se.Message);
        }

Upvotes: 2

Related Questions