CODEWITHSUNDEEP
c#
Daniel Lip
Daniel Lip

Reputation: 11329

Is there any faster way to download content from a website?

This is my code now:

private string downloadContent() 
        {
            try
            {
                WebRequest request = WebRequest.Create(url);
                request.Method = "GET";
                response = request.GetResponse();
                Stream stream = response.GetResponseStream();
                reader = new StreamReader(stream);
                string content = reader.ReadToEnd();
                return content;
            }
            catch
            {
                return error;
            }
        }

This is the site:

http://chatroll.com/testings

I did that when I'm writing something there in the chat so that every n seconds it will show me what I wrote in my program textBox1 and also write it on a text file logger on my hard disk.

The problem is that sometimes if I'm typing something very fast in the chat (for example: hello(enter),Hi(enter),Daniel(enter)) sometimes Hi will not be shown in my program. I don't think the content of what I'm typing is being read fast enough.

Is there any faster way to download the page source and handle it? Maybe the way I'm downloading it is not so fast?

You can see my project here:

https://skydrive.live.com/redir?resid=3B8A7D9F66FF985B!171&authkey=!AFO6EmoF38MtkKQ

Upvotes: 2

Views: 3122

Answers (2)

Adam
Adam

Reputation: 15805

Why not use the higher-level WebClient? I don't know if it's faster, but at least it's less error-prone. You need to pay attention to the using statement to release any resources (sockets and the like).

using (var downloader = new WebClient())
{
    string result = downloader.DownloadString(url);
}

Edit regarding performance: if the web server supports compression such as GZIP, you may want to make use of that:

  1. Set the header: 

    downloader.Headers["Accept-Encoding"] = "gzip";

  2. Use WebClient.DownloadData to load the compressed response into a byte[].

  3. Decompress it using GZipStream

Another edit: your BackgroundWorker.DoWork looks horrible: you have lots of redundant code, heaps of unnecessary loops, etc. I would strongly suggest you open up a question at Code Review and post that method. By the way, you are calling your downloading code twice on every iteration.

Upvotes: 6

L.B
L.B

Reputation: 116098

Just a few thoughts

1- Set request.Proxy to null. This may help in some speedup.

2- In function Conditions use HttpUtility.HtmlDecode instead of string operations

3- Don't use string operations to parse html (like in GetProfileNames or GetTextFromProfile). Use HtmlAgilityPack instead. For example: 

HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(page);
var text = doc.DocumentNode.Descendants("img")
    .Where(x => x.Attributes["class"].Value="????????")
    .Select(x=>x.InnerText)
    .ToArray();

Upvotes: 1

Related Questions