CODEWITHSUNDEEP
c#web-crawler
Zbone
Zbone

Reputation: 507

Web crawler time out

I am working on a simple web crawler to get a URL, Crawl first level links on the site and extract mails from all pages using RegEx...

I know it's kinda sloppy and it's just the beginning, but i always get "operation Timed Out" after 2 minutes of running the script..

 private void button1_Click(object sender, System.EventArgs e)
    {

        string url = textBox1.Text;
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();
        StreamReader sr = new StreamReader(response.GetResponseStream());
        string code = sr.ReadToEnd();
        string re = "href=\"(.*?)\"";
        MatchCollection href = Regex.Matches(code, @re, RegexOptions.Singleline);
        foreach (Match h in href)
        {

            string link = h.Groups[1].Value;
            if (!link.Contains("http://"))
            {
                HttpWebRequest request2 = (HttpWebRequest)WebRequest.Create(url + link);
                HttpWebResponse response2 = (HttpWebResponse)request2.GetResponse();
                StreamReader sr2 = new StreamReader(response.GetResponseStream());
                string innerlink = sr.ReadToEnd();


                MatchCollection m2 = Regex.Matches(code, @"([\w-]+(\.[\w-]+)*@([a-z0-9-]+(\.[a-z0-9-]+)*?\.[a-z]{2,6}|(\d{1,3}\.){3}\d{1,3})(:\d{4})?)", RegexOptions.Singleline);


                foreach (Match m in m2)
                {
                    string email = m.Groups[1].Value;

                    if (!listBox1.Items.Contains(email))
                    {
                        listBox1.Items.Add(email);
                    }
                }
            }
        }

         sr.Close();
        }

Upvotes: 2

Views: 908

Answers (2)

dknaack
dknaack

Reputation: 60516

Don't parse Html using Regex. Use the Html Agility Pack for that.

What is exactly the Html Agility Pack (HAP)?

This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don't HAVE to understand XPATH nor XSLT to use it, don't worry...). It is a .NET code library that allows you to parse "out of the web" HTML files. The parser is very tolerant with "real world" malformed HTML. The object model is very similar to what proposes System.Xml, but for HTML documents (or streams).

More Information

Upvotes: 4

Matt Cashatt
Matt Cashatt

Reputation: 24218

The comment by Oded is correct, we need to know what you need help with specifically; however, I can at least point you to the HtmlAgility Pack as it will solve most of your web scraping woes.

Good Luck!

Matt

Upvotes: 1

Related Questions