How to automate the process in c#.net for crawler?

Question

I am designing a website in ASP.NET and C# in which i am doing following task.

There is a table List_of_Site which contains list of sites to be crawled. And another table Site1_Links which contains crawled data such as links, contents etc.

I have coded a program in c# asp.net in which i have a form Home.aspx in visual studio and there is a button named Crawl. Whenever i click on the button it fetches the sites from table List_of_Site one by one and then crawl them for new links available and the save them respectively at Site1_Links table.

But to do this i need to click on the button each time but i want something that can automate the process for some time interval say X minutes.

Please suggest a way ?

Answer 1

You don't want an ASP.NET web application for this. Web applications are passive request/response systems. They lay in wait for a request, respond to it, and go back to waiting. Not suited for scheduled background tasks.

In general, you have two simple options:

1. A console application (run by a scheduler)
2. a Windows Service

A console application is historically easier to write and debug. And I believe Windows still comes with a bundled task scheduler which can run an executable on a set schedule. I usually prefer this approach. However, there are trade-offs. I don't believe it will run unless someone is logged in to the machine, for example.

A Windows Service is generally better suited for the task, but usually a little more complicated to write and maintain for an average user. However, it has the benefits that it doesn't clutter the UI (no black console window when it's running) and doesn't need someone logged in to the workstation to run.

Answer 2

Try hooking a Timer up to your button instead. You'll probably want something to make it stop as well. Here is a good tutorial on using said timer.

http://msdn.microsoft.com/en-us/library/bb386404.aspx