CODEWITHSUNDEEP
web-scraping
Duc Tran
Duc Tran

Reputation: 6294

Best approach for fetching news from websites?

I have a function which web-scraping all latest news from a website (approximately 10 news and the number of news is up to that website). Note that the news are in chronical order.

For example, yesterday I got 10 news and stored in database. Today I get 10 news but there are 3 news that are not available from yesterday (7 news stayed the same, 3 new).

My current approach is to extract each news till I find an old news (the 1st among 7 news) then I stop extracting and only update the field "lastUpdateDate" of the old news + add new news to the database. I think this approach is somehow complicated and it takes time.

Actually I'm getting news from 20 websites with same content structure (Moodle) so each request will last about 2 minutes, which my free host doesn't support.

Is it better if I delete all the news and then extracting everything from the start (this actually increments a huge amount of the ID numbers in the database)?

Upvotes: 1

Views: 1949

Answers (2)

Jitendra Gaur
Jitendra Gaur

Reputation: 778

Its depend on requirement if you want to show old news to the users or not.

For scraping you can create a custom local script for cron job which will grab the data from those news websites and will store into database.

You can also check through subject if its already exist of not.

Final make a custom news block which will show all the database feed.

Upvotes: 0

Andy Lester
Andy Lester

Reputation: 93666

First, check to see if the website has a published API. If it has one, use it.

Second, check the website's terms of service, which may specifically and explicitly disallow scraping the website.

Third, look at a module in your programming language of choice that handles both the fetching of the pages and the extraction of the content from the pages. In Perl, you would start with WWW::Mechanize or Web::Scraper.

Whatever you do, don't fall into the trap that so many who post to StackOverflow fall into: Fetching the web page, and then trying to parse the content themselves, most often with regular expressions which is an inadequate tool for the job. Surf the SO tag for tales of sorrow from those who have tried to roll their own HTML parsing systems instead of using existing tools.

Upvotes: 2

Related Questions