CODEWITHSUNDEEP
facebookscreen-scrapingbotsweb-crawler
Dan Armstrong
Dan Armstrong

Reputation: 1

Facebook Scrapes Beso.com so hard as to heavily impact site performance; how do we get FB to throttle its crawl?

Our site, Beso.com, which has 3MM UV a month, has started being hit by Facebook's crawler/scraper multiple times a day, to the point where it is causing a severe performance degradation on our site. We have recently implemented the Open Graph, and are encouraging a fair amount of sharing and liking on our site, along with Facebook Connect.

I understand that FB needs access to our site to scrape info, but we desperately need them to throttle down the rate of crawling, or we will be forced to block it entirely. This isn't a solution we wish to go with.

Would there be anyone from FB on this site who could connect us with an appropriate team member, or anyone from an established site who has battled this issue before?

Thanks.

Upvotes: 0

Views: 257

Answers (1)

Gil Birman
Gil Birman

Reputation: 35900

Facebook scrapes open graph objects once per day. I can think of two possible reasons a site would be over-scraped...

1) You have an app that implements open graph actions. The programmer who created that app accidentally set the scrape optional parameter permanentaly to true. See here: https://developers.facebook.com/docs/opengraph/actions/

2) A single page--what should be one open graph object--has multiple URL's such that Facebook is re-scraping the same material over and over.

Upvotes: 1

Related Questions