CODEWITHSUNDEEP
seleniumfacebookweb-scrapingbeautifulsouprate-limiting
PSK
PSK

Reputation: 357

Why does Facebook posts scraping become extremely slow or stop after few hundred posts are scraped?

I was trying to scrape all the community posts from 2016 onwards on a Facebook brand community page like this using Selenium and BeautifulSoup. I would like to use this data for a research project and I intend on deidentifying the data and deleting any identifiable data subsequently.

After scraping between 300-500 posts, the rest of the posts load extremely slowly or don't even load sometimes. This is probably because of the data rate limits set by Facebook for individual users.

I would like to know how I can overcome this issue. When I reach such a limit, can I wait for x minutes before which I can continue again? Is there some parameter I can set to avoid hitting a rate limit? Any suggestions would be very helpful, thanks!

Upvotes: 1

Views: 1022

Answers (3)

Kushagra A. Nalwaya
Kushagra A. Nalwaya

Reputation: 75

So I tried a paid VPN service: NordVPN, which solved the issue. You can choose from multiple VPNs like Express or Nord or whichever you find best fit.

You can refer to this link: https://stackoverflow.com/a/77028694/10495410 for detailed process.

Upvotes: 0

user17581701
user17581701

Reputation: 11

i m having the same problem, after doing multiple scrolling the feeds are stopped to be shown and and while verifying network trafic (requests and response ) i find that the error was "rate limit reacheched". The big problem is that facebook graph api documentation does not show any entry about the limit of request that can be done by the user.

Upvotes: 1

Kh4lid MD
Kh4lid MD

Reputation: 135

This problem is absolutely from your PC/Laptop specifications, I was developing an Instagram bot that scrolls on an famous person's followers to get their usernames to interact later with them, but then I realized that sometimes it is very slow, and sometimes not to get usernames, like most of times it gets 1 username every 3 seconds, but sometimes 5 usernames a second, then I got that if I switch my battery power mode (Battery saver, Better battery, Better performance, Best performance) selenium goes slower and faster, I do not know if there is something similar in MacOS but try it if there is something like that, of course your PC or Laptop would not continue for ever and will start being slow at some place.

Upvotes: 0

Related Questions