CODEWITHSUNDEEP
python-2.7twitterbeautifulsoupurllib2
CJ Sullivan
CJ Sullivan

Reputation: 246

Twitter search with urllib2 failing

I am trying to search Twitter for a given search term with the following code:

from bs4 import BeautifulSoup
import urllib2

link = "https://twitter.com/search?q=stackoverflow%20since%3A2014-11-01%20until%3A2015-11-01&src=typd&vertical=default"
page = urllib2.urlopen(link).read()
soup = BeautifulSoup(page)
first = soup.find_all('p')

(Replace "stackoverflow" in link with any search term you want.) However, when I do this (and every time I have tried for the past few days, thinking Twitter might be too bogged down), I get this error:

No results.
Twitter may be over capacity or experiencing a momentary hiccup.

(HTML in results of BS omitted for simplicity in viewing.)

This code used to work for me, but now is not. Additionally, plugging link directly into a browser gives the correct result and Twitter status shows all is well.

Thoughts?

Upvotes: 0

Views: 92

Answers (1)

dstudeba
dstudeba

Reputation: 9038

I was able to reproduce your results. I believe that Twitter is using this message to discourage people from scraping. It makes sense since they have taken the time to publish an API for people to access their data, that they discourage scraping.

My advice is to use their API which is documented here: https://dev.twitter.com/overview/documentation

Upvotes: 3

Related Questions