CODEWITHSUNDEEP
pythonseleniumweb-scraping
Suraj
Suraj

Reputation: 2477

Getting tweet-text from selenium element for twitter

I am trying to scrape the tweets from a trending tag in twitter. I tried to find the xpath of the text in a tweet, but it doesn't work.

browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
url = 'https://twitter.com/search?q=%23'+'Swastika'+'&src=trend_click'
browser.get(url)
time.sleep(1)

The following piece of code doesn't give any results.

browser.find_elements_by_xpath('//*[@id="tweet-text"]')

Other content which I was able to find where :

browser.find_elements_by_css_selector("[data-testid=\"tweet\"]") # works
browser.find_elements_by_xpath("/html/body/div[1]/div/div/div[2]/main/div/div/div/div[1]/div/div[2]/div/div/section/div/div/div/div/div/div/article/div/div/div/div[2]/div[2]/div[1]/div/div") # works

I want to know how I can select the text from the tweet.

Upvotes: 0

Views: 1893

Answers (2)

UVuuMe
UVuuMe

Reputation: 148

Applying for the API is not always successful. I used Twint, which provides a means to scrape quickly. In this case to a CSV output.

def search_twitter(terms, start_date, filename, lang):

   c = twint.Config()
   c.Search = terms

   c.Custom_csv = ["id", "user_id", "username", "tweet"]
   c.Output = filename
   c.Store_csv = True
   c.Lang = lang
   c.Since = start_date

   twint.run.Search(c)

return

Upvotes: 0

Timaayy
Timaayy

Reputation: 856

You can use Selenium to scrape twitter but it would be much easier/faster/efficient to use the twitter API with tweepy. You can sign up for a developer account here: https://developer.twitter.com/en/docs

Once you have signed up get your access keys and use tweepy like so:

import tweepy

# connects to twitter and authenticates your requests
auth = tweepy.OAuthHandler(TWapiKey, TWapiSecretKey)
auth.set_access_token(TWaccessToken, TWaccessTokenSecret)

# wait_on_rate_limit prevents you from requesting too many times and having twitter block you
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)

# loops through every tweet that tweepy.Cursor pulls -- api.search tells cursor 
# what to do, q is the search term, result_type can be recent popular or mixed, 
# and the max_id/since_id are snowflake ids which are twitters way of 
# representing time and finally count is the maximum amount of tweets you can return per request.
for tweet in tweepy.Cursor(api.search, q=YourSearchTerm, result_type='recent', max_id=snowFlakeCurrent, since_id=snowFlakeEnd, count=100).items(500):
        createdTime = tweet.created_at.strftime('%Y-%m-%d %H:%M')
        createdTime = dt.datetime.strptime(createdTime, '%Y-%m-%d %H:%M').replace(tzinfo=pytz.UTC)
        data.append(createdTime)

This code is an example of a script that pulls 500 tweets from YourSearchTerm recent tweets and then appends the time each was created to a list. You can check out the tweepy documentation here: http://docs.tweepy.org/en/latest/

Each tweet that you pull with the tweepy.Cursor() will have many attributes that you can choose and append to a list and or do something else. Even though it is possible to scrape twitter with Selenium it's realllly not recommended as it will be very slow whereas tweepy returns result in mere seconds.

Upvotes: 2

Related Questions