CODEWITHSUNDEEP
pythontwitterweb-scrapingweb-crawlerscheduler
koko
koko

Reputation: 177

Twitter Scraping Repeated Execution Code(python)

This is a Twitter scraping code that extracts tweets which contain famous keywords.

I want to repeat the entire code below every 12 hours. (Or 12 hours + 10 minutes breaks). Can you give me advice on repeating phrases?

import tweepy
import time
import os
import json
import simplejson

search_term = 'word1'
search_term2= 'word2'
search_term3='word3'

lat = "xxxx"
lon = "xxxx"
radius = "xxxx"
location = "%s,%s,%s" % (lat, lon, radius)

API_key = "xxxx"
API_secret = "xxxx"
Access_token = "xxxx"
Access_token_secret = "xxxx"

auth = tweepy.OAuthHandler(API_key, API_secret)
auth.set_access_token(Access_token, Access_token_secret)
api = tweepy.API(auth)

c=tweepy.Cursor(api.search,
                 q="{}+OR+{}".format(search_term, search_term2, search_term3),
                rpp=1000,
                geocode=location,
                include_entities=True)

data = {}
i = 1
for tweet in c.items():
    data['text'] = tweet.text
    print(i, ":", data)
    i += 1
    time.sleep(1)

wfile = open(os.getcwd()+"/workk2.txt", mode='w')   
data = {}   
i = 0       

for tweet in c.items():
    data['text'] = tweet.text   
    wfile.write(data['text']+'\n')  
    i += 1

wfile.close()

Upvotes: 4

Views: 258

Answers (1)

hatef
hatef

Reputation: 6199

You could set a Cron job that executes your script every 12 hours. To do so you should save your script with .py extension and make it executable. Then add it to your crontab:

0 0 0/12 * * ? /usr/bin/python yourscript.py

For more detail have a look at this question. Alternatively there are packages in python (e.g. APScheduler) that help you achieve this. In APScheduler you can define a job like this:

from apscheduler.schedulers.blocking import BlockingScheduler

sched = BlockingScheduler()

@sched.scheduled_job('interval', hours=12)
def timed_job():
    print('This job is run every 12 hours.')

sched.configure(options_from_ini_file)
sched.start()

Upvotes: 1

Related Questions