Python Twitter Tool simultaneous search and stream

Question

I'm trying to make a Python server that I can call from other applications to request Twitter data. I usually work with Python as a scripting language, so if there are any red flags anyone sees in my code, I'm all ears!

This is basically what I have so far, which works well when I ping the server, it gets 10 tweets from my timeline and sends them back to my other applications. My main issue is that I'd like to combine streaming and searching. That way I can have the stream open for a specific hash tag that I'd like to have sent to my other applications in real-time, but then I'd periodically search for other things that don't need to be coming down to me in real-time.

I've had success using both separately, but not sure where to start if I wanted to implement both, which in this case I'd like to bring the stream functionality into this.

I'm using Python Twitter Tools 1.10.2 - http://mike.verdone.ca/twitter/ and Python 3.3

Code below, thanks!

EDIT:I was able to get a step further by adding the twitter streaming connection after the if data == "SEARCH_NOW" if statement. But this brings up the original issue I was having. Once the twitter stream is open, the code seems to just wait there. If i put it before timeline lookup, then I can never call the timeline lookup. Updated code to reflect.

EDIT 2: Putting the search request inside of the twitter stream loop gets a little closer. I can now have the stream open and every time I get a tweet that matches the search term, then I can also do a request. But still not independently...

File: network_settings.py

#!/usr/bin/env python

#network settings
import socket

#set server variables
TCP_IP = '127.0.0.1'
TCP_PORT = 7001
BUFFER_SIZE  = 20

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)

conn, addr = s.accept()

#print connection address when someone connects
print ('Connection address:', addr)

File: twitter_settings.py

from twitter import *
import re

OAUTH_TOKEN = ''
OAUTH_SECRET = ''
CONSUMER_KEY = ''
CONSUMER_SECRET = ''

t = Twitter(auth=OAuth(OAUTH_TOKEN, OAUTH_SECRET, CONSUMER_KEY, CONSUMER_SECRET))

auth = OAuth(OAUTH_TOKEN, OAUTH_SECRET, CONSUMER_KEY, CONSUMER_SECRET)
stream = TwitterStream(auth = auth, secure = True)

File: python_server.py

#python server 

import json 

from network_settings import *
from twitter_settings import *

search_term = 'test'

while 1:

    tweet_iter = stream.statuses.filter(track = search_term)

    for tweet in tweet_iter:
        # check whether this is a valid tweet
        if tweet.get('text'):

            userName = tweet["user"]["screen_name"]
            userTweet = tweet["text"]

            # now print our tweet
            print ('user: ', userName)
            print ('tweet: ', userTweet)

            #send data back
            delivery1 = json.dumps({'type':'showdown','userName':userName,'userTweet':userTweet})
            conn.send(delivery1.encode('utf-8'))
            data = conn.recv(BUFFER_SIZE)
            data = data.decode('utf-8')

            if data == "SEARCH_NOW": 
                 print ('request newest IDS tweets')

                 x = t.statuses.home_timeline(count=10)

                for i in range(10):
                    try:
                        #print(x[i])
                        userName = x[i]['entities']['user_mentions'][0]['screen_name']
                        userTweet = x[i]['text']
                        print('username: ', userName)
                        print('tweet: ', userTweet)
                        delivery = json.dumps({'type':'display','userName':userName,'userTweet':userTweet})
                        conn.send(delivery.encode('utf-8'))
                    except:
                        print('not valid tweet')

conn.close()

Answer 1

So finally have figured out a solution for this. I ended up using threading to run the stream in it's own thread, then I open another thread every time I do a search. Not sure if I need to close each thread, or if the return takes care of that. If anyone has any thing they thing could be improved, I'm all ears!

Code below:

#!/usr/bin/env python
#python server 

import json 
import threading
import time
import socket
from twitter import *
import re

#get thread lock ready
thread_lock = threading.Lock()

#set server variables
TCP_IP = '127.0.0.1'
TCP_PORT = 7001
BUFFER_SIZE  = 20


s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)

conn, addr = s.accept()

#print connection address when someone connects
print ('Connection address:', addr)

#fill these in your app!
#twitter auth keys
OAUTH_TOKEN = ''
OAUTH_SECRET = ''
CONSUMER_KEY = ''
CONSUMER_SECRET = ''

t = Twitter(auth=OAuth(OAUTH_TOKEN, OAUTH_SECRET, CONSUMER_KEY, CONSUMER_SECRET))

auth = OAuth(OAUTH_TOKEN, OAUTH_SECRET, CONSUMER_KEY, CONSUMER_SECRET)
stream = TwitterStream(auth = auth, secure = True)


#twitter functions
def pythonSearch():
    #lock thread to not interrupt search results
    thread_lock.acquire()

    print ('request newest tweets')

    #get 10 things from timeline
    x = t.statuses.home_timeline(count=10)

    for i in range(10):
        try:
            #get username and tweet
            userName = x[i]['entities']['user_mentions'][0]['screen_name']
            userTweet = x[i]['text']

            #print out values
            print('username: ', userName)
            print('tweet: ', userTweet)

            #send json back
            delivery = json.dumps({'type':'display','userName':userName,'userTweet':userTweet})
            conn.send(delivery.encode('utf-8'))
        except:
            #not a retweet
            print('not valid tweet')

    #unlock thread when finished
    thread_lock.release()

    return


def pythonStream():
    #open stream looking for search_term
    search_term = 'TESTING'
    tweet_iter = stream.statuses.filter(track = search_term)

    for tweet in tweet_iter:
        # check whether this is a valid tweet
        if tweet.get('text'):

            #get username and tweet
            userName = tweet["user"]["screen_name"]
            userTweet = tweet["text"]

            # now print our tweet
            print ('user: ', userName)
            print ('tweet: ', userTweet)

            #send json back
            delivery1 = json.dumps({'type':'showdown','userName':userName,'userTweet':userTweet})
            conn.send(delivery1.encode('utf-8'))


#start main loop
while 1:
    #listen for calls
    data = conn.recv(BUFFER_SIZE)
    data = data.decode('utf-8')

    #if someone calls search, do a search
    if data == 'SEARCH':
        threading.Thread(target = pythonSearch).start()


    if data == 'STREAM':
        threading.Thread(target = pythonStream).start()


conn.close()