CODEWITHSUNDEEP
pythonperformancepython-2.7python-requests
vKi
vKi

Reputation: 3

Any possible way to speed up the processing of this?

I have a list of 80 usernames right now and I have my script check if each username exists or not. However it takes a little longer than I like so I was wondering if there is anything I can do to speed up how long it takes to check if each username exists or not.

# ------------------------------
# Mass Kik Username Checker
# Script Made by: Ski
# ------------------------------

import requests, threading

def check(username):
    try:
        req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code

        if req == 302:
            return False
        if req == 200:
            return True
    except Exception as e:
        print e
        exit()


def _loadList(filename):
    item_list = []
    for item in str(open(filename, "r").read()).split("\n"):
        item_list.append(item)
    return item_list

def _thread(items):
    global _usernames
    for username in _usernames[items[0]:items[1]]:
        exists = check(username)

        if exists:
            print username+" exists\n"
        if not exists:
            print username+" doesn't exist\n"

if __name__ == '__main__':
    _usernames = _loadList("usernames.txt")

    thread1 = threading.Thread(target=_thread, args=([0, 20], )).start()
    thread2 = threading.Thread(target=_thread, args=([20, 40], )).start()
    thread3 = threading.Thread(target=_thread, args=([40, 60], )).start()
    thread4 = threading.Thread(target=_thread, args=([60, 80], )).start()

Upvotes: 0

Views: 119

Answers (1)

noxdafox
noxdafox

Reputation: 15040

Try out Python 3.x Pool of threads. You can define how many workers will perform the request. Using more (ex. 32) than 4, would speed-up your code dramatically.

import requests
from concurrent.futures import ThreadPoolExecutor


NUM_OF_WORKERS=32


def check(username):
    try:
        req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code

        if req == 302:
            print(username, " does not exist.")
        if req == 200:
            print(username, "exists.")
    except Exception as error:
        print(error)


usernames = _loadList(filename)

with ThreadPoolExecutor(max_workers=NUM_OF_WORKERS) as pool:
    pool.map(check, usernames)

This makes your code way more readable as well.

EDIT: noticed now the Python 2.7 tag.

Python 2 has a Pool of threads which is available under multiprocessing module. Unfortunately it's not documented as no tests were made available.

import requests
from multiprocessing.pool import ThreadPool


NUM_OF_WORKERS=32


def check(username):
    try:
       req = requests.get("http://kik.me/"+username, allow_redirects=False).status_code

       if req == 302:
           print(username, " does not exist.")
       if req == 200:
           print(username, "exists.")
    except Exception as error:
        print(error)


usernames = _loadList(filename)


pool = ThreadPool(processes=NUM_OF_WORKERS)
pool.map_async(check, usernames)
pool.close()
pool.join()

If you want a better Pool of Threads for Python 2, you can try the Pebble module

Upvotes: 1

Related Questions