CODEWITHSUNDEEP
pythonmysql-python
PrivateUser
PrivateUser

Reputation: 4524

Python - How to prevent MySQLdb cursor update interfering with fetchmany?

if __name__ == '__main__':
    def result_generator(cursor, batch_size=10):
        while True:
            results = cursor.fetchmany(batch_size)
            if not results:
                break
            for res in results:
                yield res

    db = MySQLdb.connect(host="localhost", user="root", passwd="root", db="domains")
    # you must create a Cursor object. It will let
    #  you execute all the queries you need
    cursor = db.cursor()
    cursor.execute("SELECT domain FROM com ORDER BY id ASC")
    for result in result_generator(cursor):
        url = "http://www.{0}".format(result[0])
        print url
        w = Wappalyzer(url)
        out = w.analyze()
        cursor.execute("""UPDATE com SET frameworks=%s, is_checked=1 WHERE domain=%s""",
                       (db.escape_string(out.get('frameworks', "")), result[0]))
    # disconnect from server
    db.commit()
    db.close()

My current code runs only for the first 10 rows.

Since i'm using fetchmany function, its supposed to run continuously by selecting the next 10 rows until the end.

But the cursor.execute("""UPDATE .... interfere with the fetchmany cursor.execute("SELECT ... .

Can someone tell me what is the proper way to prevent it?

Upvotes: 0

Views: 191

Answers (1)

Pavel
Pavel

Reputation: 7552

instead of executing the UPDATE queries directly, put them in a string and run them all at once after you're done iterating through the results of the SELECT query

Upvotes: 2

Related Questions