CODEWITHSUNDEEP
pythonperformancepython-2.7sqlalchemy
user60039
user60039

Reputation: 111

Append SQLAlchemy results to python list faster

I have results (~60000 rows) from some db query that I need to add to a list.

The code looks like this :

hosts = connection.execute(some_query)
for host in hosts:
    mylist.append(host['display_name'])

However, iterating over all the items and appending them to the list is very slow. How can I make it faster ? I've looked on https://wiki.python.org/moin/PythonSpeed/PerformanceTips#loops but this doesn't seem to apply to my case.

Upvotes: 0

Views: 805

Answers (2)

user60039
user60039

Reputation: 111

I finally found out that most of the time was spent in the fetchone function of oursql (the MySQL connector I use with SQLAlchemy) which seems to be called for each iteration of the for loop.

Switching to the MySQL-Python (mysqldb) connector made the script run in ~3s instead of ~4m.

Upvotes: 1

van
van

Reputation: 76962

Use List comprehesions instead of the foor loop.

mylist = [host['display_name'] for host in hosts]

But also check if the performance is not in executing the query itself.

Upvotes: 1

Related Questions