CODEWITHSUNDEEP
pythonmysqlsqldatabase
Rakesh Singh
Rakesh Singh

Reputation: 25

Querying over large data set

I have data in excel(~10,000+ rows) which I need to match each row from a SQL table(~20,000+). The query will be like this

select field1, field2, case 
when condition 1 then 'Output 1'
when condition 2 then 'Output 2'
5-6 more such when conditions
End as Title
from Table

Since the data of both datasets is huge, should I get all the data loaded via python in an array (one time select and then do the operation in python code) or go via sql select (iterating over each row).

Which one is best in performance, and is there any other efficient solution?

Upvotes: 0

Views: 261

Answers (1)

xzelda
xzelda

Reputation: 182

Honestly, 10 000 rows or even 20 000 is pretty small, unless you have like 10000 columns. I suggest you load it in a local dataframe, and do all your transformation there.

Upvotes: 1

Related Questions