CODEWITHSUNDEEP
multithreadingperformanceloopscsvcoldfusion-9
M.A
M.A

Reputation: 478

Coldfusion Importing CSV File Million Records causing performance issue..CFThread Works

i'm importing almost one million records everyday, while importing doing filtering then adding in database table. It is causing performance issue some time loop stops on 9999.Is there a way i can do threading in loop?

is it a good idea to start million threads ??

Thanks

Upvotes: 0

Views: 346

Answers (1)

Leigh
Leigh

Reputation: 28873

is it a good idea to start million threads

Absolutely not. Even if you could open that many threads (which is doubtful) it is a terrible idea. Threads are not a cure-all. In this case, using them will only exacerbate the real problem, which is that looping is the wrong tool for this task.

Using a loop to import that much data is incredibly inefficient. Most databases provide tools specifically designed for the job of importing data in bulk. Those tools will perform MUCH, MUCH better than looping. You do not mention which dbms you are using, but some examples are MySQL's LOAD DATA, SQL Server's BULK INSERT command, or as Scott mentioned in the comments, BCP. Import the data into a work table. (Break up the file into smaller chunks if needed, and process batches of say 100K rows at a time.) Once the data is loaded, use SQL to filter/manipulate it in bulk and transfer it to wherever it needs to go.

Upvotes: 1

Related Questions