CODEWITHSUNDEEP
c#
MSHAN
MSHAN

Reputation: 11

How to improve CSV reading performance

I am reading a CSV file and saving the data from the CSV file to my database.

I'm using Streamreader's ReadLine() to read every line and then insert it into my database, which is working fine. But after profiling my code I noticed that ReadLine() is taking up too much time.

How can I improve the performance of my task.

Please provide me with alternative options.

Performance is the Main concern here.

Upvotes: 1

Views: 918

Answers (3)

Barry Kaye
Barry Kaye

Reputation: 7759

You can use the SqlBulkCopy class for this MSDN link. It is an order of magnitude faster than individual line-by-line inserts. The MSDN page has a complete example.

You can also speed up your reads with the StreamReader.Peek method - MSDN link. Again the MSDN link has a good example.

Upvotes: 2

Oded
Oded

Reputation: 499312

I suggest using one of the many CSV parsing libraries - you should test them to see how they perform.

FileHelpers is one popular library, and codeproject has several different ones too.

Chances are, however, that the issue is with inserting to the database row by row.

You should read as much of the file as you can and bulk insert to the database, instead of row by row.

Upvotes: 0

Jan
Jan

Reputation: 16038

Are you sure that StreamReader.ReadLine is the slow part here? I guess inserting data into a database is much slower than reading one line of text out of a local file!

You might try to read the file into memory in one step with System.IO.File.ReadAllLines() if its not too big.

Upvotes: 0

Related Questions