CODEWITHSUNDEEP
mysqlcsvbigdata
Moustafa Alzantot
Moustafa Alzantot

Reputation: 371

Import Very Huge CSV file into MySQL

I am having a huge file of comma separated values (CSV) that I need to import into a mysql database..

The file size is 7 GB, and the memory size on my machine is 4.00 GB..

I tried to load it using "LOAD DATA LOCAL INFILE" SQL Command, but it has been more than 12 hours and it is still running !

The ideas that I think might should be done is to either :

  1. Create a program that reads the file line by line and insert them into the database row by row.
  2. Break the CSV file into a number of other CSV files with smaller size (may be 200 Mb) and load each of them into the database using the "LOAD DATA LOCAL INFILE"

Do you think there might be a better and more efficient approach to import such huge file into the MySQL ?

Upvotes: 2

Views: 5043

Answers (1)

andrew cooke
andrew cooke

Reputation: 46892

[So you can mark the Q as closed - and I get some points!]

Dropping the index should help.

(Although I am surprised it helped so much. I guess MySQL couldn't keep the whole index in memory, which seems odd, as it should be smaller than the data.)

Upvotes: 1

Related Questions