CODEWITHSUNDEEP
sqlpython-3.xpandasvisual-studio-codedata-science
merchmallow
merchmallow

Reputation: 794

Best approach to work with .csv files with 4 gigas

I am on data science. I have a .csv file with 5kk records and 3.9gigas of size. Whats the best pratice to deal with it? I normally use vscode or jupyter and even when i set max-memory to 10gigas the operations like load etc are taking too much time to complete.

What do you recommend to improve my work?

notebook lenovo S145 20gigas ram i7-8565U - Ubuntu

Thanks

Upvotes: 1

Views: 33

Answers (1)

alexherm
alexherm

Reputation: 1362

If you want to bring a CSV into a database for reporting, one fairly quick and easy option is to use an external table. It uses syntax similar to SQLLDR in the create table definition. Once established, the latest saved CSV data will immediately be available as a table in the database.

Upvotes: 1

Related Questions