CODEWITHSUNDEEP
rbigdataregressionlogistic-regressionread.table
TongZZZ
TongZZZ

Reputation: 766

Reading big data and logistic regression in R

Situation: 1GB CSV file, 100000 rows, 4000 independent numeric variable, 1 dependent variable. R on Windows Citrix Server, with 16GB memory.

Problem: It took me 2 hours! to do:

read.table("full_data.csv", header=T, sep",")

and the glm process crashes, the program is not responding, and I have to shut it down in Task Manager.

Upvotes: 3

Views: 1174

Answers (1)

Ryogi
Ryogi

Reputation: 5617

I often resort to the package sqldf to load large .csv in memory. A good pointer is here.

Upvotes: 3

Related Questions