CODEWITHSUNDEEP
phppostgresql
gaurav parkash
gaurav parkash

Reputation: 59

I am getting issues when importing data using .csv into PostgreSQL ( greater than 50K lines in csv)

ERROR i'm getting :

This page isn’t working didn’t send any data. ERR_EMPTY_RESPONSE

I am using PHP language for reading the csv file .

My PHP approach is look like for procession the csv data :

$csvAsArray = array_map('str_getcsv', file($tmpName));

I am sure that the above code creating problem afterwords the code is not getting executing . How i can import more that greater than at least 1 million data at a time ? Can anyone help me out , which approach should i choose ?

Upvotes: 2

Views: 117

Answers (2)

Alexey
Alexey

Reputation: 3484

I would suggest using a league/csv package for CSV parsing and importing. @paulsm4 is correct that it's never needed to put the whole file into memory and then work with it, one should rather read line-by-line. But this package is well-maintained and does this all under the hood and quite effectively. And it is much more flexible than COPY postgres command to my mind. One can filter the contents, map callbacks to fields/rows and all this on PHP side.

Upvotes: 0

paulsm4
paulsm4

Reputation: 121609

It looks like you're trying to grab the entire contents of the file in one gulp. Don't do that :) PHP array_map() isn't scalable to 1000's ... or millions of lines.

SUGGESTION:

  1. Read your data into a temp file (as you appear to be doing now).

  2. Do a Postgresql COPY

EXAMPLE:

COPY my_table(my_columns, ...) 
FROM 'my_file.csv' DELIMITER ',' CSV HEADER;

Upvotes: 1

Related Questions