CODEWITHSUNDEEP
postgresqlcsv
Petros Apotsos
Petros Apotsos

Reputation: 635

Import to postgreSQL from csv filtering each line

I have the following question (even thorough research couldn't help me):

I want to import data from a (rather large) CSV/TXT file to a postgreSQL DB, but I want to filter each line (before importing it) based on specific criteria.

What command/solution can I use?

On sidenote: If I am not reading from file, but a data stream what is the relevant command/procedure?

Thank you all in advance and sorry if this has been in some answer/doc that I have missed!

Petros

Upvotes: 1

Views: 1358

Answers (1)

Chris Travers
Chris Travers

Reputation: 26464

To explain the staging table approach, which is what I use myself:

  1. Create a table (could be a temporary table) matching your csv structure

  2. Import into that table, doing no filtering

  3. Process and import your data into the real tables using SQL to filter and process.

Now, in PostgreSQL, you could also use the file_fdw to give you direct sql access to csv files. In general the staging table solution will usually be cleaner, but you can do this by essentially letting PostgreSQL treat the file as a table and going through a foreign data wrapper.

Upvotes: 2

Related Questions