CODEWITHSUNDEEP
duckdb
sojim2
sojim2

Reputation: 1317

How to update multiple parquet files with DuckDB

I have a folder with subfolders with parquet files, here's the structure:

/mainfolder/*.parquet
/mainfolder/subfolder/*.parquet

I'd like to change FirstName field in all parquet files from "John" to "Alex" How can I use DuckDB to achieve this?

Also, is there a way I can change all fields with values "e" to "f"? For example, if FirstName & LastName field both have "test", it would be changed to "tfst". I'd imagine we'd have to explicit with FieldNames, but just in case it has this capability.

Upvotes: 2

Views: 1337

Answers (1)

Andy Sawyer
Andy Sawyer

Reputation: 31

When dealing with duckdb, you will either be reading/writing to duckdb tables in a database, or reading/writing to files that duckdb imports the data from. You’re interested in the latter.

Unfortunately duckdb is not a data lakehouse format such as Iceberg, Hudi or Delta Lake. You can’t get it to update details within a file (at time of writing). You would need to save the full file back with the updates made. In database terms, you’re going to need to do a truncate and load.

Upvotes: 3

Related Questions