CODEWITHSUNDEEP
palantir-foundryfoundry-data-connection
Adil B
Adil B

Reputation: 16806

In Foundry, is it possible to extract the original file names from a concatenated import?

I need to import billions of very small image files into Foundry. I understand Foundry won't handle that amount of files very well, so best practice would be to concatenate them and make the import incremental.

However, I need to be able to get the raw files and their names to be able to process them in my pipeline. How can I get the original files back from the concatenated file?

Upvotes: 0

Views: 196

Answers (1)

amy.bananagrams
amy.bananagrams

Reputation: 110

The original file names, sizes, and modified dates are included in the transaction metadata:

enter image description here

Assuming you'll need data across multiple transactions, you likely want to do this as incrementally as possible.

Upvotes: 0

Related Questions