CODEWITHSUNDEEP
azure-blob-storageazure-cosmosdbazure-data-factory
Matthew Campbell
Matthew Campbell

Reputation: 1884

Azure Data Factory from Cosmos documents to Blob files

Want to use Azure Data Factory to pull Cosmos documents and copy each document to file (blob) in Storage where the file name == document id with file suffix == json. E.g. document content { id: "0001", name: "gary" }would be a blob named 0001.json with the same content { id: "0001", name: "gary" }.

Hava a Cosmos dataset targeting a Cosmos collection of JSON documents. Plus a Blob Storage dataset with an absolute file path (container + file name). Using a Copy Data activity in a pipeline the setup is vanilla with source == Cosmos dataset (no schema) and sink == Blob dataset (no schema) creating a single blob line-delimited per Cosmos document.

Guessing that a Lookup to a ForEach with a Copy Data activity would be right. Lookup against the Cosmos dataset should yield items (collection of JSON documents). ForEach settings ought to be Sequential and something like items == @activity('Find').output.value.id where Find is the name of the Lookup step. The items expression is probably way off (help here). Choosing an activity for the ForEach is the big question. Can Copy Data be bent to act on each item of the ForEach (only datasets available for the source + sink)?

Upvotes: 0

Views: 378

Answers (1)

Utkarsh Pal
Utkarsh Pal

Reputation: 4554

Can Copy Data be bent to act on each item of the ForEach (only datasets available for the source + sink)?

Yes, Copy activity can be use in ForEach activity, you just need to keep the Sink Dataset filename as dynamic. You can create a parameter and assign its's value equal to @activity('Find').output.value.id so that it will get new filename every time.

There is a same case scenario explained in third-party tutorial by sqlkover.com. You can refer this for better understanding.

Upvotes: 0

Related Questions