Columns gets removed while loading an excel file to cosmos dB in ADF using data flow

Question

I am new to ADF and cosmos db. I am trying to load an excel which has 34 columns from a blob to a cosmos db container using a mapping data flow . While debugging the data flow I could preview all the columns in the source and its data available in both source as well as sink. But after running the pipeline(using trigger/debug) when I check the items/documents in the cosmos db container certain columns are missing. When I do the same data load using copy activity all the columns are getting inserted and this seems to be weird. I need to change the data types of few columns before loading into the database and that's the reason I am trying to load the data using data flow activity. I have been stuck at this for so long and wasn't able to proceed further. Any help would be appreciated.

Answer 1

I had repro’d and was able to load all (34) columns into cosmos db successfully using data flow.

1. Source projection

2. Source data preview: Here C1 column is of string type.

3. Changed the datatype of the C1 column from string to an integer.

4. Sink transformation: Make sure you have included all source columns in sink mappings.

5. Cosmosdb Output: It includes all 34 columns from the source and C1 as integer type.