CODEWITHSUNDEEP
azure-data-factory
sarah
sarah

Reputation: 11

How to create YYYY/MM/DD folder dynamically in Copy activity

I was trying to create a pipeline that copy data from S3 to Azure ADLS. And the I tried to create yyyy/mm/dd folder dynamically in Copy activity.

Like this:

@concat(
  pipeline().parameters.target,
  formatDateTime(utcNow(),’yyyy’,’/‘,
  formatDateTime(utcNow(),’MM’,’/‘,
  formatDateTime(utcNow(),’dd’),’/‘,
)

I got an error “the pipeline run failed due to an activity failure. Please check the activity grid below for more detailed information” Copy data for each activity fails.

Upvotes: 0

Views: 639

Answers (3)

Usman Mushtaq
Usman Mushtaq

Reputation: 21

In my case I wanted to create main_folder and sub_folder ending with current date, following expression worked for me:

<main_folder>/<sub_folder_@{formatDateTime(utcnow(), 'yyMMdd')}>

just replace main_folder and sub_folder name in above statement and it should do the trick enter image description here

Upvotes: 0

Aswin
Aswin

Reputation: 7116

  • The expression has errors in it. Closing parenthesis are missing in the formatDateTime function.

Corrected expression:

@concat(
pipeline().parameters.target,
formatDateTime(utcNow(),'yyyy'),'/',
formatDateTime(utcNow(),'MM'),'/',
formatDateTime(utcNow(),'dd'),'/'
)

You can also use the simplified expression as suggested by community member in comments.

@concat(
pipeline().parameters.target,formatDateTime(utcNow(), '/yyyy/MM/dd/')
  • You cannot use the pipeline parameter in the folder path.

Pipeline parameters can only be used within their defining pipeline. Define dataset parameters to accept the pipeline parameter values instead.

enter image description here

  • Instead, you can create a dataset parameter for folder. Give the above expression as the value for dataset parameter.

gif1

  • When expression is given like this in dataset parameter, pipeline is executed without error.

Upvotes: 2

user18665270
user18665270

Reputation: 69

date +'%Y%m%d' : 20230830 --> try this and this will create folder with date(current date) dynamically in S3. Change the format according to you.

Upvotes: 0

Related Questions