CODEWITHSUNDEEP
google-cloud-data-fusion
Trishit Ghosh
Trishit Ghosh

Reputation: 255

Using parameters for directives of warngler

In a cloud datafusion wrangler I can perform multiple transformation using directives. The scenario that i am looking into is there are a set of directives which might change with each execution.

for e.g.

some-directives . . set-column :concat_col exp:{concat(col1,col2)} hash concat_col SHA-256 true . . some-other-directives

I want to parameterize the part of the directive. I used the below steps

some-directives . . ${custom_directive} . . some-other-directives

This works if i pass only 1 directive as the parameter value i.e. set-column :concat_col exp:{concat(col1,col2)}

However when i try to pass more than 1 directive the pipeline fails.

I have tried the below formats for the parameter value

set-column :concat_col exp:{concat(col1,col2)};hash concat_col SHA-256 true throws an error expected column name but got an identifier

set-column :concat_col exp:{concat(col1,col2)}\r\nhash concat_col SHA-256 true pipeline fails with the error expecting ; but got \r

set-column :concat_col exp:{concat(col1,col2)} \r\nhash concat_col SHA-256 true succeeds but does not perform the hash

"set-column :concat_col exp:{concat(col1,col2)}\r\nhash concat_col SHA-256 true" pipeline succeeds but doesn't perform both the steps

set-column :concat_col exp:{concat(col1,col2)},hash concat_col SHA-256 true pipeline fails with the error expected ; but got ,

Is there a way i can pass multiple lines of directives to a single parameter?

Upvotes: 1

Views: 675

Answers (1)

Trishit Ghosh
Trishit Ghosh

Reputation: 255

Supplying multiple directives to a single parameter in the wrangler works if we send the parameters via HTTP request. each line of directive needs to be separated by a \r\n character.

Strangely it does not not work when we manually enter the values in the runtime args

Upvotes: 0

Related Questions