CODEWITHSUNDEEP
azure-data-factory
databash
databash

Reputation: 646

ADF Mapping Data Flow - Sink transform dynamic Number of partitions

I have following expression to calculate the "Number of partitions" in Sink transform as dynamic content,

toInteger (round( iif(toDecimal('5671478512', 38, 2) <= 104857600, toDecimal(1.00) , toDecimal('5671478512', 38, 2)/104857600) ) )

This expression must result in integer 54, but for some reason it throws error when debugging in ADF portal.

enter image description here

When I try the exact expression in a derived column transformation, I get expected value of 54.

enter image description here

Any ideas why it is failing in "Number of partitions"? but works when tested in derived column

Following is the error I get when I add the expression in "Number of partitions" dynamic content

collectPreviewData failure on job=e97f7e77-abae-41f2-95dd-7d2d0e03800b, jobState=Failed com.microsoft.dataflow.Issues: DF-SYS-01 - requirement failed: Number of partitions (0) must be positive. - Nonejava.lang.IllegalArgumentException: requirement failed: Number of partitions (0) must be positive.
    at scala.Predef$.require(Predef.scala:224)
    at org.apache.spark.sql.catalyst.plans.logical.RepartitionByExpression.<init>(basicLogicalOperators.scala:1123)
    at com.microsoft.dataflow.TransformPlanner$$anonfun$physicalPartitionPlan$1.apply(Transformer.scala:299)
    at com.microsoft.dataflow.TransformPlanner$$anonfun$physicalPartitionPlan$1.apply(Transformer.scala:283)
    at scala.collection.immutable.Stream.map(Stream.scala:418)
    at com.microsoft.dataflow.TransformPlanner$class.physicalPartitionPlan(Transformer.scala:283)
    at com.microsoft.dataflow.transformers.ExternalCodeGenerator.physicalPartitionPlan(External.scala:126)
    at com.microsoft.dataflow.FlowRunner$$anonfun$16.apply(FlowRunner.scala:237)
    at com.microsoft.dataflow.FlowRunner$$anonfun$16.apply(FlowRunner.scala:216)
    at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
    at scala.collection.TraversableOnce$class.collectFirst(TraversableOnce.scala:145)
    at scala.collection.SeqViewLike$AbstractTransformed.collectFirst(SeqViewLike.scala:37)
    at com.microsoft.dataflow.FlowRunner$.com$microsoft$dataflow$FlowRunner$$runner(FlowRunner.scala:309)
    at com.microsoft.dataflow.FlowRunner$$anonfun$runner$2.apply(FlowRunner.scala:178)
    at com.microsoft.dataflow.FlowRunner$$anonfun$runner$2.apply(FlowRunner.scala:173)
    at scala.util.Success.flatMap(Try.scala:231)
    at com.microsoft.dataflow.FlowRunner$.runner(FlowRunner.scala:173)
    at com.microsoft.dataflow.DataflowExecutor$$anonfun$6$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6$$anonfun$apply$9$$anonfun$apply$10$$anonfun$apply$11$$anonfun$7.apply(DataflowExecutor.scala:119)
    at com.microsoft.dataflow.DataflowExecutor$$anonfun$6$$anonfun$apply$3$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6$$anonfun$apply$9$$anonfun$apply$10$$anonfun$apply$11$$anonfun$7.apply(DataflowExecutor.scala:106)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$flowCode$1.apply(DataflowJobFuture.scala:66)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$flowCode$1.apply(DataflowJobFuture.scala:66)
    at scala.Option.map(Option.scala:146)
    at com.microsoft.dataflow.DataflowJobFuture.flowCode$lzycompute(DataflowJobFuture.scala:66)
    at com.microsoft.dataflow.DataflowJobFuture.flowCode(DataflowJobFuture.scala:66)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$start$1.apply$mcV$sp(DataflowJobFuture.scala:290)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$start$1.apply(DataflowJobFuture.scala:287)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$start$1.apply(DataflowJobFuture.scala:287)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$start$1.apply$mcV$sp(DataflowJobFuture.scala:315)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$start$1.apply(DataflowJobFuture.scala:287)
    at com.microsoft.dataflow.DataflowJobFuture$$anonfun$start$1.apply(DataflowJobFuture.scala:287)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Upvotes: 1

Views: 866

Answers (1)

CHEEKATLAPRADEEP
CHEEKATLAPRADEEP

Reputation: 12798

Sharing the answer as per the comment by @Mark Kromer. Unfortunately, “We're not evaluating the dynamic content in the Number of Partitions property”.and this is confirmed as bug and product team is actively working on the fix.

Upvotes: 1

Related Questions