CODEWITHSUNDEEP
etlcdap
PARTHIPAN S
PARTHIPAN S

Reputation: 1

How can I pass dynamic runtime data (such as msisdn, name, age) from one stage to a MySQL Execute plugin in CDAP?

I am working on a pipeline in CDAP where I need to dynamically construct and execute SQL queries using data from a previous pipeline stage. The issue I am facing is that I cannot pass dynamic input from earlier stages into the MySQL Execute plugin to perform an update operation in MySQL.

Current Setup:

Plugins Used:

JavaScript Transform: To dynamically create the SQL query using fields from the input data (e.g., msisdn, name, age, etc.).

MySQL Execute: To execute the dynamically generated SQL queries, but I am unable to pass dynamic data (like the fields name, age, msisdn) from the JavaScript transform plugin to the MySQL Execute plugin.

The Problem:

The MySQL Execute plugin works when I manually pass values or use predefined inputs, but it does not accept dynamic input from the JavaScript transform plugin (like input.name, input.msisdn, etc.).

I need to update a table in MySQL with a query like:

UPDATE ${table_name} SET name = '${name}', age = ${age} WHERE msisdn = ${msisdn};

Where ${name}, ${age}, and ${msisdn} should come from the output of the JavaScript Transform plugin.

What I’ve Tried:

I’ve tried constructing the dynamic query using JavaScript, but the MySQL Execute plugin does not accept the transformed data from the previous plugin, and thus the query doesn’t execute as expected.

What I Need Help With:

How can I pass dynamic data (like msisdn, name, age) from the JavaScript Transform plugin to the MySQL Execute plugin in CDAP?

Are there any configuration settings or workarounds to achieve this dynamic query execution from CDAP?

Are there alternative approaches to dynamically execute SQL queries with parameters from the input data?

Any insights or guidance would be greatly appreciated!

Upvotes: 0

Views: 28

Answers (0)

Related Questions