CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-data-fusioncdap
code tutorial
code tutorial

Reputation: 664

Unable to upload datafusion plugin json file using CDAP RESTAPI

I am trying to upload my custom plugin to datafusion using CDAP RESTAPI reference. I followed the steps as per documentation but I still didn't find the way to add the plugin JSON file using REST API.

curl -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" "${CDAP_ENDPOINT}/api/v3/namespaces/vega_demo/artifacts/example" -H "Artifact-Extends: system:cdap-data-pipeline[6.0.0,10.0.0-SNAPSHOT)/system:cdap-data-streams[6.0.0,10.0.0-SNAPSHOT)" --data-binary @/path/to/example-1.0.0-SNAPSHOT.jar @/path/to/example-1.0.0-SNAPSHOT.json

Artifact added successfullycurl: (6) Could not resolve host:

Plugin is loaded but the config json file is not loaded causing errors in plugin

Upvotes: 1

Views: 1054

Answers (1)

ebeltran
ebeltran

Reputation: 469

Per the command used, I suggest to verify if you are setting correctly the endpoint.

export INSTANCE_ID=your-instance-id
export CDAP_ENDPOINT=$(gcloud beta data-fusion instances describe \
--location=us-central1 \
--format="value(apiEndpoint)" \
${INSTANCE_ID})

Per the official CDAP documentation, it seems that the endpoint should not include the part api before v3.

Also, if your instance belongs to Basic edition, the namespace is default; otherwise, when using Enterpise edition you can create the namespace.

When using the curl method, it seems you need to add the config information within the headers due this method doesn't include the json load

On the other hand, if you are having issues to use curl, I would suggest to use the UI.

Taking this example to upload the plugin mysql-connector-java-5.1.35.jar to Data fusion with curl, the configuration file should be like:

{
  "parents": [ "system:cdap-data-pipeline[6.1.1,6.1.1]", "system:cdap-data-streams[6.1.1,6.1.1]" ],
  "plugins": [
    {
      "name": "mysql",
      "type": "jdbc",
      "className": "com.mysql.jdbc.Driver"
    }
  ]
}

due using curl you can only upload the jar file, to include the information from the configuration file, you should use the HTTP Headers to include this information as this:

curl -X POST -H "Authorization: Bearer $(gcloud auth print-access-token)" \
"${CDAP_ENDPOINT}/v3/namespaces/default/artifacts/example" \
-H 'Artifact-Plugins: [ { "name": "mysql", "type": "jdbc", "className": "com.mysql.jdbc.Driver" } ]' \
-H "Artifact-Version: 5.1.35" \
-H "Artifact-Extends: system:cdap-data-pipeline[6.1.1, 6.1.1]/system:cdap-data-streams[6.1.1, 6.1.1]" \
--data-binary @mysql-connector-java-5.1.35.jar

Upvotes: 1

Related Questions