CODEWITHSUNDEEP
apache-sparkkubernetesairflow
Carlos Liarte
Carlos Liarte

Reputation: 3

Unable to create SparkApplications on Kubernetes cluster using SparkKubernetesOperator from Airflow DAG (Airflow version 2.0.2 MWAA)

I try to use SparkKubernetesOperator to run spark job into Kubernetes with the same DAG and yaml files as the following question:

Unable to create SparkApplications on Kubernetes cluster using SparkKubernetesOperator from Airflow DAG

But airflow shows the following error:

HTTP response headers: HTTPHeaderDict({'Audit-Id': 'e2e1833d-a1a6-40d4-9d05-104a32897deb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 10 Sep 2021 08:38:33 GMT', 'Content-Length': '462'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the object provided is unrecognized (must be of type SparkApplication): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string \"json:\\\"apiVersion,omitempty\\\"\"; Kind string \"json:\\\"kind,omitempty\\\"\" } (222f7573722f6c6f63616c2f616972666c6f772f646167732f636f6e6669 ...)","reason":"BadRequest","code":400}

Any suggestion to resolve that problem???

Upvotes: 0

Views: 680

Answers (1)

Kim Ted
Kim Ted

Reputation: 26

think u had the same problem like me

SparkKubernetesOperator(
  task_id='spark_pi_submit',
  namespace="default",
  application_file=open("/opt/airflow/dags/repo/script/spark-test.yaml").read(), #officially know bug
  kubernetes_conn_id="kubeConnTest", #ns default in airflow connection UI
  do_xcom_push=True,
  dag=dag
)

I wrapped it like this. and it works like charm

https://github.com/apache/airflow/issues/17371

Upvotes: 1

Related Questions