json.decoder.JSONDecodeError when getting string from XCOM

Question

In a task, I serialise a dict (converting a dict to string) and I pushed it to XCOM

result[data] = json.dumps({"agents": ["john.doe@example.com"], "houses": ["jane.doe@example.com"]})

In Airflow's UI it looks good as a string, and the DAG level, I get in value XCOM_DATA = "{{ task_instance.xcom_pull(task_ids='task_name', key='return_value')['data']}}"

But when a k8s pod operator, for some reason it deletes the double quotes(")

KubernetesPodOperator(
        cmds=["python", "src/send_notification.py"],
        arguments=[
            "--data",
            XCOM_DATA,
        ],
        task_id="notify",
        name="notify",
        dag=dag,
        **COMMON_TASKS_ARGS,
    )

UPDATE: This is the command passed to K8S

 ['. /secrets/env; python /home/app/src/send_notification.py '
 '"--pr" '
 '"https://gitlab.com/30675" "--data" '
 '"{"agents": ["john.doe@example.com", '
 '"jane.doe@example.com"]}" '

and when I deserialize it and inside the script, I get this error json.loads(args.data)

obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

I printed the values and the error occurs for some reason airflow deletes the double quotes to the string and that's why the error occurs. Can someone help me to correct it?

• I used jinja methods like safe or escape
• I convert to json in Jinja

Answer 1

I fixed it adding | tojson

XCOM_DATA = " \"{{ task_instance.xcom_pull(task_ids='task_name', key='return_value')['data']| tojson \" }} " 

Answer 2

You are sending string to the cli. that inside this string you need to add double quotes because of json format.

in that case you need to send it with escape literals

"{\"agents\": [\"john.doe@example.com\",\"jane.doe@example.com\"]}"