CODEWITHSUNDEEP
airflow
OCDev
OCDev

Reputation: 705

Is there a way to upload a file via REST in Airflow via the SimpleHttpOperator?

I am trying to create an Airflow DAG to query some data, save it as CSV file and upload it to a REST API endpoint. I implemented this with a shell script and a CURL command like this:

curl --location --request POST 'https://my.endpoint.com/automations-file-service/automations/fileupload/files' \
--header 'X-API-TOKEN: my_token' \
--form 'file=@/Users/myuser/myfiles/all_20200707_2.csv'

I am trying to move this to Airflow and want to convert that CURL command to Airflow via the SimpleHttpOperator. The issue is, I cannot find any parameter in the operator where I can add the path to the file, there is no "form" parameter. Has anyone implemented something like this in Airflow? Thanks!

Upvotes: 0

Views: 2006

Answers (1)

Mike Taylor
Mike Taylor

Reputation: 699

So there's no way to do this without providing or writing your own code. You have a couple ways to do this. Subclass the SimpleHttpOperator and rewrite the execute method such that you call the HttpHook with the correct arguments. Subclass the BaseOperator for your specific case, and essentially do the same. Or write the Python functionality into a Python Callable and use the PythonOperator.

Detail

If you look up the HttpHook, you can see that the execution of the SimpleHttpOperator, largely just uses the Python requests library.

And the run method takes a request_kwargs option which you can't access via the execute method on the SimpleHttpOperator. Writing an operator with your own execute method or a Callable, you can expressly use HttpHook.run(...) to pass the arguments of your Request, or you can choose to use another Python http library.

Upvotes: 1

Related Questions