Using Azure Data Factory to ingest incoming data from a REST API

Question

Is there a way to create an Azure ADF Pipeline to ingest the incoming POST requests? I have this gateway app (outside Azure) that is able to publish data via REST as it arrives from the application and this data needs to be ingested into a Data Lake. I am utilizing the REST calls from another pipeline to pull the data but this basically needs to do the reverse - the data will be pushed and i need to be constantly 'listening' to those calls...

Is this something an ADF pipeline should do or maybe there are any other Azure components able to do it?

Answer 1

If your outside application is already pushing via REST, why not have it make calls directly to the Data Lake REST APIs? This would cut out the middle steps and bring everything under your control.

Answer 2

Previous comment is right and is one of the approach to get it working but would need bit of coding (for azure function).

There could also be an alternate solution to cater to your requirement is with Azure Logic Apps and Azure data factory.

Step 1: Create a HTTP triggered logic app which would be invoked by your gateway app and data will be posted to this REST callable endpoint.

Step 2: Create ADF pipeline with a parameter, this parameter holds the data that needs to be pushed to the data lake. It could be raw data and can be transformed as a step within the pipeline before pushing it to the data lake.

Step 3: Once logic app is triggered, you can simply use Azure data factory actions to invoke the data factory pipeline created in step 2 and pass the posted data as a pipeline parameter to your ADF pipeline.

This should be it, with this - you can spin up your code-less solution.

Answer 3

Azure Data Factory is a batch data movement service. If you want to push the data over HTTP, you can implement a simple Azure Function to accept the data and write it to the Azure Data Lake.

See Azure Functions HTTP triggers and bindings overview