CODEWITHSUNDEEP
druiddata-ingestion
unknown
unknown

Reputation: 53

Refresh Data in druid

I am using the index_parallel native batch method to ingest data to Druid from s3. I have done the initial ingestion using Tasks tab from druid UI. I want to schedule another task to do delta ingestion daily.

I have gone through a lot of documentation, but I didn't find anything related to scheduling a task over druid.

Can someone help me out here what are all the ways through which we can schedule a native batch ingestion task?

Upvotes: 2

Views: 722

Answers (1)

Peter Marshall
Peter Marshall

Reputation: 196

Typically you would use something like Airflow to schedule regular ingestions, e.g. see this cool blog post:

https://www.linkedin.com/pulse/open-source-data-warehousing-druid-apache-airflow-superset-sp%C3%A4ti/

Oh and take a gander at this too to be sure you know how to configure the job to add data:

https://druid.apache.org/docs/latest/ingestion/data-management.html#adding-new-data

Upvotes: 1

Related Questions