CODEWITHSUNDEEP
druid
Kiran Krishna Innamuri
Kiran Krishna Innamuri

Reputation: 1002

Druid Cluster going into Restricted Mode

We have a Druid Cluster with the following specs

3X Coordinators & Overlords - m5.2xlarge
6X Middle Managers(Ingest nodes with 5 slots) - m5d.4xlarge
8X Historical - i3.4xlarge
2X Router & Broker - m5.2xlarge

Cluster often goes into Restricted mode Druid Cluster Services Restricted Mode

  1. All the calls to the Cluster gets rejected with a 502 error.
  2. Even with 30 available slots for the index-parallel tasks, cluster only runs 10 at time and the other tasks are going into waiting state.
  3. Loader Task submission time has been increasing monotonically from 1s,2s,..,6s,..10s(We submit a job to load the data in S3), after recycling the cluster submission time decreases and increases again over a period of time

We submit around 100 jobs per minute but we need to scale it to 300 to catchup with our current incoming load

Cloud someone help us with our queries

  1. Tune the specs of the cluster
  2. What parameters to be optimized to run maximum number of tasks in parallel without increasing the load on the master nodes
  3. Why is the loader task submission time increasing, what are the parameters to be monitored here

Upvotes: 0

Views: 610

Answers (1)

Sergio Ferragut
Sergio Ferragut

Reputation: 181

At 100 jobs per minute, this is probably why the overlord is being overloaded.

The overlord initiates a job by communicating with the middle managers across the cluster. It defines the tasks that each middle manager will need to complete and it monitors the task progress until completion. The startup of each job has some overhead, so that many jobs would likely keep the overlord busy and prevent it from processing the other jobs you are requesting. This might explain why the time for job submissions increases over time. You could increase the resources on the overlord, but this sounds like there may be a better way to ingest the data.

The recommendation would be to use a lot less jobs and have each job do more work.

If the flow of data is so continuous as you describe, perhaps a kafka queue would be the best target followed up with a Druid kafka ingestion job which is fully scalable.

If you need to do batch, perhaps a single index_parallel job that reads many files would be much more efficient than many jobs.

Also consider that each task in an ingestion job creates a set of segments. By running a lot of really small jobs, you create a lot of very small segments which is not ideal for query performance. Here some info around how to think about segment size optimization which I think might help.

Upvotes: 1

Related Questions