CODEWITHSUNDEEP
druid
Sakib Farhad
Sakib Farhad

Reputation: 130

Insert new segment when apache druid already have more them 1M segments

I have an apache druid cluster setup with 1 master, 2 data Server and 1 query server and apache cassandra as deep storage. Now my druid setup has more then 1 million segment already and 15 different data source. When ever I try to insert new data it sometimes give me OutofMemeory exception.

What I want to do is, when I want to insert new data I want to mark some of the segments unused. Here is the catch, according to druid doc if I mark any segment as unused druid will delete it from deep storage. I don't want that. After inserting new data I want to re-enable the unused segments. How can I do it?

Upvotes: 0

Views: 459

Answers (1)

q_zedd
q_zedd

Reputation: 26

  1. Druid doesn't delete unusedsegments from deep storage until you run a kill task. Read here

So basically, there are 2 steps to permanently delete a particular segment: disable segment by either segment-interval or segmentId and then post a kill task on druid server

  1. As long as a segment is only disabled, it can be reloaded. To reload unused segments post a similar task to the following on druid server: 
curl -X 'POST' -H 'Content-Type:application/json' -d '{ "interval" :
"interval-of-segment" }' http://localhost:8081/druid/coordinator/v1/datasources/datasourcename/markUsed

Upvotes: 1

Related Questions