CODEWITHSUNDEEP
vector-databasemilvus
tmandyai
tmandyai

Reputation: 85

Compaction Plan Conflict After Cluster Restart in Milvus

After restarting the Milvus cluster, I noticed that many compaction tasks are facing conflicts. Before the restart, there were a large number of ongoing compaction tasks, which seem to have remained in the task queue. As a result, many segments have their metadata state set to inCompacting.

However, after restarting the cluster, the task queue becomes empty, and the coordinator starts creating new compaction plans. Most of these newly generated compaction plans fail with the following error:

[2025/01/03 01:14:22.100 +00:00] [WARN] [datacoord/compaction_trigger.go:490] ["failed to execute compaction task"] 
[collection=455007134314668300] [triggerID=455046071905748780] [planID=455046071905748783] 
[inputSegments="[455007134350525077,455007134316823700,455007134381307513,455046071898345279]"] 
[error="segment is compacting: compaction plan conflict"]

I suspect that these input segments are still marked as inCompacting in etcd, preventing Milvus from executing the new compaction tasks. If these segments remain inCompacting, does that mean they can never be compacted again? Does Milvus have a mechanism to restore such metadata after a failure?

Upvotes: 0

Views: 15

Answers (1)

tmandyai
tmandyai

Reputation: 85

After investigating the issue, I found that the core logic related to compaction is mainly handled in datacoord, and compaction tasks are recorded in metadata. Upon restarting, any unfinished compaction tasks are read from the metadata and executed.

The relevant logic can be found in:

milvus/internal/datacoord/compaction.go

Line 90 in e6bf141

meta             CompactionMeta

When tasks are loaded from metadata at startup, the corresponding segments are marked as isCompacting. However, compaction_trigger and the recovery mechanism operate independently. The compaction_trigger scans for segments that can be compacted, but when it encounters segments already marked as isCompacting, it logs the following warning:

segment is compacting: compaction plan conflict

This conflict can be safely ignored. The compaction_trigger will skip those isCompacting segments. Once the previously recorded tasks from metadata are completed, compaction_trigger will resume normal operation, and compaction will proceed as expected.

In summary:

The warning message does not indicate a failure but a temporary state. The system will not attempt to compact isCompacting segments again until the previously unfinished compaction tasks are completed. Once those tasks finish, the new compaction plans will execute normally.

Upvotes: 0

Related Questions