MixCoord Unable to Find QueryCoord After Recent Deployment in Milvus Cluster

Question

I encountered an issue in a Milvus cluster that has been running smoothly until a recent deployment of configs/user.yaml. After this deployment, the mixcoord pod is unable to find the querycoord, and the following warnings appear in the logs:

Log Snippet:

{"level":"WARN","time":"2024/12/12 22:39:48.676 +00:00","caller":"grpcclient/client.go:474","message":"grpc client is nil, maybe fail to get client in the retry state","client_role":"querycoord","error":"empty grpc client: find no available querycoord, check querycoord state","errorVerbose":"empty grpc client: find no available querycoord, check querycoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:100\n  | github.com/milvus-io/milvus/internal/distributed/querycoord/client.(*Client).GetMetrics\n  | \t/workspace/source/internal/distributed/querycoord/client/client.go:272\n  | github.com/milvus-io/milvus/internal/rootcoord.getQueryCoordMetrics\n  | \t/workspace/source/internal/rootcoord/util.go:239\n  | github.com/milvus-io/milvus/internal/rootcoord.(*QuotaCenter).collectMetrics.func1\n  | \t/workspace/source/internal/rootcoord/quota_center.go:393\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/go/pkg/mod/golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available querycoord, check querycoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"}
{"level":"WARN","time":"2024/12/12 22:39:48.678 +00:00","caller":"grpcclient/client.go:248","message":"failed to get client address","error":"find no available querycoord, check querycoord state"}
{"level":"WARN","time":"2024/12/12 22:39:48.678 +00:00","caller":"grpcclient/client.go:460","message":"fail to get grpc client in the retry state","client_role":"querycoord","error":"find no available querycoord, check querycoord state"}

Here are what I observed

• Etcd pods appear to be running fine.
• I attempted scaling the mixcoord deployment to 0, waited for a few minutes, then scaled it back to 1, but the issue persists.

What could be causing the mixcoord pod to fail in locating the querycoord, and how can I resolve this issue? Is there a specific configuration in configs/user.yaml or other system-level changes that might need to be addressed? Any troubleshooting tips or steps would be greatly appreciated.