MLCP copy could only copy partial data from a 3 node cluster

Question

I have a most basic 3-node cluster ML environment. I need to copy the data down to other ML environment with MLCP copy command.

However, it seems it could copy about 1/3 of the documents down only.

I have no clue about what is wrong. Below is the error message in the mlcp copy process.

For me, it is strange why all 3 nodes IP addresses appear, as I only specify one node address only in the mlcp input_host parameter. How the MLCP knows all other hosts' IP?

It seems the error is related to the replica forest. However, in ML admin UI, all forests including those replicas are normal. Anyway, it is just a copy; I thought it should have nothing to do with replica DB.

 [Session: user=UserABC, cb=#288039430218036231 [ContentSource: user=UserABC, cb=XYC-content [provider: address=10.111.10.123/10.111.10.123:9001, pool=0/64]]]
 [Client: XCC/10.0-6.2, Server: XDBC/9.0-9.1]
2022-09-03 17:25:34 ERROR DatabaseContentReader:286 - QueryException:com.marklogic.xcc.exceptions.XQueryException: XDMP-FORESTNOT: Forest XYC-content-3-replica-1 not available: sync replicating
 [Session: user=UserABC, cb=#3880528255788069631 [ContentSource: user=UserABC, cb=XYC-content [provider: address=/10.111.10.121:9001, pool=0/64]]]
 [Client: XCC/10.0-6.2, Server: XDBC/9.0-9.1]
2022-09-03 17:25:34 ERROR LocalJobRunner:348 - Error running task:
2022-09-03 17:25:34 ERROR LocalJobRunner:349 - com.marklogic.xcc.exceptions.XQueryException: XDMP-FORESTNOT: Forest XYC-content-3-replica-1 not available: sync replicating
 [Session: user=UserABC, cb=#3880528255788069631 [ContentSource: user=UserABC, cb=XYC-content [provider: address=/10.111.10.121:9001, pool=0/64]]]
 [Client: XCC/10.0-6.2, Server: XDBC/9.0-9.1]
2022-09-03 17:25:46 ERROR DatabaseContentReader:286 - QueryException:com.marklogic.xcc.exceptions.XQueryException: XDMP-FORESTNOT: Forest XYC-content-1-replica-1 not available: sync replicating
 [Session: user=UserABC, cb=#9762515839579968807 [ContentSource: user=UserABC, cb=XYC-content [provider: address=/10.111.10.122:9001, pool=0/64]]]
 [Client: XCC/10.0-6.2, Server: XDBC/9.0-9.1]
2022-09-03 17:25:46 ERROR LocalJobRunner:348 - Error running task:
2022-09-03 17:25:46 ERROR LocalJobRunner:349 - com.marklogic.xcc.exceptions.XQueryException: XDMP-FORESTNOT: Forest XYC-content-1-replica-1 not available: sync replicating
 [Session: user=UserABC, cb=#9762515839579968807 [ContentSource: user=UserABC, cb=XYC-content [provider: address=/10.111.10.122:9001, pool=0/64]]]
 [Client: XCC/10.0-6.2, Server: XDBC/9.0-9.1]

Answer 1

Are you using load balancer? If yes, make sure to enable sticky sessions.