Hyperledger Caliper : Error: No valid responses from any peers when connected to remote fabric server

Question

I have setup a Hyperledger fabric node (with 3 organizations each having a single peer) on one remote server and a Hyperledger Caliper node to benchmark it using the sample chaincode (asset-transfer-basic). The same benchmark works locally. but when I change the URLs in the network config files and add SANs in certificate generation it fails:

giving the following error in the worker iterations:

error: [Transaction]: Error: No valid responses from any peers. Errors: +0000 debug [worker-message-handler] Handled "test" message for Worker (24258) error [connectors/v2/FabricGateway] Failed to perform submit transaction [DeleteAsset] using arguments [2_1], with error: Error: No valid responses from any peers. Errors: at newEndorsementError (/home/ubuntu/caliper/node_modules/fabric-network/lib/transaction.js:49:12) at getResponsePayload (/home/ubuntu/caliper/node_modules/fabric-network/lib/transaction.js:17:23) at Transaction.submit (/home/ubuntu/caliper/node_modules/fabric-network/lib/transaction.js:212:28) at async V2FabricGateway._submitOrEvaluateTransaction (/home/ubuntu/caliper/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:376:26) at async V2FabricGateway._sendSingleRequest (/home/ubuntu/caliper/node_modules/@hyperledger/caliper-fabric/lib/connector-versions/v2/FabricGateway.js:170:16) at async V2FabricGateway.sendRequests (/home/ubuntu/caliper/node_modules/@hyperledger/caliper-core/lib/common/core/connector-base.js:78:28) at async MyWorkload.cleanupWorkloadModule (/home/ubuntu/caliper/workload/readAsset.js:52:13) at async CaliperWorker.executeRound (/home/ubuntu/caliper/node_modules/@hyperledger/caliper-core/lib/worker/caliper-worker.js:214:13) at async WorkerMessageHandler._handleTestMessage (/home/ubuntu/caliper/node_modules/@hyperledger/caliper-core/lib/worker/worker-message-handler.js:235:22) at async ProcessMessenger.<anonymous> (/home/ubuntu/caliper/node_modules/@hyperledger/caliper-core/lib/worker/worker-message-handler.js:93:13)

The debug logs at the remote server also show there was a successful TLS handshake:

` DEBU [core.comm] ServerHandshake -> Server TLS handshake completed in 10.515647ms server=PeerServer remoteaddress=18.216.197.203:39598 DEBU [core.comm] ServerHandshake -> Server TLS handshake completed in 12.640673ms server=PeerServer remoteaddress=18.216.197.203:39620 DEBU [core.comm] ServerHandshake -> Server TLS handshake completed in 18.734087ms server=PeerServer remoteaddress=18.216.197.203:39610 DEBU [core.comm] ServerHandshake -> Server TLS handshake completed in 17.251823ms server=PeerServer remoteaddress=18.216.197.203:39626 DEBU [core.comm] ServerHandshake -> Server TLS handshake completed in 9.189911ms server=PeerServer remoteaddress=18.216.197.203:39628

Since it was for testing I manually imported the necessary certificate files to the host of caliper and there were no errors in incorrect path. The networkconfig file:

name: Caliper test
version: "2.0.0"

caliper:
  blockchain: fabric
  sutOptions:
    mutualTls: true

channels:
  - channelName: mychannel
    contracts:
      - id: basic

organizations:
  - mspid: Org1MSP
    identities:
      certificates:
        - name: "User1"
          clientPrivateKey:
            path: "network-profiles/peerOrganizations/org1.example.com/users/User1@org1.example.com/msp/keystore/priv_sk"
          clientSignedCert:
            path: "network-profiles/peerOrganizations/org1.example.com/users/User1@org1.example.com/msp/signcerts/User1@org1.example.com-cert.pem"
    connectionProfile:
      path: "network-profiles/peerOrganizations/org1.example.com/connection-org1.yaml"
      discover: false
  - mspid: Org2MSP
    identities:
      certificates:
        - name: "User1"
          clientPrivateKey:
            path: "network-profiles/peerOrganizations/org2.example.com/users/User1@org2.example.com/msp/keystore/priv_sk"
          clientSignedCert:
            path: "network-profiles/peerOrganizations/org2.example.com/users/User1@org2.example.com/msp/signcerts/User1@org2.example.com-cert.pem"
    connectionProfile:
      path: "network-profiles/peerOrganizations/org2.example.com/connection-org2.yaml"
      discover: false
  - mspid: Org3MSP
    identities:
      certificates:
        - name: "User1"
          clientPrivateKey:
            path: "network-profiles/peerOrganizations/org3.example.com/users/User1@org3.example.com/msp/keystore/priv_sk"
          clientSignedCert:
            path: "network-profiles/peerOrganizations/org3.example.com/users/User1@org3.example.com/msp/signcerts/User1@org3.example.com-cert.pem"
    connectionProfile:
      path: "network-profiles/peerOrganizations/org3.example.com/connection-org3.yaml"
      discover: false

`

Tried GRPCing the HF network using peer command and it worked:

eer chaincode query -C mychannel -n basic -c '{"Args":["GetAllAssets"]}'
[{"AppraisedValue":300,"Color":"blue","ID":"asset1","Owner":"Tomoko","Size":5,"docType":"asset"},{"AppraisedValue":400,"Color":"red","ID":"asset2","Owner":"Brad","Size":5,"docType":"asset"},{"AppraisedValue":500,"Color":"green","ID":"asset3","Owner":"Jin Soo","Size":10,"docType":"asset"},{"AppraisedValue":600,"Color":"yellow","ID":"asset4","Owner":"Max","Size":10,"docType":"asset"},{"AppraisedValue":700,"Color":"black","ID":"asset5","Owner":"Adriana","Size":15,"docType":"asset"},{"AppraisedValue":800,"Color":"white","ID":"asset6","Owner":"Michel","Size":15,"docType":"asset"}]