Error: error getting endorser client for channel: endorser client failed to connect to peer-govt:7051: failed to create new connection: context

Question

I have been trying to deploy a hyperledger fabric model with 3 CAs 1 orderer and 2 peer nodes. I am able to create the channel with OSADMIN command of fabric but when I try to join the channel with peer node, I get Error: error getting endorser client for channel: endorser client failed to connect to peer-govt:7051: failed to create new connection: context...... .

Here are the logs from terminal (local host machine):

2021-06-01 06:38:54.509 UTC [common.tools.configtxgen] main -> INFO 001 Loading configuration
2021-06-01 06:38:54.522 UTC [common.tools.configtxgen.localconfig] completeInitialization -> INFO 002 orderer type: etcdraft
2021-06-01 06:38:54.522 UTC [common.tools.configtxgen.localconfig] completeInitialization -> INFO 003 Orderer.EtcdRaft.Options unset, setting to tick_interval:"500ms" election_tick:10 heartbeat_tick:1 max_inflight_blocks:5 snapshot_interval_size:16777216 
2021-06-01 06:38:54.522 UTC [common.tools.configtxgen.localconfig] Load -> INFO 004 Loaded configuration: /etc/hyperledger/clipod/configtx/configtx.yaml
2021-06-01 06:38:54.712 UTC [common.tools.configtxgen] doOutputBlock -> INFO 005 Generating genesis block
2021-06-01 06:38:54.712 UTC [common.tools.configtxgen] doOutputBlock -> INFO 006 Creating application channel genesis block
2021-06-01 06:38:54.712 UTC [common.tools.configtxgen] doOutputBlock -> INFO 007 Writing genesis block
cli-dd4cc5fbf-pdcgb
Status: 201
{
    "name": "commonchannel",
    "url": "/participation/v1/channels/commonchannel",
    "consensusRelation": "consenter",
    "status": "active",
    "height": 1
}
cli-dd4cc5fbf-pdcgb
Error: error getting endorser client for channel: endorser client failed to connect to peer-govt:7051: failed to create new connection: context deadline exceeded
command terminated with exit code 1
Error: error getting endorser client for channel: endorser client failed to connect to peer-general:9051: failed to create new connection: context deadline exceeded
command terminated with exit code 1

One thing to note down here is I am using Kubernetes and service CLUSTER_IP for all the PODS.

here are logs from one of the PEER POD (same for other)

2021-06-01 06:38:42.180 UTC [nodeCmd] registerDiscoveryService -> INFO 01b Discovery service activated
2021-06-01 06:38:42.180 UTC [nodeCmd] serve -> INFO 01c Starting peer with ID=[peer-govt], network ID=[dev], address=[peer-govt:7051]
2021-06-01 06:38:42.180 UTC [nodeCmd] func6 -> INFO 01d Starting profiling server with listenAddress = 0.0.0.0:6060
2021-06-01 06:38:42.180 UTC [nodeCmd] serve -> INFO 01e Started peer with ID=[peer-govt], network ID=[dev], address=[peer-govt:7051]
2021-06-01 06:38:42.181 UTC [kvledger] LoadPreResetHeight -> INFO 01f Loading prereset height from path [/var/hyperledger/production/ledgersData/chains]
2021-06-01 06:38:42.181 UTC [blkstorage] preResetHtFiles -> INFO 020 No active channels passed
2021-06-01 06:38:56.006 UTC [core.comm] ServerHandshake -> ERRO 021 Server TLS handshake failed in 24.669µs with error tls: first record does not look like a TLS handshake server=PeerServer remoteaddress=172.17.0.1:13258
2021-06-01 06:38:57.007 UTC [core.comm] ServerHandshake -> ERRO 022 Server TLS handshake failed in 17.772µs with error tls: first record does not look like a TLS handshake server=PeerServer remoteaddress=172.17.0.1:29568
2021-06-01 06:38:58.903 UTC [core.comm] ServerHandshake -> ERRO 023 Server TLS handshake failed in 13.581µs with error tls: first record does not look like a TLS handshake server=PeerServer remoteaddress=172.17.0.1:32615

To overcome this issue, I tried disabling the TLS by setting CORE_PEER_TLS_ENABLED to FALSE

then the proposal gets submitted but the orderer POD throws the same error of TLS handshake failed.........

Here are the commands I am using to join the channel from cli pod:

kubectl -n hyperledger -it exec $CLI_POD -- sh -c "export FABRIC_CFG_PATH=/etc/hyperledger/clipod/config && export CORE_PEER_LOCALMSPID=GeneralMSP && export CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/clipod/organizations/peerOrganizations/general.example.com/peers/peer0.general.example.com/tls/ca.crt && export CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/clipod/organizations/peerOrganizations/general.example.com/users/Admin@general.example.com/msp && export CORE_PEER_ADDRESS=peer-general:9051 && peer channel join -b /etc/hyperledger/clipod/channel-artifacts/$CHANNEL_NAME.block -o orderer:7050 --tls --cafile /etc/hyperledger/clipod/organizations/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem"

I am stuck on this problem, any help will be appreciated. Thank you

Answer 1

I have fixed it. The issue I was facing was because of not setting the CORE_PEER_TLS_ENABLED = true for CLI pod.

One thing I have got learn from this whole model, whenever you see TLS issue, first to check for would be checking CORE_PEER_TLS_ENABLED variable. Make sure you have set it for all the pods or containers you are trying to interact with. The case can be false(for no TLS) or true(for using TLS) depending on your deployment. Other things to keep in mind is using the correct variables of fabric including FABRIC_CFG_PATH, CORE_PEER_LOCALMSPID, CORE_PEER_TLS_ROOTCERT_FILE, CORE_PEER_MSPCONFIGPATH and some others depending on your command.