CODEWITHSUNDEEP
sslhyperledger-fabrichyperledger
RVandersteen
RVandersteen

Reputation: 2137

Hyperledger Fabric - Peer unable to connect to (raft) Orderer with Mutual TLS

I am running a HLF on kubernetes - (3 raft orderers & 2 peers)

Now as raft requires Mutual TLS I had to setup some certificates.

The 3 raft orderers are able to communicate with eachother, as they are electing a leader, and re-electing another leader when I bring that leader down.

When I setup the peer, I used the same CA to generate the certificates. I am able to create the channel & join it from the peer. However I have to run CORE_PEER_MSPCONFIGPATH=$ADMIN_MSP_PATH prior to those commands, otherwise I get Access Denied error.

I am also forced to append the following flags to every peer channel x command I run.

--tls --cafile $ORD_TLS_PATH/cacert.pem --certfile $CORE_PEER_TLS_CLIENTCERT_FILE --keyfile $CORE_PEER_TLS_CLIENTKEY_FILE --clientauth

I am able to create, fetch, join the channel using the admin msp.

Now once the channel is joined, the peer is unable to connect with the orderer, somehow a bad certificate is given.

Orderer Logs

A bad certificate is used ?

2019-08-15 16:07:55.699 UTC [core.comm] ServerHandshake -> ERRO 221 TLS handshake failed with error remote error: tls: bad certificate server=Orderer remoteaddress=10.130.2.148:53922
2019-08-15 16:07:55.699 UTC [grpc] handleRawConn -> DEBU 222 grpc: Server.Serve failed to complete security handshake from "10.130.2.148:53922": remote error: tls: bad certificate

Peer Logs

These suggest that it could not validate it with the ca.crt ?

2019-08-15 16:10:17.990 UTC [grpc] DialContext -> DEBU 03a parsed scheme: ""
2019-08-15 16:10:17.990 UTC [grpc] DialContext -> DEBU 03b scheme "" not registered, fallback to default scheme
2019-08-15 16:10:17.991 UTC [grpc] watcher -> DEBU 03c ccResolverWrapper: sending new addresses to cc: [{orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}]
2019-08-15 16:10:17.991 UTC [grpc] switchBalancer -> DEBU 03d ClientConn switching balancer to "pick_first"
2019-08-15 16:10:17.991 UTC [grpc] HandleSubConnStateChange -> DEBU 03e pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:18.009 UTC [grpc] createTransport -> DEBU 03f grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:18.012 UTC [grpc] HandleSubConnStateChange -> DEBU 040 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:18.991 UTC [grpc] HandleSubConnStateChange -> DEBU 041 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:19.003 UTC [grpc] createTransport -> DEBU 042 grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:19.003 UTC [grpc] HandleSubConnStateChange -> DEBU 043 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:20.719 UTC [grpc] HandleSubConnStateChange -> DEBU 044 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, CONNECTING
2019-08-15 16:10:20.731 UTC [grpc] createTransport -> DEBU 045 grpc: addrConn.createTransport failed to connect to {orderer-2.hlf-orderers.svc.cluster.local:7050 0  <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority". Reconnecting...
2019-08-15 16:10:20.733 UTC [grpc] HandleSubConnStateChange -> DEBU 046 pickfirstBalancer: HandleSubConnStateChange: 0xc00260b710, TRANSIENT_FAILURE
2019-08-15 16:10:20.990 UTC [ConnProducer] NewConnection -> ERRO 047 Failed connecting to {orderer-2.hlf-orderers.svc.cluster.local:7050 [OrdererMSP]} , error: context deadline exceeded

I generated the used certificates as follows:

Orderer Admin

fabric-ca-client enroll -u https://u:[email protected] -M ./OrdererMSP

Orderer Node X

As I use the same certificates for TLS I added the used hosts here for TLS purposes

fabric-ca-client enroll -m orderer-x \
  -u https://ox:[email protected] \
  --csr.hosts orderer-x.hlf-orderers.svc.cluster.local,orderer-x.hlf-orderers,orderer-x,localhost \
  -M orderer-x-MSP

Peer Admin

fabric-ca-client enroll -u https://u:[email protected] -M ./PeerMSP

Peer Node X

fabric-ca-client enroll -m peer-x \
  -u https://ox:[email protected] \
  --csr.hosts peer-x.hlf-peers.svc.cluster.local,peer-x.hlf-peers,peer-x,localhost \
  -M peer-x-MSP

Now all of these, have the same ca.crt (/cacerts/ca.example.com.pem)

configtx.yaml

        Orderer:
            <<: *OrdererDefaults
            OrdererType: etcdraft
            EtcdRaft:
                Consenters:
                    - Host: orderer-1.hlf-orderers.svc.cluster.local
                      Port: 7050
                      ClientTLSCert: orderer-1-MSP/signcerts/cert.pem
                      ServerTLSCert: orderer-1-MSP/signcerts/cert.pem
                    - Host: orderer-2.hlf-orderers.svc.cluster.local
                      Port: 7050
                      ClientTLSCert: orderer-2-MSP/signcerts/cert.pem
                      ServerTLSCert: orderer-2-MSP/signcerts/cert.pem
                    - Host: orderer-3.hlf-orderers.svc.cluster.local
                      Port: 7050
                      ClientTLSCert: orderer-3-MSP/signcerts/cert.pem
                      ServerTLSCert: orderer-3-MSP/signcerts/cert.pem
            Addresses:
                - orderer-1.hlf-orderers.svc.cluster.local:7050
                - orderer-2.hlf-orderers.svc.cluster.local:7050
                - orderer-3.hlf-orderers.svc.cluster.local:7050

I have checked multiple times if the correct certificates are mounted on the correct places and configured.

On the peer side I made sure that:

On the orderer side I made sure that:

It seems strange to me that the orderers are able to talk to eachother (as they are electing leaders), but that the peer is not able to do so

Upvotes: 1

Views: 2420

Answers (2)

RVandersteen
RVandersteen

Reputation: 2137

So it appears to be, that the tlscacerts should be in the msp(s) directory(ies) PRIOR to creating genesis / channel block. Simply mounting them in the pod at runtime is not enough

My msp directories (used in configtx.yaml) look like:

  • admincerts
  • tlscacerts
  • cacerts
  • ...

After this it all started to work

Upvotes: 3

Narendranath Reddy
Narendranath Reddy

Reputation: 4133

seems like you have got below error

E0923 16:30:14.963567129 31166 ssl_transport_security.cc:989] Handshake failed with fatal error SSL_ERROR_SSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate.
E0923 16:30:15.964456710 31166 ssl_transport_security.cc:188] ssl_info_callback: error occured.

According to your details, All seems to be correct However check below 

certificate signed by unknown authority -> This makes me bit doubt on your certificate mapping

MAKE SURE

PEER:

  • CORE_PEER_TLS_ENABLED=true
  • CORE_PEER_TLS_CERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/tls/server.crt
  • CORE_PEER_TLS_KEY_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/tls/server.key
  • CORE_PEER_TLS_ROOTCERT_FILE=/data/maersksea-rca-maersksea-chain.pem
  • CORE_PEER_TLS_CLIENTCERT_FILE=/data/tls/maersksea-peer-maersksea-client.crt
  • CORE_PEER_TLS_CLIENTKEY_FILE=/data/tls/maersksea-peer-maersksea-client.key
  • CORE_PEER_TLS_CLIENTAUTHREQUIRED=true
  • CORE_PEER_TLS_CLIENTROOTCAS_FILES=/data/maersksea-rca-maersksea-chain.pem

Orderer:

  • ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED=true
  • ORDERER_GENERAL_TLS_CLIENTROOTCAS=[/data/maersksea-rca-maersksea-chain.pem]

Upvotes: 1

Related Questions