Hyperledger setup in Docker Swarm Mode w/ TLS

Question

I am setting up a production-ready Hyperledger network. This requires the Docker containers to run in swarm mode, so we can have orderers in different geolocations.

I have previously setup Kafka network in swarm mode (without TLS), and single-orderer networks.

We are migrating to Raft consensus as per the upgrade to version 1.4

My ordering nodes are setup and communicating with each other, and I can confirm via Raft logging. My peer node runs successfully and certificates are confirmed mounted in the container.

When I run my channel create command :

docker exec $PEER_NAME peer channel create 
 -o orderer0.company:7050 
 -c messagebus -f /etc/hyperledger/configtx/channel.tx --tls 
 --cafile /etc/hyperledger/fabric/ordererCerts/tlsca.company-cert.pem 
 --keyfile /etc/hyperledger/msp/users/Admin\@company/tls/client.key 
 --certfile /etc/hyperledger/msp/users/Admin\@company/tls/client.crt 
 --clientauth

I see the following log in the Orderer0 node:

 ServerHandshake -> ERRO 3c2 TLS handshake failed with error tls: failed to verify client's certificate: x509: certificate signed by unknown authority server=Orderer remoteaddress=10.0.0.4:50266

I see the following log in the output of the command:

[msp] getMspConfig -> DEBU 028 TLS intermediate certs folder not found at [/etc/hyperledger/msp/users/Admin@company/msp/tlsintermediatecerts]. Skipping. [stat /etc/hyperledger/msp/users/Admin@company/msp/tlsintermediatecerts: no such file or directory]
[msp] getMspConfig -> DEBU 02a crls folder not found at [/etc/hyperledger/msp/users/Admin@company/msp/crls]. Skipping. [stat /etc/hyperledger/msp/users/Admin@company/msp/crls: no such file or directory]
[msp] getMspConfig -> DEBU 02b MSP configuration file not found at [/etc/hyperledger/msp/users/Admin@company/msp/config.yaml]: [stat /etc/hyperledger/msp/users/Admin@company/msp/config.yaml: no such file or directory]

Which seems to cause a default to BCCSP-based MSP instance.. Could this be causing my certificate error?

and further down in the output from peer channel create..

Error: failed to create deliver client: orderer client failed to connect to orderer0.gmex-group:7050: failed to create new connection: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"

I noted that the msp/config.yaml is found in crypto-config/peerOrganizations/gmex-group/msp/config.yaml , but not in the users/ directory

I have picked the configs apart from the byfn scripts and docker-compose files. Below are my sample configs:

PEER

version: '3.3'
networks:
  nodex:
    driver: overlay
    attachable: true

services:
  peer0_company:
  deploy:
    replicas: 1
    restart_policy:
      condition: on-failure
      delay: 5s
      max_attempts: 3
  image: hyperledger/fabric-peer
  hostname: peer0.company
  environment:
  #Logging Fields#
  - CORE_LOGGING_GOSSIP=ERROR
  - CORE_LOGGING_MSP=DEBUG
  - FABRIC_LOGGING_SPEC=DEBUG
  #Peer Config#
  - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
  - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=nodex_nodex
  - CORE_PEER_GOSSIP_USELEADERELECTION=true
  - CORE_PEER_GOSSIP_ORGLEADER=false
  - CORE_PEER_PROFILE_ENABLED=true
  - CORE_PEER_ID=peer0.company
  - CORE_PEER_LOCALMSPID=Org1MSP
  - CORE_PEER_ADDRESS=peer0.company:7051
  - CORE_PEER_LISTENADDRESS=0.0.0.0:7051
  - CORE_PEER_CHAINCODEADDRESS=peer0.company:7052
  - CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052
  - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.company:7051
  - CORE_PEER_GOSSIP_BOOTSTRAP=peer1.company:8051
  - CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/msp/users/Admin@company/msp
  #TLS Fields#
  - CORE_PEER_TLS_ENABLED=true
  - CORE_PEER_TLS_CLIENTAUTHREQUIRED=true
  - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
  - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
  - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
  - CORE_PEER_TLS_CLIENTROOTCAS_FILES=/etc/hyperledger/msp/users/Admin@company/tls/ca.crt
  - CORE_PEER_TLS_CLIENTCERT_FILE=/etc/hyperledger/msp/users/Admin@company/tls/client.crt
  - CORE_PEER_TLS_CLIENTKEY_FILE=/etc/hyperledger/msp/users/Admin@company/tls/client.key
volumes:
  - /var/run/:/host/var/run/
  - ../crypto-config/ordererOrganizations/company/orderers/orderer0.company/msp/tlscacerts:/etc/hyperledger/fabric/ordererCerts
  - ../crypto-config/peerOrganizations/company/peers/peer0.company/tls:/etc/hyperledger/fabric/tls
  - ../crypto-config/peerOrganizations/company/peers/peer0.company/msp:/etc/hyperledger/fabric/msp
  - ../crypto-config/peerOrganizations/company/users:/etc/hyperledger/msp/users
  - ../channel-artifacts:/etc/hyperledger/configtx
command: peer node start
working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer
ports:
  - 7051:7051
  - 7052:7052
  - 7053:7053
networks:
  nodex:
    aliases:
      - peer0.company

ORDERER :

version: '3.3'

networks:
  nodex:
    driver: overlay
    attachable: true

services:
  orderer0_company:
    deploy:
        replicas: 1
        restart_policy:
            condition: on-failure
            delay: 5s
            max_attempts: 3
    image: hyperledger/fabric-orderer
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric
    command: orderer
    hostname: orderer0.company
    environment:
        - FABRIC_LOGGING_SPEC=DEBUG
        - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=nodex_nodex
        - ORDERER_GENERAL_LOGLEVEL=debug
        - ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
        - ORDERER_GENERAL_GENESISMETHOD=file
        - ORDERER_GENERAL_GENESISFILE=/etc/hyperledger/configtx/genesis.block
        - ORDERER_GENERAL_LOCALMSPID=OrdererMSP
        - ORDERER_GENERAL_LOCALMSPDIR=/etc/hyperledger/msp
        - ORDERER_GENERAL_LISTENPORT=7050
        - ORDERER_GENERAL_LEDGERTYPE=ram
        - ORDERER_GENERAL_CLUSTER_CLIENTCERTIFICATE=/etc/hyperledger/fabric/tls/server.crt
        - ORDERER_GENERAL_CLUSTER_CLIENTPRIVATEKEY=/etc/hyperledger/fabric/tls/server.key
        - ORDERER_GENERAL_CLUSTER_ROOTCAS=[/etc/hyperledger/fabric/tls/ca.crt]
        - ORDERER_GENERAL_TLS_ENABLED=true
        - ORDERER_GENERAL_TLS_PRIVATEKEY=/etc/hyperledger/fabric/tls/server.key
        - ORDERER_GENERAL_TLS_CERTIFICATE=/etc/hyperledger/fabric/tls/server.crt
        - ORDERER_GENERAL_TLS_ROOTCAS=[/etc/hyperledger/fabric/tls/ca.crt]
        - ORDERER_GENERAL_TLS_CLIENTAUTHREQUIRED=true
        - ORDERER_GENERAL_TLS_CLIENTROOTCAS=[/etc/hyperledger/msp/cacerts/ca.company-cert.pem]
    ports:
      - 7050:7050
    volumes:
      - ../channel-artifacts/genesis.block:/etc/hyperledger/configtx/genesis.block
      - ../crypto-config/ordererOrganizations/company/orderers/orderer0.company/msp:/etc/hyperledger/msp
      - ../crypto-config/ordererOrganizations/company/orderers/orderer0.company/tls/:/etc/hyperledger/fabric/tls
    networks:
      nodex:
        aliases:
            - orderer0.company

CONFIGTX

Organizations:
Name: OrdererOrg
ID: OrdererMSP

MSPDir: crypto-config/ordererOrganizations/company/msp

Policies:
  Readers:
    Type: Signature
    Rule: "OR('OrdererMSP.member')"
  Writers:
    Type: Signature
    Rule: "OR('OrdererMSP.member')"
  Admins:
    Type: Signature
    Rule: "OR('OrdererMSP.admin')"

  - &Org1
    Name: Org1MSP
    ID: Org1MSP
    MSPDir: crypto-config/peerOrganizations/company/msp

Policies:
  Readers:
    Type: Signature
    Rule: "OR('Org1MSP.admin', 'Org1MSP.peer', 'Org1MSP.client')"
  Writers:
    Type: Signature
    Rule: "OR('Org1MSP.admin', 'Org1MSP.client')"
  Admins:
    Type: Signature
    Rule: "OR('Org1MSP.admin')"

AnchorPeers:
  - Host: peer0.company
    Port: 7051

Capabilities:
  Global: &ChannelCapabilities
    V1_1: true
  Orderer: &OrdererCapabilities
    V1_1: true
  Application: &ApplicationCapabilities
    V1_4_2: true
    V1_3: false
    V1_2: false
    V1_1: false

Application: &ApplicationDefaults
  Organizations:
  Policies:
    Readers:
      Type: ImplicitMeta
      Rule: "ANY Readers"
    Writers:
      Type: ImplicitMeta
      Rule: "ANY Writers"
    Admins:
      Type: ImplicitMeta
      Rule: "MAJORITY Admins"

  Capabilities:
    <<: *ApplicationCapabilities

Orderer: &OrdererDefaults

  OrdererType: etcdraft
  EtcdRaft:
    Consenters:
      - Host: orderer0.company
        Port: 7050
        ClientTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer0.company/tls/server.crt
        ServerTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer0.company/tls/server.crt
      - Host: orderer1.company
        Port: 7050
        ClientTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer1.company/tls/server.crt
        ServerTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer1.company/tls/server.crt
      - Host: orderer2.company
        Port: 7050
        ClientTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer2.company/tls/server.crt
        ServerTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer2.company/tls/server.crt
      - Host: orderer3.company
        Port: 7050
        ClientTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer3.company/tls/server.crt
        ServerTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer3.company/tls/server.crt
      - Host: orderer4.company
        Port: 7050
        ClientTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer4.company/tls/server.crt
        ServerTLSCert: crypto-config/ordererOrganizations/company/orderers/orderer4.company/tls/server.crt

  Addresses:
    - orderer0.company:7050
    - orderer1.company:7050
    - orderer2.company:7050
    - orderer3.company:7050
    - orderer4.company:7050

  BatchTimeout: 1s
  BatchSize:
    MaxMessageCount: 50
    AbsoluteMaxBytes: 200 MB
    PreferredMaxBytes: 50 MB

  Organizations:
  Policies:
    Readers:
      Type: ImplicitMeta
      Rule: "ANY Readers"
    Writers:
      Type: ImplicitMeta
      Rule: "ANY Writers"
    Admins:
      Type: ImplicitMeta
      Rule: "MAJORITY Admins"

    BlockValidation:
      Type: ImplicitMeta
      Rule: "ANY Writers"

Channel: &ChannelDefaults
  Policies:
    Readers:
      Type: ImplicitMeta
      Rule: "ANY Readers"
    Writers:
      Type: ImplicitMeta
      Rule: "ANY Writers"
    Admins:
      Type: ImplicitMeta
      Rule: "MAJORITY Admins"
  Capabilities:
    <<: *ChannelCapabilities

Profiles:

  OneOrgOrdererGenesis:
    <<: *ChannelDefaults
    Capabilities:
      <<: *ChannelCapabilities
    Orderer:
      <<: *OrdererDefaults
      Organizations:
        - *OrdererOrg
      Capabilities:
        <<: *OrdererCapabilities
    Consortiums:
      SampleConsortium:
        Organizations:
          - *Org1
  OneOrgChannel:
    Consortium: SampleConsortium
    <<: *ChannelDefaults
    Application:
      <<: *ApplicationDefaults
      Organizations:
        - *Org1
      Capabilities:
        <<: *ApplicationCapabilities

Hyperledger setup in Docker Swarm Mode w/ TLS

Answers (1)

Related Questions