Hyperledger-fabric not working with docker swarm

Question

I 'm using the Fabric 1.1 alpha release and trying to set it up using docker swarm. I 'm using docker compose files with docker stack to deploy the containers.

The issue I 'm having is my chaincode listening port which is 7052 hardcoded somewhere in the peer container is not listening on docker swarm.

The same compose file with minor changes work, if I don't use docker swarm.

I 'm not sure if it's something wrong with the peer itself or with docker-swarm.

This is from my peer container, which clearly is not allowing any connections on 7052 port.

root@71c1b8f22052:/opt/gopath/src/github.com/hyperledger/fabric/peer# telnet 10.0.0.6 7051
Trying 10.0.0.6...
Connected to 10.0.0.6.
Escape character is '^]'.
^CConnection closed by foreign host.
root@71c1b8f22052:/opt/gopath/src/github.com/hyperledger/fabric/peer# telnet 10.0.0.6 7052
Trying 10.0.0.6...
telnet: Unable to connect to remote host: Connection refused
root@71c1b8f22052:/opt/gopath/src/github.com/hyperledger/fabric/peer# telnet 10.0.0.6 7053
Trying 10.0.0.6...
Connected to 10.0.0.6.
Escape character is '^]'.
^CConnection closed by foreign host.

But it's listening on 7052 port.

root@71c1b8f22052:/opt/gopath/src/github.com/hyperledger/fabric/peer# netstat -nalp | grep 7052
tcp        0      0 10.0.0.6:7052           0.0.0.0:*               LISTEN      7/peer         

I 'm getting this in my chaincode container logs, when I 'm instantiating the chaincode.

2018-02-06 09:45:11.886 UTC [bccsp] initBCCSP -> DEBU 001 Initialize BCCSP [SW]
2018-02-06 09:45:11.906 UTC [grpc] Printf -> DEBU 002 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.10:7052: getsockopt: connection refused"; Reconnecting to {peer0.org1.example.com:7052 <nil>}
2018-02-06 09:45:12.905 UTC [grpc] Printf -> DEBU 003 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.10:7052: getsockopt: connection refused"; Reconnecting to {peer0.org1.example.com:7052 <nil>}
2018-02-06 09:45:14.612 UTC [grpc] Printf -> DEBU 004 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 10.0.0.10:7052: getsockopt: connection refused"; Reconnecting to {peer0.org1.example.com:7052 <nil>}
2018-02-06 09:45:14.904 UTC [shim] userChaincodeStreamGetter -> ERRO 005 context deadline exceeded
error trying to connect to local peer
github.com/hyperledger/fabric/core/chaincode/shim.userChaincodeStreamGetter
        /opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/chaincode.go:111
github.com/hyperledger/fabric/core/chaincode/shim.Start
        /opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/shim/chaincode.go:150
main.main
        /chaincode/input/src/github.com/chaincode/alepomm/alepomm.go:355
runtime.main
        /opt/go/src/runtime/proc.go:195
runtime.goexit
        /opt/go/src/runtime/asm_amd64.s:2337
Error creating new Smart Contract: error trying to connect to local peer: context deadline exceeded

^^Ignore the timestamp and Ip here, the logs are from diff runs. But it happens the same everytime.

Here is my compose section for the peer.

  peer0:
    image: hyperledger/fabric-peer
    environment:
      - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
      # the following setting starts chaincode containers on the same
      # bridge network as the peers
      # https://docs.docker.com/compose/networking/
      - CORE_CHAINCODE_LOGGING_LEVEL=DEBUG
      - CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=fabric
      - CORE_LOGGING_LEVEL=DEBUG
      - CORE_PEER_TLS_ENABLED=true
      - CORE_PEER_GOSSIP_USELEADERELECTION=true
      - CORE_PEER_GOSSIP_ORGLEADER=false
      - CORE_PEER_PROFILE_ENABLED=true
      - CORE_PEER_TLS_CERT_FILE=/etc/hyperledger/fabric/tls/server.crt
      - CORE_PEER_TLS_KEY_FILE=/etc/hyperledger/fabric/tls/server.key
      - CORE_PEER_TLS_ROOTCERT_FILE=/etc/hyperledger/fabric/tls/ca.crt
      - CORE_LEDGER_STATE_STATEDATABASE=CouchDB
      - CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS=couchdb0:5984
      - CORE_LEDGER_STATE_COUCHDBCONFIG_USERNAME=
      - CORE_LEDGER_STATE_COUCHDBCONFIG_PASSWORD=
      - CORE_PEER_ID=peer0.org1.example.com
      - CORE_PEER_ADDRESS=peer0.org1.example.com:7051
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org1.example.com:7051
      - CORE_PEER_LOCALMSPID=Org1MSP
#      - CORE_PEER_ADDRESSAUTODETECT=true
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric/peer
    volumes:
        - /var/run/:/host/var/run/
        - ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/fabric/msp
        - ./crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls:/etc/hyperledger/fabric/tls
    ports:
     - 7051:7051
     - 7053:7053
    expose:
     - 7051
     - 7053
    command: peer node start
    depends_on:
      - couchdb0
    networks:
      fabric:
        aliases:
         - "peer0.org1.example.com"
    deploy:
      placement:
        constraints:
          - node.hostname == ip-172-31-22-132

This is really weird, if I remove the deploy section (which is stack specific) everything works.

My n/w is an overlay type n/w with scope of swarm.

Answer 1

@yacovm, thank you so much for the help! The issue was that the chaincode container was not launching on the same network as my peer container and hence it was not able to connect to it. To fix it,

CORE_PEER_CHAINCODELISTENADDRESS=0.0.0.0:7052

Was added to the env of my peer containers. Now it works like a charm