CODEWITHSUNDEEP
amazon-web-serviceshyperledger-fabricblockchainpeer
the_stack_over_flew
the_stack_over_flew

Reputation: 105

Hyperledger Fabric: Peer nodes fail to restart with byfn script when machine is shut down while network is running

I have a hyperledger fabric network running on a single AWS instance using the default byfn script.

ERROR: Orderer, cli, CA docker containers show "Up" status. Peers show "Exited" status.

Error occurs when:

  1. Byfn network is running, machine is rebooted (not in my control but because of some external reason).
  2. Network is left running overnight without shutting the machine. Shows same status next morning.

Error shown:

docker ps -a



CONTAINER ID        IMAGE                               COMMAND             CREATED             STATUS                      PORTS                    NAMES
b0523a7b1730        hyperledger/fabric-tools:latest     "/bin/bash"         23 seconds ago      Up 21 seconds                                        cli
bfab227eb4df        hyperledger/fabric-peer:latest      "peer node start"   28 seconds ago      Exited (2) 23 seconds ago                            peer1.org1.example.com
6fd7e818fab3        hyperledger/fabric-peer:latest      "peer node start"   28 seconds ago      Exited (2) 19 seconds ago                            peer1.org2.example.com
1287b6d93a23        hyperledger/fabric-peer:latest      "peer node start"   28 seconds ago      Exited (2) 22 seconds ago                            peer0.org2.example.com
2684fc905258        hyperledger/fabric-orderer:latest   "orderer"           28 seconds ago      Up 26 seconds               0.0.0.0:7050->7050/tcp   orderer.example.com
93d33b51d352        hyperledger/fabric-peer:latest      "peer node start"   28 seconds ago      Exited (2) 25 seconds ago                            peer0.org1.example.com



Attaching docker log: https://hastebin.com/ahuyihubup.cs
Only the peers fail to start up. Steps I have tried to solve the issue:

  1. docker start $(docker ps -aq) or manually, starting individual peers.
  2. byfn down, generate and then up again. Shows the same result as above.
  3. Rolled back to previous versions of fabric binaries. Same result on 1.1, 1.2 and 1.4. In older binaries, error is not repeated if network is left running overnight but repeats when machine is restarted.
  4. Used older docker images such as 1.1 and 1.2.
  5. Tried starting up only one peer, orderer and cli.
  6. Changed network name and domain name.
  7. Uninstalled docker, docker-compose and reinstalled.
  8. Changed port numbers of all nodes.
  9. Tried restarting without mounting any volumes.

The only thing that works is reformatting the AWS instance and reinstalling everything from scratch. Also, I am NOT using AWS blockchain template. Any help would be appreciated. I have been stuck on this issue for a month now.

Upvotes: 1

Views: 781

Answers (1)

the_stack_over_flew
the_stack_over_flew

Reputation: 105

Error resolved by adding following lines to peer-base.yaml:

GODEBUG=netdns=go
dns_search: .

Thanks to @gari-singh for the answer: https://stackoverflow.com/a/49649678/5248781

Upvotes: 1

Related Questions