Hyperledger Fabric 2.0.1: Error: Failed to connect before the deadline on Discoverer- name:

Question

I think Fabric is trying to discover peers and orderers as soon as they come to realize they do exist but not matching the machine IP address. Is there a way to specify the machine IP override for TLS enabled communication?

This is the full log

2020-04-25T02:52:32.615Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Committer- name: orderer.xxxx.com:7050, url:grpcs://orderer.xxxx.com:7050
2020-04-25T02:52:32.616Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server orderer.xxxx.com:7050 url:grpcs://orderer.xxxx.com:7050 timeout:3000
2020-04-25T02:52:32.616Z - error: [DiscoveryService]: _buildOrderer[evm] - Unable to connect to the discovered orderer orderer.xxxx.com:7050 due to Error: Failed to connect before the deadline on Committer- name: orderer.xxxx.com:7050, url:grpcs://orderer.xxxx.com:7050
2020-04-25T02:52:35.624Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer0.org1.xxxx.com:7051, url:grpcs://peer0.org1.xxxx.com:7051
2020-04-25T02:52:35.624Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.org1.xxxx.com:7051 url:grpcs://peer0.org1.xxxx.com:7051 timeout:3000
2020-04-25T02:52:35.624Z - error: [DiscoveryService]: _buildPeer[evm] - Unable to connect to the discovered peer peer0.org1.xxxx.com:7051 due to Error: Failed to connect before the deadline on Endorser- name: peer0.org1.xxxx.com:7051, url:grpcs://peer0.org1.xxxx.com:7051
2020-04-25T02:52:38.626Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer1.org1.xxxx.com:8051, url:grpcs://peer1.org1.xxxx.com:8051
2020-04-25T02:52:38.627Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.org1.xxxx.com:8051 url:grpcs://peer1.org1.xxxx.com:8051 timeout:3000
2020-04-25T02:52:38.627Z - error: [DiscoveryService]: _buildPeer[evm] - Unable to connect to the discovered peer peer1.org1.xxxx.com:8051 due to Error: Failed to connect before the deadline on Endorser- name: peer1.org1.xxxx.com:8051, url:grpcs://peer1.org1.xxxx.com:8051
2020-04-25T02:52:41.630Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer0.org2.xxxx.com:9051, url:grpcs://peer0.org2.xxxx.com:9051
2020-04-25T02:52:41.630Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer0.org2.xxxx.com:9051 url:grpcs://peer0.org2.xxxx.com:9051 timeout:3000
2020-04-25T02:52:41.630Z - error: [DiscoveryService]: _buildPeer[evm] - Unable to connect to the discovered peer peer0.org2.xxxx.com:9051 due to Error: Failed to connect before the deadline on Endorser- name: peer0.org2.xxxx.com:9051, url:grpcs://peer0.org2.xxxx.com:9051
2020-04-25T02:52:44.631Z - error: [ServiceEndpoint]: Error: Failed to connect before the deadline on Endorser- name: peer1.org2.xxxx.com:10051, url:grpcs://peer1.org2.xxxx.com:10051
2020-04-25T02:52:44.631Z - error: [ServiceEndpoint]: waitForReady - Failed to connect to remote gRPC server peer1.org2.xxxx.com:10051 url:grpcs://peer1.org2.xxxx.com:10051 timeout:3000
2020-04-25T02:52:44.631Z - error: [DiscoveryService]: _buildPeer[evm] - Unable to connect to the discovered peer peer1.org2.xxxx.com:10051 due to Error: Failed to connect before the deadline on Endorser- name: peer1.org2.xxxx.com:10051, url:grpcs://peer1.org2.xxxx.com:10051
2020-04-25T02:52:44.647Z - error: [SingleQueryHandler]: evaluate: message=Query failed. Errors: [], stack=FabricError: Query failed. Errors: []
    at SingleQueryHandler.evaluate (/home/ubuntu/application-layer/node_modules/fabric-network/lib/impl/query/singlequeryhandler.js:45:23)
    at Transaction.evaluate (/home/ubuntu/application-layer/node_modules/fabric-network/lib/transaction.js:287:49)
    at Contract.evaluateTransaction (/home/ubuntu/application-layer/node_modules/fabric-network/lib/contract.js:115:45)
    at main (/home/ubuntu/application-layer/query.js:52:39)
    at processTicksAndRejections (internal/process/task_queues.js:97:5), name=FabricError
Failed to evaluate transaction: FabricError: Query failed. Errors: []

This is what I'm using to connect:

/*
 * SPDX-License-Identifier: Apache-2.0
 */

'use strict';

const { Gateway, Wallets } = require('fabric-network');
const path = require('path');
const fs = require('fs');


async function main() {
    try {
        // load the network configuration
        const ccpPath = path.resolve(__dirname, '..', 'first-network', 'connection-org1.json');
        const ccp = JSON.parse(fs.readFileSync(ccpPath, 'utf8'));

        // Create a new file system based wallet for managing identities.
        const walletPath = path.join(process.cwd(), 'wallet');
        const wallet = await Wallets.newFileSystemWallet(walletPath);
        console.log(`Wallet path: ${walletPath}`);

        // Check to see if we've already enrolled the user.
        const identity = await wallet.get('appUser');
        if (!identity) {
            console.log('An identity for the user "appUser" does not exist in the wallet');
            console.log('Run the registerUser.js application before retrying');
            return;
        }

        // Create a new gateway for connecting to our peer node.
        const gateway = new Gateway();
        await gateway.connect(ccp, { wallet: wallet, identity: 'appUser', discovery: { enabled: true, asLocalhost: false } });

        // Get the network (channel) our contract is deployed to.
        const network = await gateway.getNetwork('evm');

        // Get the contract from the network.
        const contract = network.getContract('evmxxx');

        // Evaluate the specified transaction.
        const result = await contract.submitTransaction('getEVMAddress');
        console.log(`Transaction has been evaluated, result is: ${result.toString()}`);

    } catch (error) {
        console.error(`Failed to evaluate transaction: ${error}`);
        process.exit(1);
    }
}

main();

This is the connection-org1.json file's contents

{
    "name": "first-network-org1",
    "version": "1.0.0",
    "client": {
        "organization": "Org1",
        "connection": {
            "timeout": {
                "peer": {
                    "endorser": "300"
                }
            }
        }
    },
    "organizations": {
        "Org1": {
            "mspid": "Org1MSP",
            "peers": [
                "peer0.org1.xxxx.com",
                "peer1.org1.xxxx.com"
            ],
            "certificateAuthorities": [
                "ca.org1.xxxx.com"
            ]
        }
    },
    "peers": {
        "peer0.org1.xxxx.com": {
            "url": "grpcs://xxx.xx.xx.x:7051",
            "tlsCACerts": {
                "pem": "-----BEGIN CERTIFICATE-----xxxx-----END CERTIFICATE-----\n"
            },
            "grpcOptions": {
                "ssl-target-name-override": "peer0.org1.xxxx.com",
                "hostnameOverride": "peer0.org1.xxxx.com",
            "grpc.keepalive_timeout_ms":80000,

        "grpc-max-send-message-length": -1,
        "grpc.keepalive_time_ms": 600000,
        "grpc.http2.min_time_between_pings_ms": 120000,
        "grpc.http2.max_pings_without_data": 0,
        "grpc.keepalive_permit_without_calls": 1

            }
        },
        "peer1.org1.xxxx.com": {
            "url": "grpcs://xxx.xx.xx.x:8051",
            "tlsCACerts": {
                "pem": "-----BEGIN CERTIFICATE-----xxxx-----END CERTIFICATE-----\n"
            },
            "grpcOptions": {
                "ssl-target-name-override": "peer1.org1.xxxx.com",
                "hostnameOverride": "peer1.org1.xxxx.com",
            "grpc.keepalive_timeout_ms":80000,
        "grpc-max-send-message-length": -1,
        "grpc.keepalive_time_ms": 600000,
        "grpc.http2.min_time_between_pings_ms": 120000,
        "grpc.http2.max_pings_without_data": 0,
        "grpc.keepalive_permit_without_calls": 1

            }
        }
    },
    "certificateAuthorities": {
        "ca.org1.xxxx.com": {
            "url": "https://xxx.xx.xx.x:7054",
            "caName": "ca-org1",
            "tlsCACerts": {
                "pem": "-----BEGIN CERTIFICATE-----xxxx-----END CERTIFICATE-----\n"
            },
            "httpOptions": {
                "verify": false
            }
        }
    }
}

I'm unable to make it connect, it does, however, work within the same Fabric machine using localhost instead of the provided example ip address as the grpcs url and setting the discovery options asLocalHost to true.

Thanks in advance.

EDIT

crypto-config.yaml looks as follows:

# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
#

# ---------------------------------------------------------------------------
# "OrdererOrgs" - Definition of organizations managing orderer nodes
# ---------------------------------------------------------------------------
OrdererOrgs:
  # ---------------------------------------------------------------------------
  # Orderer
  # ---------------------------------------------------------------------------
  - Name: Orderer
    Domain: example.com
    # ---------------------------------------------------------------------------
    # "Specs" - See PeerOrgs below for complete description
    # ---------------------------------------------------------------------------
    Specs:
      - Hostname: orderer
        SANS: 
          - "xxx.xx.x.x"
      - Hostname: orderer2
        SANS: 
          - "xxx.xx.x.x"
      - Hostname: orderer3
        SANS: 
          - "xxx.xx.x.x"
      - Hostname: orderer4
        SANS: 
          - "xxx.xx.x.x"
      - Hostname: orderer5
        SANS: 
          - "xxx.xx.x.x"

# ---------------------------------------------------------------------------
# "PeerOrgs" - Definition of organizations managing peer nodes
# ---------------------------------------------------------------------------
PeerOrgs:
  # ---------------------------------------------------------------------------
  # Org1
  # ---------------------------------------------------------------------------
  - Name: Org1
    Domain: Org1.example.com
    EnableNodeOUs: true
    # ---------------------------------------------------------------------------
    # "Specs"
    # ---------------------------------------------------------------------------
    # Uncomment this section to enable the explicit definition of hosts in your
    # configuration.  Most users will want to use Template, below
    #
    # Specs is an array of Spec entries.  Each Spec entry consists of two fields:
    #   - Hostname:   (Required) The desired hostname, sans the domain.
    #   - CommonName: (Optional) Specifies the template or explicit override for
    #                 the CN.  By default, this is the template:
    #
    #                              "{{.Hostname}}.{{.Domain}}"
    #
    #                 which obtains its values from the Spec.Hostname and
    #                 Org.Domain, respectively.
    # ---------------------------------------------------------------------------
    # Specs:
    #   - Hostname: foo # implicitly "foo.org1.example.com"
    #     CommonName: foo27.org5.example.com # overrides Hostname-based FQDN set above
    #   - Hostname: bar
    #   - Hostname: baz
    # ---------------------------------------------------------------------------
    # "Template"
    # ---------------------------------------------------------------------------
    # Allows for the definition of 1 or more hosts that are created sequentially
    # from a template. By default, this looks like "peer%d" from 0 to Count-1.
    # You may override the number of nodes (Count), the starting index (Start)
    # or the template used to construct the name (Hostname).
    #
    # Note: Template and Specs are not mutually exclusive.  You may define both
    # sections and the aggregate nodes will be created for you.  Take care with
    # name collisions
    # ---------------------------------------------------------------------------
    Template:
      Count: 2
      SANS:
        - "xxx.xx.x.x"
      # Start: 5
      # Hostname: {{.Prefix}}{{.Index}} # default
    # ---------------------------------------------------------------------------
    # "Users"
    # ---------------------------------------------------------------------------
    # Count: The number of user accounts _in addition_ to Admin
    # ---------------------------------------------------------------------------
    Users:
      Count: 1
  # ---------------------------------------------------------------------------
  # Org2: See "Org1" for full specification
  # ---------------------------------------------------------------------------
  - Name: Org2
    Domain: Org2.example.com
    EnableNodeOUs: true
    Template:
      Count: 2
      SANS: 
        - "xxx.xx.x.x"
    Users:
      Count: 1

With the same IP for all the SANS related fields.

Answer 1

4 points to check when this error occurs

1. If the inbound is properly opened for the services at destination server.

ipv4 --> 7050/tcp, 7051/tcp, 8051/tcp, 9051/tcp, 10051/tcp

This setup is with ref to the test-network.

2. Enable discovery service and change asLocalHost to false, if connecting from different machine.
3. like @Badr Bellaj said add all the destination services hostnames in source server(ubuntu)

sudo vi /etc/hosts

Add below in the hosts file,

<<dest-public_ip>> orderer.xxxx.com:7050
<<dest-public_ip>> peer0.org1.xxxx.com:7051
<<dest-public_ip>> peer1.org1.xxxx.com:8051
<<dest-public_ip>> peer0.org2.xxxx.com:9051
<<dest-public_ip>> peer1.org2.xxxx.com:10051

4. As @jsonph said double check the connection files for the service names, pem file/path, ip set for the listed services.

Answer 2

To solve this issue indicate dns resolution in your /etc/hosts

192.168.1.201 peer0.org1.example.com
192.168.2.211 peer0.org2.example.com
192.168.2.251 ca.org2.example.com
192.168.1.201 ca.org1.example.com
192.168.3.241 orderer.example.com
192.168.3.209 peer1.org1.example.com

Answer 3

I encountered this error in my project as well. I put the pem incorrectly in my connection_config.json (I put the path to the file instead of the actual key, LoL..)

After putting the pem correctly, it worked fine. So, maybe double-check the pem attributes of tlsCaCerts and make sure they are correct and in correct format. Hope this will help. Good luck!!!