Hyperledger Fabric Nodejs SDK Eventhub has been shutdown

Question

I have deployed the sample fabcar network on IBM Blockchain platform and am using the code from this repository to interact with it. The interaction works great when I call every function individually.

The problem comes when I develop a node server and expose APIs for the invoke chaincode. Once I do that, I start getting the error

Failed to invoke successfully :: Error: There was a problem with the eventhub ::Error: EventHub has been shutdown

Curious thing is that when I send 3 requests simultaneously to the POST call, I get this error. But the record gets inserted on IBM cloud.

When I send 100 requests simultaneously, only around 20 get inserted in the cloud and the rest show this error and an additional error like this

error: [Orderer.js]: sendBroadcast - on error: "Error: 14 UNAVAILABLE: Connect Failed at createStatusError (/home/trs-laptop-20/Downloads/fabcar-network/node_modules/fabric-client/node_modules/grpc/src/client.js:64:15) at ClientDuplexStream._emitStatusIfDone (/home/trs-laptop-20/Downloads/fabcar-network/node_modules/fabric-client/node_modules/grpc/src/client.js:270:19) at ClientDuplexStream._readsDone (/home/trs-laptop-20/Downloads/fabcar-network/node_modules/fabric-client/node_modules/grpc/src/client.js:236:8) at readCallback (/home/trs-laptop-20/Downloads/fabcar-network/node_modules/fabric-client/node_modules/grpc/src/client.js:296:12)"

And this in some cases

error: [client-utils.js]: sendPeersProposal - Promise is rejected: Error: 14 UNAVAILABLE: Connect Failed at new createStatusError (/home/trs-laptop-20/Downloads/fabcar-network/node_modules/fabric-client/node_modules/grpc/src/client.js:64:15) at /home/trs-laptop-20/Downloads/fabcar-network/node_modules/fabric-client/node_modules/grpc/src/client.js:583:15

I can not understand why these errors are showing. I have to develop a high-performance app that can handle load, any help and guidance would be really appreciated.

This is my app.js code:

'use strict';

//get libraries
const express = require('express');
const bodyParser = require('body-parser');
const request = require('request');
const path = require('path');

//create express web-app
const app = express();
const router = express.Router();
app.use(bodyParser.json());
const invoke = require('./invokeNetwork.js');

//declare port
var port = process.env.PORT || 8000;
if (process.env.VCAP_APPLICATION) {
  port = process.env.PORT;
}

//run app on port
app.listen(port, function() {
  console.log('app running on port: %d', port);
});

app.post('/api/post', async function(req,res){
    var carID = req.body.carID;
    var make = req.body.make;
    var model = req.body.model;
    var color = req.body.color;
    var owner = req.body.owner;

  await invoke.invokeCreate(carID, make, model, color, owner).then((response) => {
    //return error if error in response
    if (response.status == 500) {
      res.status(500).send({ error: response.message });
    } else {
      //else return success
      res.status(200).send({ message: response.message });
    }
  }).catch(err => {
    console.log(err);
  });

});

And this is the invokeNetwork.js code

'use strict';

var Fabric_Client = require('fabric-client');
var path = require('path');
var util = require('util');
var os = require('os');
var fs = require('fs');

var isError = false;
var throwError = "";

//make sure we have the profiles we need
var networkConfig = path.join(__dirname, './config/network-profile.json')
var clientConfig = path.join(__dirname, './config/client-profile.json');

module.exports = {

invokeCreate: async function(carID, make, model, color, owner) {

  isError = false;

//try {

  checkProfilesExist(networkConfig, clientConfig); //terminates early if they are not found

// load the base network profile
var fabric_client = Fabric_Client.loadFromConfig(path.join(__dirname, './config/network-profile.json'));

// overlay the client profile over the network profile
fabric_client.loadFromConfig(path.join(__dirname, './config/client-profile.json'));

// setup the fabric network - get the channel that was loaded from the network profile
var channel = fabric_client.getChannel('defaultchannel');
var tx_id = null;

//load the user who is going to unteract with the network
fabric_client.initCredentialStores().then(() => {
  // get the enrolled user from persistence, this user will sign all requests
  return fabric_client.getUserContext('user1', true);
}).then((user_from_store) => {
  if (user_from_store && user_from_store.isEnrolled()) {
    console.log('Successfully loaded user1 from persistence');

  } else {
    throw new Error('Failed to get user1.... run registerUserNetwork.js');
  }

  // get a transaction id object based on the current user assigned to fabric client
  tx_id = fabric_client.newTransactionID();
  console.log("Assigning transaction_id: ", tx_id._transaction_id);

  // createCar chaincode function - requires 5 args, ex: args: ['CAR11', 'Honda', 'Accord', 'Black', 'Dave'],
  // changeCarOwner chaincode function - requires 2 args , ex: args: ['CAR11', 'MGK'],
    var request = {
        chaincodeId: 'fabcar',
        fcn: 'createCar',
        args: [carID, make, model, color, owner],
        txId: tx_id
  };

  // send the transaction proposal to the endorsing peers
  return channel.sendTransactionProposal(request);
}).then((results) => {
  var proposalResponses = results[0];
  var proposal = results[1];
  let isProposalGood = false;
  if (proposalResponses && proposalResponses[0].response &&
    proposalResponses[0].response.status === 200) {
    isProposalGood = true;
    console.log('Transaction proposal was good');
  } else {
    console.error('Transaction proposal was bad');

  }
  if (isProposalGood) {
    console.log(util.format(
      'Successfully sent Proposal and received ProposalResponse: Status - %s, message - "%s"',
      proposalResponses[0].response.status, proposalResponses[0].response.message));

    // build up the request for the orderer to have the transaction committed
    var request = {
      proposalResponses: proposalResponses,
      proposal: proposal
    };

    // set the transaction listener and set a timeout of 30 sec
    // if the transaction did not get committed within the timeout period,
    // report a TIMEOUT status
    var transaction_id_string = tx_id.getTransactionID(); //Get the transaction ID string to be used by the event processing
    var promises = [];

    var sendPromise = channel.sendTransaction(request);
    promises.push(sendPromise); //we want the send transaction first, so that we know where to check status

    // get an eventhub once the fabric client has a user assigned. The user
    // is required bacause the event registration must be signed
    console.error('Getting event hub');
    let event_hub = fabric_client.getEventHub('org1-peer1');

    // using resolve the promise so that result status may be processed
    // under the then clause rather than having the catch clause process
    // the status
    let txPromise = new Promise((resolve, reject) => {
      let handle = setTimeout(() => {
        event_hub.disconnect();
        resolve({ event_status: 'TIMEOUT' }); //we could use reject(new Error('Trnasaction did not complete within 30 seconds'));
      }, 3000);
      event_hub.connect();
      event_hub.registerTxEvent(transaction_id_string, (tx, code) => {
        // this is the callback for transaction event status
        // first some clean up of event listener
        clearTimeout(handle);
        event_hub.unregisterTxEvent(transaction_id_string);
        event_hub.disconnect();

        // now let the application know what happened
        var return_status = { event_status: code, tx_id: transaction_id_string };
        if (code !== 'VALID') {
          console.error('The transaction was invalid, code = ' + code);
          resolve(return_status); // we could use reject(new Error('Problem with the tranaction, event status ::'+code));
        } else {
          console.log('The transaction has been committed on peer ' + event_hub._ep._endpoint.addr);
          resolve(return_status);
        }
      }, (err) => {
        //this is the callback if something goes wrong with the event registration or processing
        reject(new Error('There was a problem with the eventhub ::' + err));
        throwError += err;
      });
    });
    promises.push(txPromise);

    return Promise.all(promises);
  } else {
    console.error('Failed to send Proposal or receive valid response. Response null or status is not 200. exiting...');
    throw new Error('Failed to send Proposal or receive valid response. Response null or status is not 200. exiting...');
  }
}).then((results) => {
  console.log('Send transaction promise and event listener promise have completed');
  // check the results in the order the promises were added to the promise all list
  if (results && results[0] && results[0].status === 'SUCCESS') {
    console.log('Successfully sent transaction to the orderer.');
  } else {
    console.error('Failed to order the transaction. Error code: ' + response.status);
    throw new exception("Transaction");
  }

  if (results && results[1] && results[1].event_status === 'VALID') {
    console.log('Successfully committed the change to the ledger by the peer');
    return true;
  } else {
    console.log('Transaction failed to be committed to the ledger due to ::' + results[1].event_status);
  }
}).catch((err) => {
  console.error('Failed to invoke successfully :: ' + err);
  isError = true;
  throwError += err;
});

console.log(isError);

return {
  message: (isError) ? throwError : 'Success Transaction'
}

function checkProfilesExist(networkConfig, clientConfig) {
  if (!fs.existsSync(networkConfig)) {
    console.log("Error: config file 'network-profile.json' not found.");
    console.log("Make sure 'network-profile.json' is copied into the './config' folder.");
    process.exit()
  }

  //make sure we have the client profile we need

  if (!fs.existsSync(clientConfig)) {
    console.log("Error: config file 'client-profile.json' not found.");
    console.log("Make sure 'client-profile.json' is copied into the './config' folder.");
    process.exit()
  }
}

}

}
//throw new exception("Exceoption thrown");
//return true;
// }
//   catch(err) {
//     //print and return error
//     console.log(err);
//     var error = {};
//     error.error = err.message;
//     return error;
//     }
//   }
// }

Priyav Shah · Accepted Answer

So, we've been playing around with fabric as well trying to test out some of the performance. I had all the same issues that you dealt with, so will explain our issues and hopefully they will help you.

Eventhub shutdown

Error: There was a problem with the eventhub ::Error: EventHub has been shutdown

Problem

Noticed this would come up when we sent a large number of transactions, and also would come up when we were using a dodgy internet connection. The fabric docs state:

The events are ephemeral, such that if a registered listener crashed when the event is published, the listener will miss the event

I believe that the issue was the eventhub was trying to connect to our API, but there was an issue in connection. This then causes the eventhub to timeout (since it doesn't receive a response), which then causes the eventhub to shutdown.

Solution

Again fabric docs recommend either:

listening out on a block level, so even if you miss a block, you can fetch old blocks (more info in fabric docs link above)
Build your own queue

check out this pastebin for a basic block listener -> https://pastebin.com/wCd6Ni46

Error: 14 UNAVAILABLE

Problem

We noticed when we started to send a high volume of transactions, we would also end up with a Error: 14 UNAVAILABLE, but this was after really flooding the network with alot of transactions. I think we were ok with 20 at a batch, but when we sent it 50 or more in quick succession, it couldn't handle it.

Remember blocks, at least on IBP starter addition, are limited to 10 transcations a block, so it needs to wait for that to be mined before it can process the next set of transactions. In this process, I believe the buffer becomes filled and then you cant send too many.

So your two Errors are with the orderer getting flooded or the peer getting flooded with too many transactions.

Solution

Batch up transactions into blocks, and wait until those transactions are confirmed before sending of some more. This gives your peers and orderers some space to breathe.

Hopefully this helps!

Hyperledger Fabric Nodejs SDK Eventhub has been shutdown

Answers (2)

Eventhub shutdown

Problem

Solution

Error: 14 UNAVAILABLE

Problem

Solution

Related Questions