CODEWITHSUNDEEP
javaapache-storm
AbtPst
AbtPst

Reputation: 8008

Unable to Submit storm topology

I am trying to submit a Storm topology on a remote host using Eclipse.

Here is my code:

Config conf = new Config();
conf.setDebug(false);
conf.setNumWorkers(1);
conf.put(Config.NIMBUS_HOST, "hostName");
conf.put(Config.NIMBUS_THRIFT_PORT,6627);
conf.put(Config.STORM_ZOOKEEPER_SERVERS,Arrays.asList(new String[]{"hostName"}));
conf.put(Config.STORM_ZOOKEEPER_PORT,2181);

// Remote submission
StormSubmitter.submitTopology("classMain", conf, topology);

But I get this exception:

Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.TApplicationException: Binary field exceeded string size limit
  at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:250)
 at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:271)
  at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:157)
  at com.rbc.rbccm.hackathon.Countersearch.submitTopology(Countersearch.java:111)
  at com.rbc.rbccm.hackathon.Countersearch.main(Countersearch.java:37)
Caused by: org.apache.thrift7.TApplicationException: Binary field exceeded string size limit
  at org.apache.thrift7.TApplicationException.read(TApplicationException.java:111)
  at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:71)
  at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:184)
  at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:168)
  at backtype.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:236)
... 4 more

is there a string size limit on the parameters that we can pass to the submitTopology function?

When I follow the trail a bit more, it leads to:

public void submitTopology(String name, String uploadedJarLocation, String jsonConf, StormTopology topology) throws AlreadyAliveException, InvalidTopologyException, AuthorizationException, org.apache.thrift.TException
{
    send_submitTopology(name, uploadedJarLocation, jsonConf, topology);
    recv_submitTopology();
}

The recv is causing the issue. Any thoughts?

Upvotes: 2

Views: 1333

Answers (2)

Nav
Nav

Reputation: 20658

If you see the code in Storm's source code at StormSubmitter.java, it's this: 

public static void submitTopology(String name, Map stormConf, StormTopology topology)
        throws AlreadyAliveException, InvalidTopologyException, AuthorizationException {
    submitTopology(name, stormConf, topology, null, null);
}

The Thrift error is because either the name you specified is too long (more than 2MB?) or stormConf has too much of info or there's the more likely cause, which is that when the topology is created, you're filling the Spout or Bolt instance with too much info.

In my case, I was creating a bolt into which I was initializing too much data. 

builder.setBolt(genBolt, new GenBolt(myTable1.getHashMap(), myTable2.getHashMap(), myTable3.getHashMap(), myTable4.getHashMap()), 2)
  .fieldsGrouping(iterSpout, new Fields(con.BATCH_ID))

Upvotes: 0

Matthias J. Sax
Matthias J. Sax

Reputation: 62330

You need to increase nimbus.thrift.max_buffer_size parameter. You can either set it in storm.yaml or in Config object.

Upvotes: 2

Related Questions