CODEWITHSUNDEEP
javaapache-stormtopology
Judking
Judking

Reputation: 6381

Why can't I write result to a file in Bolt when using Storm in distributed mode? works fine in LocalCluster

I changed the class WordCount in WordCountTopology as follows:

public static class WordCount extends BaseBasicBolt {
    Map<String, Integer> counts = new HashMap<String, Integer>();

    @Override
    public void execute(Tuple tuple, BasicOutputCollector collector) {
        String word = tuple.getString(0);
        Integer count = counts.get(word);
        if(count==null) count = 0;
        count++;
        counts.put(word, count);
        OutputStream o;
        try {
            o = new FileOutputStream("~/abc.txt", true);
            o.write(word.getBytes());
            o.close();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        collector.emit(new Values(word, count));
    }

    @Override
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("word", "count"));
    }
}

in which I write the word to file abc.txt.

When I ran the WordCountTopology in local mode(which used the LocalCluster), it just works fine. But when running in distributed mode(which used the StormSubmitter.submitTopology() method), the WordCount class didn't write the word to abc.txt as if the execute() method has not run at all. Could anyone give me some idea? Thanks a lot!

P.S. I'm sure my nimbus, supervisor, ui, zookeeper is running normally, and I can see the task in 127.0.0.1:8080.

Upvotes: 5

Views: 2906

Answers (1)

user2435082
user2435082

Reputation: 295

The main problem is location of abc.txt file.This file will be created in the system from where you are submitting the topology.So this file wont be available in other cluster machines.You can check supervisors log for file not found error.To resolve this issue you need some NFS configuration through which common location can be shared by all cluster machines.After configuring NFS create new file in common location so that this file wound be available for all supervisors.

Upvotes: 3

Related Questions