CODEWITHSUNDEEP
hadoopapache-pig
HM9527
HM9527

Reputation: 121

In pig, how do I count the number of lines that contained a specific string?

Suppose I have a group of target words:

a b c d

and an input file:

a d f s g e
12399
c a d i f
a 2

then I should return:

a 3
b 0
c 1
d 2

How can I do that in pig? Thank you!

Upvotes: 3

Views: 910

Answers (1)

Rahul Sharma
Rahul Sharma

Reputation: 5834

First remove the duplicate words from each line then run word count.
Pig steps:

REGISTER 'udf-1.0-SNAPSHOT.jar'
define tuple_set com.ts.pig.UniqueRecords();
data = load '<file>' using PigStorage();

remove duplicate words from each line

unique= foreach data generate tuple_set($0) as line;
words= foreach unique generate flatten(TOKENIZE(line,' ')) as word;
grouped = group words BY word;
count= foreach grouped GENERATE group, COUNT(words);
dump count;

Pig UDF sample code:

/**
 * This udf removes duplicate words from line
 */
public class UniqueRecords extends EvalFunc<String> {
    @Override
    public String exec(Tuple tuple) throws IOException {
        if (tuple == null || tuple.size() == 0)
            return null;
        String[] splits=tuple.get(0).toString().split(" ");
        Set<String> elements = new HashSet<String>(Arrays.asList(splits));
        StringBuilder sb = new StringBuilder();
        for(String element:elements ){
            sb.append(element+" ");
        }
        return sb.toString();
    }
}

Upvotes: 1

Related Questions