CODEWITHSUNDEEP
apache-pig
Harish Swamy
Harish Swamy

Reputation: 1

Pig Latin Word Count

I am trying to count number of lines that contain the following words: 'jack', 'hack', 'mat', 'throttle' in a pig script. I am using Cloudera quickstart vm.

The input file is:

09-jack-17,5:00PM;#slowmotion,Tribune Logic hack: how is life in temrs of money Creative hack

14-June-18,7:15PM;#Indiacalling,Horton-NJ Strategic/Halloween One World at Application Deployment

12-jack-16,jfh:er;#temporary, accomodation, osteoporosis, juxtapose, don't misinterpret this awaiting throttle jack

The output should be: hack 2 jack 2 throttle 1 mat 0 I am unable to extract those words and calculate it's count. What should I do? I tried the following script which was given by inquisitive_mind:

A = LOAD 'Input.txt'AS(line: chararray);
SPLIT A INTO M IF line matches'hackathon,N IF line matches'dec', O IF line matches'chicago',P IF line matches'java';
M1 = GROUP M ALL;
M2 = FOR EACH M1 GENERATE COUNT(M);
M3 = FOREACH M2 GENERATE CONCAT('hackathon',(chararray)M2.$0);
N1 = GROUP N ALL;
N2 = FOREACH N1 GENERATE COUNT(N);
N3 = FOREACHN2 GENERATE CONCAT('dec',(chararray)N2.$0);
O1 = GROUP O ALL;
O2 = FOREACH O1 GENERATE COUNT(O);
O3 = FOR EACH O2 GENERATE CONCAT('chicago',(chararray)O2.$0);
P1 = GROUP P ALL;
P2 = FOR EACH P1 GENERATE COUNT(P);
P3 = FOREACH P2 GENERATE CONCAT('java',(chararray)P2.$0);
DUMP M3;
DUMP N3;
DUMP O3;
DUMP P3;

But when I run it in mapreduce or local mode I get the following error:

2016-10-11 09:43:44,084 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Lexical error at line 19, column 0. Encountered: after : "" Details at logfile: /home/cloudera/pig_1476204218406.log

Here is the logfile:

Pig Stack Trace

ERROR 1000: Error during parsing. Lexical error at line 19, column 0.  Encountered: <EOF> after : ""

org.apache.pig.tools.pigscript.parser.TokenMgrError: Lexical error at line 19, column 0.  Encountered: <EOF> after : ""
    at org.apache.pig.tools.pigscript.parser.PigScriptParserTokenManager.getNextToken(PigScriptParserTokenManager.java:3326)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.jj_ntk(PigScriptParser.java:1379)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:106)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:613)
    at org.apache.pig.Main.main(Main.java:158)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Upvotes: 0

Views: 1284

Answers (1)

nobody
nobody

Reputation: 11080

If the words are fixed and small in number you can use split and count the lines.

A = LOAD 'data.txt' AS (line:chararray);
SPLIT A INTO M IF line matches 'jack', N IF line matches 'hack',O IF line matches 'throttle', P IF line matches 'mat';

M1 = GROUP M ALL;
M2 = FOREACH M1 COUNT(M); 
M3 = FOREACH M2 GENERATE CONCAT('jack ',(chararray)M2.$0); 

N1 = GROUP N ALL;
N2 = FOREACH N1 COUNT(N); 
N3 = FOREACH N2 GENERATE CONCAT('hack ',(chararray)N2.$0);

O1 = GROUP O ALL;
O2 = FOREACH O1 COUNT(O); 
O3 = FOREACH O2 GENERATE CONCAT('throttle ',(chararray)O2.$0);

P1 = GROUP P ALL;
P2 = FOREACH P1 COUNT(P); 
P3 = FOREACH P2 GENERATE CONCAT('mat ',(chararray)P2.$0);

DUMP M3;
DUMP N3;
DUMP O3;
DUMP P3;

Upvotes: 0

Related Questions