CODEWITHSUNDEEP
hadoopmapreduceapache-pig
user3662937
user3662937

Reputation: 245

"Invalid scalar projection" error after doing a JOIN and then FILTER in Pig

I am using Pig 0.12.1. Why am I getting an error when I FILTER after doing a JOIN by LEFT OUTER?

Here is a modified example:

A = LOAD '$file1' USING PigStorage('\t') AS (idA:int, manufacturer:chararray);
B = LOAD '$file2' USING PigStorage('\t') AS (idB:int, price:float);
C = JOIN A BY idA LEFT OUTER, B BY idB;
D = FILTER C BY price > 2.0;

Why do I get this "Invalid scalar projection" error on D:

ERROR pig.PigServer: exception during parsing: Error during parsing. Pig script failed to parse:
Invalid scalar projection: D

Upvotes: 1

Views: 3470

Answers (2)

Rengasamy
Rengasamy

Reputation: 1043

Try this, You can access the price column by,

D = FILTER C BY $3 > 2.0;
or
D = FILTER C BY B::price > 2.0;

Upvotes: 3

o-90
o-90

Reputation: 17585

Try accounting for any NULLs in the relation.

D = FILTER C BY price > 2.0 AND price is not null;

Upvotes: 0

Related Questions