CODEWITHSUNDEEP
apache-pig
hese
hese

Reputation: 3417

Using IN clause with PIG FILTER

Does PIG support IN clause?

filtered = FILTER bba BY reason not in ('a','b','c','d');

or should i split it up into multiple OR's?

Thanks!

Upvotes: 5

Views: 21604

Answers (6)

Tarivs
Tarivs

Reputation: 19

you can do this likes:

X = FILTER bba BY NOT reason IN ('a','b','c','d');

more info

Upvotes: 1

Prashant_M
Prashant_M

Reputation: 3134

We can use IN clause as follows:

A = FILTER alias_name BY col_name IN (val1, val2,...,valn);

DUMP A;

Upvotes: 1

seanv507
seanv507

Reputation: 1306

Pig 0.12 added In operator http://www.edureka.co/blog/operators-in-apache-pig-diagnostic-operators/ see bottom of page..release notes. Haven't located it in official docs (apart from bare mention in release notes)

Upvotes: 3

kris
kris

Reputation: 171

You can use below udf from Apache DataFu instead. This will help you to avoid writing lot of OR.

https://github.com/linkedin/datafu/blob/master/src/java/datafu/pig/util/InUDF.java

Upvotes: 6

Tapan Avasthi
Tapan Avasthi

Reputation: 425

No, Pig doesn't support IN Clause. I had a similar situation. Though you can use AND operator and filter keyword as a work around. like

A= LOAD 'source.txt' AS (user:chararray, age:chararray);

B= FILTER A BY ($1 matches 'tapan') AND ($1 matches 'superman');

However, if the number of filtering required is huge. Then, probably, you can just create a relation that contains all these keywords and do a join to filter wherever the occurrence matches. Hope this helps.

Upvotes: 1

Icarus
Icarus

Reputation: 63970

I didn't find it in any of the samples in the documentation.

You can get by using AND/OR/NOT

Upvotes: 2

Related Questions