CODEWITHSUNDEEP
apache-pig
YuliaPro
YuliaPro

Reputation: 305

Referencing field in nested tuple in PIG;

I have been stuck on this for several hours and I cannot figure out what I am doing wrong. I have a relation "grouped" with the schema of

    grouped: {seedword: chararray,baggy: {outertup: (groupy: (seedword: chararray,coword: chararray))}}

A sample of what the relation looks like is: (auto,{((auto,car)),((auto,truck))})

I need to generate just the seedword and a tuple of cowords. In my example I would want

(auto, (car, truck)).

I have tried:

 FOREACH grouped GENERATE baggy::outertup.groupy.coword;

 FOREACH grouped GENERATE baggy.outertup.groupy.coword;
 FOREACH grouped GENERATE baggy.groupy.coword;

and none of these work, and give me error messages saying there is no such field. Please help! !!

HEre's some more of my code:

keywords = LOAD 'merged' USING as ( seedword:chararray, doc:chararray);

---COUNT HOW MANY DOCUMENTS EACH WORD IS IN
group_by_seedword = GROUP keywords BY $0;

invert_index = FOREACH group_by_seedword GENERATE $0 as seedword:chararray, keywords.$1;
word_doc_count= FOREACH invert_index GENERATE seedword, COUNT($1);

-- map words to document
words_in_doc= GROUP keywords BY doc;
word_docs = FOREACH words_in_doc GENERATE group AS doc, keywords.seedword;
--(document:(keyword, keyword, keyword...))

--map words to their cowords in doc
temp_join = JOIN keywords BY doc,word_docs BY doc;
--DUMP temp_join;
cowords_by_doc = FOREACH temp_join GENERATE $0 as seedword:chararray, $3 as cowords;

cowords_interm=  FOREACH cowords_by_doc GENERATE seedword, FLATTEN(cowords);
cowords = FILTER cowords_interm BY (seedword!=$1);---GETS RID OF SINGLE DOC WORD; 
temp_join_count1 = JOIN cowords BY $0, word_doc_count BY seedword; 

-- GETS WORDS THAT OCCURE BY THEMSELVES IN A SINGLE DOCUMENT
G = JOIN cowords_interm BY $0 LEFT OUTER, cowords by $0;
orph_word = FILTER G BY $2 is null;
orph_word_count = FOREACH orph_word GENERATE $0,null, 0;

temp_join_count= UNION temp_join_count1, orph_word_count; 

inter_frac = FOREACH temp_join_count GENERATE $0 as seedword:chararray, $1 as coword:chararray, 1.0/$3 as frac:double;
inter_frac_combine = GROUP inter_frac BY (seedword, coword); 
inter_frac_sum = FOREACH inter_frac_combine GENERATE $0 , SUM(inter_frac.frac) as frac:double;

filtered = FILTER inter_frac_sum BY ($1 >=$relatedness_ratio);
grouped= GROUP filtered by $0.seedword;
g = FOREACH grouped GENERATE group as seedword:chararray, filtered.$0;   
named = FOREACH g GENERATE $0 as seedword:chararray, $1 as baggy:bag{(outertup:tuple(groupy:tuple(seedword:chararray, coword:chararray)))};

the input file you can try should be like this:

car doc1.txt
auto doc1.txt
bunny doc2.txt
ball doc2.txt
toy car doc2.txt
random doc3.txt

plane doc3.txt

Upvotes: 1

Views: 4723

Answers (1)

user2019983
user2019983

Reputation: 11

I'd had a similar issue where I couldn't reference inner tuples. My solution was to flatten the data and then some more filtering and grouping. Cheers V

Upvotes: 1

Related Questions