using TOKENIZE in PIG

Question

I am trying to use the TOKENIZE function in PIG with a document that is comma separated. I would like to split on the commas, but NOT on white space. For example I would like for a list of (car, toy car, bunny) to be ((car), (toy car), (bunny) not ((car), (toy), (car), (bunny)). Is there a way to this?

Answer 1

Alternative way,

You can try with Flatten operator as well

Example:

Input -> (a,(b,c))

B = foreach A generate $0 , flatten ($1)

Output -> (a,b,c)

Use of Flatten and tokenize together

You can read the word count problem Here

Answer 2

Have you had a look to STRSPLIT for splitting just on the comma?

(it works for CHARARRAY like TOKENIZE)