CODEWITHSUNDEEP
hadoopapache-pig
Ironman
Ironman

Reputation: 1410

Invalid format: "19690321" is too short

I am trying to convert yyyyMMdd format to yyyy/MM/dd format using pig for that i have written below code.

Code:

STOCK_A = LOAD '/user/root/xxxx/*' USING PigStorage('|'); 
data = FILTER STOCK_A BY ($1 matches '.*ID.*'); 
MSH_DATA = FOREACH data GENERATE ToDate($8,'yyyy/MM/dd','UTC') AS dob;

When i am trying to dump the result i am getting below error.

ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(org.apache.pig.builtin.ToDate3ARGS)[datetime] - scope-209 Operator Key: scope-209) children: null at []]: java.lang.IllegalArgumentException: Invalid format: "19690321" is too short

Sample:

EXVORV@#PDULD21F|ID|1|483|1020783||EXVORV@#PDULD||19690321|F|

$8 seems valid to me i am not able to locate the reason the issue is coming. Any help would be really appreciated.

Upvotes: 0

Views: 2996

Answers (2)

nobody
nobody

Reputation: 11080

The issue is most likely because of the load statement.Since you are not specifying the schema the datatype by default will be bytearray. You will have to convert it to chararray before passing the field to ToDate

STOCK_A = LOAD '/user/root/xxxx/*' USING PigStorage('|'); 
data = FILTER STOCK_A BY ($1 matches '.*ID.*'); 
MSH_DATA = FOREACH data GENERATE ToDate((chararray)$8,'yyyy/MM/dd','UTC') AS dob;

Upvotes: 1

wargre
wargre

Reputation: 4753

You use :

ToDate($8,'yyyy/MM/dd','UTC')

but the format is 

19690321

so you should have

ToDate($8,'yyyyMMdd','UTC')

Upvotes: 1

Related Questions