CODEWITHSUNDEEP
hadoopamazon-ec2hiveamazon-emr
pavan kumar
pavan kumar

Reputation: 111

EMR 5.16.0 - Too many Mappers being launched

Trying to upgrade EMR EC2 instance from 5.3 C4.4Xlarge(HIVE ver 2.1.1) to EMR 5.16.0 C5.4XLarge (Hive 2.3.3) My simple query select max(date) from table in 2.1.1 EMR 5.3.0 launches just 1 mapper and 1 reducer where as when I upgrade and run the same query it is launching 1300+ mappers.

Tried to compare the set properties of both and found no difference.

Could someone help me here.

Upvotes: 1

Views: 70

Answers (1)

Jiten Pai
Jiten Pai

Reputation: 26

Try the setting below. Somewhere along the way from hive 2.1.1 to 2.3.3, they flipped it from true to false.

set hive.optimize.metadataonly=true;

Upvotes: 1

Related Questions