CODEWITHSUNDEEP
performancehadoophive
alex_bondar
alex_bondar

Reputation: 442

Hive query getting stuck in the middle of execution

Colleagues,

I'm facing with an issue while execution of bash file with sql script in hive - it always stuck at the same place map=100%, reduce=67%

I tried to play with mappers and reducers number with different variations and other tunning characteristics:

SET hive.exec.parallel=true;
SET hive.default.fileformat=RCFILE;
SET hive.stats.autogather=false;
SET hive.exec.compress.output=true; 
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec; 
SET mapred.output.compression.type=BLOCK;
SET hive.input.format=org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;

But result still the same - it continuously return 67% of reducing progress.

Hive version - 0.13.0.2.X.X.X

SQL contains table creation from several source - simple join of several tables.

Any ideas how to tune this?

Any help will be appreciated.

Upvotes: 0

Views: 2524

Answers (2)

alex_bondar
alex_bondar

Reputation: 442

So, as found out there were several duplicated records and this was the cause why execution was stuck.

Upvotes: 0

WestCoastProjects
WestCoastProjects

Reputation: 63259

The phases of a Reducer are:

  • Shuffle
  • Sort
  • Reduce

Getting stuck @ 67% indicates that the Shuffle and Sort have completed but none of your partitions are able to succeed in the Reduce phase. The Reduce phase is your actual Reducer code. This indicates your code is unable to complete. You should examine your code and also look at the hive logs to see why your code is unable to be run.

Upvotes: 2

Related Questions