CODEWITHSUNDEEP
bashshellhive
Parthi
Parthi

Reputation: 41

Track success or failure for hive queries running through shell script

I am running few hive queries in a single shell script like below

#!/bin/bash  
HQLSource='/home/hql/'  
hive  -f $HQLSource/query1.hql               
hive  -f $HQLSource/query2.hql

query1.hql and query2.hql have statements to create a table and insert data - if anything fails, I would consider the whole .hql as failed. Can I write to a text file whether the queries are all successful or failed?

Expected Output: 

query1.hql - Success  
query2.hql - Failed

Upvotes: 3

Views: 6364

Answers (3)

kartik
kartik

Reputation: 2135

We can do two checks with in bash file to know status of hql execution.

  1. Return result should be equal to 0 else declare hql excution fail.

    beeline -u jdbc:hive2://IP:PORT -n USER -p PASSWORD -f hql_file.hql 2> ./hql_file.logs
query1result=$?
if [ $query1result -eq 0 ]; then
       # no issue
fi

  2. Scan generated log file for pattern Error/Exception, if pattern matches declare hql fail.

    for logfile in ./hql_file.logs; do
    errorFound="false"
    while read -r line; do
        # check for exception or error.Add grep search pattern based on need.
        excep=$(echo $line|grep "Error/|Exception")
        # if no space means no error or exception
        if [ -n "${excep##+([[:space:]])}" ]; then
            errorFound="true";                   
        fi
    done < "$logfile"
done

Upvotes: 0

Umer
Umer

Reputation: 25

I have checked and verified the code so use it without any doubt.

  1. Put each hive statement in a separate file e.g setup_step_4.hive
  2. In the shell script call this file. setup_step_4.hive load data local inpath 'nyse.csv' into table nyse_data;

code in shell script will look like this :

hive -f setup_step_3.hive
query1result=$?
echo $query1result
if [ "$query1result" -ne 0 ]
    then
      echo "  Error."
      exit 4
      else 
      echo "Done.Successfully"
    fi

hive -f setup_step_4.hive
query1result=$?
echo $query1result
if [ "$query1result" -ne 0 ]
    then
      echo "  Error"
      exit 4
      else 
      echo "Done.Successfully"
    fi

For example the step 3 executed successfully and the step 4 failed because file does not exists.

Output

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties OK

Time taken: 1.413 seconds

0

Done.Successfully

Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties OK

Time taken: 1.574 seconds

FAILED: SemanticException Line 3:23 Invalid path ''nyse.csv'': No files matching path file:/root/Data/nyse.csv

64

Error,

Upvotes: 0

miken32
miken32

Reputation: 42719

The exit code rom the previous command is stored in the variable $?. Any non-zero value is considered a failure of some kind.

#!/bin/bash  
HQLSource='/home/hql/'  
hive -f "$HQLSource/query1.hql"               
query1result=$?
hive -f "$HQLSource/query2.hql"               
query2result=$?
# do stuff with $query1result and $query2result

The exit code is also used by if statements to determine a code path, so you could also do this:

#!/bin/bash  
HQLSource='/home/hql/'  
if hive -f "$HQLSource/query1.hql"
then
    echo Query 1 succeeded
else
    echo Query 1 failed
fi
if hive -f "$HQLSource/query2.hql"
then
    echo Query 2 succeeded
else
    echo Query 2 failed
fi

Upvotes: 4

Related Questions