CODEWITHSUNDEEP
bashshellshaws-cli
arj
arj

Reputation: 983

Syntax error: EOF in backquote substitution shell script

I have shell script executing some Athena queries When i run the script ,it showing below error

Syntax error: EOF in backquote substitution

Script code

#!/bin/bash
date ;
ruby <filename>.rb ;
echo "delete existing <table name>";
aws configure set region us-east-1;
aws athena start-query-execution \
--query-string "DROP TABLE IF EXISTS <table name>" \
--work-group "primary" \
--query-execution-context Database=<database>\
--result-configuration "OutputLocation=s3://<path>/";
echo "create external table "
SQL="CREATE EXTERNAL TABLE <table name>( user_id string, file_name string, file_type string, count string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ("separatorChar" = ",", "quoteChar" = "`", "escapeChar" = "\\") STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 's3://<path>/\$client_id/'";
FINALSQL=$(echo $SQL | sed -e "s/\$client_id/$client_id/g");

echo "FINALSQL--> ${FINALSQL}";
aws athena start-query-execution \
--query-string "$FINALSQL"\
--work-group "primary" \
--query-execution-context Database=<dbname>\
--result-configuration "OutputLocation=s3://<path>/"
echo "generating report"
query="select  user_id,case file_type  when 'networkclick' then 'Click' when 'networkactivity' then 'Activity' when 'networkimpression' then 'Impression' when 'matchtables' then 'Sync Pixel'  else 'NA' end as file_type ,sum(cast(count as integer)) as count from gdpr.gdpr_access_result group by file_type,user_id";
output_location="s3://<path>/"
query_execution_id=$(aws athena start-query-execution \
    --region "$region" \
    --query-string "$query" \
    --result-configuration "OutputLocation=$output_location" \
    --query QueryExecutionId \
--output text)
echo "query_execution_id was $query_execution_id"

while true; do
    status=$(aws athena get-query-execution \
        --query-execution-id "$query_execution_id" \
        --query QueryExecution.Status.State \
    --output text)
    if [[ $status != 'RUNNING' ]]; then
        break
    else
        sleep 5
    fi
done

if [[ $status = 'SUCCEEDED' ]]; then
    result_location=$(aws athena get-query-execution \
        --query-execution-id "$query_execution_id" \
        --query QueryExecution.ResultConfiguration.OutputLocation \
    --output text)
    echo "result_location was $result_location"
    exec aws s3 cp "$result_location" -
else
    reason=$(aws athena get-query-execution \
        --query-execution-id "$query_execution_id" \
        --query QueryExecution.Status.StateChangeReason \
    --output text)
    echo "Query $query_execution_id failed: $reason" 1>&2;
    exit 1
fi
date ;

Am I missing something? I've spent some time for looking things.

Couldn't identify the issue yet. I have tried below command

bash -n <filename.sh>

getting two issues

enter image description here

How can I resolve it?

Upvotes: 0

Views: 2240

Answers (1)

choroba
choroba

Reputation: 241968

This is wrong:

SQL="CREATE EXTERNAL TABLE <table name>( user_id string, file_name string, file_type string, count string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde' WITH SERDEPROPERTIES ("separatorChar" = ",", "quoteChar" = "`", "escapeChar" = "\\") STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 's3://<path>/\$client_id/'";

Even the syntax highlighter here shows you the problem: double quotes don't nest. Moreover, ` inside double quotes still undergoes the command substitution expansion. Backslash the inner double quotes as well as the backquote.

Upvotes: 2

Related Questions