CODEWITHSUNDEEP
sqlpython-2.7boto3aws-cliamazon-athena
Rajeev
Rajeev

Reputation: 1061

AWS CLI Athena,Python - pass query programmatically

I am trying to run a set of Athena SQL statements through the AWS CLI commands and I need to know the following

The set of SQL queries are in a file and I need to sequentially iterate the --query-string in the following CLI command for each query through a python script

I can use FOR loop to get the sql commands from the file but I don't know how to parameterize it in AWS CLI command.

AWS CLI Command

aws athena start-query-execution \
   --query-string **"select count(*) from tablename;"** \
   --query-execution-context Database=test45 \
   --result-configuration OutputLocation=s3://test/ \
   --output text

Python Code

import boto3

client = boto3.client('athena')
fd = open('Athenacli.sql', 'r')
sqlFile = fd.read()
fd.close()

sqlCommands = sqlFile.split(';')
for command in sqlCommands:
    print command
    response = client.start_query_execution(
        QueryString=command,
        QueryExecutionContext={
            'Database': 'Test45'
        },
        ResultConfiguration={
            'OutputLocation': 's3://test/'
        }
    )

print response

File content for Athenacli.sql

select count(*) CNT from Test45.table1;

Error Message

Traceback (most recent call last):
     pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:/Python27/mypycode/CLIFrame.py", line 18, in <module>
    'OutputLocation': 's3://test/'
  File "C:\Users\AppData\Roaming\Python\Python27\site-packages\botocore\client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\AppData\Roaming\Python\Python27\site-packages\botocore\client.py", line 531, in _make_api_call
    api_params, operation_model, context=request_context)
  File "C:\Users\AppData\Roaming\Python\Python27\site-packages\botocore\client.py", line 586, in _convert_to_request_dict
    api_params, operation_model)
  File "C:\Users\AppData\Roaming\Python\Python27\site-packages\botocore\validate.py", line 291, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid length for parameter QueryString, value: 0, valid range: 1-inf

Process finished with exit code 1

Upvotes: 6

Views: 13385

Answers (2)

Frederic Henri
Frederic Henri

Reputation: 53713

#!/bin/bash
while IFS='' read -r sql || [[ -n "$sql" ]]; do
    aws athena start-query-execution \
    --query-string "$sql" \
    --query-execution-context Database=test45 \
    --result-configuration OutputLocation=s3://test/ \
    --output text
done < "$1"

You can call your file run_athena_query.sh:

$ chmod +x run_athena_query.sh
$ ./run_athena_query.sh <file_with_query>

Upvotes: 4

Bill King
Bill King

Reputation: 72

Not sure if this is your whole problem, but you are splitting on ";" so you will get two queries, the first:

select count(*) CNT from Test45.table1

and the second:

""

This might explain the error message:

Invalid length for parameter QueryString, value: 0, valid range: 1-inf

Do you get any output before the error?

Upvotes: 2

Related Questions