Max Ferreira
Max Ferreira

Reputation: 719

How to Pass SQS Message Body as Environment Variables to AWS Batch Containers Using EventBridge

I am configuring a workflow where messages from an SQS queue trigger AWS Batch jobs, and the content of these messages is passed as environment variables to the container executing the job. I am using AWS EventBridge Pipes to connect the SQS queue to AWS Batch.

enter image description here

# 📌 EventBridge Pipes - Ligação SQS -> AWS Batch
resource "aws_pipes_pipe" "pipe" {
  name     = "sqs-to-batch-pipe"
  role_arn = aws_iam_role.pipes.arn

  source = aws_sqs_queue.queue.arn
  target = aws_batch_job_queue.queue.arn

  source_parameters {
    sqs_queue_parameters {
      batch_size = 1
    }
  }

  target_parameters {
    batch_job_parameters {
        job_definition = aws_batch_job_definition.job.arn
        job_name       = "my-batch-job"
        parameters = {
            MESSAGE = "$.body" <<<<< I can't send the message to the container
        }
    }
  }
}

Here is the configuration I am using:

  1. SQS Queue: I have an SQS queue named my-batch-queue that receives JSON messages.
  2. AWS Batch Job Definition: I created an AWS Batch job definition (batch-job-definition) specifying a Docker container hosted on Amazon ECR. Within this definition, I configured an environment variable named MESSAGE with a default value.
  3. EventBridge Pipe: I set up an EventBridge pipe with the SQS queue as the source and the AWS Batch job queue (batch-job-queue) as the target. In the target_parameters of the pipe, I configured batch_job_parameters with parameters to pass the SQS message body as the value of the MESSAGE environment variable.

In my Node.js application, I am attempting to access the MESSAGE environment variable as follows:

console.log("Received message:", process.env.MESSAGE);

However, the displayed value is always 'default_value', indicating that the SQS message body is not being correctly passed to the container.

Question:

How can I configure EventBridge Pipes to correctly pass the content of SQS messages as environment variables to the AWS Batch container? Is there an appropriate way to override the MESSAGE variable defined in the aws_batch_job_definition with the SQS message content when the job is triggered by EventBridge Pipes?

Additional Details:

I understand that AWS Batch allows defining environment variables in the job_definition, but I am unable to override the MESSAGE variable with the SQS message content at runtime.

I have attempted to use the parameters attribute in the aws_pipes_pipe, but have not been successful in passing the SQS message content to the container.

Any guidance or configuration examples would be greatly appreciated.

Upvotes: -1

Views: 42

Answers (0)

Related Questions