CODEWITHSUNDEEP
amazon-web-servicesamazon-ec2user-data
AlexLordThorsen
AlexLordThorsen

Reputation: 8498

EC2 user-data Script Not Redirect `user-data.log` Output to Console

Description

Looking at this AWS EC2 Doc It should be possible to add exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 to a user-data script that is run on EC2 initialization.

When running aws ec2 --region eu-west-1 get-console-output --instance-id i-<id> | grep "user-data" (or searching for other patterns that should be present) none are found after the ec2 initialization.

Goal

To read the results and debug information from this initialization script without needing to SSH into the ec2 instance and poll the logs for the shutdown statement. Using the Instance shutdown as the "finished" state has a significant simplification on the deployment process for this repository.

Question

What about this particular setup is not correct such that we are not getting logs out of the aws ec2 get-console-output command.

Alternative answer: What's a better method of retrieving the logs from an EC2 instance.

User-Data Script

#!/bin/bash -xe
# redirect output to log file and console
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1

if [ -n "postgresql,jq" ]
then
        yum -y -q update
        echo "complete: yum update"
        IFS=', ' read -r -a REQS <<< "postgresql,jq"
        for REQ in "${REQS[@]}"
        do
                yum -y -q install $REQ
                echo "complete: yum install $REQ"
        done
fi

echo "EC2P: ENTERING BOOTSTRAP SCRIPT....."

export PGPASSWORD=$( aws secretsmanager get-secret-value --secret-id <secret_arn> --query SecretString --region eu-west-1 | jq -r . | jq -r .password)

aws s3 cp s3://<query_object> /tmp/postgres-query.sql

echo  "EC2P: Starting PSQL Execution"

psql    -h <host> \
                -p 5432 \
                -U <user> \
                -o /tmp/postgres-query-result.txt \
                <db_name>_db \
                < /tmp/postgres-query.sql \
                > /tmp/postgres-query-output.txt 2>&1

echo "psql exit code is $?"

echo  "EC2P: PSQL Execution Complete"


# since instance_initiated_shutdown_behavior = "terminate"
echo "shutdown"
shutdown

Terraform Declaration Of EC2 Instance

resource "aws_instance" "ec2_provisioner" {
  count                                = var.enabled ? 1 : 0
  ami                                  = data.aws_ami.ec2_provisioner.id
  iam_instance_profile                 = var.iam_instance_profile
  instance_initiated_shutdown_behavior = "terminate"
  instance_type                        = var.instance_type
  root_block_device {
    volume_type           = "gp2"
    volume_size           = "16"
    delete_on_termination = "true"
  }
  subnet_id = var.subnet_id
  tags = merge(
    {
      "es:global:component-name" = "${var.component_name}-ec2-provisioner",
      "Name"                     = var.name
    },
    jsondecode(var.additional_tags)
  )
  user_data = templatefile(
    "${path.module}/user_data.tpl",
    {
      BASH_SCRIPT = var.bash_script,
      PACKAGES    = join(",", var.packages)
    }
  )
  volume_tags = {
    "Name" = var.name
  }
  vpc_security_group_ids = [var.security_group_id]
}

Upvotes: 1

Views: 672

Answers (1)

AlexLordThorsen
AlexLordThorsen

Reputation: 8498

The core issue here is that our initialization of the instance is producing too much console output

74295 # > 64 KBs

Which is hitting a size limit built into the get-console-output command

By default, the console output returns buffered information that was posted shortly after an instance transition state (start, stop, reboot, or terminate). This information is available for at least one hour after the most recent post. Only the most recent 64 KB of console output is available

The solution we're going with to fix this issue to to enable SSM and to log into parse the log output.

Upvotes: -1

Related Questions