AWS Chatbot and EventBridge for Glue Job State Changes Error - Event received is not supported

Question

I am trying to set up the AWS Chatbot with Slack integrations to display error messages for changes in states (errors) for AWS Glue. I have set up AWS EventBridge event pattern to catch Glue Job State Changes as follows:

{
  "source": ["aws.glue"],
  "detail-type": ["Glue Job State Change"],
  "detail": {
    "state": [
      "FAILED"
    ]
  }
}

This successfully catches all failed Glue Jobs and I have set up an AWS SNS topic as the target using the input transformer.

Input Transformer Input Path

{"jobname":"$.detail.jobName","jobrunid":"$.detail.jobRunId","jobstate":"$.detail.state"}

Input Transformer Input Template

"{\"detail-type\": \"Glue Job <job-name> has entered the state <job-state> with the message <message>.\"}"

AWS SNS has a subscriptions endpoint to the AWS Chatbot which fails to send the notification to Slack.

AWS Chatbot CloudWatch logs after an event using Input Transformer

Event received is not supported (see https://docs.aws.amazon.com/chatbot/latest/adminguide/related-services.html ): 
{
    "subscribeUrl": null,
    "type": "Notification",
    "signatureVersion": "1",
    "signature": <signature>,
    "topicArn": <topic-arn>,
    "signingCertUrl": <signing-cert-url>,
    "messageId": <message-id>,
    "message": "{\"detail-type\": \"Glue Job MyJob has entered the state FAILED with the message SystemExit: None.\"}",
    "subject": null,
    "unsubscribeUrl": <unsubscribe-url>,
    "timestamp": "2022-03-02T12:17:16.879Z",
    "token": null
}

When the input is set to 'Matched Events' in the AWS EventBridge Select Target, the Slack Notification will send however it lacks any details.

Slack Notification

Glue Job State Change | eu-west-1 | Account: <account>
Glue Job State Change

AWS EventBridge Matched Events JSON Output

{
    "Type" : "Notification",
    "MessageId" : <message-id>,
    "TopicArn" : <topic-arn>,
    "Message" : "{\"detail-type\": [\"Glue Job State Change\"]}",
    "Timestamp" : "2022-03-02T11:17:52.443Z",
    "SignatureVersion" : "1",
    "Signature" : <signature>,
    "SigningCertURL" : <signing-cert-url>,
    "UnsubscribeURL" : <unsubscribe-url>
  }

There are very little differences between the two JSON outputs however the input transformer is considered an unsupported event. Is it possible to generate a custom message when using the AWS Chatbot for errors?

Answer 1

This example demonstrates how to send alerts using Google Chat webhooks. The same method applies for Slack webhooks.

from json import dumps
from urllib import request
import json

def handler(event, context):
    url = "https://chat.googleapis.com/v1/spaces/123123"
    
    print("Whole event printed: ")
    print(event)
    
    raw_SNS_message = event['Records'][0]['Sns']['Message']
    sns_message = json.loads(raw_SNS_message)

    formatted_message = ''
    
    try:         
        job_name = sns_message['detail']['jobName']
        severity = sns_message['detail']['severity']
        job_run_id = sns_message['detail']['jobRunId']
        message = sns_message['detail']['message']
        
        formatted_message = (f"AWS GLUE ALERT!\n"
                             f"Job Name: {job_name}\n"
                             f"Severity: {severity}\n"
                             f"Job Run ID: {job_run_id}\n"
                             f"Message: {message}")    
    except Exception as e:
        formatted_message = raw_SNS_message
        
    print(formatted_message)
    
    message = {
        "text": formatted_message,
        "thread": {"threadKey": "THREAD_KEY_VALUE"}
    }
    print("Request message content:")
    print(message)
    
    headers = {"Content-Type": "application/json; charset=UTF-8"}
    data = dumps(message).encode('utf-8')

    req = request.Request(url, data=data, headers=headers, method='POST')
    response = request.urlopen(req)

    print(response.read().decode('utf-8'))

Answer 2

I recently came across this issue.

The event not support error is because of two possible issues https://docs.aws.amazon.com/chatbot/latest/adminguide/chatbot-troubleshooting.html

• The notification's originating service is not supported by AWS Chatbot.
• The service event is modified.

You are using Glue which is supported.

Because you are modifying the event your Input Transformer Input Template needs to comply with the AWS chatbot custom notification schema https://docs.aws.amazon.com/chatbot/latest/adminguide/custom-notifs.html

For example, your template could look like below

{
  "version": "1.0",
  "source": "custom",
  "content": {
      "title": "Glue Job",
      "description": "Glue Job <job-name> has entered the state <job-state> with the message <message>."
  }
}

Answer 3

The best solution was to create a Lambda function as the target of the AWS EventBridge which performs a POST to a Slack Webhook.

# Import modules
import logging
import json
import urllib3

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)


# Define Lambda function
def lambda_handler(event, context):
    
    http = urllib3.PoolManager()
    
    url = <url>
    link = <glue-studio-monitoring-link>
    message = f"A Glue Job {event['detail']['jobName']} with Job Run ID {event['detail']['jobRunId']} has entered the state {event['detail']['state']} with error message: {event['detail']['message']}. Visit the link for job monitoring {link}"
    logger.info(message)
    headers = {"Content-type": "application/json"}
    data = {'text': message}
    response = http.request('POST',
                        url,
                        body = json.dumps(data),
                        headers = headers,
                        retries = False)
    logger.info(response.status)