how to view aws log real time (like tail -f)

Question

I can view the log using the following command.

aws logs get-log-events --log-group-name groupName --log-stream-name streamName --limit 100

what is the command to get feature like tail -f so that i can see the log real time

Answer 1

For example, I use the command

aws logs tail group_name --profile abc --follow --since 20m --filter-pattern "{ $.level = 50 }" --format json

to tail all messages with level=50. The result looks like this:

2024-12-19T09:22:56.057000+00:00 group_name 
{
    "level": 50,
    "time": 1734600176057,
    "pid": 1,
    "hostname": "...",
    "group": "error",
    "err": {
        "type": "HttpException",
        "message": "Forbidden",
        "stack": "Error: Forbidden\n    at Object.findUserByAuth (/api/src/middleware/utils.js:49:11)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async apiAuth (/api/src/middleware/apiAuth.middleware.js:7:16)",
        "status": 403
    },
    "msg": "Forbidden"
}
2024-12-19T09:33:09.788000+00:00 group_name 
{
    "level": 50,
    "time": 1734600789788,
    "pid": 1,
    "hostname": "...",
    "group": "error",
    "err": {
        "type": "HttpException",
        "message": "Forbidden",
        "stack": "Error: Forbidden\n    at Object.findUserByAuth (/api/src/middleware/utils.js:49:11)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async apiAuth (/api/src/middleware/apiAuth.middleware.js:7:16)",
        "status": 403
    },
    "msg": "Forbidden"
}

All options can be found here: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/tail.html

Answer 2

Amazon recently announced Live tail option to see the real time streaming experience to monitor logs as they occur. Live tail is focused on real-time monitoring and troubleshooting, while log groups are more about log storage, organization, and analysis over a longer period.

Navigate to AWS CloudWatch and select Live tail option under logs.

Answer 3

AWS launched a console page specifically to support near real time tailing across multiple logGroups and across linked accounts checkout Cloud watch Live tail

Answer 4

After checking many options and testing some custom tools aws logs tail worked the best for me.

Here there is a simple example command:

aws logs tail <GROUP_NAME> --follow

and here is the official doc which was very useful:

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/logs/tail.html

Answer 5

AWS allows you to tail the logs now. Exactly like tail -f. use the following command

aws logs tail <log group name> --follow

E.g. if you are using ElasticBeanStalk with app name myapp-prd and want to tail web1.log it would be

aws logs tail /aws/elasticbeanstalk/myapp-prd/var/log/web-1.log --follow

Answer 6

Because CloudWatch logs can be delayed (i.e. not "realtime" by precise definition) you parse the previous events for the last timestamp and start the next iteration there. This script uses aws logs get-log-events for which you must specify a valid stream_name.

#!/bin/bash
    
group_name='<log-group-name>'
stream_name='<log-stream-name>'
start_seconds_ago=300

start_time=$(( ( $(date -u +"%s") - $start_seconds_ago ) * 1000 ))
while [[ -n "$start_time" ]]; do
    loglines=$(aws logs get-log-events --log-group-name "$group_name" --log-stream-name "$stream_name" --start-time $start_time --output text)
    [ $? -ne 0 ] && break
      next_start_time=$( sed -nE 's/^EVENTS.([[:digit:]]+).+$/\1/ p' <<< "$loglines" | tail -n1 )
    [ -n "$next_start_time" ] && start_time=$(( $next_start_time + 1 ))
    echo "$loglines"
    sleep 15
done

Or if you want to tail an entire log group, this script uses aws logs filter-log-events without a stream name:

#!/bin/bash

group_name='<log-group-name>'
start_seconds_ago=300
  
start_time=$(( ( $(date -u +"%s") - $start_seconds_ago ) * 1000 ))
while [[ -n "$start_time" ]]; do
    loglines=$(aws logs filter-log-events --log-group-name "$group_name" --interleaved --start-time $start_time --output text)
    [ $? -ne 0 ] && break
    next_start_time=$( sed -nE 's/^EVENTS.([^[:blank:]]+).([[:digit:]]+).+$/\2/ p' <<< "$loglines" | tail -n1 )
    [ -n "$next_start_time" ] && start_time=$(( $next_start_time + 1 ))
    echo "$loglines"
    sleep 15
done

I've also put up the scripts that I use as GitHub gists: https://gist.github.com/tekwiz/964a3a8d2d84ff4c8b5288d9a703fbce.

Warning: the above code & scripts are written for my macOS system which is customized (bastardized??) with Homebrew and GNU coreutils, so some command options may need to be tweaked for your system. Edits are welcome :)

Answer 7

Here's a bash script that you can use. The script requires the AWS CLI and jq.

#!/bin/bash

# Bail out if anything fails, or if we do not have the required variables set
set -o errexit -o nounset

LOG_GROUP_NAME=$1
LOG_BEGIN=$(date --date "${2-now}" +%s)
LOG_END=$(date --date "${3-2 minutes}" +%s)
LOG_INTERVAL=5
LOG_EVENTIDS='[]'

while (( $(date +%s) < $LOG_END + $LOG_INTERVAL )); do
  sleep $LOG_INTERVAL
  LOG_EVENTS=$(aws logs filter-log-events --log-group-name $LOG_GROUP_NAME --start-time "${LOG_BEGIN}000" --end-time "${LOG_END}000" --output json)
  echo "$LOG_EVENTS" | jq -rM --argjson eventIds "$LOG_EVENTIDS" '.events[] as $event | select($eventIds | contains([$event.eventId]) | not) | $event | "\(.timestamp / 1000 | todateiso8601) \(.message)"'
  LOG_EVENTIDS=$(echo "$LOG_EVENTS" | jq -crM --argjson eventIds "$LOG_EVENTIDS" '$eventIds + [.events[].eventId] | unique')
done

Usage: save the file, chmod +x it, and then run it: ./cloudwatch-logs-tail.sh log-group-name. The script also takes parameters for begin and end times, which default to now and 2 minutes respectively. You can specify any strings which can be parsed by date --date for these parameters.

How it works: the script keeps a list of event IDs that have been displayed, which is empty to begin with. It queries CloudWatch Logs to get all log entries in the specified time interval, and displays those which do not match our list of event IDs. The it saves all of the event IDs for the next iteration.

The script polls every few seconds (set by LOG_INTERVAL in the script), and keeps polling for one more interval past the end time to account for the delay between log ingestion and availability.

Note that this script is not going to be great if you want to keep tailing the logs for more than a few minutes at a time, because the query results that it gets from AWS will keep getting bigger with every added log item. It's fine for quick runs though.

Answer 8

Note that tailing an aws log is now a supported feature of the official awscli, albeit only in awscli v2, which is not released yet. Tailing and following the logs (like tail -f) can now be accomplished by something like:

aws logs tail $group_name --follow

To install the v2 version, see the instructions on this page. It was implemented in this PR. To see it demonstrated at the last re:Invent conference, see this video.

In addition to tailing the logs, it allows viewing the logs back to a specified time using the --since parameter, which can take an absolute or relative time

aws logs tail $group_name --since 5d

To keep the v1 and v2 versions of awscli separate, I installed awscli v2 into a separate python virtual environment and activate it only when I need to use awscli v2.

Answer 9

The aws cli does not provide a live tail -f option.

Those other tools mentioned above do provide a tailing feature, however, I tried all these tools, awslogs, cwtail and found them frustrating. They were slow to download events, often unreliable and not helpful in displaying JSON log data and were primitive with query options.

I wanted an extremely fast, simple log viewer that would allow me to instantly and easily see application errors and status. The CloudWatch logs viewer is slow and CloudWatch Insights can take > 1m for some pretty basic queries.

So I created SenseLogs, a free AWS CloudWatch Logs viewer that runs entirely in your browser. There is no server-side services required. SenseLogs transparently downloads log data and stores events in your browser application cache for immediate viewing, smooth infinite scrolling and full text queries. SenseLogs has live tail with infinite back scrolling. See https://github.com/sensedeep/senselogs/blob/master/README.md for details.

Answer 10

To tail CloudWatch Logs effectively I created a tool called cw.

It's super easy to install (it supports brew, snap and scoop), fast (it targets the specific hardware architecture, no intermediate runtime) and it has a set of features that make life easier.

Your example with cw would be:

cw tail -f groupName:streamName

Answer 11

You can use awslogs, a python package to tail aws logwatch logs.

Install it with

pip install awslogs

List all the groups with

awslogs groups        

Then select a stream and watch it with

awslogs get staging-cluster --watch

You can also filter logs with matching patterns.

# tail logs of a cluster
awslogs get staging-cluster --watch

# tail logs of a lambda function
awslogs get /aws/lambda/some-service --watch

# print all logs containg "error"
awslogs get staging-cluster --watch --filter-pattern="error"

# print all logs *not* containg "error"
awslogs get staging-cluster --watch --filter-pattern="-error"

See project readme for more information on using awslogs.

Answer 12

I was really disappointed with awslogs and cwtail so I made my own tool called Saw that efficiently streams CloudWatch logs to the console (and colorizes the JSON output):

You can install it on MacOS with:

brew tap TylerBrock/saw
brew install saw

It has a bunch of nice features like the ability to automatically expand (indent) the JSON output (try running the tool with --expand):

saw watch my_log_group --expand

Got a Lambda you want to see error logs for? No Problem:

saw watch /aws/lambda/my_func --filter error 

Saw is great because the output is easily readable and you can stream logs from entire log group, not just a single stream in the group. Filtering and watching streams with a certain prefix is also just as easy!

Answer 13

I created a JetBrains plugin called awstail to do this :)

Answer 14

I've just discovered cwtail and it works well (to watch a lambda function's CloudWatch logs).

To install:

npm install -g cwtail

To list log groups:

cwtail -l

Then, once you've picked which log group to 'tail':

cwtail -f /aws/lambda/ExampleFunction

Answer 15

Have a look at awslogs.

If you happen to be working with Lambda/API Gateway specifically, have a look at apilogs.

Answer 16

This is not currently a feature of the CLI since it just exposes the HTTP API for CloudWatch Logs. You could fairly trivially emulate the functionality with a shell script:

#! /bin/sh

end_time=$(($(date +"%s") * 1000))
aws logs get-log-events --log-group-name groupName --log-stream-name streamName --end-time $end_time

while :
do
    start_time=$end_time
    end_time=$(($(date +"%s") * 1000))
    aws logs get-log-events --log-group-name groupName --log-stream-name streamName --start-time $start_time --end-time $end_time
    sleep 1
done

Disclaimer: this won't work on Windows, and there may be a better way to get the time in milliseconds.