CODEWITHSUNDEEP
amazon-web-servicesaws-sdkamazon-cloudwatchlogs
Pramod Jacob
Pramod Jacob

Reputation: 101

AWS SDK for JavaScript CloudWatch Logs - GetLogEventsCommand isn't fetching logs, potentially due to a log stream size issue?

I have multiple Node.js applications deployed via AWS Elastic Beanstalk on the Docker platform. I can manually download the full logs for every environment without trouble via the AWS console. Let's say I have two AWS Elastic Beanstalk Environments: env-a and env-b.

I've started using the AWS SDK for JavaScript, specifically @aws-sdk/client-cloudwatch-logs, in a Node app so that I can programmatically fetch logs, render them in a custom UI, and do my own analysis as needed.

I'm running the following code in order to fetch the log events for a given app (pseudocode):

// IMPORTS

const { 
  CloudWatchLogsClient, 
  DescribeLogStreamsCommand,
  GetLogEventsCommand
} = require("@aws-sdk/client-cloudwatch-logs");

// SETUP

const awsCloudWatchClient = new CloudWatchLogsClient({
  region: process.env.AWS_REGION,
});

// APPLICATION CODE

const logGroupName = getLogGroupName(); 

// Get the log streams for the given log group. 
const logStreamRes = await awsCloudWatchClient.send(new DescribeLogStreamsCommand({
  descending: true,
  logGroupName,
  orderBy: 'LastEventTime',
  limit: 50,
}))

// For testing purposes, I'll just use the first log stream name I find. 
const logStreamName = logStreamRes.logStreams[0].logStreamName;

// Get the log events for the first log stream. 
const logEventRes = await awsCloudWatchClient.send(new GetLogEventsCommand({
  logGroupName,
  logStreamName,
}));

const logEvents = logEventRes.events;

Now, I can fetch the log events for env-a without trouble using this code. However, GetLogEventsCommand always returns an empty collection when I attempt to fetch the logs for env-b. If I download the logs manually via the AWS console, I can definitely see that logs exist - yet for a reason that isn't clear to me yet, the AWS SDK doesn't seem to recognize that.

Here's some interesting details that may help diagnose the issue.

env-a is configured in Elastic Beanstalk so that each new deploy (which happens potentially multiple times a day) replaces EC2 instances. On the other hand, env-b is configured so that new application code is deployed to existing EC2 instances without actually replacing them. Since log streams map to EC2 instances, env-a has a high number of pretty small log streams whereas env-b` has three extremely large log streams for each of its long-lived EC2 instances. The logs are easily >1 MBs in size.

Considering that GetLogEventsCommand returns responses up to 1 MB in size, am I hitting some size limit and the AWS SDK is handling it by returning 0 log events for env-b? I tried setting a limit on the GetLogEventsCommand above, but still causes the AWS SDK to return 0 events for env-a.

Another interesting note: if I go to Amazon CloudWatch > Log Group and select env-a's Log Group, I can see the log events for every log stream without trouble. If I try to view the log events for env-b's three very large log streams, I run into "Rate exceeded" errors on the console. This seems to confirm that the log stream's event count is simply too large for both the AWS console and AWS SDK to process, though I'm not certain.

Is there anything I can do to get the AWS SDK to fetch env-b's logs? How can I further confirm that excessive log stream size is the culprit here? And if that's the case, is there anything I can do about it, e.g. purge logs?

Or could this be some other issue that I'm not seeing?

Upvotes: 2

Views: 1446

Answers (1)

user19009041
user19009041

Reputation: 77

Another potential cause for your issues could be IAM permissions. Do the environments run as different IAM roles? Ideally, your logs are only permitted to various developer and beanstalk accounts. The two environments' log groups will need to explicitly permit the other environment's IAM roles.

I suspect since you are getting an empty list (ostensibly finding the log group, but not the streams), this is not the issue, but felt it was worth mentioning.

Upvotes: 0

Related Questions