CODEWITHSUNDEEP
pythonamazon-web-servicesamazon-cloudwatchaws-batch
Sud
Sud

Reputation: 101

AWS : Cloudwatch logs only published after Batch job is completed

I am using a AWS Batch job , which triggers an ECR Image ( Docker Image containing python code) and the batch logs to Cloudwatch.

As per the code inside Docker, I am using print command to log as shown below. The issue is all the print statements are shown in the CloudWatch only when the Batch is completed, which takes around 2-3 hours to complete.

The prints are not logged as per code executes, instead all prints are shown only when the whole process completes. Is there a way we can flush the print as soon as the line executes.

Does the CloduWatch/Batch store the logs in memory and flushes only when the job is completed?

Python Code:

print("Process Started..")
#some code 
print("Process Completed.")

Upvotes: 10

Views: 1794

Answers (1)

Elisabeth Strunk
Elisabeth Strunk

Reputation: 566

You can add the environment variable PYTHONUNBUFFERED to your Dockerfile:

ENV PYTHONUNBUFFERED=1

This will assure that the Python output is sent straight to your log instead of being buffered.

You can read more about PYTHONUNBUFFERED here:

Upvotes: 1

Related Questions