CODEWITHSUNDEEP
ray
Pablo
Pablo

Reputation: 11041

Logs missing from console from a local ray session

I am running a series of unit tests with ray, where I am logging various events within Ray tasks.

I am finding that frequently, the logs occurring towards the end of my test do not show up in my console logs. This is making me unsure about whether the code is running at all.

How can I check what's going on in this case?

Upvotes: 1

Views: 609

Answers (1)

Pablo
Pablo

Reputation: 11041

Ray does its best to collect logs from worker processes (be it print statements or actual logging-module logs.

That being said, I've noticed that they do not always make it to the console, especially when the Ray session terminates quickly.

My advice to verify whether the log was reached is to look at Ray's log directory (/tmp/ray/session-${SESSION_ID}/logs/) - from there, specifically the worker-* logs contain output from the worker processes, which are in charge of executing Ray tasks.

Upvotes: 1

Related Questions