CODEWITHSUNDEEP
pythontensorflowkerasmemory-leaks
Itamar Katz
Itamar Katz

Reputation: 9645

Keras / Tensorflow suspected memory leak

I am using Keras (2.4) with Tensorflow (2.3.1) backend. I am tuning some meta-parameters with grid-search so there are multiple calls to model creation and model.fit method (the training is done on a machine with a single GPU).

Tensorflow has a documented memory leak issue (e.g see here but there are more reports, just search Google for tensorflow memory leak). I've encountered the issue in the past and managed to keep it at a minimal level using a combination of small batch size and calling Python's garbage collector manually. But now on a different project I suspect I am encountering it again.

I'm attaching an image of monitoring RAM usage over approx. 15 hours, and my question is simple - does it look like it can be the result of a memory leak? On one hand, the maximal usage grows with time. But, on the other hand, it seems that de-allocation (maybe when model instance is destroyed and created with new parameters?) frees most of the used memory back (almost) to the initial level. enter image description here

Upvotes: 2

Views: 3201

Answers (1)

FancyXun
FancyXun

Reputation: 1298

How about tf.keras.backend.clear_session()
Follow the document

Keras manages a global state, which it uses to implement the Functional model-building API and to uniquify autogenerated layer names.

If you are creating many models in a loop, this global state will consume an increasing amount of memory over time, and you may want to clear it. Calling clear_session() releases the global state: this helps avoid clutter from old models and layers, especially when memory is limited.

Upvotes: 4

Related Questions