CODEWITHSUNDEEP
c++tensorflowgpu
pseudomarvin
pseudomarvin

Reputation: 1627

Tensorflow C++ not releasing GPU resources after closing the session

I compiled tensorflow C++ with GPU support from source (branch r1.7) on Windows 10. Upon creating a new session 3GBs of memory are allocated on the GPU. Closing the session does not seem to result in the memory being released from the GPU as confirmed by the nvidia-smi command. GPU memory is only released when the C++ program exits. How can I release the memory in code?

int main()
{
    SessionOptions options;
    Session* session;
    tensorflow::Status status = NewSession(SessionOptions(), &session); // returns ok

    status = session->Close(); // returns ok
    delete session;

    // GPU memory is still occupied at this point
    std::string s;
    std::cin >> s;

    return 0;
}
// GPU memory is released when process exits

Upvotes: 1

Views: 735

Answers (1)

Stargateur
Stargateur

Reputation: 26757

From doc:

Note that we do not release memory, since that can lead to even worse memory fragmentation.

Current issue to find a solution. It seems that to avoid having to close your own process, you need to create a child process that will do the job for you.

Upvotes: 1

Related Questions