CODEWITHSUNDEEP
python
d33tah
d33tah

Reputation: 11561

ResourceWarning: unclosed file <_io.BufferedReader name=4>

Consider the following program:

import tempfile
import unittest
import subprocess

def my_fn(f):
    p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
    yield p.stdout.readline()
    p.kill()
    p.wait()

def my_test():
    with tempfile.TemporaryFile() as f:
        l = list(my_fn(f))

class BuggyTestCase(unittest.TestCase):
    def test_my_fn(self):
        my_test()

my_test()
unittest.main()

Running it results in the following output:

a.py:13: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  l = list(my_fn(f))
ResourceWarning: Enable tracemalloc to get the object allocation traceback
.
----------------------------------------------------------------------
Ran 1 test in 0.005s

OK

What is the actual cause of the warning and how to fix it? Note that if I comment out unittest.main() the problem disappears, which means that it's specific to subprocess+unittest+tempfile.

Upvotes: 24

Views: 22339

Answers (3)

neu242
neu242

Reputation: 16575

You can just change my_fn to use with, then you won't need kill and wait. Python'll close it for you:

def my_fn(f):
    with subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f) as p:
        yield p.stdout.readline()

Upvotes: 1

Leonardo
Leonardo

Reputation: 1891

I also got some ResourceWarning: unclosed file <_io.FileIO name=7 mode='wb' closefd=True> messages when running unit tests. I couldn't use the Popen object as a context manager, so I did this in my classes __del__ method:


import subprocess

class MyClass:
    def __init__(self):
        self.child= subprocess.Popen(['dir'],
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE)

    def __del__(self):
        self.child.terminate()
        self.child.communicate()

Upvotes: 1

Martijn Pieters
Martijn Pieters

Reputation: 1122532

You should be closing the streams associated with the Popen() object you opened. For your example, that's the Popen.stdout stream, created because you instructed the Popen() object to create a pipe for the child process standard output. The easiest way to do this is by using the Popen() object as a context manager:

with subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f) as p:
    yield p.stdout.readline()
    p.kill()

I dropped the p.wait() call as Popen.__exit__() handles this for you, after having closed the handles.

If you want to further figure out exactly what cause the resource warning, then we can start by doing what the warning tells us, and enable the tracemalloc module by setting the PYTHONTRACEMALLOC environment variable:

$ PYTHONTRACEMALLOC=1 python a.py
a.py:13: ResourceWarning: unclosed file <_io.BufferedReader name=4>
  l = list(my_fn(f))
Object allocated at (most recent call last):
  File "/.../lib/python3.8/subprocess.py", lineno 844
    self.stdout = io.open(c2pread, 'rb', bufsize)
.
----------------------------------------------------------------------
Ran 1 test in 0.019s

OK

So the warning is thrown by a file opened by the subprocess module. I'm using Python 3.8.0 here, and so line 844 in the trace points to these lines in subprocess.py:

if c2pread != -1:
    self.stdout = io.open(c2pread, 'rb', bufsize)

c2pread is the file handle for one half of a os.pipe() pipe object created to handle communication from child process to Python parent process (created by the Popen._get_handles() utility method, because you set stdout=PIPE). io.open() is exactly the same thing as the built-in open() function. So this is where the BufferedIOReader instance is created, to act as a wrapper for a pipe to receive the output the child process produces.

You could also explicitly close p.stdout:

p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
yield p.stdout.readline()
p.stdout.close()
p.kill()
p.wait()

or use p.stdout as a context manager:

p = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stdin=f)
with p.stdout:
    yield p.stdout.readline()
p.kill()
p.wait()

but it's easier to just always use subprocess.Popen() as a context manager, as it'll continue to work properly regardless of how many stdin, stdout or stderr pipes you created.

Note that most subprocess code examples don't do this, because they tend to use Popen.communicate(), which closes the file handles for you.

Upvotes: 30

Related Questions