CODEWITHSUNDEEP
pythonmultiprocessingcpupython-os
Right leg
Right leg

Reputation: 16720

Difference between multiprocessing.cpu_count and os.cpu_count

Both os and multiprocessing modules define a cpu_count function.

os.cpu_count is documented as follows:

Return the number of CPUs in the system. Returns None if undetermined.

and multiprocessing.cpu_count's documentation says:

Return the number of CPUs in the system. May raise NotImplementedError. See also os.cpu_count()

On my machine, they both return the same result:

>>> import os
>>> import multiprocessing as mp
>>> os.cpu_count()
8
>>> mp.cpu_count()
8

I would have thought that multiprocessing.cpu_count would be a mere reference to os.cpu_count, but it is not:

>>> os.cpu_count is mp.cpu_count
False

So what is the difference between them? Am I guaranteed that they'll always return the same result? Moreover, if I want to specify a number of processes to create for multiprocessing.Pool, should I use os or multiprocessing's function?

Upvotes: 15

Views: 7042

Answers (1)

Right leg
Right leg

Reputation: 16720

The answer lies in multiprocessing.context, which defines BaseContext.cpu_count as follows:

# cpython/Lib/multiprocessing/context.py

class BaseContext(object):
    def cpu_count(self):
        '''Returns the number of CPUs in the system'''
        num = os.cpu_count()
        if num is None:
            raise NotImplementedError('cannot determine number of cpus')
        else:
            return num

Then, this cpu_count method is exposed by multiprocessing:

# cpython/Lib/multiprocessing/__init__.py

__all__ = [x for x in dir(context._default_context) if not x.startswith('_')]
globals().update((name, getattr(context._default_context, name)) for name in __all__)

So in the end, multiprocessing.cpu_count is nothing but a wrapper around os.cpu_count.

Upvotes: 19

Related Questions