CODEWITHSUNDEEP
pythonsystem-information
phihag
phihag

Reputation: 288230

How to find out the number of CPUs using python

I want to know the number of CPUs on the local machine using Python. The result should be user/real as output by time(1) when called with an optimally scaling userspace-only program.

Upvotes: 842

Views: 626431

Answers (15)

Ciro Santilli OurBigBook.com
Ciro Santilli OurBigBook.com

Reputation: 384124

len(os.sched_getaffinity(0)) is what you usually want

https://docs.python.org/3/library/os.html#os.sched_getaffinity

os.sched_getaffinity(0) (added in Python 3) returns the set of CPUs available considering the sched_setaffinity Linux system call, which limits which CPUs a process and its children can run on.

0 means to get the value for the current process. The function returns a set() of allowed CPUs, thus the need for len().

multiprocessing.cpu_count() and os.cpu_count() on the other hand just returns the total number of logical CPUs, that is e.g. the number of CPUs considering hyperthreading.

The difference is especially important because certain cluster management systems such as Platform LSF limit job CPU usage with sched_getaffinity.

Therefore, if you use multiprocessing.cpu_count(), your script might try to use way more cores than it has available, which may lead to overload and timeouts.

We can see the difference concretely by restricting the affinity with the taskset utility, which allows us to control the affinity of a process.

Minimal taskset example

For example, if I restrict Python to just 1 core (core 0) in my 16 core system:

taskset -c 0 ./main.py

with the test script:

main.py

#!/usr/bin/env python3

import multiprocessing
import os

print(multiprocessing.cpu_count())
print(os.cpu_count())
print(len(os.sched_getaffinity(0)))

then the output is:

16
16
1

Vs nproc

nproc does respect the affinity by default and:

taskset -c 0 nproc

outputs:

1

and man nproc makes that quite explicit:

print the number of processing units available

Therefore, len(os.sched_getaffinity(0)) behaves like nproc by default.

nproc has the --all flag for the less common case that you want to get the physical CPU count without considering taskset:

taskset -c 0 nproc --all

os.cpu_count documentation

The documentation of os.cpu_count also briefly mentions this https://docs.python.org/3.8/library/os.html#os.cpu_count

This number is not equivalent to the number of CPUs the current process can use. The number of usable CPUs can be obtained with len(os.sched_getaffinity(0))

The same comment is also copied on the documentation of multiprocessing.cpu_count: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count

From the 3.8 source under Lib/multiprocessing/context.py we also see that multiprocessing.cpu_count just forwards to os.cpu_count, except that the multiprocessing one throws an exception instead of returning None if os.cpu_count fails:

    def cpu_count(self):
        '''Returns the number of CPUs in the system'''
        num = os.cpu_count()
        if num is None:
            raise NotImplementedError('cannot determine number of cpus')
        else:
            return num

3.8 availability: systems with a native sched_getaffinity function

The only downside of this os.sched_getaffinity is that this appears to be UNIX only as of Python 3.8.

cpython 3.8 seems to just try to compile a small C hello world with a sched_setaffinity function call during configuration time, and if not present HAVE_SCHED_SETAFFINITY is not set and the function will likely be missing:

psutil.Process().cpu_affinity(): third-party version with a Windows port

The third-party psutil package (pip install psutil) had been mentioned at: https://stackoverflow.com/a/14840102/895245 but not the cpu_affinity function: https://psutil.readthedocs.io/en/latest/#psutil.Process.cpu_affinity

Usage:

import psutil
print(len(psutil.Process().cpu_affinity()))

This function does the same as the standard library os.sched_getaffinity on Linux, but they have also implemented it for Windows by making a call to the GetProcessAffinityMask Windows API function:

So in other words: those Windows users have to stop being lazy and send a patch to the upstream stdlib :-)

Tested in Ubuntu 16.04, Python 3.5.2.

os.process_cpu_count() (Python 3.13)

https://docs.python.org/3.13/library/os.html#os.process_cpu_count

This seems to be the same as len(os.sched_getaffinity(0)), except that:

  • it can be overridden by the -X cpu_count Python CLI argumentor the PYTHON_CPU_COUNT environment variable
  • TODO confirm: it might fall back nicely on platforms where devs were too lazy to implement sched_getaffinity rather than blow an exception

Mentioned by jfs in the comments.

Upvotes: 174

Ranjeet R Patil
Ranjeet R Patil

Reputation: 501

If you are looking for printing the number of cores in your system.

Try this:

import os 
no_of_cores = os.cpu_count()
print(no_of_cores)

This should help.

Upvotes: -3

Jabir Ali
Jabir Ali

Reputation: 698

For python version above 3.4 you can use

import os
os.cpu_count()

If you are looking for an equivanlent of linux command nproc. You have this option

len(os.sched_getaffinity(0))

Upvotes: 28

Charlie Parker
Charlie Parker

Reputation: 5209

If you are using torch you can do:

import torch.multiprocessing as mp

mp.cpu_count()

the mp library in torch has the same interface as the main python one so you can do this too as the commenter mentioned:

python -c "import multiprocessing; print(multiprocessing.cpu_count())"

hope this helps! ;) it's always nice to have more than 1 option.

Upvotes: 2

Davoud Taghawi-Nejad
Davoud Taghawi-Nejad

Reputation: 16826

If you want to know the number of physical cores (not virtual hyperthreaded cores), here is a platform independent solution:

psutil.cpu_count(logical=False)

https://github.com/giampaolo/psutil/blob/master/INSTALL.rst

Note that the default value for logical is True, so if you do want to include hyperthreaded cores you can use:

psutil.cpu_count()

This will give the same number as os.cpu_count() and multiprocessing.cpu_count(), neither of which have the logical keyword argument.

Upvotes: 64

Konchog
Konchog

Reputation: 2188

This may work for those of us who use different os/systems, but want to get the best of all worlds:

import os
workers = os.cpu_count()
if 'sched_getaffinity' in dir(os):
    workers = len(os.sched_getaffinity(0))

Upvotes: 14

yangliu2
yangliu2

Reputation: 421

These give you the hyperthreaded CPU count

  1. multiprocessing.cpu_count()
  2. os.cpu_count()

These give you the virtual machine CPU count

  1. psutil.cpu_count()
  2. numexpr.detect_number_of_cores()

Only matters if you works on VMs.

Upvotes: 32

phihag
phihag

Reputation: 288230

If you're interested into the number of processors available to your current process, you have to check cpuset first. Otherwise (or if cpuset is not in use), multiprocessing.cpu_count() is the way to go in Python 2.6 and newer. The following method falls back to a couple of alternative methods in older versions of Python:

import os
import re
import subprocess


def available_cpu_count():
    """ Number of available virtual or physical CPUs on this system, i.e.
    user/real as output by time(1) when called with an optimally scaling
    userspace-only program"""

    # cpuset
    # cpuset may restrict the number of *available* processors
    try:
        m = re.search(r'(?m)^Cpus_allowed:\s*(.*)$',
                      open('/proc/self/status').read())
        if m:
            res = bin(int(m.group(1).replace(',', ''), 16)).count('1')
            if res > 0:
                return res
    except IOError:
        pass

    # Python 2.6+
    try:
        import multiprocessing
        return multiprocessing.cpu_count()
    except (ImportError, NotImplementedError):
        pass

    # https://github.com/giampaolo/psutil
    try:
        import psutil
        return psutil.cpu_count()   # psutil.NUM_CPUS on old versions
    except (ImportError, AttributeError):
        pass

    # POSIX
    try:
        res = int(os.sysconf('SC_NPROCESSORS_ONLN'))

        if res > 0:
            return res
    except (AttributeError, ValueError):
        pass

    # Windows
    try:
        res = int(os.environ['NUMBER_OF_PROCESSORS'])

        if res > 0:
            return res
    except (KeyError, ValueError):
        pass

    # jython
    try:
        from java.lang import Runtime
        runtime = Runtime.getRuntime()
        res = runtime.availableProcessors()
        if res > 0:
            return res
    except ImportError:
        pass

    # BSD
    try:
        sysctl = subprocess.Popen(['sysctl', '-n', 'hw.ncpu'],
                                  stdout=subprocess.PIPE)
        scStdout = sysctl.communicate()[0]
        res = int(scStdout)

        if res > 0:
            return res
    except (OSError, ValueError):
        pass

    # Linux
    try:
        res = open('/proc/cpuinfo').read().count('processor\t:')

        if res > 0:
            return res
    except IOError:
        pass

    # Solaris
    try:
        pseudoDevices = os.listdir('/devices/pseudo/')
        res = 0
        for pd in pseudoDevices:
            if re.match(r'^cpuid@[0-9]+$', pd):
                res += 1

        if res > 0:
            return res
    except OSError:
        pass

    # Other UNIXes (heuristic)
    try:
        try:
            dmesg = open('/var/run/dmesg.boot').read()
        except IOError:
            dmesgProcess = subprocess.Popen(['dmesg'], stdout=subprocess.PIPE)
            dmesg = dmesgProcess.communicate()[0]

        res = 0
        while '\ncpu' + str(res) + ':' in dmesg:
            res += 1

        if res > 0:
            return res
    except OSError:
        pass

    raise Exception('Can not determine number of CPUs on this system')

Upvotes: 237

Bakuriu
Bakuriu

Reputation: 102029

Another option is to use the psutil library, which always turn out useful in these situations:

>>> import psutil
>>> psutil.cpu_count()
2

This should work on any platform supported by psutil(Unix and Windows).

Note that in some occasions multiprocessing.cpu_count may raise a NotImplementedError while psutil will be able to obtain the number of CPUs. This is simply because psutil first tries to use the same techniques used by multiprocessing and, if those fail, it also uses other techniques.

Upvotes: 136

Nadia Alramli
Nadia Alramli

Reputation: 115011

If you have python with a version >= 2.6 you can simply use

import multiprocessing

multiprocessing.cpu_count()

http://docs.python.org/library/multiprocessing.html#multiprocessing.cpu_count

Upvotes: 1295

amit12690
amit12690

Reputation: 167

You can also use "joblib" for this purpose. 

import joblib
print joblib.cpu_count()

This method will give you the number of cpus in the system. joblib needs to be installed though. More information on joblib can be found here https://pythonhosted.org/joblib/parallel.html

Alternatively you can use numexpr package of python. It has lot of simple functions helpful for getting information about the system cpu.

import numexpr as ne
print ne.detect_number_of_cores()

Upvotes: 11

jfs
jfs

Reputation: 414745

In Python 3.4+: os.cpu_count().

multiprocessing.cpu_count() is implemented in terms of this function but raises NotImplementedError if os.cpu_count() returns None ("can't determine number of CPUs").

Upvotes: 72

Alkero
Alkero

Reputation: 35

Another option if you don't have Python 2.6:

import commands
n = commands.getoutput("grep -c processor /proc/cpuinfo")

Upvotes: 1

Douglas B. Staple
Douglas B. Staple

Reputation: 10946

multiprocessing.cpu_count() will return the number of logical CPUs, so if you have a quad-core CPU with hyperthreading, it will return 8. If you want the number of physical CPUs, use the python bindings to hwloc:

#!/usr/bin/env python
import hwloc
topology = hwloc.Topology()
topology.load()
print topology.get_nbobjs_by_type(hwloc.OBJ_CORE)

hwloc is designed to be portable across OSes and architectures.

Upvotes: 22

Ben Scherrey
Ben Scherrey

Reputation: 349

Can't figure out how to add to the code or reply to the message but here's support for jython that you can tack in before you give up:

# jython
try:
    from java.lang import Runtime
    runtime = Runtime.getRuntime()
    res = runtime.availableProcessors()
    if res > 0:
        return res
except ImportError:
    pass

Upvotes: 10

Related Questions