CODEWITHSUNDEEP
pythonmultiprocessingcpu
tjmgis
tjmgis

Reputation: 1661

Python CPU Count works on one windows server but not on another?

I have written come code that has worked on both Windows XP and Windows Server 2008 64-Bit. However I have just launched an Amazon Windows 64-bit instance and the code fails.

Very simple it looks like this

import multiprocessing

processors = multiprocessing.cpu_count()
print processors

I receive a NotImplementedError which I do not understand and the docs are not that helpful in explaining.

I just do not understand why it would work on one server and not on another with identical installations of Python 2.7

Anyone else run into this problem/error?

Upvotes: 5

Views: 3787

Answers (3)

user1786283
user1786283

Reputation:

It might be just the multiprocessing module, try using the psutil module it might work. SO in your case just do: 

import psutil
processors = psutil.cpu_count()
print processors
>>> 4

I tried this on the Amazon Windows 64-bit and it works quite well.

Upvotes: 4

Craig McDaniel
Craig McDaniel

Reputation: 5700

The failure is due to multiprocessing.cpu_count() relying on the NUMBER_OF_PROCESSORS environment variable on Windows, which can be occasionally be missing. Using wmi works as a substitute. Thanks for suggestion eryksun.

if sys.platform == 'win32':
    import wmi
    c = wmi.WMI(find_classes=False)
    return sum(x.NumberOfLogicalProcessors for x in c.Win32_Processor())
else:
    return multiprocessing.cpu_count()

Upvotes: 1

Eryk Sun
Eryk Sun

Reputation: 34260

If getting the CPU count is all you need to do with psutil, you could use ctypes instead:

import ctypes
from ctypes import wintypes

class SYSTEM_INFO(ctypes.Structure):
    _fields_ = [
        ('wProcessorArchitecture', wintypes.WORD),
        ('wReserved', wintypes.WORD),
        ('dwPageSize', wintypes.DWORD),
        ('lpMinimumApplicationAddress', wintypes.LPVOID),
        ('lpMaximumApplicationAddress', wintypes.LPVOID),
        ('dwActiveProcessorMask', ctypes.c_size_t),
        ('dwNumberOfProcessors', wintypes.DWORD),
        ('dwProcessorType', wintypes.DWORD),
        ('dwAllocationGranularity', wintypes.DWORD),
        ('wProcessorLevel', wintypes.WORD),
        ('wProcessorRevision', wintypes.WORD),
    ]

GetSystemInfo = ctypes.windll.kernel32.GetSystemInfo
GetSystemInfo.restype = None
GetSystemInfo.argtypes = [ctypes.POINTER(SYSTEM_INFO)]

def cpu_count():
    sysinfo = SYSTEM_INFO()
    GetSystemInfo(sysinfo)
    num = sysinfo.dwNumberOfProcessors
    if num == 0:
        raise NotImplementedError('cannot determine number of cpus')
    return num

Edit:

Here's an alternative to try that might return the same value as the NUMBER_OF_PROCESSORS environment variable. Note that the documentation says to use GetSystemInfo instead, which is what psutil uses. This is also using a native NT API, which is generally discouraged.

import ctypes
from ctypes import wintypes

SystemBasicInformation = 0

class SYSTEM_INFORMATION(ctypes.Structure): pass
PSYSTEM_INFORMATION = ctypes.POINTER(SYSTEM_INFORMATION)

class SYSTEM_BASIC_INFORMATION(SYSTEM_INFORMATION):
    _fields_ = [
        ('Reserved1', wintypes.BYTE * 24),
        ('Reserved2', wintypes.LPVOID * 4),
        ('NumberOfProcessors', ctypes.c_ubyte),
    ]

ntdll = ctypes.windll.ntdll
NtQuerySystemInformation = ntdll.NtQuerySystemInformation
NtQuerySystemInformation.argtypes = [
    wintypes.LONG,       # SystemInformationClass
    PSYSTEM_INFORMATION, # SystemInformation
    wintypes.ULONG,      # SystemInformationLength
    wintypes.PULONG]     # ReturnLength

def cpu_count():
    info = SYSTEM_BASIC_INFORMATION()
    retlen = wintypes.ULONG()
    status = NtQuerySystemInformation(SystemBasicInformation,
                                      info, 
                                      ctypes.sizeof(info), 
                                      retlen)
    num = info.NumberOfProcessors
    if status < 0 or num == 0:
        raise NotImplementedError('cannot determine number of cpus')
    return num

Upvotes: 3

Related Questions