CODEWITHSUNDEEP
pythonmultiprocessingcpu
xyhuang
xyhuang

Reputation: 424

why my python multiprocessing can't use all cpu?

i have a server with 48 cores, so i decide to use python multiprocess to run my code with 46 process.

here is the multiprocessing wrapper function:

import multiprocessing
def MPRun(func, args, dicts={}, n_pool=max(1, multiprocessing.cpu_count()-2), ret=True, show_process=True, show_num=100):
  print('submit a multiprocess task with cpu', n_pool)
  q = Queue()
  if not isinstance(args, list) or not isinstance(dicts, dict):
    print('args should be list, dicts should be dict', type(args), type(dicts))
    sys.exit(1)
  pool = multiprocessing.Pool(processes=n_pool)
  res = []
  for i, arg in enumerate(args):
    if not isinstance(args[0], list):
      q.put(pool.apply_async(func, tuple([arg]), dicts))
    else:
      q.put(pool.apply_async(func, tuple(arg), dicts))
  count = 0 
  r = []
  while count < len(args):
    r.append(q.get().get())
    count += 1
    if show_process and count % show_num == show_num - 1:
      print('MPRUN process[%d/%d]'%(count, len(args)))
  pool.close()                                                                                                                                                                               
  pool.join()
  return r

I call it like this:

MPRun(myfunc, [[task] for task  in tasks])

when i call top command in linux, i found it do has 46 python process, but among them, only 8-12 is running, the left's state are s(which means wait cpu to run). and many cpu's idle is 100%(which means it is idle)

there is no other code running, so, i am confused, why those idle cpu dont run the state s tasks?

Is there any problems with my MPRun()? or is it related with my task type?

Upvotes: 1

Views: 496

Answers (1)

Vlad
Vlad

Reputation: 9481

You didn't show what your tasks actually do. There could be many reasons why they are not using full CPU. 's' - means that the task is waiting on something to happen it could be one of the following:

  • Waiting for response to the network request
  • Waiting on synchronization object
  • Waiting for swapped out page to load from memory ( this is very common on very busy systems, did you allocate more memory than your system can handle?)
  • Waiting on the IO request ( are your tasks IO bound or CPU bound )
  • Other things that a program can wait for

Try to profile your processes with 'strace' and see if there something going on there.

Also look at your /proc/interrupts to see if you have unsually high IO activity. You have to take a baseline first to determine the interrupts volume on your system when it's idle, and then sample it again while your program is running

Upvotes: 2

Related Questions