Use All Cores with Python Multiprocessing

Question

I wrote a test-purpose snippet to use multiprocessing to work on all cores of my laptop. I have a 8 core cpu. Below the (basic) code:

import os
import time
import multiprocessing


def worker(n):
    pid = os.getpid()
    for x in range(0, 10):
        print("PID: %s   INPUT: %s" % (str(pid), str(n)))
        time.sleep(2)


input_params_list = [1, 2, 3, 4, 5, 6, 7, 8]
pool = multiprocessing.Pool(8)
pool.map(worker, input_params_list)
pool.close()
pool.join()

Basically it should start 8 processes which should just print their pid and the integer they get as input parameter. I just added a sleep to introduce some delay and make all of them running in parallel. When I run the script this is what I get:

PID: 811   INPUT: 1
PID: 812   INPUT: 2
PID: 813   INPUT: 3
PID: 814   INPUT: 4
PID: 815   INPUT: 5
PID: 816   INPUT: 6
PID: 817   INPUT: 7
PID: 818   INPUT: 8
PID: 811   INPUT: 1
PID: 812   INPUT: 2
PID: 813   INPUT: 3
PID: 814   INPUT: 4
PID: 815   INPUT: 5
PID: 816   INPUT: 6
PID: 817   INPUT: 7
PID: 818   INPUT: 8
... ... ... ... ...
... ... ... ... ...

I see that I have 8 different processes (plus the "father") running at the same time. The problem is that I think they're not running on 8 different cores. This is what I get from htop (I get the same with top too):

As I understood, the CPU column should contain the number of the core the process is running on. In this case I think that something is not working as expected since it is 1 for all of them. Otherwise I suppose there's something I misunderstood or something wrong in my code.

Accepted Answer

MisterMiyagi is right. To show that CPUs are working, I changed your code a bit, and add a bit of CPU-bound task to calculate the factorial of a big number to last for a few seconds (CPUs are on fire). Also, I use the private variable _identity to see what core the worker runs on.

import os
import time
import multiprocessing
import numpy as np

def worker(n):
    factorial = np.math.factorial(900000)
    # rank = multiprocessing.current_process().name one can also use
    rank = multiprocessing.current_process()._identity[0]
    print(f'I am processor {rank}, got n={n}, and finished calculating the factorial.')


cpu_count = multiprocessing.cpu_count()
input_params_list = range(1, cpu_count+1)


pool = multiprocessing.Pool(cpu_count)
pool.map(worker, input_params_list)
pool.close()
pool.join()

output

I am processor 4, got n=4, and finished calculating the factorial.
I am processor 2, got n=2, and finished calculating the factorial.
I am processor 1, got n=1, and finished calculating the factorial.
I am processor 3, got n=3, and finished calculating the factorial.

Use All Cores with Python Multiprocessing

1 Answer

Must Read

Your Answer

David Miller

Use All Cores with Python Multiprocessing

1 Answer

Must Read

Your Answer

Sign up or log in

Post as a guest

David Miller