How to Send Data Use Mpi4Py Across Multiple Nodes?

everyone: I run my code on the hpc,but I can't transmit data between nodes. I write a simple code to test the communications between cores across nodes. First, I use one node 8 cores, my code is(test.py)

from mpi4py import MPI
import sys
import numpy as np

def print_hello(rank, size, name):
  msg = "Hello World! I am process {0} of {1} on {2}.\n"
  sys.stdout.write(msg.format(rank, size, name))

if __name__ == "__main__":
    comm = MPI.COMM_WORLD
    size = MPI.COMM_WORLD.Get_size()
    rank = MPI.COMM_WORLD.Get_rank()
    name = MPI.Get_processor_name()
    if rank == 0:
        data = np.random.random((8,64,64,64))
        print(data.shape)
    else:
        data = None

    data = comm.scatter(data,root=0)
    print(data.shape)

    print_hello(rank, size, name)

and I run it use the srun -N 1 -n 8 python3 test.py 2>&1 | tee out.txt, it just like mpirun -np 8 python3 test.py 1>&1 | tee out.txt, it runs only 5 seconds The out.txt file is :

(64, 64, 64)
Hello World! I am process 4 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 5 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 6 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 7 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 1 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 2 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 3 of 8 on cn3478.
(8, 64, 64, 64)
(64, 64, 64)
Hello World! I am process 0 of 8 on cn3478.

everything looks good! However, When I use two nodes 48 cores, it went wrong! The file is that(test48.py):

from mpi4py import MPI
import sys
import numpy as np

def print_hello(rank, size, name):
  msg = "Hello World! I am process {0} of {1} on {2}.\n"
  sys.stdout.write(msg.format(rank, size, name))

if __name__ == "__main__":
    comm = MPI.COMM_WORLD
    size = MPI.COMM_WORLD.Get_size()
    rank = MPI.COMM_WORLD.Get_rank()
    name = MPI.Get_processor_name()
    if rank == 0:
        data = np.random.random((48,64,64,64))
        print(data.shape)
    else:
        data = None

    data = comm.scatter(data,root=0)
    print(data.shape)

    print_hello(rank, size, name)

I run it through yhrun -N 2 -n 48 python3 test48.py 2>&1 | tee out2.txt It runs a lot of time(more than 1 hour) and did't print anything. I guess that the data transmission went wrong because I comment out the two lines:

    data = comm.scatter(data,root=0)
    print(data.shape)

The code is completed quickly and the output is :

Hello World! I am process 25 of 48 on cn3598.
Hello World! I am process 28 of 48 on cn3598.
Hello World! I am process 30 of 48 on cn3598.
Hello World! I am process 31 of 48 on cn3598.
Hello World! I am process 32 of 48 on cn3598.
Hello World! I am process 40 of 48 on cn3598.
Hello World! I am process 41 of 48 on cn3598.
Hello World! I am process 44 of 48 on cn3598.
Hello World! I am process 24 of 48 on cn3598.
Hello World! I am process 26 of 48 on cn3598.
Hello World! I am process 27 of 48 on cn3598.
Hello World! I am process 29 of 48 on cn3598.
Hello World! I am process 33 of 48 on cn3598.
Hello World! I am process 34 of 48 on cn3598.
Hello World! I am process 35 of 48 on cn3598.
Hello World! I am process 36 of 48 on cn3598.
Hello World! I am process 37 of 48 on cn3598.
Hello World! I am process 38 of 48 on cn3598.
Hello World! I am process 42 of 48 on cn3598.
Hello World! I am process 43 of 48 on cn3598.
Hello World! I am process 45 of 48 on cn3598.
Hello World! I am process 46 of 48 on cn3598.
Hello World! I am process 47 of 48 on cn3598.
Hello World! I am process 39 of 48 on cn3598.
Hello World! I am process 1 of 48 on cn3597.
Hello World! I am process 3 of 48 on cn3597.
Hello World! I am process 4 of 48 on cn3597.
Hello World! I am process 9 of 48 on cn3597.
Hello World! I am process 12 of 48 on cn3597.
Hello World! I am process 15 of 48 on cn3597.
Hello World! I am process 16 of 48 on cn3597.
Hello World! I am process 17 of 48 on cn3597.
Hello World! I am process 20 of 48 on cn3597.
Hello World! I am process 2 of 48 on cn3597.
Hello World! I am process 5 of 48 on cn3597.
Hello World! I am process 6 of 48 on cn3597.
Hello World! I am process 7 of 48 on cn3597.
Hello World! I am process 8 of 48 on cn3597.
Hello World! I am process 10 of 48 on cn3597.
Hello World! I am process 11 of 48 on cn3597.
Hello World! I am process 13 of 48 on cn3597.
Hello World! I am process 14 of 48 on cn3597.
Hello World! I am process 18 of 48 on cn3597.
Hello World! I am process 19 of 48 on cn3597.
Hello World! I am process 21 of 48 on cn3597.
Hello World! I am process 22 of 48 on cn3597.
Hello World! I am process 23 of 48 on cn3597.
(48, 64, 64, 64)
Hello World! I am process 0 of 48 on cn3597.

What's wrong is the code? or the data transmission between nodes is not allowed? Any suggestion is welcomed!

2
James H. Sterling

James H. Sterling

Environmental Science & Climate Journalist

James Sterling reports on renewable energy developments, climate policy, ecological conservation, and green tech innovations around the globe.

Share this article