How to Send Data Use Mpi4Py Across Multiple Nodes?
Everyone: I Run My Code on the Hpc, but I Can't Transmit Data Between Nodes. I Write a Simple Code to Test the Communications Between Cores Across Nodes...
everyone:
I run my code on the hpc,but I can't transmit data between nodes. I write a simple code to test the communications between cores across nodes. First, I use one node 8 cores, my code is(test.py)
from mpi4py import MPI
import sys
import numpy as np
def print_hello(rank, size, name):
msg = "Hello World! I am process {0} of {1} on {2}.\n"
sys.stdout.write(msg.format(rank, size, name))
if __name__ == "__main__":
comm = MPI.COMM_WORLD
size = MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()
name = MPI.Get_processor_name()
if rank == 0:
data = np.random.random((8,64,64,64))
print(data.shape)
else:
data = None
data = comm.scatter(data,root=0)
print(data.shape)
print_hello(rank, size, name)
and I run it use the srun -N 1 -n 8 python3 test.py 2>&1 | tee out.txt, it just like mpirun -np 8 python3 test.py 1>&1 | tee out.txt, it runs only 5 seconds
The out.txt file is :
(64, 64, 64)
Hello World! I am process 4 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 5 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 6 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 7 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 1 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 2 of 8 on cn3478.
(64, 64, 64)
Hello World! I am process 3 of 8 on cn3478.
(8, 64, 64, 64)
(64, 64, 64)
Hello World! I am process 0 of 8 on cn3478.
everything looks good!
However, When I use two nodes 48 cores, it went wrong! The file is that(test48.py):
from mpi4py import MPI
import sys
import numpy as np
def print_hello(rank, size, name):
msg = "Hello World! I am process {0} of {1} on {2}.\n"
sys.stdout.write(msg.format(rank, size, name))
if __name__ == "__main__":
comm = MPI.COMM_WORLD
size = MPI.COMM_WORLD.Get_size()
rank = MPI.COMM_WORLD.Get_rank()
name = MPI.Get_processor_name()
if rank == 0:
data = np.random.random((48,64,64,64))
print(data.shape)
else:
data = None
data = comm.scatter(data,root=0)
print(data.shape)
print_hello(rank, size, name)
I run it through yhrun -N 2 -n 48 python3 test48.py 2>&1 | tee out2.txt
It runs a lot of time(more than 1 hour) and did't print anything. I guess that the data transmission went wrong because I comment out the two lines:
data = comm.scatter(data,root=0)
print(data.shape)
The code is completed quickly and the output is :
Hello World! I am process 25 of 48 on cn3598.
Hello World! I am process 28 of 48 on cn3598.
Hello World! I am process 30 of 48 on cn3598.
Hello World! I am process 31 of 48 on cn3598.
Hello World! I am process 32 of 48 on cn3598.
Hello World! I am process 40 of 48 on cn3598.
Hello World! I am process 41 of 48 on cn3598.
Hello World! I am process 44 of 48 on cn3598.
Hello World! I am process 24 of 48 on cn3598.
Hello World! I am process 26 of 48 on cn3598.
Hello World! I am process 27 of 48 on cn3598.
Hello World! I am process 29 of 48 on cn3598.
Hello World! I am process 33 of 48 on cn3598.
Hello World! I am process 34 of 48 on cn3598.
Hello World! I am process 35 of 48 on cn3598.
Hello World! I am process 36 of 48 on cn3598.
Hello World! I am process 37 of 48 on cn3598.
Hello World! I am process 38 of 48 on cn3598.
Hello World! I am process 42 of 48 on cn3598.
Hello World! I am process 43 of 48 on cn3598.
Hello World! I am process 45 of 48 on cn3598.
Hello World! I am process 46 of 48 on cn3598.
Hello World! I am process 47 of 48 on cn3598.
Hello World! I am process 39 of 48 on cn3598.
Hello World! I am process 1 of 48 on cn3597.
Hello World! I am process 3 of 48 on cn3597.
Hello World! I am process 4 of 48 on cn3597.
Hello World! I am process 9 of 48 on cn3597.
Hello World! I am process 12 of 48 on cn3597.
Hello World! I am process 15 of 48 on cn3597.
Hello World! I am process 16 of 48 on cn3597.
Hello World! I am process 17 of 48 on cn3597.
Hello World! I am process 20 of 48 on cn3597.
Hello World! I am process 2 of 48 on cn3597.
Hello World! I am process 5 of 48 on cn3597.
Hello World! I am process 6 of 48 on cn3597.
Hello World! I am process 7 of 48 on cn3597.
Hello World! I am process 8 of 48 on cn3597.
Hello World! I am process 10 of 48 on cn3597.
Hello World! I am process 11 of 48 on cn3597.
Hello World! I am process 13 of 48 on cn3597.
Hello World! I am process 14 of 48 on cn3597.
Hello World! I am process 18 of 48 on cn3597.
Hello World! I am process 19 of 48 on cn3597.
Hello World! I am process 21 of 48 on cn3597.
Hello World! I am process 22 of 48 on cn3597.
Hello World! I am process 23 of 48 on cn3597.
(48, 64, 64, 64)
Hello World! I am process 0 of 48 on cn3597.
What's wrong is the code? or the data transmission between nodes is not allowed? Any suggestion is welcomed!