Introduction to MPI with mpi4py

100. Introduction to MPI with mpi4py#

MPI (abbreviation for message passing interface) is the standard library for communication within a cluster of processors. The library is available for many languages, such as C or Python. In this tutorial we use MPI within jupyter notebooks

100.1. Latest installation instructions are found here:#

Installing parallel NGSolve

older alternative below:

100.2. The installation happens in three steps:#

  • install MPI including mpi4py

  • install PETSc including petsc4py (needed for advanced tutorials)

  • install MPI-parallel ngsolve (shipped as binaries starting from NGSolve 24.04)

100.2.1. Installing with conda#

The quickest path is using conda. Conda-forge provides binary packages for MPI and PETSc for many platforms and versions (TODO: list versions). Install anaconda (or mini-conda), and run something like

~/miniconda3/bin/conda install mpi4py petsc4py
~/miniconda3/bin/python3 -m pip install --pre --upgrade ngsolve

100.2.2. Installing conda-packages using pip#

  • Linux/MacOS 

    pip3 install -i https://pypi.anaconda.org/mpi4py/simple mpi4py==4.0.0.dev0 openmpi

  • Windows 

    pip install impi_rt
pip install -i https://pypi.anaconda.org/mpi4py/simple mpi4py==4.0.0.dev0

100.2.3. Installing MPI and PETSc without conda#

  • Linux:
    install either one with your package manager: openmpi, mpich, or IntelMPI
    pip install mpi4py

  • MacOS:

    download [openmpi4.1.6](https://www.open-mpi.org//software/ompi/v4.1/)
./configure
make all
sudo make all install
pip install mpi4py

  • Windows:
    Install IntelMPI

Install PETSc from a source-wheel:

export PETSC_CONFIGURE_OPTIONS="--with-fc=0 --with-debugging=0 --download-hypre"
pip cache remove petsc 
pip install --upgrade --no-deps --force-reinstall mpi4py petsc petsc4py

100.3. Using ipyparallel#

The ipyparallel module let us communicate with the cluster. The Cluster class represents the cluster. Every processor starts its own Python instance. You have to install the ipyparallel package:

!pip install ipyparallel

from ipyparallel import Cluster
c = await Cluster(engines="mpi").start_and_connect(n=4, activate=True)
c.ids

Starting 4 engines with <class 'ipyparallel.cluster.launcher.MPIEngineSetLauncher'>

[0, 1, 2, 3]

import mpi4py.MPI

[p238-029.vps.tuwien.ac.at:46717] shmem: mmap: an error occurred while determining whether or not /var/folders/_l/x5rb66s96zd96123z8dzgl5r0000gn/T//ompi.p238-029.501/jf.0/4028104704/sm_segment.p238-029.501.f0180000.0 could be created.

We can define variables on the i\(^{th}\) process:

for i in c.ids:
    c[i]['a'] = i

Cells tagged with the %%px - magic are executed by the cluster processes:

%%px
b = a*a

query the results from the processes:

c[:]['b']

[0, 1, 4, 9]

100.4. The MPI library#

MPI provides functions to communicate within the cluster. The back-bone is a communicator object which knows about the participating processors. The standard communicator is the world-communicator where all processors of the cluster are included.

The communicator knows the own number (called rank) within this group, and the size of the group:

%%px
from mpi4py import MPI
comm = MPI.COMM_WORLD
print ('I am proc', comm.rank, 'within the team of', comm.size)

[stdout:0] I am proc 0 within the team of 4

[stdout:1] I am proc 1 within the team of 4

[stdout:2] I am proc 2 within the team of 4

[stdout:3] I am proc 3 within the team of 4

We can send messages between processes using send and recv. With send we give the destination process where to send, and with recv we give the source processor number from where we expect data. We can send one Python object, which may also be a tuple or list.

In this example, process \(i\) is sending data to all processes with \(j > i\). Then process \(i\) is expecting data from processes with smaller ranks. This kind of communication is called point-to-point communication. Depending on the implementation of the mpi-library, and the object, the send and recv may or may not block.

%%px
fruits = ['apple', 'banana', 'clementine', 'durian', \
          'elderberries', 'figs', 'grapes', 'honeydew melon']
for dst in range(comm.rank+1, comm.size):
    comm.send(fruits[comm.rank%8], dest=dst)
for src in range(comm.rank):
    print ("got a", comm.recv(source=src))

[stdout:3] got a apple
got a banana
got a clementine

[stdout:2] got a apple
got a banana

[stdout:1] got a apple

If we want to send data to all processes in the cluster we use collective communication. A broadcast operation sends the same data from the root to everyone, a scatter operation splits a list to all processes. The default root is process 0.

%%px
if comm.rank == 0:
    comm.bcast("Hello from the boss!!")
    comm.scatter( ["personal note to "+str(i) for i in range(comm.size)])
else:
    print (comm.bcast(None))
    print (comm.scatter(None))

[stdout:1] Hello from the boss!!
personal note to 1

[stdout:2] Hello from the boss!!
personal note to 2

[stdout:3] Hello from the boss!!
personal note to 3

The technology behind sending of Python objects is pickling. Every class which knows how to convert itself to and from a byte-stream (called serialization) can be exchanged via mpi communication.

c.shutdown(hub=True)
Contents