Chapter 3: Distributed Memory Programming with MPI

In this chapter, we will be focusing on using the OpenMPI library with C to create parallel programs (SPMD) that run on distributed memory systems (multiple processes as opposed to multithreading).

OpenMPI Cheatsheet

Import

#include <mpi.h>

Compilation

mpicc -g -Wall -o path/to/executable path/to/source/code

Execution

mpiexec -n <number of processes> path/to/executable

Initialization

int MPI_Init(
    int* argc_p,
    char*** argv_p
);

Takes in the arguments given to the main function (the values passed in the CLI when executing). Sincle usually the main function will take no input, you will most likely call it in the following manner

MPI_Init(NULL, NULL);

Finalization

int MPI_Finalize(void);

Usually called

MPI_Finalize();

MPI_COMM_WORLD

The most commonly used communication world in MPI is MPI_COMM_WORLD (all caps).

Number of Processors in Communicator

int MPI_Comm_size(
    MPI_Comm comm,
    int* comm_sz_p
);

You give it an MPI communicator and an integer pointer. It places the number of processes in the passed pointer. Usually used in the following manner

int comm_sz;
MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);

The variable commm_sz now stores the number of processes in MPI_COMM_WORLD.

Current Process Rank

The rank acts like the id of the process in its communicator, and it is a value in between 0 and \(p-1\) where \(p\) is the number of process in the communicator.

int MPI_Comm_rank(
    MPI_Comm comm,
    int* my_rank_p
);

Its interface is very similar to MPI_Comm_size. It is usually used in the following manner

int my_rank;
MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

Datatypes

MPI Datatype	C Datatype
MPI_CHAR	signed char
MPI_SHORT	signed short int
MPI_INT	signed int
MPI_LONG	signed long int
MPI_LONG_LONG	signed long long int
MPI_UNSIGNED_CHAR	unsigned char
MPI_UNSIGNED_SHORT	unsigned short int
MPI_UNSIGNED	unsigned int
MPI_UNSIGNED_LONG	unsigned long int
MPI_FLOAT	float
MPI_DOUBLE	double
MPI_LONG_DOUBLE	long double
MPI_BYTE
MPI_PACKED

Sending Data

int MPI_Send(
    void* msg_buffer_pointer,
    int msg_size,
    MPI_Datatype msg_type
    int dest_rank,
    int tag,
    MPI_Comm communicator
);

MPI_Send is used to send a message from one process to another. It takes 6 parameters

1. A pointer to the variable containing the message to be sent
2. The number of msg_types to be sent. For example, if you are sending a string s, the message type will be MPI_CHAR and the value of msg_size will be strlen(s) + 1, where +1 is for the \0 character in the end.
3. The type of the data being sent.
4. The rank of the destination process
5. Tag of the message to be sent
6. The communicator; usually MPI_COMM_WORLD

Example of using it:

MPI_Send(greeting, strlen(greeting)+1, MPI_CHAR, 0, 0, MPI_COMM_WORLD);

Receiving Data

int MPI_Recv(
    void* msg_buffer_pointer,
    int buffer_size,
    MPI_Datatype buffer_type
    int dest_rank,
    int tag,
    MPI_Comm communicator,
    MPI_Status* status_pointer
);

7 parameters. The first 6 are the same as MPI_Send with the only difference being that the second parameter is now the buffer size and can be bigger (but not smaller) than the received message. The last parameter will have its own section. If you are not interested in recieveing the status, just pass MPI_STATUS_IGNORE to it.

Example of using it:

const int MAX_STRING = 100;
char greeting[MAX_STRING];
MPI_Recv(greeting, MAX_STRING, MPI_CHAR, 5, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);

The Stauts of MPI_Recv

The last paramater of MPI_Recv is a pointer to a struct that will carry information related to the received message. At least, it includes

• status.MPI_SOURCE: the rank of the message sender
• status.MPI_TAG: the tag of the recieved message
• status.MPI_ERROR

You can also use it to extract the amount of data that has been recieved by passing it to the MPI_Get_count function.

int count;
MPI_Get_count(&status, recv_type, &count);

Wildcards

You could pass MPI_ANY_SOURCE to the source parameter of MPI_Recv to receive a message from any process.

You could pass MPI_ANY_TAG to the tag parameter of MPI_Recv to receive a message from any process.

There is no wildcard for the communicator.

General Structure of an MPI program

#include <mpi.h>

int main(){

    MPI_Init(NULL, NULL);

    int comm_sz;
    int my_rank;

    MPI_Comm_size(MPI_COMM_WORLD, &comm_sz);
    MPI_Comm_rank(MPI_COMM_WORLD, &my_rank);

    /*
    The actual algorithm code
    Uses MPI_Send and MPI_Recv
    */

    MPI_Finalize();
    return 0;
}

Tips

• All MPI functions start with MPI_ and only the first follwing character is capitalized
• Constants are all caps