When using MPI to send data between distributed processes, you must specify the type of data being sent so that the MPI library can bundle up the information and ship it out to all receiving processes and a receiving process can know how to place the data into local memory. The library comes with predefined primitive data types — character, integer, floating-point. I can send an array of a primitive data type by setting the parameter in send and receive functions specifying the number of objects being communicated.

For many instances of data exchanges these primitive types are sufficient. But what if you need to send data that is composed of two or more primitive types? For example, think of a simple personnel record that contains a name, address, contact phone number, salary, number of dependents, and an indicator of the retirement plan that the employee is enrolled in. (I'm a bit troubled that I keep defaulting to employee records when I want to give an example of heterogeneous record types. It must stem from learning COBOL as a first programming language.) This record (structure) contains character strings, integers, and a floating-point value.

If two processes are required to share such a record, you could send and receive each different data type with a separate message. This would quickly clutter up your code and be a drag on performance since each message would have the overhead associated with sending a message. Alternatively, especially if the data were gathered into a logical record within memory, you can define a derived data type describing the structure of the data and send/receive a single message to communicate the record. The process for setting up a derived data type in an MPI code is the subject of this post. The example I will be using is a derived type that corresponds to the List structure I defined for my list-based version of Game of Life for a distributed-memory environment.

Every MPI derived type is ultimately made up of MPI primitive types. Besides specifying what primitive types make up a user-defined type, I need to describe how those are primitive types are composed together. This is done through a typemap, which is simply the relative displacement of each primitive type element from the base address of the derived type in memory. Before I describe how to define the myList MPI derived type, here is the type declaration that creates a List data structure in my Game of Life code.

 typedef struct cell {
        int row, col;    //grid coordinates
} Cell;
typedef Cell ListEntry;

typedef struct list {
	int count;
	ListEntry entry[MAXLIST];
} List;

For the Game of Life simulation, a list holds pairs of coordinate indices of cells that are of interest. The ListEntry type has two integers, row and col, for this pair. The List type itself is an array of ListEntry elements and a count of the number of pairs currently on the list. The constant MAXLIST is the maximum number of coordinate pairs that a list can hold. The fixed size of the List data structure has some obvious drawbacks, but it is what allows me to define a derived type that can be sent and received between MPI processes.

If you remember my previous post that used myList, there was a call to MakeList() to create the derived type. Before I describe that function, I want to go over the MakeListEntry() function, called from MakeList(), to define the derived type for an individual grid cell coordinate pair element. Mimicking the definition of the List type above, the function calls I employ are modeled on the structure of the struct declarations.

 void  MakeListEntry  ( MPI_Datatype*  myListEntry )
{
   // Set up cell datatype for communication
   MPI_Datatype    entryType[1] = { MPI_INT };
   int             count[1]     = { 2 };
   MPI_Aint        addr[1]      = { 0 };
      
   MPI_Create_type_struct( 1, count, addr, entryType, myListEntry );
   MPI_Type_commit( myListEntry );
} // MakeListEntry

The purpose of the MakeListEntry() function is to define an MPI derived type for a single grid coordinate pair structure and return that new derived type through the parameter. As I mentioned above, I need to define a derived type by noting the primitive types that make up the new type, the number of items, and the displacement of each different primitive type from the start address of the structure. In this case, I have integers (row and col) that are both MPI_INT types, thus I only need one displacement for the pair. The relative address of the first of these two elements is zero bytes from the base address. This information is encoded in the three declarations of the function.

The entryType array (of one element) holds a collection of MPI_Datatype primitive types in the order that they are defined within the derived type structure. The count array (one corresponding element for each element in the entryType array) holds the number of copies for each primitive type item within the derived type structure. The MPI_Aint type is used to hold a valid C type address. In my case, it is used as the declared type of the array of displacements for each type listed in the entryType array. With only one pair of MPI_INT elements that make up my derived type, this displacement is zero bytes, which is held in the (one element) array addr.

(Note: I'm making the assumption that two integers declared as they were in ListEntry will be right next to each other, address-wise. I don't expect consecutively declared int types will be separately aligned on some 8-byte boundary. When I create the derived type that holds an array of the myListEntry types, I will be sure to compute displacements that may be offset by "blank" bytes to satisfy memory boundaries between the count integer and array of list entries.)

Once all the information is assembled, I call MPI_Create_type_struct() to create the typemap for my new derived type in the myListEntry. Before the new type can actually be used in communication functions, it must be committed. This is done with a call to MPI_Type_commit(). Plus, after a derived type has been committed, it can be used as a primitive type in the definition of other derived types as I show now.

With the single grid coordinate pair as a derived type, I can create a type holding an array of coordinate pairs that will be the data section of the myList type. Along with the array of coordinates, the list also contains a count of the number of items on the list. Below is the function I used to construct the myList derived type. The first thing that MakeList() does is to call MakeListEntry() to define and commit the type for a single coordinate pair.

 void  MakeList  ( MPI_Datatype*  myList )
{
   MPI_Datatype    myListEntry;
   MakeListEntry( &myListEntry );

   // Set up list datatype for communication
   MPI_Aint        displs[2]; 
   MPI_Datatype    type[2]       = { MPI_INT, myListEntry };
   int             blockCount[2] = { 1,       MAXLIST };
   List            list;

   MPI_Get_address( &list.count, &displs[0] );
   MPI_Get_address( &list.entry, &displs[1] );

   displs[1] -= displs[0];
   displs[0]  = 0;

   MPI_Type_struct( 2, blockCount, displs, type, myList );
   MPI_Type_commit( myList );
} // MakeList

After defining the coordinate pair type, I declare and assign some two element arrays to hold the displacement, the types and the counts for the two parts of the List data structure; one item in the newly declared arrays for the count and one for the coordinate array. Next, a List is declared in order to determine the offsets of the two parts of the structure. I call MPI_Get_address() to find these displacements.

When I was defining the grid point pair type, I assumed that the two integers would be placed in memory one immediately after the other. Since the List structure has a single integer and an array of listEntry, I'm not completely sure what word or byte boundaries the compiler or the hardware will enforce when the structure is declared and allocated in memory. Thus, I create a List and use the MPI function to determine the exact offsets of the allocated data structure. Of course, since I want the offset from a relative base of zero, I subtract the address of the array from the address of the count integer and set the count displacement to zero.

Because I preloaded the data into the arrays holding the component MPI data types (type) and the count of each type (blockCount), I now simply call MPI_Type_struct() to define the myList derived type composed of the two primitive types and then MPI_Type_commit() to complete the definition.

After breaking it down, it all seems rather easy and straightforward. As I said before, the only way that this works is due to the fact that the List data structure is going to be a fixed size. If I were to use a dynamic list structure that changes size based on the actual number of entries in the list, I could not create a derived type. For that implementation of a distributed Game of Life simulation, I would recommend using the ghost cell idea presented in my previous post, which will ensure that a known quantity of data will be exchanged between neighboring processes.