# 10 Advanced MPI Communication Techniques Part 2
###### tags: `SS2021-IN2147-PP`

## Datatypes

### Central MPI Types and Objects

// Communicators
MPI_Comm

// handle for non-blocking communication
MPI_Request

// object
// Object object type for a key/value store
// Passed to many routines to pass additional information
MPI_Info

// structured datatypes for communication
MPI_Datatype

### Derived MPI Datatypes and Management

// Datatype construction
int MPI_Type_contiguous(int count, MPI_Datatype oldtype,
                        MPI_Datatype *newtype)
// Datatype commit
int MPI_Type_commit(MPI_Datatype *datatype)

// Datatype free
int MPI_Type_free(MPI_Datatype *datatype)


### Constructing a Vector Datatype

int MPI_Type_vector(int count, int blocklength, int stride,
                    MPI_Datatype oldtype, MPI_Datatype *newtype)


### Constructing an Indexed Datatype

int MPI_Type_indexed(int count,
                     const int array_of_blocklengths[],    // AoB
                     const int array_of_displacements[],   // AoD
                     MPI_Datatype oldtype,
                     MPI_Datatype *newtype)


### Creating a Struct Datatype

int MPI_Type_create_struct(int count,
                           const int array_of_blocklengths[],         // AoB
                           const MPI_Aint array_of_displacements[],   // AoD
                           const MPI_Datatype array_of_types[],       // AoT
                           MPI_Datatype *newtype)


### Constructing a Subarray Datatype

int MPI_Type_create_subarray(int ndims,
                             const int array_of_sizes[],
                             const int array_of_subsizes[],
                             const int array_of_starts[],
                             int order,
                             MPI_Datatype oldtype, MPI_Datatype *newtype)


### Some Other Datatype Construction Functions

MPI_Type_dup
// Duplication of datatype

MPI_Type_create_hvector
// Like MPI_Type_create_vector only with Byte-sized strides

MPI_Type_create_hindexed
// Like MPI_Type_create_indexed only with Byte-sized strides

MPI_Type_create_indexed_block
// Like MPI_Type_create_indexed only with fixed block size

MPI_Type_create_hindexed_block
// Like MPI_Type_create_indexed_block only with Byte-sized strides

MPI_Type_create_darray
// Create a distributed array datatype

### Inspecting Datatypes

// Query metadata information on a type
int MPI_Type_get_envelope(MPI_Datatype datatype,
                          int *num_integers, int *num_addresses,
                          int *num_datatypes, int *combiner)

// Query actual type information
int MPI_Type_get_contents(MPI_Datatype datatype,
                          int max_integers, int max_addresses,
                          int max_datatypes, int array_of_integers[],
                          MPI_Aint array_of_addresses[],
                          MPI_Datatype array_of_datatypes[])

### Datatype Discussions
* Warning
* Datatype creation/destruction can lead to overhead
* Not many MPI implementations are optimized for datatypes

## One-Sided Communication

### Communication Modes

#### Two-sided communication
* Explicit sender and receiver for `P2P`
* All processes call `collectives`
* Advantages
* `Simple`
* Clear communication locations
* `Implicit synchronization`
* Disadvantages
* `Implicit synchronization`
* Requires active involvement on both sides
* Receiver can delay sender

#### One-sided communication
* Decouple data movement with process synchronization
* Exposes a part of its memory
* Directly read
* 
* Advantages
* `Direct memory access`
* Receiver does not get involved
* Requires `extra synchronization`
* Remote Memory Access (`RMA`) in MPI
* Shared memory models
* Explicit get and put calls
* Need hardware support

### Two-sided Communication v.s. One-sided Communication

### Creating Public Memory
* MPI terminology for `remotely accessible memory` is a “**`Window`**”
* Several Window Creation Models

    // Make existing/allocated memory regions remotely accessible
    MPI_WIN_CREATE
    
    // Allocate memory region and make it accessible
    MPI_WIN_ALLOCATE
    
    // No buffer yet, but will have one in the future
    MPI_WIN_CREATE_DYNAMIC
    
    // Add memory to a dynamic window
    MPI_WIN_ATTACH
    
    // Remove memory from a dynamic window
    MPI_WIN_DETACH

### Making Local Memory Accessible

// Expose a region of memory in an RMA window
int MPI_Win_create(void *base, MPI_Aint size, int disp_unit,
                   MPI_Info info, MPI_Comm comm, MPI_Win *win)
                   
// Create a remotely accessible memory region in an RMA window
int MPI_Win_allocate(MPI_Aint size, int disp_unit,
                     MPI_Info info, MPI_Comm comm,
                     void *base, MPI_Win *win)

// Create an RMA window, to which data can later be attached
int MPI_Win_create_dynamic(MPI_Info info, MPI_Comm comm,
                           MPI_Win *win)

// Attaching and Detaching Memory
int MPI_Win_attach(MPI_Win win, void *base, MPI_Aint size)
int MPI_Win_detach(MPI_Win win, void *base)

### Data Movement

// Move data to origin, from target
int MPI_Get(void *origin_addr, int origin_count,
            MPI_Datatype origin_datatype, int target_rank,
            MPI_Aint target_disp, int target_count,
            MPI_Datatype target_datatype, MPI_Win win)

// Move data from origin, to target
int MPI_Put(void *origin_addr, int origin_count,
            MPI_Datatype origin_datatype, int target_rank,
            MPI_Aint target_disp, int target_count,
            MPI_Datatype target_datatype, MPI_Win win)

// Move data from origin, to target
// Special case: MPI_REPLACE - Leads to atomic update
int MPI_Accumulate(void *origin_addr, int origin_count,
                   MPI_Datatype origin_datatype, int target_rank,
                   MPI_Aint target_disp, int target_count,
                   MPI_Datatype target_datatype,
                   MPI_Op op, MPI_Win win)

### Additional RMA Operations
* Get-accumulate
* Compare-and-swap
* Useful for linked list
* Fetch-and-Op
* Faster for the hardware to implement

### Ordering of Operations in MPI RMA
* `Put`/`Get` operations
* Ordering does not matter
* Write-after-Write
* Additional consistency issues
* `Accumulate` operations
* Use `MPI_REPLACE` for atomic `PUTs`
* All accumulate operations are ordered by default

### RMA Synchronization Models
* MPI RMA model allows data to be accessed only within an “epoch”
* Three types of epochs
* Fence (active target)
* Post-start-complete-wait (active target)
* Lock/Unlock (passive target)

#### Fence Synchronization

// Simplest model
MPI_Win_fence(assert, win)


#### Post/Start/Complete/Wait Synchronization

// Target: Exposure epoch
// Target may allow a smaller group of processes to access its data
MPI_Win_post
MPI_Win_wait

// Origin: Access epoch
// Origin can indicate a smaller group of processes to retrieve data from 
MPI_Win_start
MPI_Win_complete


#### Lock/Unlock Synchronization
* Passive mode: One-sided, asynchronous communication
* Not mutex!!!
* Non-blocking
* Similar to release consistency

### Which Communication to Use?

#### Two-sided communication
* `Easier` to handle
* `Explicit send/recv` points
* Easier to reason about data and memory model
* Easy to use if `send/recv points are known` (e.g., SPMD codes)

#### RMA
* Good for `low protocol overheads` architecture
* Especially if supported from `underlying interconnect` (e.g. `InfiniBand`)
* **If not, RMA can be really slow**

#### Passive mode: `asynchronous` one-sided communication
* Data characteristics
* Big data analysis
* Requiring memory aggregation
* Asynchronous data exchange
* Data-dependent access pattern
* Computation characteristics
* Adaptive methods
* E.g. AMR, MADNESS
* Asynchronous dynamic load balancing

### Shared Memory in MPI

// Create an MPI window that is also accessible with **local load/store operations**
// * Function returns base pointer
// * Needs manual synchronization
// * Only part of address space is shared
int MPI_Win_allocate_shared(MPI_Aint size, int disp_unit,
                            MPI_Info info, MPI_Comm comm,
                            void *baseptr, MPI_Win *win)

## Hybrid Programming
* Use a shared memory model within an MPI process
* MPI + OpenMP as a typical option

### Threading and MPI
* Accesses from several threads can cause problems during execution
* Shared data structures
* Coordinate access to NIC
* Callback coordination

### MPI’s Four Levels of Thread Safety

// New MPI initialization routine
int MPI_Init_thread(int *argc, char ***argv,
                    int required, int *provided)

MPI_THREAD_SINGLE
// Only one thread exists in the application

MPI_THREAD_FUNNELED
// Multithreaded, but only the main thread makes MPI calls

MPI_THREAD_SERIALIZED
// Multithreaded, but only one thread at a time makes MPI calls

MPI_THREAD_MULTIPLE
// Multithreaded and any thread can make MPI calls at any time

### Consequences of using MPI_THREAD_MULTIPLE

#### Each call to MPI completes on its own

#### User is responsible for making sure `racing calls are avoided`
* Collective operations must be ordered correctly among threads

### Thread Safe Probing and Matching Receive

// MPI_Probe/MPI_Iprobe inspects unexpected message queue (UMQ)

// Matching probes and receive
int MPI_Mprobe(int source, int tag, MPI_Comm comm,
               MPI_Message *message, MPI_Status *status)
int MPI_Improbe(int source, int tag, MPI_Comm comm,
                int *flag, MPI_Message *message, MPI_Status *status)
// Wait for/Check if matching message
// * Returns handle to matched message

// Matching receive: 
// If two threads call MPI_Probe and both return
// We need matching receive!!!
int MPI_Mrecv(void* buf, int count, MPI_Datatype datatype,
              MPI_Message *message, MPI_Status *status)
int MPI_Imrecv(void* buf, int count, MPI_Datatype datatype,
               MPI_Message *message, MPI_Request *request)
// Receive message identified by Mprobe
// * Handle of message passed into receive

### Hybrid Programming MPI+OpenMP
* Danger: Amdahl’s Law in the `non-OpenMP regions`
* Need to split data structures (for MPI), then share data parts (for OpenMP threads)
* Hierarchical data decomposition

### When to Use Hybrid?
* Depends on application
* Option
* MPI only with one MPI process per HW thread
* One MPI process per node, one OpenMP thread per core on node
* Several MPI processes per node, rest of the concurrency in OpenMP
* **`One MPI process per socket`** often fits well
* Large degree of parallelism in OpenMP
* Naturally **`avoids NUMA problems`**
* Need to pin threads to that socket

### What‘s Left in MPI: Other Topics
* MPI I/O
* Parallel I/O reading and writing files
* Often used in connection with a parallel file system
* Dynamic Process Control
* Spawning of additional processes
* Intra. Vs. Inter-Communicators
* Unfortunately, rarely implemented or fully supported
* Additional concepts
* Communicator attributes
* Generalized requests
* Topologies
* MPI Tool Interfaces
* Transparent interception of MPI calls
* Query interface for internal state

## Advanced Message Passing Interface (MPI)
...