Introducing Endpoints into the EMPI4Re MPI library

Endpoints is a topic under discussion within the MPI forum [3] that proposes to introduce a new communicator creation function that creates a communicator with multiple ranks for each MPI process in a parent communicator. In this talk we will discuss our initial implementation of endpoints in our research MPI library, called EMPI4Re. The Exascale MPI library for Research (the EMPI4Re library) is an MPI-1 library created by EPCC at The University of Edinburgh as a vehicle for research into new MPI functionality. The library adopts the conceptual model of Partitioned Global Address Space (PGAS) and assumes hardware support for Remote Direct Memory Access (RDMA) operations. This conceptual model enables efficient implementation of remotely accessible double-buffered first-in-first-out (FIFO) queues, used for point-to-point operations, and distributed state control structures, used for collective operations. Modern machine architectures, such as the Cray XC30, support RDMA operations only for exposed segments of process memory. Data movement of arbitrary application buffers is enabled by collective window creation operations (using MPI terminology) or symmetric memory allocation operations (using DMAPP terminology). The code-base for the EMPI4Re library currently consists of 55,495 lines of C code (c.f. 933,889 for OpenMPI version 1.8.6) including source code and header files for both the library and its test suite. The endpoint function, MPI_COMM_CREATE_ENDPOINTS, is collective over all the MPI processes in the parent communicator and returns an array of handles for the new communicator. Each handle in the array corresponds to one of the ranks in the new communicator. The ranks represented by the handles in the array are contiguous and the starting rank is defined by the rank of the calling MPI process in the