Irregular Redistribution Scheduling by Partitioning Messages

Dynamic data redistribution enhances data locality and improves algorithm performance for numerous scientific problems on distributed memory multi-computers systems. Regular data distribution typically employs BLOCK, CYCLIC, or BLOCK-CYCLIC(c) to specify array decomposition. Conversely, an irregular distribution specifies an uneven array distribution based on user-defined functions. Performing data redistribution consists of four costs: index computational cost, schedule computational cost, message packing/unpacking cost, and data transfer cost. Previous results focus on reducing the former three costs. However, in irregular redistribution, messages with varying sizes are transmitted in the same communication step. Therefore, the largest sized messages in the same communication step dominate the data transfer time required for this communication step. This work presents an efficient algorithm to partition large messages into multiple small ones and schedules them by using the minimum number of steps without communication contention and, in doing so, reducing the overall redistribution time. When the number of processors or the maximum degree of the redistribution graph increases or the selected size of messages is medium, the proposed algorithm can significantly reduce the overall redistribution time to 52%. Moreover, the proposed algorithm can be applied to arbitrary data redistribution while slightly increasing the communication scheduling time.