Overcoming Distributed Debugging Challenges in the MPI+OpenMP Programming Model

There is a general consensus that exascale computing will employ a wide range of programming models to harness the many levels of architectural parallelism [1], including models to exploit parallelism in CPUs and devices, such as OpenMP. To aid programmers in managing the complexities arising from multiple programming models, debugging tools must enable programmers to identify errors at the level of the programming model. However, the question of what the effective levels for debugging in hybrid distributed models are, remains unanswered. In this work, we present a novel framework to build an intuitive stack trace view of MPI+OpenMP programs. We develop a methodology to reconstruct call stacks for OpenMP threads and share our lessons learned from incorporating OpenMP awareness into a highly-scalable, lightweight debugging tool for MPI applications: the Stack Trace Analysis Tool (STAT) [2]. Our framework leverages OMPD [3], an emerging debugging interface for OpenMP, so that we can evaluate the effective levels of debugging for MPI+OpenMP. Our easy-to-understand stack trace views help users debug MPI+OpenMP programs at the user code level by mapping the stack traces to the high-level abstractions provided by programming models.