A Distributed Continuous Quality Assurance Process to Manage Variability in Performance-intensive Software

Performance-intensive software is increasingly being used on heterogeneous combinations of OS, compiler, and hardware platforms. Examples include reusable middleware that forms the basis for scientific computing grids and distributed realtime and embedded systems. Since this software has stringent quality of service (QoS) requirements, it often provides a multitude of configuration options that can be tuned for specific application workloads and run-time environments. This paper describes the architecture of Skoll, which is a DCQA environment containing software QA processes and tools that leverage the extensive computing resources of worldwide user communities to significantly and rapidly improve software quality. It describes novel modeling tools and modeling language BGML that allow Skoll users to capture the system’s axes of variability (such as configuration options, QoS strategies, and platform dependencies) to generate scaffolding code needed to conduct QA tasks on remote machines. It describes experiments that apply BGML to systematically evaluate and improve the performance of DRE component middleware on a range of platforms and configuration options. The results show that automatic analysis of QA task results can significantly improve software quality by capturing the impact of software variability on performance and providing feedback to help developers optimize performance.