Fine-Grain Software Distributed Shared Memory on SMP Clusters

Commercial SMP nodes are an attractive building block for software distributed shared memory systems. The advantages of using SMP nodes include fast communication among processors within the same SMP node and potential gains from clustering where remote data fetched by one processor is used by other processors on the same node. The widespread availability of SMP servers with small numbers of processors has led several researchers to consider their use as building blocks for Shared Virtual Memory (SVM) systems. These systems exploit the SMP cache-coherence hardware to support fine-grain communication within a node, and use software to support communication across nodes at a coarser page-size granularity. Our goal is to explore the use of SMP nodes in the context of the Shasta system. A unique aspect of Shasta compared to most other software distributed shared memory systems is that shared data can be kept coherent at a fine granularity. Shasta implements this coherence by inserting inline code that checks the cache state of shared data before each load or store. In addition, Shasta allows the coherence granularity to be varied across different shared data structures in a single application. This approach alleviates potential inefficiences that arise from the fixed large granularity of communication typical in most software systems. This paper describes a major extension to the Shasta system that supports fine-grain shared memory across SMP nodes. Allowing processors to efficiently share memory within the same SMP is complicated by race conditions that arise because the inline state check is non-atomic with respect to the actual load or store of shared data. We present a novel and efficient protocol that avoids such race conditions without the use of costly synchronization in the inline checking code. The above protocol is fully functional and runs on a prototype cluster of Alpha multiprocessors connected through Digital’s Memory Channel network. To characterize the benefits of using SMP nodes in the context of Shasta, we also present detailed performance results for nine SPLASH-2 applications running on this cluster.