A Commodity High Performance PC Cluster Project Centered on a Programmable Network Processor

This project aims to construct a non-dedicated, high performance PC cluster using an advanced network processor for solving CPUand memory-intensive applications. Early cluster projects either use commonplace network technologies such as Ethernet or use specialized, more advanced network technologies such as Myrinet to connect commodity computers to form a single parallel processing system. Regardless what network technologies are used, these cluster projects typically implement their resource management and scheduling mechanisms at the application level, due to the limited programmability of ASIC-based switches, and hence sacrifice performance. While using specialized network technologies can help achieve higher throughput and lower latency at the message passing layer, clusters that use these technologies also require additional hardware investment on new switches and adapters (a new adapter for each node), which would compromise adoptability. The proposed cluster project balances the tradeoff between performance and additional hardware costs. It uses one programmable network processor as a switch to connect commodity computers via commonplace Fast Ethernet. In addition to being used as a regular network switch, the programmable network processor is used to implement the global resource manager, the job scheduler, and the distributed shared memory mechanism at the network level. This approach will boost the overall cluster performance by simplifying the algorithms for distributed environment and minimizing the number of control messages passed between cluster nodes. The advanced parallel architecture of programmable network processors, which delivers a combination of wirespeed deep packet inspection and full programmability, provides a solid platform and tools to achieve high cluster performance. An IXP2400 programmable network processor will be used to demonstrate the feasibility and effectiveness of this new approach. In addition, the proposed project will develop an execution environment transparent to applications. Thus, existing CPUand memory-intensive application programs can be directly executed in the cluster environment without to need to be reprogrammed. Also, the execution environment will combine global resource manager and workload migration mechanism to provide flexibility for nodes to join and leave the cluster at will.