HiFi: A New Monitoring Architecture for Distributed Systems Management

With the increasing complexity of large-scale distributed (LSD) systems, an efficient monitoring mechanism has become an essential service for improving the performance and reliability of such complex applications. This paper presents a scalable, dynamic, flexible and non-intrusive monitoring architecture for managing large-scale distributed (LSD) systems. This architecture, which is is referred to as the HiFi monitoring system, detects and classifies interesting primitive and composite events and performs either a corrective or steering action. When appropriate, information is also disseminated to management applications, such reactive control tools. The monitoring architecture employs a hierarchical event filtering approach that distributes the monitoring load and limits event propagation. This significantly improves scalability and performance while minimizing the monitoring intrusiveness. The architecture provides dynamic monitoring capabilities through subscription policies that enable users to add, delete and modify monitoring demands on-the-fly, and a programmable environment that facilitates development of self-directed monitoring tasks. Increased flexibility is achieved through a declarative and comprehensible monitoring language, simple code instrumentation process, and automated monitoring agents administration. The outlined solution offers improvements over related works by supporting new monitoring techniques such as hierarchical filtering-based monitoring and filter incarnation that improve the monitoring scalability and dynamism which are required for managing large-scale distributed systems. The HiFi monitoring system has been implemented and used at the Old Dominion University for monitoring and steering Interactive Remote Instruction (IRI) which is a large-scale distributed multimedia system for distance learning.