Use of InfoSleuth to Coordinate Information Acquisition, Tracking and Analysis in Complex Applications

Today’s world is characterized by accessibility to a wide variety of information sources. This provides both the luxury of being able to know and use more information and the problem of accessing it in the manner required by our work and our computer applications. Many applications, including simulation-type applications, are no longer hampered by the availability of data, but rather are concerned with the accessibility of that data. We see three separate needs that an informationoriented infrastructure can provide to such applications. These are: 1. Similar information may exist in many places, but in incompatible forms or formats. Applications need to view it as if it were coming from a single source. 2. Information processing must integrate both computer-based applications and real machines, such as sensors, giving a uniform methodology to deal with all kinds of information sources and processes. 3. Applications often must track the changing state of the information to develop up-to-date information feeds, summaries and analyses. InfoSleuth employs intelligent agent technology to provide for integrated concept-based access to and awareness of unstructured, heterogeneous, and distributed information in a dynamically changing network of servers, data-collecting machines, and the World Wide Web. InfoSleuth agents can leverage existing legacy information resources, obviating the need for migrating information out of those sources and into new technologies. InfoSleuth agents collaborate to combine speedy information retrieval tasks with more cumbersome analysis and processing tasks. Furthermore, InfoSleuth is especially capable of monitoring changing or continuous data sources and converting event patterns to the appropriate level of abstraction for the users of the system. From the user perspective, InfoSleuth uses concept-based access to information sources, analysis tools, and machines. Concept-based access is far more accurate and focused than that provided by currently available technologies. With InfoSleuth, users receive information at the level of abstraction and integration appropriate to their task. We believe the InfoSleuth technology will help an application to go far beyond the reach of current information gathering and analysis technologies by facilitating the defining and building up of new applications from diverse, existing components. InfoSleuth Functionality InfoSleuth is a very powerful agent based software application that performs information retrieval and fusion, event detection, data analysis, knowledge discovery and trend analysis using existing databases or the internet as data sources. Summaries of the core functionality, representing important advantages of this technology, include: • Concept-based access paradigm: InfoSleuth uses concept based addressing, rather than syntactic features such as keywords. • User perspective: The InfoSleuth user formulates queries using his or her own vocabulary. An "ontology", in InfoSleuth terminology, is the vocabulary appropriate to the user's domain of interest, rather than the (often cryptic) local names and structures that happen to appear in any given database or text resource. Thus different users may use entirely different vocabularies to reference identical information, which in turn may be maintained under some schematic vocabulary unfamiliar to both users. • User Power: The InfoSleuth user has at his or her disposal the expressive power of the standard structured database language SQL, rather than the limitations of structured keyword search. Supplementing this, the user may interact with the system through any of a variety of commercial or user supplied graphical user interfaces. • Information retrieval and fusion: InfoSleuth agents access and fuse information from a wide variety of types of information sources, including external machines, databases, text and image repositories and the World Wide Web. • Monitoring capabilities: InfoSleuth gives the user, on request, dynamic focused notification as the world of data changes. The user only need specify the style of information to be monitored. InfoSleuth transparently maps this to event monitoring on the appropriate resources. • Distributed processing: InfoSleuth processes data where the data is. It enhances efficiency by distributing the processing of queries and data manipulations among multiple agents, each responsible for some subpart of the entire world of information. • Collaborative Processing: InfoSleuth Agents cooperate with each other by pooling their resources to answer complex queries. • Dynamic Architecture: InfoSleuth agents can come and go, i.e. be initiated, killed or moved, and InfoSleuth increases (or degrades) gracefully, using whatever services are available through the currently available set of agents. • Scalability: InfoSleuth is extensible to a changing distributed world of information under a paradigm similar to that allowing growth of the Internet. Technical Discussion and Design Method: Agents, Clients, & Tools InfoSleuth is designed as an agent-based, objectoriented system. It was designed using certain hierarchical methodologies so that users can create and modify InfoSleuth agents easily using a standard set of interfaces and underlying components. The InfoSleuth system consists of agents, clients and tools. Clients are user interfaces built using a common API. Agents are designed as instances of a set of Java classes called the generic agent shell. Agents communicate via conversations (which are specified using finite state automata) using a language called KQML (Knowledge Query and Manipulations Language) co-developed by academic and commercial participants interested in forging a de facto standard agent communications language. Tools such as the ontology creation and maintenance tools are built independently and with no overriding architectural hierarchy or relationship. Clients allow users to update or query information via InfoSleuth and/or define event streams that are dynamically instantiated. Clients can subscribe to complex events extracted from the event stream by application of an event algebra. Changes to data anywhere in the information space can be thought of as an event. Thus, any client can ask to be notified when any datum changes values, or when some formally specified combination of data values changes. Other events might include a specific user logging on, or more than some number of users logging on simultaneously, a new data source joining the system (or an existing one leaving) etc. Users may create applications using Info/Sleuth’s SleuthClient application programming Interface (API) or using the supplied TQML (Template Query Markup Language) interface. Agents initiate, translate, decompose, receive, and synthesize queries and data. They are the "engine" of the system. All of the agents are designed as instances of a generic agent shell and thus all contain a common set of capabilities. This fixed portion of the agent architecture includes a set of finite state automata-based conversations that describe the kinds of conversations that are allowed. These conversations do not specify which agents participate in a conversation, but only which types of agents can participate. Thus any agent who knows how to produce some part of the answers to a request may join a conversation, or choose not to do so. Fundamental to the operation of this dynamic architecture is a set of broker agents that advise agents on how to locate other agents that provide required functionality to complete a particular task or subtask. The broker keeps track of the different agents in the system, making it an authority on who is out there. Other agents access the broker as they need to; thus, the brokers effectively facilitate the construction of dynamic, short-lived cooperative communities of agents to perform specific tasks. They also can balance application loads across similar agents. Particular classes of agents extend the capabilities inherited from the generic agent shell to perform specific tasks. The list of agent classes that have been implemented and evaluated under the InfoSleuth R&D project is quite large and only a few will be described here. For descriptions of other agents, and more detail, visit the InfoSleuth web pages that can be found by asking through the MCC web site located at http://www.mcc.com/projects. • Portal Agents provide an interface tailored to the user’s needs and maintain a persistent state for the user so that requests can continue to execute and return results even when the user has logged off the system. • Resource Agents advertise the contents of the information resource that they manage, and translate between the InfoSleuth lingua franca and the language spoken locally by the information resource technology. Resource agents may interface to data, image, or document bases, information-gathering machines such as sensors and specialized machines, and informationgenerating programs such as operating systems or web crawlers. • Broker Agents collectively maintain an awareness of the dynamically changing set of available agents, including the content of the currentlyavailable information space. Brokers offer advice to other agents that are attempting to discover or monitor information or events. • Query Agents accept queries from users or other agents and decompose them into subqueries that each target an individual resource agent, and integrate the results as requested. This provides complex, distributed query processing to the users. • Subscription Agents accept monitoring requests from users or other agents and maintain an active watch over the requested data or event specifications. • Analysis Agents monitor streams of data performing different types of analysis as needed by the application. For instance, Deviation Detection Agents monitor streams of data for significant variations from the normal values for that data, or from variations from the current trends in that data. They use thresholds that may have been statically specified, or learned over time. • Control Agents execute complex, multi-step tasks, where each task can be a query, a subscription, or the running of an analysis task such as deviation detection. Implementation Methodology Implementation of several large domain specific applications using the InfoSleuth technologies has provided proof of the concept that agent-based systems can significantly reduce the time and cost of developing and deploying systems that access large amounts of distributed, heterogeneous information. The first step in creating any new application is agreeing on the "domain of discourse" of the users which allows an ontology to be specified and implemented. Implementation of the ontology requires no programming as such, but only the specification of a structure similar to an entityrelationship diagram needs to be produced. Tools exist for easily converting the specification to an implementation usable by InfoSleuth. Once the ontology(s) have been created and added to the system, it is necessary to create a mapping and advertisement for each of the information resources that will participate in the application. For databases, the administrator of each information resource uses a set of point and click tools to map (some or all) of the information to the ontology, and to create an advertisement of the information that the information resource claims to be willing to export. For other types of information resources, which have non-standard or legacy interfaces, this mapping requires some direct implementation over a Resource Agent Shell. InfoSleuth provides such implementations for operating system commands and for web crawlers. The final step is creating a specialized user interface that meets the needs of the user community. Creation of these interfaces can be done with the InfoSleuth tool suite with only minor programming required. Legacy GUIs can often be interfaced to the InfoSleuth system.