Helping Intelligence Analysts Make Connections

Discovering latent connections between seemingly unconnected documents and constructing “stories” from scattered pieces of evidence are staple tasks in intelligence analysis. We have worked with government intelligence analysts to understand the strategies they use to make connections. Beyond techniques like clustering that aim to provide an initial broad summary of large document collections, an important goal of analysts in this domain is to assimilate and synthesize fine grained information from a smaller set of foraged documents. Further, analysts’ domain expertise is crucial because it provides rich contextual background for making connections and thus the goal of KDD is to augment human discovery capabilities, not supplant it. We describe a visual analytics system we have built— Analyst’s Workspace (AW)—that integrates browsing tools with a storytelling algorithm in a large screen display environment. AW helps analysts systematically construct stories of desired fidelity from document collections and helps marshall evidence as longer stories are constructed. Introduction What do the April’07 shootings at Virginia Tech, Bernard Madoff’s Ponzi scheme uncovered in Dec’08, and the March’09 recall of Zencore plus have in common? They are all extreme happenings that lead us to question: ‘Why didn’t somebody connect the dots?’ Our ongoing failures to do so have led to these and many other, arguably avoidable, catastrophes. Yet, piecing together a story between seemingly disconnected information remains an elusive skill and an understudied task. Storytelling is an accepted metaphor in analytical reasoning and in visual analytics (Thomas and Cook (eds.) 2005). Many software tools exist to support story building activities (Eccles et al. 2008; Hsieh and Shipman 2002; Wright et al. 2006). Analysts are able to lay out evidence according to spatial cues and incrementally build connections between them. Such connections can then be chained together to create stories which either serve as end hypotheses or as templates of reasoning that can then be prototyped. Copyright c © 2011, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. However, there are severe limitations to human sensemaking capabilities, even on gigapixel-sized displays, when confronted with massive haystacks of data. Algorithmic support to help sift through the myriad of possibilities is crucial here. At the same time, storytelling is not entirely automatable since it is an exploratory activity and the analyst brings in valuable intuition and contextual cues to direct the story building process. Hence it is imperative that we view storytelling as a collaborative enterprise between algorithmic and human capabilities. The focus of this paper is on exploring document collections and we present a visual analytics system called Analyst’s Workspace (AW) that aids intelligence analysts in exploring connections and building stories between possibly disparate end points. Our key contributions are: 1. Design considerations that have emerged from a detailed user study with five analysts working on intelligence analysis tasks. 2. New algorithms that find stories through document collections and also help marshall evidence to support discovered stories. 3. Implementation of both interactive visualization and algorithmic storytelling support in AW; and a case study over a public domain dataset. How Analysts make Connections We recently had the opportunity to interview and perform a study with five intelligence analysts currently employed at a government organization. The detailed results are presented and discussed in [Andrews et al. 2010]. We begin by describing qualitative lessons from the interviews followed by a study of their strategies in solving analysis tasks. Interviews with Analysts For the purpose of this paper, it suffices to note that the goal of the interviews was to attempt to typify how analysts approached the large quantities of data they were required to sift through, and to learn what tools they used and how they used them. From these interviews, the most interesting fact that emerged was that the analysts largely used software tools only at the beginning and at the end of their analysis. Basic search tools were used to filter down a dataset at the start of their analysis. At the end of the analysis, presenFigure 1: How intelligence analysts make connections (from (Pirolli and Card 2005).) tation tools (such as PowerPoint) would be used to create reports. For the middle of the analytic process, where the actual sensemaking occurs, the analysts in our study reported that they tended to print out reports and other source materials. This allowed them to easily read them, annotate them with notes and highlights, sort them into physical folders, stack them in meaningful ways on the desk, and even lay all the documents out on a large table where they could be organized and rapidly skimmed. A formal way to characterize the above observations is with reference to the schematic of Pirolli and Card (Pirolli and Card 2005). As Fig. 1 shows, the process by which intelligence analysts make connections is frequently tentative and evolutionary, with structures developing as understanding of the data increases. There are two ‘subloops’ in Fig. 1: information foraging and sense-making. Most analytic systems, such as IN-SPIRE (PNNL ), Jigsaw (HCII ), ThemeRiver (Havre et al. 2002), NetLens (Kang et al. 2007) focus on support for the information foraging loop, leaving the sensemaking to the analyst. Other tools, such as Analyst’s Notebook (i2group ), Sentinel Visualizer (FMS, Inc. ), Entity Workspace (Bier et al. 2006), and Palantir (Khurana et al. 2009) focus more on the the sensemaking loop, and while many of them ostensibly support foraging, the analysts reported using these tools primarily for late stage sensemaking and presentation. The key problem with this separation of the two halves of the sensemaking process is that the schematic is not meant to be a state diagram – it is a representation of some of the thought processes and structures that are identifiable during sensemaking and a description of how they relate. There is an overall trend from a collection of raw data to a final report, but inbetween, the analyst should be ranging widely across the entire process, building up an understanding through progressive foraging and structuring.