Towards a Common Evaluation Framework for Cyber Security Visualizations

With the increasing number of cyber security visualization tools, literature has repeatedly highlighted the need for a common evaluation framework to help assessing and quantifying the effectiveness of proposed tools and validate their adequacy to targeted usages. However, there is no research yet that supports the development of such framework. We present a work in progress and preliminary ideas towards building a common evaluation framework for cyber security visualizations. Many authors have contributed to the state of the art of evaluation in the field of cyber security visualization. Staheli et al. [4] systematized a taxonomy for evaluable components, which have also been derived from evaluation work in other research fields. The methodologies described by Sethi et al. [3] and Suo et al. [5] reflect the different forms of an evaluation framework for cyber security visualization and methods to address its development. Moreover, through an analysis of research works from the VizSec venue, Staheli et al. [4] highlighted that among the forms of evaluation that are included in the selected works, some dimensions and evaluation techniques (e.g., psychophysiological methods) are not yet considered. This is especially because doing so would require knowledge that is not necessarily covered by the expertise of cyber security analysts. Besides, considering a target user’s requirements as evaluation metrics, as described by Sethi et al. [3], is undoubtedly an important form of evaluation but covers only qualitative aspects. In order to build a comprehensive framework, this could also be completed with other relevant aspects such as quantitative ones.