One prevalent method for evaluating the results of automated software analysis tools is to compare the tools’ output to the judgment of human experts. This evaluation strategy is commonly assumed in the field of software clone detector research. We report our experiences from a study using several human judges who tried to establish “reference sets” of function clones for several medium-sized s...

Online knowledge repositories typically rely on their users or dedicated editors to evaluate the reliability of their content. These evaluations can be viewed as noisy measurements of both information reliability and information source trustworthiness. Can we leverage these noisy evaluations, often biased, to distill a robust, unbiased and interpretable measure of both notions? In this paper, w...

Defence organisations perform information security evaluations to confirm that electronic communications devices are safe to use in security-critical situations. Such evaluations include tracing all possible dataflow paths through the device, but this process is tedious and error-prone, so automated reachability analysis tools are needed to make security evaluations faster and more accurate. Pr...

Writing tasks are one way students can apply science concepts, yet evaluating students’ writing can be difficult in large classes. With the web-based Calibrated Peer Review* (CPR) system, students submit written work and evaluate each other. Students write a response to a prompt, read and evaluate responses prepared by the curriculum developers, and receive feedback on their evaluations, allowi...

Electronic communications devices intended for government or military applications must be rigorously evaluated to ensure that they maintain data confidentiality. High-grade information security evaluations require a detailed analysis of the device’s design, to determine how it achieves necessary security functions. In practice, such evaluations are labour-intensive and costly, so there is a st...

The current research used the contexts of U.S. presidential debates and negotiations to examine whether matching the linguistic style of an opponent in a two-party exchange affects the reactions of third-party observers. Building off communication accommodation theory (CAT), interaction alignment theory (IAT), and processing fluency, we propose that language style matching (LSM) will improve su...

This study focuses on the antecedents and consequences of Internet buyer regret in the overall purchasing process. We examine the roles that search effort, service-attribute evaluations, product-attribute evaluations and post-purchase price perceptions play in determining buyer regret and satisfaction in e-commerce. Furthermore, the study examines the consequences of regret and satisfaction in ...