1.0 Introduction In recent years the federal government has mounted several large -scale evaluations of the effectiveness of various educational programs. For a variety of practical, ethical, and political reasons, strict adherence to the canons of experimental research design has generally been impossible. Our interest in the general problem of analyzing non experimental data has grown out of ...

This research introduces a new approach to model the non-linear relations among different design and manufacturing evaluation measures for multiobjective optimal concurrent design. In this approach, different design and manufacturing evaluation measures are mapped to comparable evaluation indices. The non-linear relation between an evaluation measure and its evaluation index is identified based...

As more evaluations of ubiquitous computing (ubicomp) applications are being undertaken, researchers are finding that the evaluations are more difficult than those for typical desktop computing applications. In this paper, we discuss some aspects of ubicomp that make these evaluations challenging. We present some properties unique to ubiquitous computing applications and suggest metrics to eval...

This paper surveys current notions of social and cultural presence as they may help the evaluation of cultural heritage projects. We argue that cultural heritage requires specialized evaluation, as key issues both connect and separate the aims of presence researchers and cultural heritage experts. To support this argument, three case studies of virtual heritage evaluations are summarized, and r...

Selecting and adopting the appropriate assessment method for conducting subjective quality assessments is a challenging task. The method decides whether the assessment is successful in delivering the correct answers to previously set up hypotheses. Therefore, in this paper we provide recommendations on test methods used in the domain of Sensory Experience. The proposed test methods comprise sin...

We describe a technique for the design and analysis of a simple asynchronous microprocessor from a Labelled Petri Net speciication. The implementation is obtained by means of reenement, transformation and translation. Several versions of the microprocessor design are presented, evaluated and compared. The Petri net based approach allows an interplay of diierent formal tasks, such as synthesis, ...

Flying Message Service (FMS) is a design project promoting community-based messaging in public spaces in the city. We have built a system that enables people to attach electronic messages to pigeons. From input stations located in public plazas, pigeons would transmit text messages to be displayed on large public screens at plazas throughout the city. The project evolved through ethnographic an...

We present a performance evaluation framework for Spoken Language Understanding (SLU) modules, focusing on three elements: (1) characterization of spoken utterances, (2) experimental design, and (3) quantitative evaluation metrics. We then describe the application of our framework to Scusi?— our SLU system that focuses on referring expressions.

Inspired by the success of evidence-based medicine, environmental scholars and practitioners have grown enthusiastic about applying a similar evidencebased approach to solve some of the world’s most pressing environmental problems. An important component of the evidence-based movement is the empirical evaluation of program and policy impacts. Impact evaluations draw heavily from recent advances...

This paper argues that the common practice of benchmarking is inadequate as a scientific evaluation methodology. It further attempts to introduce the empirical tradition of the physical sciences by using techniques from Statistical Design of Experiments applied to the example of SPARQL endpoint performance evaluation. It does so by studying full as well as fractional factorial experiments desig...