In recent years, there has been much advancement in the field of ontology modularity. However, work on evaluation metrics for modules is lacking. For the studies that do exist on evaluation metrics, they specialise on only a few metrics, and there is not always a quantitative approach to calculate them. Furthermore, the metrics are not comprehensive enough to apply to the different types of mod...

The MUC-5 Scoring System is evaluation software that aligns and scores the templates produced by th e information extraction systems under evaluation in comparison to an "answer key" created by humans . The Scoring System produces comprehensive summary reports showing the overall scores for the templates in the test set ; these may be supplemented by detailed score reports showing scores for ea...

The MUC-4 evaluation metrics measure the performance of the message understanding systems . This paper describes the scoring algorithms used to arrive at the metrics as well as the improvements that were made to th e MUC-3 methods . MUC-4 evaluation metrics were stricter than those used in MUC-3. Given the differences in scoring between MUC-3 and MUC-4, the MUC-4 systems' scores represent a lar...

This paper discusses various metrics for game play evaluation based on user categories and market expectation. The use of various metrics improves the game testing ahead of the regular testing methods and also pinpoints the paradigm shift in the user taste. The use of metrics improves game play in conjunction with the user's psychological pattern that is usually age independent. We have al...

The MUC-3 evaluation metrics are measures of performance for the MUC3 template fill task. Obtaining summary measures of performance necessitates the los s of information about many details of performance . The utility of summary measures for comparison of performance over time and across systems should outweigh thi s loss of detail . The template fill task is complex because of the varying natu...

Certain generation applications may profit from the use of stochastic methods. In developing stochastic methods, it is crucial to be able to quickly assess the relative merits of different approaches or models. In this paper, we present several types of intrinsic (system internal) metrics which we have used for baseline quantitative assessment. This quantitative assessment should then be augmen...

In order to ensure that technology supports business needs and that IT investments deliver the desired value, it is fundamental to define an Information System Architecture (ISA) and measure its accurateness to the business model and existing technologies. Thus, in this paper we are concern on evaluating ISA by measuring its qualities (relevant at enterprise level). Since software architecture ...

The technical skill of surgeons directly impacts patient outcomes. Advanced tracking systems enable the development of objective motion-based metrics for skill evaluation, but these metrics are not sufficient to evaluate the performance in complex surgical tasks. In this study, we developed metrics for surgical skill evaluation that are based on the orientation of the surgical instruments. Expe...

This paper presents a set of metrics and algorithms for performance evaluation of object tracking systems. Our emphasis is on wide-ranging, robust metrics which can be used for evaluation purposes without inducing any bias towards the evaluation results. The goal is to report a set of unbiased metrics and to leave the final evaluation of the evaluation process to the research community analyzin...

The concept of a negative class does not apply to many problems for which classification is increasingly utilized. In this study we investigate the reliability of evaluation metrics when the negative class contains an unknown proportion of mislabeled positive class instances. We examine how evaluation metrics can inform us about potential systematic biases in the data. We provide a motivating c...