Robust semantic confidence scoring

Comparing confidence-based and conventional scoring methods: The case of an English grammar class

This study aimed at investigating the reliability, predictive validity, and self-esteem and gender bias of confidence-based scoring. This is a method of scoring in which the test takers receive a positive or negative point based on their rating of their confidence in an answer. The participants, who were 49 English-major students taking their grammar course, were given 8 multiple-choice tests d...

Evaluating semantic-level confidence scores with multiple hypotheses

In any dialogue manager, confidence scores play a central role in ensuring robust operation. Recently, dialogue managers have attempted to exploit N-best lists of alternatives for the semantics rather than the single most likely interpretation. Each alternative in the N-best list must have an associated confidence score and it is very useful to be able to evaluate the utility of these scored li...

Improving rejection with semantic slot-based confidence scores

This paper introduces a new confidence scoring mechanism which takes advantage of the semantic parsing provided by a natural language understanding system. As a result, different segments of the user input which fill individual semantic slots are identified and individual semantic slot confidence scores are generated. With slot-based confidence scores, each individual slot receives a confidence...

A Comparative Study on the Effect of the Formative Use of Confidence-Based Scoring and Conventional Scoring on Iranian EFL Learners’ Grammar Improvement

Confidence Scoring for Speech Understanding Systems1

This research investigates the use of utterance-level features for confidence scoring. Confidence scores are used to accept or reject user utterances in our conversational weather information system [10]. We have developed an automatic labeling algorithm based on a semantic frame comparison between recognized and transcribed orthographies. We explore recognition-based features along with semant...