Bayesian student modeling, user interfaces and feedback: A sensitivity analysis

The Andes physics tutoring system has a student modeler that uses Bayesian networks. Although the student modeler was evaluated once with positive results, in order to better understand it and student modeling in general, a sensitivity analysis was conducted. That is, we studied the effects on accuracy of varying both numerical parameters of the student modeler (e.g., the prior probabilities) and structural parameters (e.g., whether the tutor uses feedback; whether the tutor insists that students correct errors; whether missing entries are counted as errors). Many of the results were surprising. For instance: Leaving feedback on when testing students improved the assessor’s accuracy; Long tests harmed accuracy in certain circumstances; CAI-style user interfaces often yielded higher accuracy than ITS-style user interfaces. Furthermore, we discovered that the most important problem confronted by the Andes student modeler was not the classic assignment of credit and blame problem, which is what Bayesian student modeling was designed to solve. Rather, it is that if students do not keep moving along a solution path, knowledge that they have mastered may not get a chance to apply, and thus the student modeler can not detect it. This factor had more impact on assessment accuracy than any other numerical or structural parameter. It is arguably a problem for all student modelers, and other assessment technology as well.