This paper reviews ve statistical tests for determining whether one learning algorithm out-performs another on a particular learning task. These tests are compared experimentally to determine their probability of incorrectly detecting a diierence when no diierence exists (type 1 error). Two widely-used statistical tests are shown to have high probability of Type I error in certain situations an...

In this paper we make two contributions. First, we show by example that empirical likelihood and other commonly used tests for parametric moment restrictions, including the GMM-based J-test of Hansen (1982), are unable to control the rate at which the probability of a Type I error tends to zero. From this it follows that, for the optimality claim for empirical likelihood in Kitamura (2001) to h...

We show by example that empirical likelihood and other commonly used tests for moment restrictions are unable to control the (exponential) rate at which the probability of a Type I error tends to zero unless the possible distributions for the observed data are restricted appropriately. From this, it follows that for the optimality claim for empirical likelihood in Kitamura (2001) to hold, addit...

1) Be able to perform the key steps to hypothesis testing 2) Understand the concept and use of the alpha value 3) Understand in general terms how the p value is derived 4) Correctly interpret the p value 5) Distinguish between type I and type II error 6) Understand the relationships among sample size, power, type I and type II error 7) Have a general notion of the concept of power as it relates...

Abstract Background Linear mixed models (LMM) are a common approach to analyzing data from cluster randomized trials (CRTs). Inference on parameters can be performed via Wald tests or likelihood ratio (LRT), but both approaches may give incorrect Type I error rates in finite sample settings. The impact of different combinations size, number clusters, intraclass correlation coefficient (ICC), an...