Flawed external validation study of the ADNEX model to diagnose ovarian cancer

External validation studies of prediction models are of utmost importance in order to assess the performance of a predictionmodel in different locations (Altman et al., 2009).We therefore readwith interest the recent external validation study of the ADNEX model (Szubert et al., 2016). For patients with a persistent adnexal tumor who are scheduled for surgery, the ADNEXmodel predicts the risk of five tumor types: benign, borderline malignant, stage I cancer, stage II–IV cancer, or secondary metastatic cancer (Van Calster et al., 2014). The model was developed on data from 5909 patients collected at 24 centers, in 10 countries, between 1999 and 2012. ADNEX aims to assist cliniciansmake appropriate clinical decisions for patients presenting with an adnexal mass. When validating the ADNEXmodel, it is natural to first evaluate the prediction ofmalignancy, followed by themulticlass prediction ofmalignancy subtypes, in a similar way to other validation studies of multiclass models (Steyerberg et al., 1998). This approach is followed in the recent paper, but there are a number of important issues around the design, analysis, and reporting we wish to raise. First, validation studies should be designed to reliably assess performance in terms of discrimination and calibration (Steyerberg, 2009). In this particular case, the authors report a sample size calculation for testing the hypothesis that the AUC of themodel is higher than 0.5. Assuming an AUC of 0.94 leads to a very low required sample size (n = 22). This approach is at odds with methodological guidance and the result is that the precision of performance measures will be low: for dichotomous prediction, previous studies have suggested that at least 100, and preferably at least 200 individuals with the event (in this case ovarian malignancy) are required for a meaningful validation (Steyerberg, 2009; Vergouwe et al., 2005; Collins et al., 2016). Here, center 1 has 70 malignant tumors, whilst center 2 has only 34, leading to unreliable per center results. Validation would therefore best be done on all patients, with center-specific results as an exploratory addition. Furthermore, statistical tests to compare results between centers are provided throughout the text. Although heterogeneity of performance across locations is important (Riley et al., 2016), p-values to compare two specific centers are uninformative. It is useful to observe that the AUCs were 0.955 and 0.907, since this is in line with the center-specific values reported in the original publication describing the ADNEX model (Van Calster et al., 2014). A detailed investigation of heterogeneity should however involve a larger dataset with patients from many different