In Search of Black Swans: Identifying Students at Risk of Failing Licensing Examinations

Authors

Cassandra Barber, Western UniversityFollow
Robert Hammond, Western University
Lorne Gula, Western University
Gary Tithecott, Western University
Saad Chahine, Western University

Document Type

Article

Publication Date

3-1-2018

Journal

Academic Medicine

Volume

93

Issue

3

First Page

478

Last Page

485

URL with Digital Object Identifier

https://https:doi.org/10.1097/ACM.0000000000001938

Abstract

Purpose

To determine which admissions variables and curricular outcomes are predictive of being at risk of failing the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1), how quickly student risk of failure can be predicted, and to what extent predictive modeling is possible and accurate in estimating future student risk.

Method

Data from five graduating cohorts (2011-2015), Schulich School of Medicine & Dentistry, Western University, were collected and analyzed using hierarchical generalized linear models (HGLMs). Area under the receiver operating characteristic curve (AUC) was used to evaluate the accuracy of predictive models and determine whether they could be used to predict future risk, using the 2016 graduating cohort. Four predictive models were developed to predict student risk of failure at admissions, year 1, year 2, and pre-MCCQE1.

Results

The HGLM analyses identified gender, MCAT verbal reasoning score, two preclerkship course mean grades, and the year 4 summative objective structured clinical examination score as significant predictors of student risk. The predictive accuracy of the models varied. The pre-MCCQE1 model was the most accurate at predicting a student's risk of failing (AUC 0.66-0.93), while the admissions model was not predictive (AUC 0.25-0.47).

Conclusions

Key variables predictive of students at risk were found. The predictive models developed suggest, while it is not possible to identify student risk at admission, we can begin to identify and monitor students within the first year. Using such models, programs may be able to identify and monitor students at risk quantitatively and develop tailored intervention strategies.

Notes

Article available at Academic Medicine, Vol. 93(3).

https://https:doi.org/10.1097/ACM.0000000000001938