Date of Award
2007
Degree Type
Thesis
Degree Name
Master of Science
Program
Epidemiology and Biostatistics
Supervisor
Dr. John Kova
Second Advisor
Dr. Allan Donner
Abstract
This thesis deals with robustness of confidence interval construction for the kappa statistic based on a dichotomous response in the presence of rater bias. Two methods of construction are considered: The Goodness-of-fit (GOF) ap proach proposed by Donner and Eliasziw [10] and the Modified Wald method proposed by Donner and Zou [11] using estimates (MLE’s) of 7 and K under the common correlation model. These approaches produce asymmetric confidence intervals. Bloch and Kraemer [4] proposed the common correlation model which is a simplification of models proposed by Landis and Koch [22], Kraemer [20] and Mak [24]. This model assumes the same marginal probability of success for each observer i.e. no rater bias. Kraemer claims that the assumption of a common success rate holds if the observers are picked randomly from a population of ob servers [21]. This can be due to reasons such as differences in training, experience etc of the observers. The model was therefore extended to a heterogenous form to accommodate differences in observer success rates and has the parameters K, the coefficient of agreement, 71 and 72, the true success rates for each observer and n, the sample size. The evaluation and comparisons were done using bias and confidence interval coverage and calculations were carried out using exact calculation methods. The overall general results suggest that as K and n increase, the interval coverage decreases, and moreover, the coverage is also dependent on the indi vidual success rates of the judges and their distance from a success rate of 0.5. Cohen’s kappa consistently has lower bias than the intraclass kappa and is the iii preferred estimator. The two confidence interval construction approaches per form well and similarly when the difference in rater bias is less than 10% in terms of coverage and confidence interval width. However, as the difference in success rates increases, the coverage of the two methods decreases below nominal levels and moreover, the bias for the intraclass kappa and Cohen’s kappa also increase. When success rates differ by 10% or less, we recommend the use of the Goodness-of-fit approach. As the the difference in success rates increases beyond 10%, the estimation of confidence intervals should not be calculated using the investigated methods.
Recommended Citation
Parpia, Sameer N., "EVALUATION OF CONFIDENCE INTERVAL CONSTRUCTION METHODS FOR THE KAPPA STATISTIC IN THE PRESENCE OF RATER BIAS" (2007). Digitized Theses. 5117.
https://ir.lib.uwo.ca/digitizedtheses/5117