A novel agreement statistic using data on uncertainty in ratings
Date
2023-07-15
Journal Title
Journal ISSN
Volume Title
Publisher
Journal of the Royal Statistical Society: Series C
Abstract
Many existing methods for estimating agreement correct for chance agreement by adjusting the observed proportion agreement by the probability of chance agreement based on different assumptions. These assumptions may not always be appropriate, as demonstrated by pathologists’ ratings of kidney biopsy descriptors. We propose a novel agreement statistic that accounts for the empirical probability of chance agreement, estimated by collecting additional data on rater uncertainty for each rating. A standard error estimator for the proposed statistic is derived. Simulation studies show that in most cases, our proposed statistic is unbiased in estimating the probability of agreement after removing chance agreement.
Description
This is a pre-copyedited, author-produced version of an article accepted for publication in Journal of the Royal Statistical Society Series C: Applied Statistics following peer review. The version of record Zee, Jarcy, Laura Mariani, Laura Barisoni, Parag Mahajan, and Brenda Gillespie. “A Novel Agreement Statistic Using Data on Uncertainty in Ratings.” Journal of the Royal Statistical Society Series C: Applied Statistics, July 15, 2023, qlad063. https://doi.org/10.1093/jrsssc/qlad063 is available online at: https://doi.org/10.1093/jrsssc/qlad063. © The Royal Statistical Society 2023. All rights reserved. This article will be embargoed until 07/15/2024.
Keywords
agreement, chance agreement, rater assessment, reproducibility, uncertainty
Citation
Zee, Jarcy, Laura Mariani, Laura Barisoni, Parag Mahajan, and Brenda Gillespie. “A Novel Agreement Statistic Using Data on Uncertainty in Ratings.” Journal of the Royal Statistical Society Series C: Applied Statistics, July 15, 2023, qlad063. https://doi.org/10.1093/jrsssc/qlad063.