A novel agreement statistic using data on uncertainty in ratings
| Author(s) | Zee, Jarcy | |
| Author(s) | Mariani, Laura | |
| Author(s) | Barisoni, Laura | |
| Author(s) | Mahajan, Parag | |
| Author(s) | Gillespie, Brenda | |
| Date Accessioned | 2023-09-06T14:33:23Z | |
| Date Available | 2023-09-06T14:33:23Z | |
| Publication Date | 2023-07-15 | |
| Description | This is a pre-copyedited, author-produced version of an article accepted for publication in Journal of the Royal Statistical Society Series C: Applied Statistics following peer review. The version of record Zee, Jarcy, Laura Mariani, Laura Barisoni, Parag Mahajan, and Brenda Gillespie. “A Novel Agreement Statistic Using Data on Uncertainty in Ratings.” Journal of the Royal Statistical Society Series C: Applied Statistics, July 15, 2023, qlad063. https://doi.org/10.1093/jrsssc/qlad063 is available online at: https://doi.org/10.1093/jrsssc/qlad063. © The Royal Statistical Society 2023. All rights reserved. This article will be embargoed until 07/15/2024. | |
| Abstract | Many existing methods for estimating agreement correct for chance agreement by adjusting the observed proportion agreement by the probability of chance agreement based on different assumptions. These assumptions may not always be appropriate, as demonstrated by pathologists’ ratings of kidney biopsy descriptors. We propose a novel agreement statistic that accounts for the empirical probability of chance agreement, estimated by collecting additional data on rater uncertainty for each rating. A standard error estimator for the proposed statistic is derived. Simulation studies show that in most cases, our proposed statistic is unbiased in estimating the probability of agreement after removing chance agreement. | |
| Sponsor | The Nephrotic Syndrome Study Network Consortium (NEPTUNE) is part of the Rare Disease Clinical Research Network (RDCRN), funded by the National Institutes of Health (NIH) and led by the National Center for Advancing Translational Sciences (NCATS) through its Division of Rare Diseases Research Innovation (DRDRI). NEPTUNE is funded under grant number U54DK083912 as a collaboration between NCATS and the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK). Additional funding and/or programmatic support for this project has also been provided by the University of Michigan, NephCure Kidney International, Alport Syndrome Foundation, the Halpin Foundation, and Arbor Research Collaborative for Health. | |
| Citation | Zee, Jarcy, Laura Mariani, Laura Barisoni, Parag Mahajan, and Brenda Gillespie. “A Novel Agreement Statistic Using Data on Uncertainty in Ratings.” Journal of the Royal Statistical Society Series C: Applied Statistics, July 15, 2023, qlad063. https://doi.org/10.1093/jrsssc/qlad063. | |
| ISSN | 1467-9876 | |
| URL | https://udspace.udel.edu/handle/19716/33289 | |
| Language | en_US | |
| Publisher | Journal of the Royal Statistical Society: Series C | |
| Keywords | agreement | |
| Keywords | chance agreement | |
| Keywords | rater assessment | |
| Keywords | reproducibility | |
| Keywords | uncertainty | |
| Title | A novel agreement statistic using data on uncertainty in ratings | |
| Type | Article |
Files
Original bundle
1 - 2 of 2
Loading...
- Name:
- A novel agreement statistic using data on uncertainty in ratings.pdf
- Size:
- 1.04 MB
- Format:
- Adobe Portable Document Format
- Description:
- Main article
Loading...
- Name:
- A novel agreement statistic using data on uncertainty in ratings suppl.pdf
- Size:
- 734.49 KB
- Format:
- Adobe Portable Document Format
- Description:
- Supplementary files
License bundle
1 - 1 of 1
No Thumbnail Available
- Name:
- license.txt
- Size:
- 2.22 KB
- Format:
- Item-specific license agreed upon to submission
- Description: