Synergistic human-machine prediction: active error analysis and mitigation with Gaussian process regression
| Author(s) | Claros-Olivares, Claudio Cesar | |
| Date Accessioned | 2021-02-11T13:55:06Z | |
| Date Available | 2021-02-11T13:55:06Z | |
| Publication Date | 2020 | |
| SWORD Update | 2020-10-12T19:03:34Z | |
| Abstract | Before deployment, a machine learning model is evaluated on both training and validation sets. Assuming the latter is a representative sample, the validation performance offers an estimate of how well the model will perform on a test set. Even if the performance meets specifications, there may be cases of systematic errors caused by model underfitting, poor model design or even overfitting. We propose to perform error analysis of the training and validation set during deployment to alert a user when instances similar to previous systematic errors arise. Triggering user vigilance during deployment will improve the synergistic operation of the machine and the user. Our model-agnostic approach interpolates the distribution of errors (taking cues from both the training and validation sets), optimizes the threshold for alerting a user, and requests verification for possibly erroneous predictions that exceed the threshold. Under the assumption that the user would make the correct decision, the approach is evaluated by the reduction of loss, while seeking to maintain a budget of interventions. The framework is tested on illustrative examples and real-world data sets where machine learning models have systematic errors. We conclude with a discussion of the limitations and areas for future work. | en_US |
| Advisor | Brockmeier, Austin J. | |
| Degree | M.S. | |
| Department | University of Delaware, Department of Electrical and Computer Engineering | |
| DOI | https://doi.org/10.58088/s01s-ty68 | |
| Unique Identifier | 1237265681 | |
| URL | https://udspace.udel.edu/handle/19716/28665 | |
| Language | en | |
| Publisher | University of Delaware | en_US |
| URI | https://login.udel.idm.oclc.org/login?url=https://www.proquest.com/dissertations-theses/synergistic-human-machine-prediction-active-error/docview/2455762029/se-2?accountid=10457 | |
| Keywords | Collaborative AI | en_US |
| Keywords | Machine learning | en_US |
| Keywords | Safe AI | en_US |
| Title | Synergistic human-machine prediction: active error analysis and mitigation with Gaussian process regression | en_US |
| Type | Thesis | en_US |