Methods and systems for targeted evaluations of clinical machine learning models on the deployment population
Abstract/Contents
- Abstract
- The development of clinical machine learning applications frequently follows a sequential path from ideation to deployment — where each step is only considered after completion of the prior. A common consequence is that the distribution of data used to validate a model and permit eventual translation (the evaluation population) fails to reflect the distribution of data on which the model is served (the deployment population). In this dissertation I discuss methods and systems for enabling targeted evaluations of clinical machine learning models on the deployment population. I detail the implications of label selection (censoring) on the evaluation of binary classifiers, showing that traditional weighted estimators from causal inference literature recover performance over the deployment population when selection probabilities are properly specified. I emphasize the importance of silent prospective evaluations of machine learning models to appraise performance in the intended production environment. Silent trial evaluations require integrated deployment infrastructure with modern electronic medical record vendors — for which I develop and blueprint. I finally discuss the implications of feedback effects, where use of deployed models induce differences between the deployment population and the population that is prospectively observed. I provide a taxonomy of feedback effects, and recommend feedback aware model monitoring strategies. Instead of following a linear path from model ideation, to development and eventual deployment, researchers and machine learning practitioners should first consider the deployment population, and use it to contextualize model development and evaluation throughout its entire life-cycle.
Description
| Type of resource | text |
|---|---|
| Form | electronic resource; remote; computer; online resource |
| Extent | 1 online resource. |
| Place | California |
| Place | [Stanford, California] |
| Publisher | [Stanford University] |
| Copyright date | 2023; ©2023 |
| Publication date | 2023; 2023 |
| Issuance | monographic |
| Language | English |
Creators/Contributors
| Author | Corbin, Conor Kirkham |
|---|---|
| Degree supervisor | Chen, Jonathan H |
| Degree supervisor | Shah, Nigam |
| Thesis advisor | Chen, Jonathan H |
| Thesis advisor | Shah, Nigam |
| Thesis advisor | Baiocchi, Michael |
| Thesis advisor | Tibshirani, Robert |
| Degree committee member | Baiocchi, Michael |
| Degree committee member | Tibshirani, Robert |
| Associated with | Stanford University, School of Medicine |
| Associated with | Stanford University, Department of Biomedical Informatics |
Subjects
| Genre | Theses |
|---|---|
| Genre | Text |
Bibliographic information
| Statement of responsibility | Conor Kirkham Corbin. |
|---|---|
| Note | Submitted to the Department of Biomedical Informatics. |
| Thesis | Thesis Ph.D. Stanford University 2023. |
| Location | https://purl.stanford.edu/hg979cs4589 |
Access conditions
- Copyright
- © 2023 by Conor Kirkham Corbin
- License
- This work is licensed under a Creative Commons Attribution Non Commercial 3.0 Unported license (CC BY-NC).
Also listed in
Loading usage metrics...