The Fairness and Generalizability Assessment Framework

Abstract/Contents

Abstract

As artificial intelligence makes continuous progress to improve quality of care for some patients by leveraging ever increasing amounts of digital health data, others are left behind. Empirical evaluation studies are required to keep biased AI models from reinforcing systemic health disparities faced by minority populations through dangerous feedback loops. We developed a broadly applicable fairness and generalizability assessment framework and used it to perform a case study on a MIMIC-trained benchmarking model. While open-science benchmarks are crucial to overcome many study limitations today, this case study revealed a strong class imbalance problem as well as fairness concerns for Black and publicly insured ICU patients. Therefore, we advocate for the widespread use of comprehensive fairness and performance assessment frameworks to effectively monitor and validate benchmark pipelines built on open data resources.

Description

Type of resource	software, multimedia
Date created	2020 - 2021
Date modified	September 8, 2022
Publication date	September 2, 2022

Creators/Contributors

Author	Röösli, Eliane	https://orcid.org/0000-0003-0557-6239 (unverified)
Advisor	Bozkurt, Selen	https://orcid.org/0000-0003-1234-2158 (unverified)
Research team head	Hernandez-Boussard, Tina	https://orcid.org/0000-0001-6553-3455 (unverified)

Subjects

Subject	Stanford University. School of Medicine
Subject	Stanford Center for Biomedical Informatics Research
Subject	Fairness
Subject	Bias
Subject	Generalizability
Subject	Artificial intelligence
Subject	MIMIC
Subject	STARR
Genre	Software/code

Bibliographic information

Related item	Title GitHub repository Preferred citation Röösli, E., Bozkurt, S. & Hernandez-Boussard, T. Peeking into a black box: the fairness and generalizability of a MIMIC-III benchmarking model. Sci Data (2021)
DOI	https://doi.org/10.25740/tb877wd0973
Location	https://purl.stanford.edu/tb877wd0973

Access conditions

Use and reproduction

User agrees that, where applicable, content will not be used to identify or to otherwise infringe the privacy or confidentiality rights of individuals. Content distributed via the Stanford Digital Repository may be subject to additional license and use restrictions applied by the depositor.

License

This work is licensed under a Mozilla Public License 2.0.

Preferred citation

Preferred citation

Röösli, E., Bozkurt, S. & Hernandez-Boussard, T. (2021). The Fairness and Generalizability Assessment Framework. Stanford Digital Repository. Available at: https://purl.stanford.edu/tb877wd0973 https://doi.org/10.25740/tb877wd0973

Collection

Stanford Research Data

View other items in this collection in SearchWorks

Contact information

Contact

boussard@stanford.edu

Also listed in

View in SearchWorks

Loading usage metrics...