Improving foundation models on electronic health records
Date
2025
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Recent advances in foundation models have opened up new possibilities for healthcare applications, particularly by utilizing transformer-based models to take advantage of the longitudinal nature of both natural language and electronic health records (EHRs). While these models have shown promise, existing approaches face challenges related to multi-task learning; knowledge transfer from pre-training to finetuning stages; simultaneous representation of medical codes and visits; and potential social biases in predictions. ☐ The primary goal of this dissertation is to tackle these issues by presenting multiple transformer-based models while investigating and mitigating their issues related to fairness. Our proposed solutions have been evaluated on a multitude of popular medical predictive tasks. We first propose a transformer-based model tailored for multi-task learning, used for the primordial prevention of cardiovascular disease. Second, we tackle the issue of decreasing performance on small datasets with a semi-supervised transformer model that leverages both in- and out-of-cohort patients in the context of few-shot learning. Third, we propose a hybrid model that leverages graph neural networks to extract the structure of medical visits, and a transformer encoder to extract the temporal relationships of visits. Fourth, we investigate the fairness implications of our models and propose a bias mitigation technique based on federated learning principles. Lastly, we investigate the specific challenges of fairness in medical large language models (LLMs), conducting a comprehensive evaluation of the bias patterns. We then present a novel bias mitigation technique for medical LLMs based on model alignment ideas within a knowledge distillation framework.
Description
Keywords
Electronic health records, Multi-task learning, Large language models, Healthcare applications