Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples

Steven J. Holfinger*, M. Melanie Lyons, Brendan T. Keenan, Diego R. Mazzotti, Jesse Mindel, Greg Maislin, Peter A. Cistulli, Kate Sutherland, Nigel McArdle, Bhajan Singh, Ning Hung Chen, Thorarinn Gislason, Thomas Penzel, Fang Han, Qing Yun Li, Richard Schwab, Allan I. Pack, Ulysses J. Magalang

*Corresponding author for this work

Research output: Contribution to journalJournal Article peer-review

12 Scopus citations


Background: Prediction tools without patient-reported symptoms could facilitate widespread identification of OSA. Research Question: What is the diagnostic performance of OSA prediction tools derived from machine learning using readily available data without patient responses to questionnaires? Also, how do they compare with STOP-BANG, an OSA prediction tool, in clinical and community-based samples? Study Design and Methods: Logistic regression and machine learning techniques, including artificial neural network (ANN), random forests (RF), and kernel support vector machine, were used to determine the ability of age, sex, BMI, and race to predict OSA status. A retrospective cohort of 17,448 subjects from sleep clinics within the international Sleep Apnea Global Interdisciplinary Consortium (SAGIC) were randomly split into training (n = 10,469) and validation (n = 6,979) sets. Model comparisons were performed by using the area under the receiver-operating curve (AUC). Trained models were compared with the STOP-BANG questionnaire in two prospective testing datasets: an independent clinic-based sample from SAGIC (n = 1,613) and a community-based sample from the Sleep Heart Health Study (n = 5,599). Results: The AUCs (95% CI) of the machine learning models were significantly higher than logistic regression (0.61 [0.60-0.62]) in both the training and validation datasets (ANN, 0.68 [0.66-0.69]; RF, 0.68 [0.67-0.70]; and kernel support vector machine, 0.66 [0.65-0.67]). In the SAGIC testing sample, the ANN (0.70 [0.68-0.72]) and RF (0.70 [0.68-0.73]) models had AUCs similar to those of the STOP-BANG (0.71 [0.68-0.72]). In the Sleep Heart Health Study testing sample, the ANN (0.72 [0.71-0.74]) had AUCs similar to those of STOP-BANG (0.72 [0.70-0.73]). Interpretation: OSA prediction tools using machine learning without patient-reported symptoms provide better diagnostic performance than logistic regression. In clinical and community-based samples, the symptomless ANN tool has diagnostic performance similar to that of a widely used prediction tool that includes patient symptoms. Machine learning-derived algorithms may have utility for widespread identification of OSA.

Original languageEnglish
Pages (from-to)807-817
Number of pages11
Issue number3
StatePublished - 03 2022
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2021 American College of Chest Physicians


  • OSA
  • artificial neural network
  • electronic medical record
  • kernel support vector machine
  • machine learning
  • prediction model
  • random forest


Dive into the research topics of 'Diagnostic Performance of Machine Learning-Derived OSA Prediction Tools in Large Clinical and Community-Based Samples'. Together they form a unique fingerprint.

Cite this