Subgroup deployment-readiness audit of calibration and clinical utility in perioperative infection prediction
In brief
A 0.876 AUROC masked subgroup calibration gaps in infection predictions
In 10,719 procedures, the model discriminated well overall, but calibration and decision-making value differed by physiological reserve. Recalibration brought overall calibration close to ideal, yet predictions still overestimated infection risk in a later time period. Infection coding was largely absent before 2018 and differed across reserve groups, so prospective validation with comparable outcome tracking is needed before clinical use.
- Journal
- Journal of the American Medical Informatics Association : JAMIA (Q1)
- Published
- 30 October 2026
- Study design
- Prospective / inception cohort
- Evidence level
- Level 2, Moderate (CEBM 2b)
- Authors
- Hugo Guillen-Ramirez, Katharina Lucia Lucas, Yves Max Wintsch, Tobias Ueli Blatter, Karen Triep, Olga Endrich, et al.
- PMID
- 42962674
- DOI
- 10.1093/jamia/ocag184
Why clinicians should know about it
- Picked for Health Informatics (paper of the day, 2 November 2026): Deployment-readiness audit examines calibration, subgroup safety, and clinical utility
Abstract
OBJECTIVE: We audited deployment readiness of an end-of-surgery postoperative infection model across clinically relevant subgroups and assessed mitigation of subgroup miscalibration. MATERIALS AND METHODS: We analyzed out-of-fold predictions from 10 719 procedures (2014-2022) with 504 present-on-admission (POA)-coded postoperative infection events across recorded sex, age, and a clinical physiological-reserve proxy. We assessed discrimination, calibration, threshold-specific errors, and decision-curve net benefit, treating t = 0.10 as a reference anchor. We compared no recalibration with common and group-specific cross-fitted isotonic recalibration, evaluated a refit without the Synthetic Minority Oversampling Technique (SMOTE), and performed a temporal-holdout analysis. RESULTS: Overall area under the receiver operating characteristic curve (AUROC) was 0.876, with similar sex-marginal discrimination in women and men (0.878 vs 0.875). Calibration-in-the-large was -1.02 overall, -0.86 in frail, and -1.34 in prefrail patients; the 6-event nonfrail stratum was not estimable. At t = 0.10, net benefit was positive in frail and negative in prefrail patients in the source cohort, but negative prefrail utility did not persist in the ascertainable-period (2018-2022) and temporal analyses. Common recalibration moved overall calibration-in-the-large from -1.020 to -0.007. Removing SMOTE shifted overall calibration but did not reduce the reserve gradient. Temporal-holdout AUROC was 0.851 with persistent overprediction. Present-on-admission coding was effectively absent before 2018; prevalence-dependent results were therefore additionally reported for the ascertainable 2018-2022 period, and ascertainment differed by physiological reserve. DISCUSSION: Discrimination-only or sex-marginal evaluation would have missed deployment-relevant subgroup heterogeneity, while endpoint ascertainment affected prevalence-dependent interpretation. CONCLUSION: Joint subgroup evaluation can identify deployment-relevant limitations missed by aggregate metrics. Prospective validation with comparable outcome ascertainment remains necessary before clinical deployment.
Abstract as published, via PubMed.
For healthcare professionals. The summary is generated by AI from the published abstract, and the evidence level is assigned automatically from the study design on the Oxford CEBM hierarchy. Neither is medical advice. Read the full paper before changing practice.