· Publications · 2 min read
How recording inputs shape AI assessment of physical exams
New in BMC Medical Education: a UTSW study examines what audio, video, and transcripts contribute to AI-assisted OSCE scoring.
Our retrospective study of recording inputs for AI-assisted physical-examination scoring is now published online in BMC Medical Education, as an Article in Press on November 13, 2026.
Read the journal article · Download the publisher PDF
The study examined 300 objective structured clinical examination (OSCE) encounters across six UT Southwestern stations, comparing Gemini 2.5 Pro scores with a locally constructed composite reference as recording inputs were removed.
Three-camera video with audio achieved the highest agreement with that reference (quadratic-weighted κ = 0.832, 95% CI 0.775–0.885). Removing audio substantially reduced agreement. This matters because OSCE speech includes students describing their actions and standardized patients responding to them: agreement with rubric scores does not by itself show that a model can visually judge examination technique.
The comparison with routine standardized-patient evaluators also requires care. The AI advantage was sensitive to one rubric item with different scoring instructions; after excluding that item, the difference was not statistically significant. The composite reference incorporated standardized-patient and emergency medical technician ratings, with joint physician–EMT adjudication of disagreements. It does not independently establish physical-examination competence, and generalization to unscripted clinical encounters remains untested.
For UT-REAL, this UTSW study informs recording choices and the questions that further validation must address. It reports a single-institution retrospective analysis, rather than multi-site consortium outcomes.
Citation: Kang S, Holcomb MJ, Shakur AH, Hein D, Ngo H-T, Schuler H, Jarrett PC, Dalton TO, Jamieson AR. How recording inputs affect multimodal AI agreement with reference scores in OSCE physical examinations: a retrospective ablation study. BMC Medical Education (2026). doi:10.1186/s12909-026-10779-6.
The peer-reviewed Article in Press is citable and may receive further production edits. Springer Nature will replace it with the final Version of Record at the same DOI. The earlier medRxiv preprint remains available for the publication history.
Explore Research & Publications for related work, including the prospective MAPLES deployment study.


