Perspective Video Interview: Benchmarks for AI Agents and Medical Trainees

Added: 2026-05-19

[00:00:00]Researchers and AI developers have been very creative at developing very effective benchmarking systems to test their tools ability to make a diagnosis, to come up with a management plan, and a lot of it is published. Some of it is open, some of it is actually owned by the AI developers, which is potentially problematic.

[00:00:17]But, as I've looked at these benchmarks, including the benchmarks of creating clinical vignettes that specifically test a model's ability to say I don't know, that allowed me that made me reflect on our own education, meaning it would be it would be useful to have similar benchmarks also for trainees, where, for example, trainees could be taught could be given cases or clinical vignettes embedded in which there's embedded falsehoods or false test, and where the expected behavior that you assess for is could a student say I don't know when there is a factitious element in a particular case.

[00:00:59]And so, developing these benchmarks both for the human clinician and the the AI co-pilot, so to speak, is is I think the next the next step forward. And these are being developed right now. There's a very robust line of research um defining benchmarks across a multitude of clinical tasks, including the ability to say I don't know.