When humans work together, “a team of individual experts does not necessarily make for an expert team,” cautioned Rice Computer Science Assistant Professor Vaibhav Unhelkar.
He was talking about that necessary ingredient for successful teamwork–understanding the bigger picture, having the wide-angle perspective that takes in everything. In short, the role of a coach.
There are many instances, however (such as in hospitals and during natural disasters), where people need to work together well without a coach present. To help team members to train for such situations, Unhelkar and Rice CS PhD student Sangwon Seo have developed an AI coach, whose origins and purpose they explain in the paper “Socratic: Enhancing Human Teamwork via AI-enabled Coaching.”
Seo and Unhelkar are scheduled to present the paper at the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS25) in May.
What’s in a name
This new AI coach is named after the Socratic method, said Unhelkar. “This system is called Socratic [short for ‘the System for Objective Coaching through Automated Task-time Interventions for Collaboration’] because it doesn't tell a team what to do. It asks them, and then the team figures out what to do by themselves,” just as the Socratic method is founded on asking a series of questions.
Because each person on a team is focused on their individual responsibilities, “sometimes they lose the big picture and cannot tell whether they are on the same page,” said Unhelkar. “Our system is filling in that gap,” he said. Socratic “gauges whether not being on the same page is going to lead to suboptimal performance, and if it is, then it pings the team,” asking them to reconsider their actions to better align.
To prevent alert fatigue, said Seo, Socratic tries to time its interventions appropriately and “not to provide advice intervention too much.” In addition, text outputs from the coach are worded carefully, so team members “do not feel obligated to follow the instructions every time. We also want to respect the autonomy of humans,” he said.
Humans seem receptive, so far. “In general, [study] participants perceived Socratic positively,” Seo said. “They felt the Socratic system was intelligent and very effective.” Unhelkar added, “The gold standard of whether it works or not is humans.”
Real-time interventions
Using AI to bolster teamwork is not new. There are currently “fantastic tools” that generate summaries and create action item lists for teams, said Unhelkar, but these “contributions have historically occurred in the planning or evaluation stages of a task.” Seo agreed, noting there are “many approaches to improve teamwork, but mostly they can only be applied before or after the task.”
The beauty of Socratic is its real-time interaction. “With our system, it's an on-the-fly feedback. It's looking at the team as they are doing the task and intervening,” Unhelkar explained.
Something else that sets Socratic apart is that its suggestions are not based on video footage or audio transcripts, but on observed human behavior and inferred intent. To develop a system to infer team member intent, researchers mathematically modeled task and team dynamics and collected data from training sessions and practice sessions.
“This is a big part of my research,” said Seo, because, in the context of teams and teamwork, “there was no existing work that modeled human behavior that is dependent on intent.”
To develop how and when Socratic intervened with members, researchers said they drew from existing frameworks from multi-agent systems (that is, computerized systems with lots of autonomous parts interacting), modifying them for Socratic’s human teams.
Interdisciplinary knowledge
Developing an AI coach that understands human team interaction and tries to predict human intent demanded expertise well beyond the field of computer science. To begin with, that of medical doctors.
“The genesis of this project was teamwork in the operating room,” said Unhelkar. The researchers’ ultimate goal is to employ Socratic in “training environments–surgical simulation facilities” to catch preventable medical errors, Unhelkar said.
“Our co-author Dr. [Marco] Zenati, a cardiac surgeon and professor of surgery from Harvard Medical, has been studying medical teams for a long time, and he has been investigating inefficiencies and ways to address them in medical teams,” Unhelkar explained. Other co-authors from Harvard Medical School are Roger D. Dias and Rayan E. Harari.
Co-author Bing (Tim) Han, a 2024 Rice graduate with a BS in Computer Science, helped develop intervention strategies and analyzed the efficacy of the intervention algorithm.
“In addition to expertise in medicine and AI, we need somebody who understands teams themselves, right?” Unhelkar said. “For that, we were very fortunate to work with a preeminent expert in teaming, Dr. Eduardo Salas.” Co-author Salas is professor and chair of Psychological Sciences at Rice whose research specialities include teamwork, team training and development, and human-computer interaction. “The research has really been made possible from folks who understand teams from a psychological standpoint,” said Unhelkar.
Success, limitation, and future direction
“AI can assist humans as they perform collaborative tasks. That's the key takeaway” of this research, said Unhelkar. “I think it's really amazing that we are able to show this; there are a lot of things that needed to work just right to make this work.”
He added, “This is possible even with very small amounts of data. We try to make systems which are ‘sample efficient,’ so they don't require too many data samples to do the training.”
He also admitted that the current system has far to go. “Our experiments are limited to a very specific environment, a game-like environment, and they aren’t happening in the real world, so that's a limitation and opportunity for future work,” Unhelkar said. “In a game, we can measure everything, we can get that data, but in practice, there’s still a gap.”
The ultimate goal of using Socratic during surgical simulation is currently “a very grand vision, and there are a lot of problems to be solved” before that is possible, Unhelkar said. “Our project collaborators at Harvard Medical are working on how you can measure surgical teams using physiological sensors and cameras, and that's the piece we haven't put together yet.”
But this “first prototype” already carries “a lot of implications in terms of how human teams are trained…. The motivation is the operating room, but the solutions we are building are more general. Similar challenges can also occur in other domains,” such as disaster response, he said. “Our system can be a game changer in terms of how teams are trained.”
This research is funded by the National Science Foundation (award id number 2205454).