Are we All on the Same Page? Capturing and Visualizing Real-time Multi-source Evaluations using MIRROR in Team-Based Simulations

Are we All on the Same Page? Capturing and Visualizing Real-time Multi-source Evaluations using MIRROR in Team-Based Simulations

Academic Year:
2022 - 2023 (June 1, 2022 through May 31, 2023)
Funding Requested:
$10,000.00
Project Dates:
-
Overview of the Project:
To enhance the current high-stakes learning environment, technological advances are needed to provide team- and learner-specific feedback to coach learners in retaining and improving cognitive and behavioral skills required for team-based clinical care (Kozlowski & Chao, 2018; Rosen et al., 2018a,b). This project aims to fill this gap by building upon the previously validated Radar graphical tool to develop and evaluate a novel real-time 3600 assessment and feedback system: MIRROR (MultIsource RadaR Tool On Team Reflexivity). MIRROR maps the four constructs of clinical decision-making (seeking information from the patient, seeking information from the team, evaluating information, and planning and acting on a decision), where output is a visualization of multisource evaluations from the instructor, peer-observers, learner, and team on a given simulated situation. Through this, instructors will be able to provide new insights and personalized feedback during reflection-based debriefing sessions to allow for more meaningful reflection, targeted intervention, and rapid development of these complex skills.
Final Report Fields
Project Objectives:

This project was set out to determine the effects of a novel MIRROR (MultIsource RadaR Tool On Team Reflexivity) debriefing system using dental care related simulation scenarios (respiratory and cardiovascular cases). MIRROR maps the four constructs of clinical decision making (seeking information from the patient, seeking information from the team, evaluating information, and planning and acting on decision), where output is a visualization of multisource evaluations from instructor, peer-observers, learner, and team on a given simulated situation. We employed a 2x2 factorial design to investigate two key factors in simulation-based debriefing: multi-source feedback (MSF) and instructor-guided facilitation with video review.

The primary objectives are:

  1. To evaluate how the addition of the MIRROR feedback tool to post-simulation debriefing affects participants' satisfaction with debriefing and reflection, as well as the overall quality of the debriefing process.
  2. To compare the quality of feedback and trainees' reflective behavior during debriefings that use the MIRROR feedback tool versus those that rely on conventional, observational practices.

 

Project Achievements:

With the generous support from CRLT, our project team developed and implemented the MIRROR feedback tool, which maps four constructs of clinical decision-making. This tool allows for semi-automated support in problem identification and strategy development, specifically targeting clinical management and teamwork skills. This new tool enables instructors to provide more personalized and insightful feedback during reflection-based debriefing sessions.

Our team successfully piloted and implemented two iterations of this tool in the Human Simulation Laboratory that combined the 2nd year dental Respiratory and cardiovascular system courses in the clinical simulation setting. This learning experience is designed to help students better understand and apply difficult concepts in pulmonary and cardiovascular physiology to their future clinical practice.

This study makes several theoretical, methodological, and educational contributions:

First, our findings enhance our understanding of how teams process discrepant feedback information and how facilitation can improve team-level reasoning and co-construction of new understandings. MSF brings feedback processing to a higher level (in terms of depth of processing and team processing). Feedback discrepancies contained in MSF seem to create a learning opportunity for teams to co-construct new understandings. By contrast, when teams only receive feedback from their instructor, they keep moving to different types of information or components without sharing and building upon those different components. They act more as individuals and less as teams. Adding human facilitation to a debriefing seems to help team members engage in high reasoning with each other, build upon each other, and co-construct new understandings (higher and team processing). If teams are only given feedback data (and no facilitation), they tend to make sense of those data, but they stick to individual-level reasoning. In other words, they rather focus on “what do those data mean to me?” than on “what can we learn from each other as a team”?

Second, methodological innovation. The study introduced Ordered Network Analysis (ONA) to analyze team communication patterns in post-simulation debriefings, providing a more comprehensive understanding of team processes (more details in the attached files).

Third, the integration of the Human Simulation Laboratory into challenging dental courses has significantly enhanced the curriculum design. This approach addresses conceptually difficult topics, particularly in respiratory and cardiovascular physiology, helping students better translate critical concepts to their future clinical practice. Building on this, the introduction of multi-source evaluations and the MIRROR tool not only provides a more comprehensive view of individual learner and team performance but also has the potential to promote clinical decision-making skills. By providing an empirical basis for designing future large-scale studies aimed at promoting clinical decision-making skills, the project positions the department as a leader in evidence-based dental education.

Furthermore, this project has fostered interdisciplinary collaboration, bringing together experts from various fields such as dental education, emergency medicine, and learning health sciences. This collaborative effort has strengthened interdisciplinary ties across the schools and departments generating to new knowledge in teaching and learning practices that can be applied beyond dental education.

Continuation:
The project is continuing beyond the grant period by expanding its focus to include reflection-in-action during simulations, addressing a significant research gap in high-fidelity simulation studies. This expansion involves developing and implementing Cognitive Aids with Multisource Feedback to support students in acquiring crucial metacognitive skills for effective reflection during simulations.
Dissemination:
The project's activities and findings have been disseminated through multiple channels, with plans for further dissemination in the near future. The research team has already presented preliminary findings at the INGRoup’ 2024 conference, sharing insights with colleagues in the field of group dynamics and team research. To reach a broader audience within dental education, we plan to present the work at the American Dental Education Association conference and at the International Meeting on Simulation in Healthcare, which will expose the project to a wide range of health profession educators and simulation experts. Our team is currently developing a manuscript for publication in a peer-reviewed journal.
Advice to your Colleagues:
One of the key factors that facilitated the success of the MIRROR feedback tool project was the interdisciplinary collaboration. By bringing together experts from dental education, emergency medicine, and learning health sciences, the project team was able to draw on diverse perspectives and expertise, leading to a richer and more innovative outcome. Another factor is piloting and implementing the MIRROR tool across two iterations, which required meticulous planning and coordination within the simulation lab setting. Potential advice is to establish clear protocols and maintain regular communication with all instructors, PI team, and students.