A holistic approach to calibrating clinical dental faculty for assessments that support a “growth mind-set”

A holistic approach to calibrating clinical dental faculty for assessments that support a “growth mind-set”

Academic Year:
2018 - 2019 (June 1, 2018 through May 31, 2019)
Funding Requested:
$9,960.00
Project Dates:
-
Overview of the Project:
Standardization of faculty, which occurs as a result of calibration activities, is a particular challenge across all of dental education. Dental students must learn and be assessed on literally hundreds of procedures, techniques and clinical activities. At the UMSD as in other dental schools, dental faculty are diverse, graduating from clinical training programs around the world, with variations in clinical philosophies of care. The UMSD employs hundreds of full-time and adjunct-faculty often teaching in different clinical disciplines and in multiple locations, who must be calibrated on teaching and assessment; these factors create barriers to successful and sustainable calibration programs using traditional approaches. Calibration activities seek to achieve “consistent application of protocols, techniques, and philosophies, so the student experience is as consistent as possible.” (McAndrew, 2016). In this proposal, we present a much broader approach to calibration that supports and broadens faculty members’ knowledge of educational principles and their ability to facilitate a growth mind-set culture and humanistic learning environment within the UMSD learning environment. This program will utilize a variety of technology-based solutions to make learning opportunities more accessible to faculty and provide innovative ways to track and communicate the outcomes of calibration activities. Program goals will include increasing faculty intra and inter-rater reliability for selected student assessments and increasing faculty participation and collaboration in calibration activities. Study design and analysis will include a needs assessment, preliminary assessment of current systems using mixed methods, a calibration training intervention and a program outcomes assessment.
Final Report Fields
Project Objectives:

1. Provide consistent foundational knowledge to faculty on the evidence based educational principles supporting the growth mindset.

2. Increase faculty intra and inter-rater reliability for selected student assessments.

3. Increase faculty participation and collaboration in calibration activities.

Project Achievements:
  1. Needs assessment was completed using restorative faculty test case ratings of student performance using Tableau to identify assessments with low inter-rater reliability. These were targeted for calibration workshops.
  2. Programs were planned to meet the objectives of this study and increase faculty calibration:
    1. Calibration workshops in restorative dentistry were implemented (to date 11 workshops have been conducted) including topics on restorative policy, alignment of clinical faculty with what is taught preclinically and grading of test cases.
    2. Calibration workshops in prosthodontics, endodontics, periodontics and other disciplines have been conducted.
    3. Two workshops on growth mindset have been conducted.
    4. An online, self-paced calibration program was implemented in Canvas. 
      • Topics:  importance of calibration, growth mindset, providing effective feedback, preclinical/clinical assessment basics and 14-point assessment of overall competency.
      • To date 153 faculty have completed the program.
  3. A new faculty cohort program was established that includes foundation knowledge of educational best practices and growth mindset.  To date, two cohorts with a total of 18 faculty have participated in the program.
  4. The number of clinical faculty participating in calibration has increased since the beginning of the study.  Approximately 60% of predoctoral clinical faculty have participated in one or more calibration activities.
Continuation:
The calibration activities developed through this project will continue into the foreseeable future. This will include educational principles and growth mindset. In addition, we plan to continue using Tableau to track faculty inter-rater reliability for calibration.
Dissemination:
The online calibration program was presented during the American Dental Education Association Annual Session in 2022.
Advice to your Colleagues:
Faculty calibration is an important and worthwhile activity, however it is challenging to measure its impact. Factors facilitating the success of this project were the contributions of dental informatics staff (setting up Tableau) and the efforts of discipline faculty to develop online and in-person calibration sessions.
Our greatest challenge was the COVID-19 pandemic. Time and resources were diverted to address the impacts of the shutdown. On the other hand, the pandemic did increase our comfort with online instruction. The implementation of an online calibration program and virtual calibration workshops helped us overcome the challenges of social distancing. Tableau is a useful tool in helping assess inter- and intra-rater reliability in clinical assessments.