Competency Assessment Tool (CAT)

Grant:

Gilbert Whitaker Fund for the Improvement of Teaching

Academic Year:

2011 - 2012 (June 1, 2011 through May 31, 2012)

Funding Requested:

$14,960.00

Project Dates:

Sat, 01/01/2011 - Mon, 12/31/2012

Applicant(s):

mbeleh

Overview of the Project:

This project plans to implement and evaluate a Competency Assessment Tool (CAT) that will be used to track individual student's competency level for each of course outcomes. This tool will allow students to self evaluate their competency level for each course outcome and provide evidence and artifacts that support such evaluation. Instructors can then use the tool to support the students' self evaluation or make changes, providing students with a rationale for the changes. This tool allows for constant evaluation of students' progression and provides them with instant feedback as they progress through the course. The tool may serve as an alternative to examinations and other traditional evaluation methods particularly in case of certain skills such as communication or clinical skills and as a learning tool that allows instant and continuous evaluation of students' competency. The project plans to assess the use of the CAT tool in two courses. In the first course, the CAT will be used as part of a remediation plan for struggling students in the course to track their competency level for course outcomes and provide feedback to help them succeed in the course. In the second course, the CAT will serve as an integral teaching and assessment tool, where the instructor will examine students' self evaluation on the competency scales of the CAT and make changes providing feedback to students. This will continue throughout the semester providing an excellent opportunity for constant assessment and interaction between student and instructor. To assess the usefulness of this CAT tool both students and faculty will be surveyed to explore perspectives, perceptions and usefulness of the tool. The study will also compare individual students' scores in the course with the level of achievement of students regarding outcomes as set by the CAT tool. Conclusions from these comparative studies may help in providing evidence for expanding the role of CAT in student assessment and making changes in how graded tasks are designed.

Final Report Fields

Project Objectives:

1. To design and construct the Competency Assessment Tool (CAT) 2. To implement CAT in a four courses, with an eye to include in other clinically-related courses. 3. To evaluate the correlation between examination scores and competency levels as measured by the CAT. 4. To evaluate the efficacy of the CAT tool as a learning tool. 5. To assess students’ level of satisfaction and perception of the CAT and faculty's evaluation of its usefulness and applicability.

Project Achievements:

In its initial implementation of the tool, it has been highly successful. As one of the tools for the remediation program in the Medicinal Chemistry/Pharmacology course sequence, studies have shown that remediated students perform better on final cumulative examination than non-remediating students, including those who scored better on midterm examinations. These results were presented at the 2012 annual AACP meeting in Kissimmee, Florida and was well received. A manuscript has been submitted for publication based on these results. The tool was also used in the first year of the drug discovery course as the main evaluation method. Students received the tool well, in particular the constant feedback and the ability to demonstrate competency of course outcome, versus answering questions, which are either correct or incorrect. A number of faculty at the College of Pharmacy have inquired about the tool, and are interested in using it as an assessment method to certain skills in their course. Once the changes planned for the tool are complete, we plan on including the tool in their courses.

Continuation:

We plan to continue making changes to the tool and evaluating its use and usefulness during the stage II of the grant.

Dissemination:

We have already presented the use of the tool as part of the remediation program in the Medicinal Chemistry/Pharmacology sequence at the annual AACP meeting in July 2012. We have also submitted a manuscript for the same work in September 2012. We have been active presenting the tool to other faculty at the College of Pharmacy to explore its use beyond the courses where it is currently used. We have been successful in attracting interest from a number of courses at the College. During the stage II period, we plan on presenting the work in the drug discovery course, where the tool plays a central role in the assessment process and provided we have supporting data, submit a manuscript for publication. We will continue to actively share the tool and our finding with other faculty at the college and possibly in other units in the university. The possible inclusion of this tool in CTools would help with such dissemination.

Advice to your Colleagues:

The tool relies on evaluation of competency level rather than test students based on a question/answer format or completing an assignment or task. Beyond its intended use, we have found this method to better reflect student knowledge and ability to apply this knowledge. Constant feedback to students regarding their competency level leads to a high motivation level to improve on their knowledge level and explore ways to show such level. Students find this mode of assessment of high education value. Even if examinations or assignments are returned to students, they do not review them carefully. This tool motivates students to investigate their weaknesses and work towards correcting them. The tool also helps instructor focus on what they want students to take out of their courses and to evaluate the actual outcomes of the course. Constant, timely and constructive feedback is the major factor that facilitates the success of the tool. Students also must feel that the tool plays an important role in their final grade. Finally challenging assignments are essential for the success of the tool. The main challenge of this method of evaluation is to have students work on the tool throughout the semester instead of waiting till the end of the marking period, since most of the outcomes are continuously measured. This was overcome by including landmark dates where students must reach a specific competency level to be eligible to receiving an A grade.

PRINTER VERSION

SEND BY EMAIL