The Canadian Ophthalmology Assessment Tool for Surgery: Development and Validation of a Surgical Assessment Tool for Ophthalmology Training Programs

Theme: Medical Education
What: Medical Education
Part of: Symposium for Young Ophthalmologists and Residents: Visionary Horizons - Preparing for a Future in Eye Care / Symposium pour les jeunes ophtalmologistes et les résidents: Horizons visionnaires – Préparer l’avenir des soins oculaires
When: 5/30/2024, 11:30 AM - 05:00 PM
Where: Room | Salle 716 B

Abstract

Purpose

Competency-based medical education emphasizes frequent work-place based assessment and requires valid and reliable assessment tools for this purpose. In ophthalmology, one area that requires assessment is surgical competence. Although multiple assessment tools have been proposed to assess surgical competence, the majority are not validated. Furthermore, all the previously proposed tools are specific to only surgical procedure or skill. The primary objectives of this study were to develop a surgical assessment tool to evaluate the breadth of surgical procedures in which ophthalmology residents are required to gain competence during their postgraduate training and to validate this tool for cataract surgery.

Study Design

Prospective Cohort Validation Study

Methods

The Canadian Ophthalmology Assessment Tool for Surgery (COATS) was developed by a group of experts by modifying the O-SCORE, a previously validated tool, to make it more specific to ophthalmology. Ophthalmology residents and faculty in the University of Ottawa Department of Ophthalmology were invited to participate in the study. Data collection was performed over a two-year period from July 1, 2021 to June 30, 2023. To examine validity, the effect of stage of training, number of cataract surgeries performed, and overall procedural independence on total COATS score was evaluated. To examine reliability, a generalizability analysis was performed.

Results

There were 160 COATS assessments collected across 5 residents. Mean total COATS scores were higher for the last three blocks of cataract surgery training compared to the first two blocks (4.81±0.03 vs. 4.33±0.25, p=0.012) and for procedures where the resident was rated “independent” compared to procedures where the resident was rated as “not independent” (4.74±0.06 vs. 4.26±0.13, p=0.006). There was a significant correlation between the number of cataract surgeries performed and mean total COATS score (Pearson r = 0.20, p=0.02). The overall reliability of the COATS measured using the G-coefficient was 0.39. A subsequent D-study revealed that 45 COATS assessments per resident would be required to obtain a G-coefficient of 0.70, the accepted threshold for intermediate stakes assessments.

Conclusions

The results of this study suggest that the COATS is a valid tool for the assessment of surgical competence in cataract surgery. There is also evidence that the COATS is a reliable tool when completed multiple times per resident over the course of training. Future research to further determine the sources of variability in COATS assessments and to model learning curves for cataract surgery is warranted.

Presenter(s)

Presenting Author: Jeffrey Mah

Additional Author(s):

Meghan McConnell, Department of Innovation in Medical Education, University of Ottawa

Kaisra Esmail, Alberta Health Services

Irfan Kherani, Department of Ophthalmology & Vision Sciences, University of Toronto

Stephanie Baxter, Department of Ophthalmology, Queen's University

Bernard Hurley, Department of Ophthalmology, University of Ottawa, Ottawa Hospital Research Institute

Michael O'Connor, Department of Ophthalmology, University of Ottawa, Ottawa Hospital Research Institute

Chloe Gottlieb, Department of Ophthalmology, University of Ottawa, Ottawa Hospital Research Institute

---