Simulation for Assessment of Milestones in Emergency Medicine Residents

Abstract

Introduction

All residency programs in the United States are required to report their residents' progress on the Milestones to the Accreditation Council for Graduate Medical Education (ACGME) biannually. Since the development and institution of this competency-based assessment framework, residency programs have been attempting to ascertain the best ways to assess resident performance on these metrics. Simulation was recommended by the ACGME as one method of assessment for many of the milestone subcompetencies. We developed 3 simulation scenarios with scenario specific Milestone based assessment tools. We aimed to gather validity evidence for this tool.

Methods

We conducted a prospective observational study to investigate the validity evidence for 3 mannequin-based simulation scenarios for assessing individual residents on Emergency Medicine (EM) milestones. The subcompetencies (i.e. PC1, PC2, PC3) included were identified via a modified Delphi technique using a group of experienced EM simulationists. The scenario specific checklist items were designed based on the individual milestone items within each EM subcompetency chosen for assessment and reviewed by experienced EM simulationists. Two independent live raters who were EM faculty at the respective study sites scored each scenario following brief rater training. The interrater reliability (IRR) of the assessment tool was determined by measuring intraclass correlation coefficient (ICC) for the sum of the checklist (CL) items as well as the global rating scales (GRS) for each scenario. Comparing GRS and CL scores between various PGY levels was performed with ANOVA.

Results

Eight subcompetencies were chosen to assess with 3 simulation cases, using 118 subjects. Evidence of test content, internal structure, response process and relations with other variables were found. The ICCs for the sum of the CL items and the GRSs were > 0.8 for all cases, with one exception (clinical management GRS 0.74 in sepsis case). The Sum of Checklist items and GRSs (p<0.05) discriminated between post graduate (PGY) levels on all cases. However, when the specific CL items were mapped back to milestones in various proficiency levels, the milestones in the higher proficiency levels (level 3 & 4) did not often discriminate between various PGY levels. Level 3 milestone items discriminated between PGY levels on 5 out of 12 occasions they were assessed, and Level 4 items discriminated only 2 out of 12 times they were assessed.

Conclusion

Three simulation cases with scenario specific assessment tools allowed evaluation of EM residents on proficiency levels 1-4 within 8 of the EM Milestone subcompetencies. Evidence of test content, internal structure, response process and relations with other variables were found. Good to excellent IRR and the ability to discriminate between various PGY levels was found for both the sum of CL items and GRSs. However, there was a lack of a positive relationship between advancing PGY level and the completion of higher level milestone items (levels 3 and 4).

This article is protected by copyright. All rights reserved.

http://ift.tt/2vOWQ9l