Resident Self-Assessment on a Case-Based Oral Examination

Objective: We developed a multi-institutional, case-based resident oral examination with standardized questions and evaluation instrument. We sought to assess the accuracy of trainee self-evaluation on their performance and to evaluate the examination coverage of the core competencies.

Methods: A competency-based oral examination was administered to sixteen PGY3-6 trainees from two different plastic surgery training programs. Each case had eight to eleven designed evaluation points scored zero to three. Two examiners also graded the overall performance on each case dichotomously for mastery of the subject. Post-examination, trainees were asked to evaluate their own performance and quality of the exam via an separate assessment tool.

Results: Trainee self-assessment on performance was skewed toward estimation of a poorer performance compared to the actual evaluation by the examiners. Trainees underestimated their raw average score at least 0.5 points below (on a 0 to 3 scale) the examiners 61% of the time. Trainees overall predicted “not mastered” on 61% of cases compared to only 33% of “not mastered” grades given by examiners. The positive predictive value for trainee self-evaluation of “mastered” was 78 to 90% and was between 27 to 60% for “not mastered” depending on how “split” voters were counted.  Overall trainee feedback on the exam experience was overwhelming positive. All six core-competencies were felt to be covered with Medical Knowledge, Patient Care, and Communication being better covered than Professionalism, Systems Based Practice, and Practice Based Learning.

Conclusions: We found that trainees tend to underestimate their own performance on a case-based oral examination. While self-assessed mastery of a concept is recognized by trainees up to  90% of the time, trainee prediction of not mastering a topic was poor. The overall case-based oral examination experience was evaluated as a positive experience and was felt to include all six core competencies.