Background: Tracking surgical skill progression through “real-time” feedback could provide valuable information to learners and teachers alike but has proven challenging. Construct validity testing of such tracking is critically important.
Methods: OBGYN and FPMRS learners and teachers used the myTIPreport program for work-place feedback on surgical skills. The correlation of learners’ and teachers’ assessments of learners’ procedural abilities was examined by PGY-year. Additionally, senior learner performance, PGY-4s (residents) and PGY-7s (fellows), was compared to junior learner performance, as assesed by both learners and teachers, to begin the process of myTIPreport construct validity testing.
Results: From October, 2014-May, 2016, 12 OBGYN residency and 7 FPMRS fellowship programs participated. There were 439 unique learners and 442 unique teachers. Learners and teachers generated/recorded feedback on 4427 surgical procedures. Learner- and teacher-assigned performance curves were generated by PGY-year for all procedures rated in myTIPreport. Procedural levels assigned ranged from 1 to 5 (5 representing “expert”). Twenty of 26 procedures in the OBGYN myTIPreport had > 10 observations for index and subsequent procedures. Statistically significant correlations were seen between learner and teacher assessments for these 20 procedures with strong/very strong correlation noted (r > 0.71). As only 3 of 29 procedures in the FPMRS myTIPreport had > 10 observations for index/subsequent procedures, learner- and teacher-reported performance level correlations were not completed. Construct validity testing of myTIPreport to distinguish amongst PGY level performance was initiated for procedures with > 10 observations. For all 18 of 26 OBGYN procedures meeting this threshold, PGY-4s performed at a statistically significant higher level than PGY-1s as assessed by both learners and teachers. Similarly, for the 8 FPMRS procedures meeting this threshold, PGY-7s performed at a significantly higher level than PGY-5s.
Discussions: Strong correlation was found between OBGYN resident and teacher assessments of procedural ability using myTIPreport. The demonstrated significant effect of PGY level on procedural ratings by both learners and teachers supports emerging construct validity for myTIPreport.
Keywords: Assessment, Communication Skills, Faculty Development, Instructional Materials/Methods, Teaching Skills, Technology
Topics: CREOG & APGO Annual Meeting, 2017, Resident, Faculty, Residency Director, Residency Coordinator, Patient Care, Medical Knowledge, Professionalism, Systems-Based Practice & Improvement, Interpersonal & Communication Skills, Practice-Based Learning & Improvement, GME, Assessment, Faculty Development, Female Pelvic Medicine & Reconstructive Surgery, General Ob-Gyn,
AnnaMarie Connolly, MD, University of North Carolina
