APGO

Study Objective: Characterize the psychometric properties of an Obstetric Communication Assessment Tool (OCAT) for a novel OB communication module.

Methods: We developed four difficult OB Standardized Patient (SP) scenarios: Religious Beliefs (RB), Angry Father (AF), Maternal Smoking (MS), and Intimate Partner Violence (IPV). A trial run was performed with SPs and twenty-one third year OB/GYN clerkship students. Five trained Standardized Patient Reviewers (SPRs) independently scored twenty-four randomized video-recorded encounters. Internal consistency (IC) was estimated with Cronbach’s alpha. Inter-rater reliability (IRR) was estimated using the Intraclass Correlation Coefficient-2 (ICC-2) test. Systematic variability in reviewer responses was assessed using the Stuart-Maxwell test.

Results: IC was acceptable to excellent with alpha values (95% Confidence Intervals [CI]): RB 0.91 (0.86, 0.95), AF 0.76 (0.62, 0.87), MS 0.91 (0.86, 0.95), and IPV 0.94 (0.91, 0.97). IRR was unacceptable with ICC-2: RB 0.46 (0.40, 0.53), AF 0.48 (0.41, 0.54), MS 0.52 (0.45, 0.58), and IPV 0.67 (0.61, 0.72). Stuart-Maxwell analysis indicated systematic variability in reviewer scoring.

Conclusions: We have developed the OCAT with high IC but unacceptable IRR. Our findings suggest that the OCAT effectively discriminates learner performance but systematic differences in SPR rigor require a revised training regimen to improve reliability. Dedicated OB communication training is lacking, despite the wealth of uniquely sensitive and difficult clinical topics unaddressed by conventional communication curricula. We propose the optimization of the OCAT as an essential step toward development of a focused OB communication module and discuss future directions of the project