Comparing the Borderline Group Method with the Borderline Regression Method in Standard Setting and Measuring OSCE Quality Assurance Metrics in a Malaysian Family Medicine Examination

Abstract

Objective: To determine if there were any significant difference in cut scores and any advantage in using the borderline group method (BGM) versus the borderline regression method (BRM) in standard setting the Objective Structured Clinical Examination (OSCE) in a high stakes Family Medicine examination. Methods: All the 2018 and 2019 Part 2 Conjoint MAFP/FRACGP examination OSCE marks, together with the cut scores obtained from BGM, were compiled and entered into SPSS version 23. BRM cut scores were obtained by plotting linear regression graphs using the candidates’ total marks against their global assessment grade in each OSCE station. The cut scores from both methods were compared to see if there were any significant difference (p value<0.05). OSCE quality assurance metrics (Cronbach’s alpha, R2 coefficient, intergrade discrimination, number of failures, and between-group variation percentage, for each OSCE station) were calculated and the results analysed. Results: No significant difference in cut scores were found using both methods. Using BRM, additional OSCE quality assurance metrics (R2 coefficient, intergrade discrimination) were obtained from the linear regression graphs as compared to BGM. Conclusions: BRM required more work to compute but had the advantage of producing more OSCE quality assurance metrics. Cut scores could be obtained even when no candidates were assessed as borderline in any particular station, making it suitable for small scale OSCEs with fewer candidates. As a result, BRM is now used in standard setting for Part 2 conjoint examinations.