Development and Validity Evidence for a Holistic Admissions Assessment Rubric

Abstract

Background: Few evidence-based tools exist to holistically assess applicants during the admissions process for pre-doctoral dental school. Prior studies focus more on the interview process itself rather than the application that the candidate submits. Without validated assessment tools, institutions risk selecting applicants that may not fit their mission or values. This study aimed to assess the validity of a Holistic Admissions Assessment Rubric (HAAR) created for Kansas City University College of Dental Medicine for the 2024-2025 application cycle. Methods: Messick’s universal validation framework was used to assess the validity of the HAAR in the five domains of content, response process, internal structure, relationship to other variables, and consequence. A nominal group process was first held with admissions content experts to solidify the final version of the HAAR for application cycle use. The HAAR was used to assess 100 applicants during the 2024-2025 pre-doctoral dental admissions cycle at Kansas City University College of Dental Medicine, and data were collected for each item score as well as overall score between two separate raters. Admissions committee members were blind to the HAAR results when making decisions of admission. Results: Validity evidence was collected in all five validity domains. Inter-rater reliability with two-raters was excellent with ICC= 0.85 (ICC 95% confidence interval (CI) 0.70-0.90) for overall HAAR score. Internal consistency was high with item discrimination scores ranging from 0.14-0.67, and inter-item correlation scores ranging from 0.05-0.08. Logistic regression showed that a higher HAAR score was significantly associated with a greater likelihood of admission (OR=1.28, p.001). Conclusions: These data results indicate that the HAAR has a high predictive value for admissions acceptance and there is high inter-rater reliability throughout the 16-criteria in the rubric. This initial validity evidence in Messick’s five validity domains strongly supports the use of the HAAR in the pre-doctoral dental admissions process.