LEVERAGING ARTIFICIAL INTELLIGENCE IN PRECLINICAL MEDICAL EDUCATION: ENHANCING FORMATIVE ASSESSMENT FEEDBACK THROUGH THE DEVELOPMENT OF AN AUTOMATED LEARNING ASSESSMENT SYSTEM (ALAS)

Abstract

The increasing demands of competency-based medical education have amplified the need for scalable, timely, and pedagogically sound formative assessment tools. This thesis explores the design, implementation, and evaluation of the Automated Learning Assessment System (ALAS), a generative AI-based platform developed to automate grading of short-answer responses in Harvard Medical School’s Case-Based Collaborative Learning (CBCL) curriculum. ALAS aims to reduce grading burden for faculty while improving feedback quality and enabling data-informed instruction. This study evaluated ALAS’s grading reliability, usability, and impact on instructional decision-making. Two datasets informed the quantitative analysis. The Multi-Grade Dataset (n=822) included student responses independently graded by ten teaching assistants (TAs), allowing comparison of ALAS scores to a human consensus standard. The Single-Grade Dataset (n=8,964) included responses each graded by one TA, allowing direct comparison of ALAS scores to individual human-assigned grades. Inter-rater agreement was measured using Spearman correlation and Fleiss’ Kappa. A faculty survey (N=6) explored ALAS’s usability, feedback utility, and integration into teaching workflows. ALAS demonstrated strong alignment with collective human judgment, achieving higher correlation with average human scores (Spearman ρ=0.678) than human raters did with each other (ρ=0.422). Faculty found the system moderately to highly usable and appreciated its ability to surface both known and previously unrecognized student learning gaps. Visual dashboards and performance summaries were highlighted as useful tools for refining instruction. Despite a small survey sample and technical limitations, ALAS was noted for reducing grading workload, mitigating cognitive load, and enhancing formative assessment responsiveness. ALAS represents a novel and effective application of generative AI in medical education. It supports formative assessment theory by providing timely, individualized feedback and aligns with cognitive load theory by offloading repetitive grading tasks. Future goals include Canvas integration, real-time student feedback, interactive faculty dashboards, and broader adoption supported by AI literacy training. ALAS offers a promising model for ethical and scalable AI integration in health professions education.