Improving efficacy attribution in a self-directed learning environment using prior knowledge individualization

Models of learning in EDM and LAK are pushing the boundaries of what can be measured from large quantities of historical data. When controlled randomization is present in the learning platform, such as randomized ordering of problems within a problem set, natural quasi-randomized controlled studies can be conducted, post-hoc. Difficulty and learning gain attribution are among the factors of interest that can be studied with secondary analysis under these conditions; however, much of the content that we might like to evaluate for learning value is not administered as a random stimulus to students but instead is being self-selected, such as a student choosing to seek help in the discussion forums, wiki pages, or other pedagogically relevant material in online courseware. Help seekers, by virtue of their motivation to seek help, tend to be the ones who have the least knowledge. When presented with a cohort of students with a bi-modal or uniform knowledge distribution, this can present problems with model interpretability when a single point estimation is used to represent cohort prior knowledge. Since resource access is indicative of a low knowledge student, a model can tend towards attributing the resources with low or negative learning gain in order to better explain performance given the higher average prior point estimate. In this paper we present several individualized prior strategies and demonstrate how learning efficacy attribution validity and prediction accuracy improve as a result. Level of education attained, relative past assessment performance, and the prior per student cold start heuristic were employed and compared as prior knowledge individualization strategies.