2024 ASEE Annual Conference & Exposition

Paper type: Work in Progress

Abstract: We present an open-source and highly configurable web application for posing coding exercises to students, keylogging their attempted solutions, and administering surveys and tutorials between attempts. The application is aimed at assessment and analysis of the student problem-solving process. Its multi-language (Python and JavaScript) support and open and portable design remove barriers for both experimenters and participants, potentially enabling significant expansion of and collaboration across recent educational data mining efforts. We validate the application in a small pilot study involving three students and 16 coding exercises each, and demonstrate how the collected data can be used for analysis. Although small-scale, the preliminary pilot results suggest that coding performance is highly bimodal, imperfectly aligned with student perceptions of problem difficulty, and can be predicted in advance based on early cursor movements in the beginning of an attempt. We conclude with a discussion of future work to scale up our data collection efforts towards a more comprehensive and robust analysis.

Keywords: student assessment, undergraduate first-year curriculum, computer science education, problem solving, data collection