2023 ASEE Annual Conference & Exposition

Due to the difficulty in assessing programming skills that arise from the open-ended nature of programming, in 2017, researchers conducted a major literature review on IDE-based learning analytics. The results of this review led researchers to put forth a call to action to expand the ability of IDEs to collect and analyze different types of data. Through the development of Instrumented IDEs, we can acquire complex programming process data, however, this approach is hindered by the complexity of developing and deploying an API for multiple IDEs. This complexity and the cross-compatibility of APIs is the primary limitation in conducting cross-IDE research, followed by the inconsistent structure and collection of data and a lack of variety in the types of metrics used to instrument IDEs.
In response to the call to action, we developed a web-based IDE known as the Archimedes platform for capturing flowcharts and a persistent trace of student programming and design data. Using this application, we conducted an investigation of introductory and intermediate students’ programming process patterns using the Python programming language. Student programming event data was collected based on a custom event compression system for capturing events such as CREATE, UPDATE, DELETE, RUN_SUCCESS, RUN_FAIL, and various browser-based events for detecting external behavior, such as copying and pasting from external sources. Using this data, we seek to validate an additional IDE-based metric called the Timeline of Program Development. We define this as a sequence of events for categorize programming skills by looking at students’ programming behavior and actions taken over time. A timeline of events records events such as time spent designing, writing, updating, running, or deleting code. This poster illustrates the programming process patterns captured and analyzed through the Archimedes platform. It is our hope that this data will be used as a method to better understand student’s programming behavior.