2024 ASEE Annual Conference & Exposition

This study explores the design, implementation, and evaluation of computer-based examination methods within the context of the System Programming course. The System Programming course is designed for students who have already completed prerequisite two programming courses. As the foundational course for system-based courses, its goal is to acquaint students with the underlying processes of computer systems.

Computer-based exams were introduced to the course in the Spring of 2022. Feedback from the course evaluation survey prompted the formulation of two research questions:
1. Is resource restriction necessary during computer-based exams?
2. If we allow students to access online resources during the exam, is there a link between students' scores and their frequency of resource access during exams?
To find the answer to these questions, two computer-based exam types were introduced in Fall 2022 (next semester):
- First Type (Closed Digital Resources): The exam has two parts: multiple-choice/true-false and short answers/C++ solutions. Students could only move forward in the first part and both ways in the second. Only handwritten notes are allowed.
- Second Type (Open Digital Resources): The exam has the same structure as the closed digital resources exam format. Additionally, students could access specific online resources, including the online textbook, online lecture notes, lab codes, and two C++ reference sites during the exam. To ensure compliance with allowed resources, a proctoring software embedded in the test platform records students' network traffic and screens and generates post-exam reports for violations.

In the Fall of 2022, seven sections of system programming courses administered computer-based exams. Among these sections, four allowed handwritten notes in their four closed digital resource exams, while the remaining three sections conducted four open digital resource exams.

Both types of computer-based exams are conducted in the classroom. Students used their own devices. They were responsible for installing or keeping up-to-date necessary software (Chrome browser, proctoring plug-in to browser, authentication for test-platform and wireless connection) and bringing hardware utilities such as charge adaptor.

To analyze the exam results, we computed the average, standard deviation, and median values for two distinct types of exams. The results of exams where open resources were allowed indicate that the median score of three sections is higher than the average, showing that most students scored above the average. In contrast, the closed-resource exam exhibited nearly identical average and median values of four sections. Still, both were lower than those of the open-resource exam, suggesting better performance in the open-resource setting.

Regarding the correlation between exam scores (0-50) and open resource access frequency, the first exam showed a moderate negative correlation (-0.38): lower scores correlated with more frequent resource access. Subsequent exams (Exam 2, 3, 4) had weak negative correlations (-0.12, -0.03, -0.18). The variation might stem from changing exam topics for four exams, with students accessing resources more as subjects became unfamiliar.

The full paper will be structured as follows: Section 1 introduces the purpose of the study by summarizing the related work on the computer-based exams. Section 2 outlines the system programming course content and its assessments. Section 3 explains exam design methods for System Programming and lists the research questions. The study findings are detailed in Section 4, while Section 5 concludes the paper.

By presenting detailed analyses, statistical comparisons, and insights from student feedback, we aim to contribute to the broader discussion on adapting computer-based examination methods in computer science education.