2023 ASEE Annual Conference & Exposition

Evaluating the Low-Stakes Assessment Performance: Student-Perceived Accessibility, Belongingness, and Self-Efficacy in Connection to the Use of Digital Notes in Engineering and Computing Courses

Presented at Cohort-Based Postdoctoral Scholars Program, Transforming the National Engineering Education, Defining Accountability, and Evaluating the Low-Stakes Assessment Performance

Course content plays a critical role in student success. Among all college students, students with disabilities (SWD) face numerous additional challenges when digital content is inaccessible or difficult to use. Digital note-taking has been increasingly implemented and studied in higher education for its potential to further develop universal design for learning (UDL) techniques to benefit all students, especially SWD. This study combines low-stakes assessment results with surveys about accessibility, belongingness, self-efficacy and perceived learning in connection to the use of digital notes. The multi-discipline, multi-site, and multi-timepoint research design includes two junior-level undergraduate probability and statistics courses in the disciplines of Computer Science and Industrial Engineering at the University of Illinois Urbana Champaign (UIUC). The study investigated the effect of digital notes introduced at week 10, produced using ClassTranscribe developed at UIUC. The digital notes allow for transcripts, screenshots, and mathematical equations taken directly from video recordings. The digital notes were provided to the students of the two classes in both .pdf and .epub formats with included links to the video. Additionally, we considered Students With Accessibility Needs (SWAN) if they reported conditions that prevented them from attending class at some point while not having an officially recognized disability by the university. The students in the classes were surveyed three times: a baseline survey followed by two posttest surveys. The baseline survey was conducted early in the Fall 2022 semester by measuring the students’ perceived accessibility of the course using the Web Content Accessibility Guidelines (WCAG)-based POUR model in the context of mathematically-rich engineering courses. The POUR model (perceivable, operable, understandable, and robust subscales) was found previously to be a significant predictor of perceived learning, while disability type was not a significant predictor of perceived learning. After the digital notes were introduced, two posttest surveys were conducted in the 12th and 16th week, respectively, to record changes in students’ responses for the POUR scales, perceived learning, belongingness, self-efficacy, and their association with low-stakes assessment including homework assignments and quizzes.
The results from the responses of 285 students (including 22 SWD) for the first survey found that students generally reported positive scores for the courses’ accessibility in all four POUR subscales (scores average > 4.40 out of 5). SWAN and students without disabilities (SWOD) differed in perceivable subscales (p < 0.006), with SWAN finding course material less
perceivable. As the course content’s difficulty level increased, SWD differed from SWOD significantly regarding perceivable, understandable and robustness subscales at the second survey with lower scores for SWD (FDR p < 0.02). Even though the perceived accessibility scales for the classes decreased in the second survey, the scores for all scales were still relatively high (score average > 4.3 out of 5). We found all four subscales did not change significantly for digital note users while SWOD who didn’t use digital notes reported significantly decreased understandable, robust and operable subscales (FDR p < 0.05). The change of perceivable subscale was positively correlated to the change of low stakes assessments in SWD (r = 0.427, p < 0.08). Analysis of the second (n = 255, 18 SWD) and third survey (n = 224, 16 SWD) also found that digital notes helped increase the belongingness of SWAN (p < 0.02) and increase the self-efficacy for SWAN (p < 0.003). In addition, SWAN who used digital notes reported higher perceived learning than those who didn’t (p < 0.03). This study provides empirical evidence of the benefits of UDL based digital notes in promoting the belongingness, self-efficacy and perceived learning of SWAN. It also shows the evidence SWD benefited from digital notes in their low stakes assessments. Results further show the close relationship among belongingness, self-efficacy and perceived learning and yet distinctness of these learning outcome metrics. In addition, the perceived accessibility is confirmed to be uniquely useful for understanding the needs of SWD, and thus deserves more attention to help them succeed.

Authors
  1. Xiuhao Ding University of Illinois, Urbana-Champaign [biography]
  2. Kang Sun University of Illinois at Urbana - Champaign
  3. Zhiyuan Xiao University of Illinois, Urbana-Champaign [biography]
  4. Sujit Varadhan University of Illinois, Urbana-Champaign [biography]
  5. Jiaxi Li Orcid 16x16http://orcid.org/0009-0004-3678-2240 University of Illinois, Urbana-Champaign [biography]
  6. Meghana Gopannagari University of Illinois at Urbana - Champaign
  7. Alan Tao University of Illinois, Urbana-Champaign
  8. David Dalpiaz University of Illinois, Urbana-Champaign
  9. Dr. Jennifer R Amos Orcid 16x16http://orcid.org/0000-0002-9437-8201 University of Illinois, Urbana-Champaign [biography]
  10. Dr. Lawrence Angrave Orcid 16x16http://orcid.org/0000-0001-9762-7181 University of Illinois, Urbana-Champaign [biography]
  11. Dr. Hongye Liu University of Illinois, Urbana-Champaign [biography]
Download paper (740 KB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.