2023 ASEE Annual Conference & Exposition

This Complete Research paper investigates the holistic assessment of creativity in design in engineering education. Design is a key element of engineering education given the emphasis on its development through the ABET criteria. As such, design projects play a central role in many first-year engineering courses. Creativity is a vital component of design capability which can influence design performance; however, it is difficult to measure using traditional assessment rubrics and holistic assessment approaches may be more suitable to assess these solutions. One such holistic assessment approach is Adaptive Comparative Judgement (ACJ). In this system, student designs are presented to judges in pairs, and they are asked to select the design that they deem to have demonstrated the greatest level of specified criteria. Each judge is asked to make multiple judgements where the work they are presented with is adaptively paired in order to create a ranked order of all solutions in the sample. The use of this approach in technology education has demonstrated high levels of reliability among judges (~0.9) irrespective of whether the judges are students or faculty. This research aimed to investigate the use of ACJ to holistically assess the creativity of design solutions in engineering education. The research also sought to explore the differences, if any, that would exist between the rank order produced by first-year engineering students and faculty who regularly teach first-year students.
Forty-six first-year engineering students and 23 faculty participated in this research. A separate ACJ session was carried out with each of these groups; however, both groups were asked to assess the same items of work. Participants were instructed to assess the creativity of 101 solutions to a design task, a “Ping Pong problem,” where undergraduate engineering students had been asked to design a ping pong ball launcher. In both ACJ sessions each item of work (solution) was included in at least 11 pairwise comparisons, with the maximum number of comparisons for a single item being 29 in the faculty ACJ session and 50 in the student session. The data from the ACJ sessions were analyzed to determine the reliability of using ACJ to assess creativity of design solutions in first-year engineering education, and to explore whether the rankings produced from the first-year engineering students ACJ session differed significantly from those of the faculty. The results indicate a reasonably high level of reliability in both sessions as measured by the Scale Separation Reliability (SSR) coefficient, SSRfaculty = 0.65 ± 0.02, SSRstudents = 0.71 ± 0.02. Further a strong correlation was observed between the ACJ ranks produced by the students and faculty both when considered in terms of the relative differences between items of work, r = .533, p < .001, and their absolute rank position, σ = .553, p < .001. These findings indicate that ACJ is a promising tool for holistically assessing design solutions in engineering education. Additionally, given the strong correlation between ranks, ACJ could be used to include students in their own assessment to reduce the faculty grading burden or to develop a shared construct of capability to increase the alignment of teaching and learning.