2024 ASEE Annual Conference & Exposition

At ASEE 2023, the study authors presented a new method of evaluating student engagement which combines the Miville-Guzman Universality-Diversity Scale – Short Form (MGUDS-S) with the 'Slack' online messaging tool. In this paper we build on this research, employing cross-analysis to comprehensively investigate the correlations between factors like students' overall satisfaction with online workshops, students' final workshop grades, 'Slack' usage times, 'Slack' message quality, the subject major of the students, and their nationality, to learn more about patterns of student engagement and satisfaction in this type of online learning program. Furthermore, the authors provide details on how the 'Slack-based evaluation' tool was planned and developed.

Since the 2020 academic year (AY2020) our university which specialises in Engineering and related fields, has been conducting online robotics workshops in collaboration with partner universities in Malaysia and Thailand as part of our institutional Global Project-Based Learning (referred to as gPBL hereafter). After running these workshops for several years we have observed that there are a number of recurring issues, one of which is the necessity and difficulty of assessing students' levels of engagement in group work.

In order to try and increase the level of student engagement, we first need to be able to measure it. To this end we conducted a study on how modern tools and resources can be used to assess levels of involvement of course participants. We chose the ‘Slack’ messaging platform for a potential assessment tool because our institution has been using it as the main way for students to communicate with each other in online group projects. We decided to investigate whether detailed analysis of student activity on this platform (henceforth referred to as 'Slack-based evaluation') could deliver a clear picture of how fully students are engaged with their work.

Study authors A and B, who are students from the Department of Engineering Science and Mechanics and had served as teaching assistants on the online robotics workshop in AY2022, were chosen to be part of this Slack-based evaluation team. They worked under the supervision of authors C and D to explore and decide on data sorting methods, data analysis, and development of other processes and workflows for this task. From a data point of view the study authors drew from assessments of students' 'global competence' using the MGUDS-S tool, student satisfaction surveys, and categorisation and evaluation of behaviour in group work particularly focusing on students' Slack messages.

The study authors concluded that Slack itself is suitable for analyzing group work; on a practical level it provides an environment for the smooth sharing of video, images, programs, and other elements necessary for group work, and it facilitates the acquisition of detailed conversation logs and granular data on user behaviour which is necessary for 'engagement', 'involvement' and other related aspects to be measured and analysed. This new approach is a valuable tool for assessing student’s participation rate in group work, and it is expected to contribute to increasing student motivation to participate in the program.