2024 ASEE Annual Conference & Exposition

This evidence – based practice paper explores the use of Artificial Intelligence to improve mathematical skills of First Year Engineering Students, particularly Modelling Competence, which is relevant to Our University curricula as a fundamental competence for understanding engineering phenomena [1].

Competence based educational models are centered in developing three dimensions of students: theoretical frameworks, application skills, and values [2]. A coherent integration of these three aspects allow the students to demonstrate consistent behaviors when performing tasks or assessments. Moreover, literature show that use of technology improves engagement of students and allows a better development of competences [3], hence, developing technological – based solutions for student’s learning is important to motivate competence development.

We present a quasi – experimental study with a pre – test post – test analysis. The experiment involved 34 students that were enrolled in a Mathematical Modelling course during 2023 August – September period. Students were randomly divided in 2 groups: a control group of 8 students in which the modelling competence was evaluated without any exposure to Artificial Intelligence; and an experimental group with 26 students that were exposed to Game – Based Learning using Artificial Intelligence.

During the experiment, it was of our interest to analyze the performance of students with respect to the Mathematical Modelling Competence. We focused in three behaviors of the competence: Understanding the meaning of rates of change, Interpretation of graphs, and decision making according to a mathematical model. Each aspect was measured using an argumentative questionnaire, in which students had to demonstrate modelling skills by understanding and applying mathematical concepts, without the requirement of performing mathematical calculations.

The experiment was divided into four phases: Pre – test, Game Play, Exit Survey, and Post – test. During the Pre – test, both groups, control and experimental group, solved the argumentative questionnaire, and their outcomes were recorded. Then, for the Game Play, a prompt engineering process with 7 iterations took place. During this process, we developed a prompt that would allow the students to play a text roleplay game, in which the students took place of an epidemiologist facing a world pandemic. The objective was to select the optimal alternatives by correctly interpreting functions’ behaviors and rates of change according to the Susceptible – Infected – Recovered (SIR) model [5].

During implementation of Game Play, students entered the prompt into ChatGPT and selected what they considered the best alternatives according to their analysis of functions and rates of change. The game duration was about 15 minutes, after which they received feedback from same ChatGPT to improve their decisions, and they discussed in teams what were the improvement opportunities. Immediately after playing the game, the students answered an Exit Survey that measured their perception of Easiness of playing the game, the Interactivity achieved, how Enjoyable it was, and if their Curiosity was triggered. Finally, the day after the game took place, students in the experimental group solved another argumentative questionnaire, involving the same behaviors as before. Their performance was recorded to compare those results with the ones of the pre – test.

The results of this study show that there is a significant difference of means between the pre – test scores and the post – test scores, implying that the proposed methodology positively impacts the mathematical modelling competence of students. Moreover, the results also show that the variability of results in the control group is significantly more disperse than the results of the experimental group, which points out that the game in ChatGPT was useful to level students’ competence, reducing variability of learning as expected in a homogeneous group [6].

Additionally, obtained results show a strong positive correlation between the perception of Interactivity, Enjoyable, and Curiosity. This relationship supports that introducing fun elements into instructional scheme generates engagement of students [7], and therefore contributes to the development of competences [8].