2025 ASEE Annual Conference & Exposition

Large language models (LLM) such as ChatGPT have the ability to generate code in response to prompts providing specifications for a program. Because LLM are easy to access, students are likely to use them to complete programming assignments. Programming instructors need to consider the impact of these tools on their courses.
As the instructor of an undergraduate introductory programming course for Biological Engineering majors, I decided to investigate the capabilities and limitations of ChatGPT for generating code in the context of my course. The research objectives are: i) To assess the ability of a student-prompted generative AI tool, ChatGPT, to produce R code for assignments in a biological engineering programming course, ii) To document the time, number and type of prompts required for students to be satisfied with the AI-generated code.

Three undergraduate students who completed the introductory course in a previous semester were tasked to enter the description of the course programming assignments into the free version of ChatGPT to obtain a script. They did not receive training on how to use ChatGPT to generate code. They were allowed to update the assignment descriptions when pasting them into ChatGPT, prompt ChatGPT as much as they wanted to refine the script and to execute the script to check for errors. They were not permitted to make large changes to the code by hand. The students were given access to the current course learning management system and submitted their scripts as if they were enrolled in the course.
To quantify the amount time taken, number and type of prompts required by a student to effectively utilize ChatGPT for completing R programming assignments, a record of all the interactions between the students and ChatGPT was kept. For each assignment, the students reported how long it took them to complete it and rated how straightforward it was to prompt ChatGPT to obtain the desired output code. Teaching assistants in the current course unknowingly graded these codes generated by ChatGPT along with the codes that were written in class by current students in the course (without ChatGPT).

ChatGPT provided correct code on the first attempt 77% of the time and achieved 100% accuracy after follow-up prompts. An unexpected finding is that the subjects asked ChatGPT to modify the code by replacing commands not covered in class with simpler approaches, making the output more similar to that of a student in the course to prevent it from being flagged as non-original work. The findings from this study offer insights into strategies to minimize opportunities for students to use generative AI to directly obtain complete solutions to their coding assignments.