2024 ASEE Annual Conference & Exposition

The rate at which generative AI is entering the education space is vastly outpacing the rate at which institutional policies are being crafted to address how best to handle these disruptive tools in courses. Consequently, both faculty and students alike are left to their own devices and judgements on how to best navigate the role (if any) that generative AI tools will play in the experience and learning outcomes in their courses. It is yet to be revealed what impact such tools will have and how course dependent the impact may be. Accordingly, the proposed work described herein aims to evaluate the possible roles and impacts generative AI will have in Biomedical Engineering (BME) courses that require extensive technical and scientific writing. In such courses, a policy on student use of generative A has been adopted in which students fill out and submit a “Generative AI Assistance Disclosure” (GAIA) form with every assignment. On the form, the students indicate whether they used generative AI tool(s) as part of working on a given assignment. If the answer is “No”, then students are not required to fill out the rest of the form. If they select “Yes”, then students must fill out the remaining sections of the form, which include the following: (i) naming the generative AI tool(s) used, (ii) explaining how and why the tool(s) were used for the assignment, and (iii) providing the entire exchange with the generative AI tool(s). Assessments generally involve writing technical reports and conducting literature searches. When students report the usage of AI tools, the GAIA form offers options such as writing and/or coding support, aiding in understanding and explaining concepts, and saving time or reducing stress. The purpose of the proposed work is to establish a baseline understanding of student use of generative AI tools and their impact specifically in the context of BME courses with technical and scientific writing components, which will involve both quantitative and qualitative analyses and insights.

The authors are interested in how generative AI tools can enhance students’ experiences and learning outcomes related to course objectives. The long-term goal of the study is to collect and analyze cross-institutional data to investigate and compare AI usage by BME students from different institutions, and the investigators are currently working towards a multi-institution Institutional Review Board approval for this purpose. Preliminary results from this proposed work will prove useful in starting to gather data and extract insights around the role and impact of generative AI in BME courses with a technical writing component. We aim to conduct qualitative and quantitative analyses on the GAIA disclosure form responses. Some quantitative explorations include but are not limited to: (i) analyzing what percentage of students used versus did not use generative AI tools and how those percentages might change between assignments and topics; (ii) comparing individual assignment performance between students who used to students who did not use generative AI tools; and (iii) assessing overall student performances in the courses as a function of percent AI usage across all assignments. Some qualitative explorations include but are not limited to: (i) investigating the main ways in which students were using generative AI tools (the how and why on the GAIA forms), and thus the main ways in which the students potentially found the tools most useful with respect to course material and assessments; (ii) identifying potential areas where students need the most help with respect to core skills and competencies in the courses based on how students used generative AI tools; (iii) understanding the degree of complexity with respect to the exchanges between students and generative AI tools.

Outcomes from this study will also contribute to an ever-evolving understanding of the opportunities and challenges generative AI will present in BME education and education more broadly, and what frameworks and policies should be considered as the community navigates these pervasive and powerful tools.