2026 ASEE Annual Conference & Exposition

This Complete Research paper evaluates incoming first-year engineering students’ generative AI (GenAI) literacy. Unlike most information sources, GenAI tools generate information on-demand using advanced machine learning methodologies. This technology has unique capabilities (e.g., the ability to write programming code in response to a prompt in everyday language) and limitations (e.g., generating incorrect information). GenAI tools like ChatGPT have widespread use among students; recent surveys in May 2025 and July 2025 found that 84% of high school students and 85% of college students were using GenAI tools for schoolwork [1], [2]. Early research on GenAI technologies indicated that students were more optimistic about tools’ capabilities and less worried about the limitations than their professors [3]. However, given that GenAI tools are no longer novel (e.g., [4]) and students are regularly using them for school-related tasks, there is a need to assess students’ understanding the capabilities, limitations, and ethical risks of GenAI tools to support responsible and ethical use.

To responsively adapt curriculum for GenAI literacy in a first-year engineering course, we sought to answer the following research questions:
1. What was the extent of GenAI literacy of incoming students in Fall 2025?
2. How did GenAI literacy scores differ between students with high versus low self-reported prior experience with GenAI tools?

First-year engineering students (N=226) completed a survey during the first week of an introductory engineering course. The survey included questions adapted from published GenAI literacy assessments: the Generative AI Literacy Assessment Test (GLAT [5]), a second AI Literacy framework [6], and a subjective GenAI literacy scale (adapted from [7]). Objective questions asked about how AI works and limitations and capabilities of AI (e.g., Which of the following statements best describes how an LLM (Large Language Model) works?). Subjective questions evaluated students’ perceptions of their own knowledge and skills using GenAI (e.g., I can skillfully use AI applications or products to help me with my daily work.)

The survey evaluated the following subcomponents of GenAI literacy:
• Objective items (multiple choice)
o Basic knowledge & understanding
o How generative models work
o Ability to evaluate & create with GenAI
o Ethics
• Subjective items (Likert scale)
o Awareness
o Usage
o Evaluation
o Ethics
Students additionally rated their prior GenAI experience on a 7-point scale (from “None – I’ve actively avoided generative AI” to “Explorer – I continue to explore new generative AI tools to see what they can do and use them regularly whenever possible”).

To analyze the data, average scores were first computed for each subscale. Data was separated into low-experience vs. high-experience groups using a median split based on their response to the prior experience question. Two ANOVAs, one for objective and one for subjective literacy components, were conducted to evaluate mean differences between experience levels.

Preliminary results revealed a significant main effect of prior experience in both objective and subjective GenAI literacy measures, but in opposite directions (statistics will be included in the full paper). Students with higher self-reported GenAI experience demonstrated higher subjective GenAI literacy but lower objective GenAI literacy than students in the low-experience group. In other words, frequent users feel more confident and capable with GenAI than the low-experience group. However, their knowledge about GenAI was lower than the low-experience group.
The study presents a potential misalignment between students’ confidence and competence in GenAI. High-experience students display greater self-efficacy but not greater factual knowledge, indicating potential overconfidence and gaps in understanding. These findings point to the need for first-year engineering curricula to bridge this gap by explicitly teaching GenAI fundamentals (to build students’ foundational understanding of these tools) and embedding ethical framing in GenAI activities. Tailored instructional strategies should ensure that frequent GenAI users gain necessary conceptual grounding and that all students develop both the confidence and competence to use generative AI tools responsibly.

References
[1] C. Flaherty, “How AI Is Changing—Not ‘Killing’—College,” Inside Higher Ed. Accessed: Oct.14, 2025 Available: https://www.insidehighered.com/news/students/academics/2025/08/29/survey-college-students-views-ai
[2] A. Adair, J. Howell, A. Jacklin, and A. W. Radford, “U.S. High School Students’ Use of Generative Artificial Intelligence - New Evidence from High School Students, Parents, and Educators,” College Board Research, Oct. 2025.
[3] C. K. Y. Chan and K. K. W. Lee, “The AI generation gap: Are Gen Z students more interested in adopting generative AI such as ChatGPT in teaching and learning than their Gen X and millennial generation teachers?,” Smart Learn. Environ., vol. 10, no. 1, p. 60, Nov. 2023, doi: 10.1186/s40561-023-00269-3.
[4] “AI as Normal Technology,” Knight First Amendment Institute. Accessed: Oct. 14, 2025. [Online]. Available: http://knightcolumbia.org/content/ai-as-normal-technology
[5] Y. Jin, R. Martinez-Maldonado, D. Gašević, and L. Yan, “GLAT: The generative AI literacy assessment test,” Comput. Educ. Artif. Intell., vol. 9, p. 100436, Dec. 2025, doi: 10.1016/j.caeai.2025.100436.
[6] L. Ding, S. Kim, and R. A. Allday, “Development of an AI literacy assessment for non-technical individuals: What do teachers know?,” Contemp. Educ. Technol., vol. 16, no. 3, p. ep512, July 2024, doi: 10.30935/cedtech/14619.
[7] H. Wang, A. Dang, Z. Wu, and S. Mac, “Generative AI in higher education: Seeing ChatGPT through universities’ policies, resources, and guidelines,” Comput. Educ. Artif. Intell., vol. 7, p. 100326, Dec. 2024, doi: 10.1016/j.caeai.2024.100326.