2026 ASEE Annual Conference & Exposition

WIP: Low Effort, High Grades? Benchmarking LLMs on Various Engineering Assignments

Presented at Computers in Education (CoED): Poster Session - Division Special Events (1 of 4) -- M208

Large language models (LLMs) are advancing rapidly. Recent systems support explicit reasoning through chain-of-thought prompting, multimodal understanding of files and images, programming code generation, and document production (e.g., slide decks, spreadsheets, and PDFs). Instructors are increasingly concerned that students can use LLMs to complete a wide range of assignments with minimal effort, yet still earn high grades, potentially diminishing meaningful learning. On the other hand, systematic evidence remains limited on whether currently widely used LLMs can complete authentic coursework end-to-end (from prompt to final solution without human intervention).

In this paper, we evaluate six state-of-the-art LLMs commonly used by students on 24 authentic assignments drawn from a variety of engineering courses at a public research university, including introductory programming, numerical methods, and computational algorithms. Courses range from introductory to upper-division. To assess the effects of prompt input modality, we compare text-based versus image-based prompts. To capture assignment type diversity, we organize tasks into multiple-choice questions (MCQ), programming code-writing tasks, and constructed-response questions, which we further divide into long-form algorithm design questions and short-form conceptual analysis questions. MCQ and code-writing submissions are both auto-graded through an online assessment platform used by the university. Algorithm-design responses are graded by two engineering graduate students with advanced training in algorithms using a predefined rubric; any discrepancies are reviewed together and resolved. Conceptual-analysis responses are added to each course’s regular grading pool, mixed with student submissions, and evaluated by the course teaching assistant without disclosure of authorship to minimize bias.

We observe near-ceiling performance on multiple-choice, code-writing, and long-form algorithm design questions across all six models. Performance on conceptual analysis questions is more variable, though generally high. Text-based and image-based prompts perform similarly overall. These preliminary findings provide insight into the capabilities and limitations of current LLMs when applied to authentic engineering coursework.

Authors

Mr. Yuxuan Chen http://orcid.org/0009-0009-2159-8746 University of Illinois Urbana-Champaign [biography]

Yuxuan Chen is a Master’s student in Computer Science at UIUC. His primary research interests focus on computer science education and artificial intelligence. He is dedicated to enhancing student learning experiences and accessibility in computing education through both innovative technology and research-driven teaching practices.
Siegfried Eggl University of Illinois Urbana-Champaign [biography]

Siegfried Eggl (Ph.D. ’13 Univ. of Vienna, Austria) is Assistant Professor in the Grainger College of Engineering, Department of Aerospace Engineering at the University of Illinois Urbana-Champaign, where he is leading the Astrodynamics and Planetary EXploration (APEX) group at UIUC conducting research in Space Situational Awareness, Celestial Navigation, Planetary Science and Planetary Defense. As part of the NSF-Simons Foundation-funded SkAI institute, he is engaged in AI research across astronomy and aerospace engineering.
Dr. Abdussalam Alawini University of Illinois Urbana-Champaign [biography]

Dr. Alawini is a Teaching Associate Professor in the School of Computing and Data Science at the
University of Illinois at Urbana-Champaign. He holds a bachelor’s degree from the University of Tripoli
and master’s and doctoral degrees in Computer Science from Portland State University. His research focuses on data management systems and computing education. Dr. Alawini is passionate about building data-driven, AI-based systems for improving teaching and learning.
Prof. Mariana Silva University of Illinois Urbana-Champaign [biography]

Mariana Silva is a Teaching Associate Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign. Silva is known for her teaching innovations and educational studies in large-scale assessments and collaborative learning. She has participated in two major overhauls of large courses in the College of Engineering: she played a key role in the re-structure of the three Mechanics courses in the Mechanical Science and Engineering Department, and the creation of the new computational-based linear algebra course, which was fully launched in Summer 2021. Silva research focuses on the use of web-tools for class collaborative activities, and on the development of online learning and assessment tools. Silva is passionate about teaching and improving the classroom experience for both students and instructors.
Max Fowler http://orcid.org/https://0000-0002-4730-447X University of Illinois Urbana-Champaign [biography]

Max Fowler is a Teaching Assistant Professor in the Siebel School of Computing and Data Science at the University of Illinois Urbana-Champaign. Much of Max's research is in the space of skills and assessment: specifically identifying the core competencies of programming (and, increasingly, skills for working with generative AI) while developing strategies for fair, secure assessment-at-scale. His work in this space includes leveraging GenAI for autograding and feedback provision, intersecting with "LLM-as-judge" methodologies.
Abhishek Umrawal http://orcid.org/0000-0003-4460-7499 University of Illinois Urbana-Champaign [biography]

Dr. Abhishek Umrawal is a Teaching Assistant Professor of Electrical and Computer Engineering (ECE) at the University of Illinois Urbana–Champaign, with affiliations in the Coordinated Science Laboratory (CSL) and the National Center for Supercomputing Applications (NCSA). He earned his Ph.D. in Industrial Engineering (Operations Research) from Purdue University, where his doctoral work developed efficient machine learning algorithms for influence maximization on social networks. His research integrates machine learning, operations research, and causal inference to design intelligent and trustworthy decision making systems, with applications in algorithmic marketing, digital platforms, and safe and controllable generative AI. His scholarship also includes teaching‑integrated work in computing and algorithms education, including the responsible use of AI in instruction and assessment. He also holds an M.S. in Economics from Purdue University’s Daniels School of Business and an M.S. in Statistics from the Indian Institute of Technology (IIT) Kanpur.
Melkior Ornik University of Illinois Urbana-Champaign [biography]

Melkior Ornik (mornik@illinois.edu) is an Assistant Professor with the Dept. of Aerospace Engineering and the Coordinated Science Laboratory, University of Illinois Urbana-Champaign, Urbana, IL, USA. He received a Ph.D. degree in electrical and computer engineering from the University of Toronto, Toronto, ON, Canada, in 2017. In the context of education, he is interested in exploring the impact of generative artificial intelligence on the postsecondary enterprise. He is the principal investigator of the effort "Artificially Successful? Investigating and Responding to Generative AI’s Capability to Solve Course Assignments" supported by the Strategic Instructional Innovations Program of the University of Illinois Urbana-Champaign's Grainger College of Engineering. His technical research interests include developing theory and algorithms for learning and planning of autonomous systems operating in uncertain, complex, and changing environments, as well as in scenarios where only limited knowledge of the system is available.

Note

The full paper will be available to logged in and registered conference attendees once the conference starts on June 21, 2026, and to all visitors after the conference ends on June 24, 2026

« View session

For those interested in:

computer science
engineering technology
Faculty