2025 ASEE Annual Conference & Exposition

An Evaluation of Prompt Engineering Strategies by College Students in Competitive Programming Tasks

Presented at Computers in Education Division (COED) Track 5.C

Generative AI, powered by Large Language Models (LLMs), has the potential to automate aspects of software engineering. This study examined how computer science students utilized these tools during an AI-assisted competitive programming competition across multiple campuses. Participants used generative AI tools such as ChatGPT, GitHub Copilot, and Claude and submitted transcripts documenting their interactions for analysis. Drawing from prompt engineering literature, the study mapped six key strategies to 14 areas of best practices for competitive programming. These practices included clarifying instructions, streamlining prompt context, employing chain-of-thought prompting, providing feedback to refine solutions, and leveraging LLM meta-capabilities for problem-solving. The transcripts were analyzed to assess adherence to these practices. Findings revealed significant variability in adherence, with an average compliance rate of 34.2% across practices. While simpler practices achieved adherence rates as high as 98%, eight practices saw minimal or no usage. These results highlight that students often adopt basic prompt engineering techniques but struggle with more complex strategies, suggesting the need for structured prompt engineering instruction in computer science curricula to maximize the potential of generative AI tools.

Authors

Sita Vaibhavi Gunturi Pennsylvania State University, Harrisburg, The Capital College
Dr. Jeremy Joseph Blum Pennsylvania State University, Harrisburg, The Capital College [biography]

Dr. Jeremy Blum is an associate professor of Computer Science at the Pennsylvania State University, Harrisburg, PA, USA. Prior to joining Penn State Harrisburg, Dr. Blum worked as a research scientist at the Center for Intelligent Systems Research at the George Washington University. Dr. Blum received a D.Sc. in Computer Science and an M.S. in Computational Sciences, both from the George Washington University, as well as a B.A. in Economics from Washington University. His research interests include computer science education and transportation safety.
Dr. Tyler S. Love http://orcid.org/0000-0002-1161-1443 University of Maryland Eastern Shore [biography]

Dr. Love is Professor and Director of Career and Technology Education Studies for the University of Maryland Eastern Shore (UMES) at the Baltimore Museum of Industry. He is also the Coordinator for Technology and Engineering (T&E) education. Dr. Love earned his master's and Ph.D. in Integrative STEM Education from Virginia Tech. His bachelors degree is in Technology Education from UMES. He previously taught T&E courses in Maryland’s Public School System. He is nationally recognized for his work related to the safer design of makerspaces and collaborative STEM labs. Dr. Love is an Authorized OSHA Trainer for General Industry. He has also served on committees at state and national levels that developed P-12 engineering education standards. In recognition of his work, Dr. Love was awarded the 2022 ASEE mid-Atlantic section Best Paper Award, and has received numerous awards from the International Technology and Engineering Educators Association.

Note

The full paper will be available to logged in and registered conference attendees once the conference starts on June 22, 2025, and to all visitors after the conference ends on June 25, 2025

« View session