2025 ASEE Annual Conference & Exposition

This paper presents an investigation into the use of Generative AI (GenAI), specifically ChatGPT, for automating quiz generation in higher education by conducting a case study in a graduate Artificial Intelligence (AI) course. The study aims to compare the quality and relevance of AI-generated quizzes with manually created quizzes, addressing a critical question in computer science education: Can Generative AI effectively support educators in creating assessments that align with course learning objectives?

We conducted the study in a graduate-level AI course, which involves 47 students, one instructor and one teaching assistant (TA). The manual quizzes were created by the instructor who is knowledgeable about the domain, and the GAI-based quizzes were created by the TA who is knowledge about GenAI. Both ways of quiz generation focused on the same course topics, including machine learning, neural networks, and search algorithms. These quizzes were then assessed through a combination of qualitative and quantitative measures, focusing on alignment with learning outcomes, question relevance, and student performance. This approach allowed for a comprehensive comparison of the effectiveness of AI-generated quizzes versus those created manually.

Our findings indicate that while AI-generated quizzes display creativity and cover a broad range of topics, they often lack the depth and precision required for specialized subject areas. This shortcoming is further highlighted by our quantitative analysis, which shows notable differences in both student scores and relevance or alignment with course objectives when comparing AI-generated quizzes to manually created ones. These results underscore the need for more tailored AI customization. Additionally, we examine ethical considerations, such as the impact of AI on educational assessments and the potential for bias in AI-generated content. To address these concerns, we propose strategies to better align Generative AI tools with specific course objectives, thereby improving both the precision and relevance of quiz generation. Ultimately, our goal is to guide educators in thoughtfully adopting AI tools, maximizing their benefits while addressing the challenges they present.