2026 ASEE Annual Conference & Exposition

This research study investigates how well commonly available generative AI (gen AI) tools synthesize empirical literature in engineering education research. Gen AI and other LLMs are touted as time saving tools for conducting research and writing. To establish their credibility, these tools go through benchmarking for their ability to search information and interpolate complex problems. Yet, their effectiveness and trustworthiness for academic tasks like seeking information in journal publications, synthesizing literature, and aiding in writing are less commonly scrutinized. As gen AI tools are being integrated into everyday life, engineering education researchers and students using these tools may question how useful these tools can be for academic purposes. We examined gen AI tools for their usefulness for academic research and writing tasks and were guided by the questions: Which tools perform best in synthesizing research findings, and does their accuracy vary by research methodology (qualitative, quantitative, or mixed methods)?
Six gen AI and LLM tools were tested for their ability to synthesize empirical engineering education articles across qualitative, quantitative and mixed methodologies. Set questions, determined by a review of gen AI evaluation and bench marking criteria, were asked of each gen AI model. Each model was compared to a pre-set solution developed by the researchers for its accuracy. Findings will include a table of the synthesized results for the six models and pre-set researcher determined solution. Findings will compare results and discuss which gen AI tools are best suited for synthesizing empirical research articles. Researchers and students may use the findings of this study to guide their decided workflow based on the strengths of multiple gen AI tools.