2026 ASEE Annual Conference & Exposition

Examining Accuracy in AI-Powered Metadata Extraction for Engineering Education Research

Presented at Engineering Libraries Division (ELD) Poster Session

Examining Accuracy in AI-Powered Metadata Extraction for Engineering Education Research

Abstract

Background: Engineering libraries are drowning in paperwork. Every time a researcher publishes a paper, someone has to record key details about it: who wrote it, where they work, and what the paper is about. As the volume of engineering education research grows, doing this by hand is no longer realistic. Many libraries are turning to automated tools to do this job, but no one has tested how well these tools actually work or where they fail.
Purpose: This study asks a simple but urgent question: which automated tool does the best job of pulling key information from research papers, and can a smarter combination of tools do better than any single one alone?
Method: We tested four different automated approaches on 50 peer-reviewed engineering education papers, comparing each tool's output against information that was verified by humans. The four tools ranged from a free, widely used academic document reader, to a rule-based system, to a modern AI assistant, to a custom hybrid approach that assigns each task to whichever tool handles it best.
Results: The hybrid approach, AlphaMind-Extrator, outperformed all others. More importantly, the study uncovered a critical blind spot: one of the most popular free tools used in libraries today gets author affiliations, which university or department an author belongs to, wrong more than 96% of the time, while appearing reliable on everything else. The hybrid approach fixed this failure, achieving 86% accuracy on affiliations at a cost of less than two cents for 50 papers.
Conclusions: No single automated tool does everything well. The safest and most cost-effective approach is to match each task to the tool best suited for it, and to test tools field by field before trusting them with your data.

Authors

Mr. Joshua Owusu Ansah Arizona State University [biography]

Joshua Owusu Ansah is a researcher and entrepreneur pursuing a PhD in Engineering Education Systems and Design at Arizona State University. His work explores AI-powered metadata extraction and VR-based learning tools to improve research accessibility and laboratory experiences in underfunded universities, bridging technology, education, and social impact.
Dr. Brooke Charae Coley Arizona State University, Polytechnic Campus [biography]

Dr. Brooke Coley, an Assistant Professor of Engineering at Arizona State University, is a pioneering force in disrupting the status quo of engineering to create a more equitable and inclusive field where all individuals can thrive. As the Founding Executive Director of the Center for Research Advancing Racial Equity, Justice, and Sociotechnical Innovation Centered in Engineering (RARE JUSTICE), Dr. Coley leads transformative efforts to challenge systemic barriers and promote equity in academia. Her research focuses on amplifying the lived experiences of racially minoritized scholars, dismantling anti-Blackness in STEM, graduate student education, and fostering awareness of, and ultimately, accountability for, the lived realities of individuals navigating STEM through immersive virtual reality experiences. Collaborating with mental health experts, she also is intentional to integrate a head-on focus on the implications for wellness and wholeness in academic environments. Dr. Coley's transparent and culturally responsive approaches, coupled with her dedication and fortitude, have positioned her as a recognized leader in the field. Since 2017, she has secured millions of dollars in grant funding from the National Science Foundation, employing critical qualitative and arts-based methodologies in her work. She received the Wickenden Award and Betty Vetter Award in 2024 and was named a Virtual Visiting Scholar by the ARC Network in 2023. Launching from the Ph.D. in Bioengineering from the University of Pittsburgh oriented to the challenges of navigating STEM as an underrepresented and minoritized scholar, she continues to lead change and advocate for institutional transformation and accountability through novel applications and approaches.
Monica Lynn Miles http://orcid.org/0000-0003-0006-1842 University at Buffalo, The State University of New York [biography]

Monica L. Miles, Ph.D. is an early career Assistant Professor of Engineering Education at the University at Buffalo in the School of Engineering and applied sciences. Dr. Miles considers herself a scholar-mother-activist-entrepreneur where all her identities work in harmony as she reshapes her community. She is a critical scholar who seeks transformative solutions to cultivate liberated and environmentally just environments for Black people, and other minoritized individuals. She believes in fostering racial solidarity and finding her own path in the movement.

Note

The full paper will be available to logged in and registered conference attendees once the conference starts on June 21, 2026, and to all visitors after the conference ends on June 24, 2026

« View session