2025 ASEE Annual Conference & Exposition

Project-based experiential learning has a proven track record of success in engineering education across disciplines. For the last several years, our lab has run such a course, Diagnostic Intelligent Assistance for Global Health (DIAG), which teaches undergraduate and first-year master’s students about biomedical engineering and bioinformatics through multi-year longitudinal projects inspired by global health issues. DIAG’s projects incorporate knowledge across fields such as the biomedical sciences, computer science, bioinformatics, and AI. As part of DIAG’s inclusionary and interdisciplinary approach, students from widely varying backgrounds in these fields are encouraged to join. While this diversity comes with countless benefits, it makes it challenging to provide each student with the specific learning resources and support they need to efficiently and confidently get started and progress on new projects.

The recent and rapid development of large language model (LLM) capabilities to comprehensively answer user queries by synthesizing vast quantities of information across varying domains and modalities has made it a highly promising option to assist in engineering education. However, a major limitation to LLMs’ utility as an educational tool is their propensity to “hallucinate” incorrect responses and an inability for the user to readily detect such hallucinations or even verify the LLM’s source of information. Retrieval augmented generation (RAG) integrates the LLM with a corpus of domain specific information that is used to inform and guide the LLM’s response and enables the citation of sources from which it drew its response, thus addressing the greatest shortcomings of standard LLM applications as an educational tool. Here we propose the DIAG student-lead development and assessment of custom RAG-LLM based applications for assisting students from diverse educational backgrounds to confidently and efficiently get started and progress on team projects.

For our first aim, DIAG students will develop a working suite of custom RAG-LLM based applications. These applications will operate as chat-bot assistants for the user, answering questions they have as they “onboard” and then work on a project. This development phase includes compiling and organizing the open-source learning resource corpus for the RAG systems to draw upon. Applications covering three general categories will be developed to assess the value of application complexity and domain specificity. In the first category, the application will be built around a single general purpose LLM agent, such as ChatGPT. In the second, applications will be built around a single domain-specific fine-tuned LLM, such as OpenBioLLM for biomedical knowledge assistance. In the third category, the application will be built around multiple expert LLM agents using tools such as LangChain or LangGraph.

For our second aim, the utility of the custom RAG-LLM based applications will be assessed against the most popular existing standards such as ChatGPT and Claude in a double-blinded fashion by DIAG students starting on new projects. We plan to assess new students joining our course at the start of the winter semester in January 2025. We will assess the amount of time needed before starting a project; their attitude towards the “onboarding” process; and self-reported measures including their confidence and competency before and after extended use of the application.

Finally, all code and data will be shared freely and in such a way that enables others to re-implement our applications and validate our assessments. If resources are available, we intend to host the application online to enhance its usability by interested groups.