2025 ASEE Annual Conference & Exposition

Participant-based research remains essential in education-based experimental designs. However, there are many barriers to optimizing these studies prior to experimental deployment. While it would be ideal to include human participants in every stage of research design including early development, this is often not possible. In this study, we propose that Large Language Models (LLMs) can serve as preliminary participants during the early phases of experimental design. The robust role-playing capabilities of LLMs allow researchers to conduct trials with a simulated population representative of the target audience. We note that this approach is intended to optimize study designs prior to student involvement, rather than to replace human participants.

By leveraging LLMs in this manner, researchers can gauge potential pitfalls in and refine various aspects of their studies prior to deployment, such as question phrasing, question ordering, and response types. This is especially valuable in personalized learning environments powered by LLMs as the performance of these models is highly dependent on the prompts used. Using LLMs to simulate the experiment in early stages allows researchers to conduct as many trials as needed to tweak prompting, hyperparameters, and study design wherever necessary. We applied these principles in a small-scale text personalization study, where LLMs were used to adapt academic texts to users’ learning preferences.

Using LLMs in place of human participants in the preparatory stage allowed us to account for potential flaws in our earliest experimental designs. Ultimately, we propose that these tools can be used to improve the design of participant-based studies. While not a replacement for humans, LLMs can serve as a valuable tool in the development and optimization of human subject studies. Future work will explore the scalability of this approach among different types of educational research.