2026 ASEE Annual Conference & Exposition

Learning analytics (LA) enables the identification of learning patterns, monitoring of student behaviors, and support for targeted interventions, which are particularly important in online learning environments. However, traditional LA approaches based on data from learning management system (LMS) often rely on metrics such as content access frequency and assignment completion rates, which provide an incomplete view of student behavior and limit their usefulness for informing instructional decisions. Given their responsibility for designing assignments, structuring learning activities, and providing formative feedback, faculty and instructional designers are instrumental in shaping the online learning environment. This study explores the gap between LMS-derived quantitative data and the qualitative understanding required to interpret student behavior and preferences effectively through the perspectives of online learning facilitators, that is, faculty and instructional designers. Using a mixed-methods approach that integrates thematic analysis (TA) with the community of inquiry (CoI) model, the qualitative interviews of online learning facilitators were analyzed to gain a comprehensive understanding of the effectiveness and challenges of online STEM instruction. The preliminary analysis shows that faculty rely on contextual, experience-based interpretations of student behavior such as motivation, self-pacing, course design quality, and interpersonal dynamics that are not captured by LMS data. Faculty view flexibility and pacing as essential to cognitive engagement, perceive tools such as discussion boards as insufficient for building authentic social presence, and see teaching presence as strongly shaped by institutional infrastructure and support. Comparing large language model (LLM)-generated themes with human-coded themes revealed that while LLMs are effective at detecting explicit instructional elements such as structured activities, resources, and communication channels, they are less sensitive to the implicit, contextual, and relational nuances that human coders identify.