2025 ASEE Annual Conference & Exposition

Researchers have long faced challenges in drawing insights from complex and detailed customer feedback. Often, they rely on lexicon-based sentiment tools like VADER (Hutto & Gilbert, 2014) and AFINN (Nielsen, 2011) to classify words as positive, neutral, or negative, and aggregate scores to derive insights. However, these models often miss the nuances in reviews, especially when sentiments are context-specific (Dalipi et al., 2021). This research explores an advanced sentiment analysis technique called BERT (Devlin et al., 2019) to extract insights from lengthy reviews.
Traditional sentiment analysis tools are less challenging to apply but often struggle with the full context of reviews, especially when emotions, sarcasm, or complex expressions are present. To understand whether an alternative approach can help us overcome these limitations, we apply BERT (Bidirectional Encoder Representations from Transformers), a transformer-based model that belongs to the family of large language models (LLMs). Building on studies by Dalipi et al. (2021) and Sung et al. (2019), our research extends BERT’s use to analyze Coursera reviews across courses and institutions, providing a novel contribution to EdTech. Our dataset, sourced from Kaggle, contains over 1.45 million Coursera reviews, capturing variables such as review text, rating, date, and course ID (Muhammad, 2023). These reviews, some up to 2,470 characters long, allow us to explore patterns across courses and institutions, offering insights beyond the scope of traditional numerical ratings.
We compared the performance of VADER, AFINN, and BERT in predicting course ratings. BERT, by processing entire sentences and using this to understand context, provided more accurate sentiment predictions. BERT (R-squared: 0.26) consistently outperformed lexicon-based models (R-squared: 0.07) according to our regression models. When combined with traditional methods, the R-squared improved marginally to 0.27, reflecting a slight enhancement in prediction accuracy.
We also conducted t-tests to assess how sentiment varied over time, comparing weekend and weekday reviews. Results indicated that weekend reviews were generally more positive (t-statistic = 2.33, p < 0.05). Additionally, ANOVA revealed that similar courses at different institutions produced varying sentiment scores, influenced by the institution's reputation, with top-tier institutions averaging 0.99 in sentiment compared to 0.63 for lower-tier institutions, on a normalized scale of 0-1.
From a managerial perspective, BERT’s insights are valuable for identifying areas in need of improvement, such as course content or instructor effectiveness, each of these is often missed in any traditional models. For example, consistently low sentiment regarding instructors (coefficient = 0.21, p < 0.05) may signal the need for training or course redesign. Our analysis also indicated that temporal trends in feedback can inform strategies for collecting reviews or launching new content. Furthermore, understanding differences across courses (e.g., STEM vs. non-STEM course clusters) and their associated sentiment scores can help institutions assign developmental priority to courses most popular, better satisfying student preferences on platforms like Coursera.
In conclusion, this study underscores the potential of the BERT model to enhance sentiment analysis in the EdTech space. By providing more actionable and context-aware insights than traditional tools, BERT offers a more practical and accurate approach to improving educational platforms like Coursera. Future research could apply this approach to other educational settings, such as K-12 or corporate training programs, and explore how real-time feedback systems using BERT can enhance student outcomes. This research contributes to the broader discourse on how LLM tools can improve educational feedback mechanisms, offering both practical and academic insights into the future of EdTech and student satisfaction. Additionally, the approach outlined in this research can also be implemented across other business areas as well where platforms or brands intend to offer better services based on customers’ detailed feedback.