2023 ASEE Annual Conference & Exposition

Students often evaluate their professors’ teaching, both formally (e.g., managed by the institution) and informally (e.g., on websites such as Rate My Professor). However, quantifying the students’ views from their written comments about their professors' abilities can be difficult, because current text processing methods tend to only capture generic concepts such as “positivity,” rather than teaching-specific concepts such as “helpfulness.” To measure students' perception of their professors' teaching, we create lexicons to represent different desired aspects of professors’ performance including quality, helpfulness, clarity, and difficulty. Each lexicon is correlated with one quality and de-correlated with others using a dataset from the popular website Rate My Professor, which contains written evaluations of professors and numerical scores corresponding to different teacher qualities. For example, the correlation method identifies words associated with “helpful” ratings but not “difficulty.”
To ensure that lexicons can extend to a more formal domain of text, we incorporate a dataset including more than 20,000 engineering teaching evaluations, which was provided by the diversity, equity and inclusion center of a large public university in the Midwest United States (henceforth “university dataset”). Using the university dataset, we expand the lexicons by identifying nearest-neighbors to given words according to semantic representations of the words (e.g. “helper” is a neighbor of “helpful” in semantic space; “good” is a neighbor of “wonderful” and “superb”). We validate the final expanded lexicons by correlating the lexicons' use in written reviews with the numerical rating assigned in the review. Professors whose written evaluations contain positive lexicon words (high quality, helpful) also tend to receive higher teaching evaluation scores, while lexicons representing negative teaching qualities (low quality, unhelpful, unclear) are negatively correlated with teaching evaluation scores (p < 0.001).
As a case study for the lexicons, we use them to measure differences in the language used in course evaluations before and during the COVID-19 pandemic. We look at these differences overall, as well as considering differences based on the sex of the instructor. We find changes in the frequency of words representing the quality of instruction, and find that students refer to instructor's helpfulness more often during COVID-19. However, we do not find statistically significant differences in how students discuss male and female instructors, either before or during the pandemic.