2024 ASEE Annual Conference & Exposition

(This is a Methods Paper.) In this study, we aim to provide some insights into what item characteristics are related to item stability through the newly developed exploratory graph analysis (EGA; Golino & Epskamp, 2017) and bootstrap exploratory graph analysis (bootEGA; Christensen & Golino, 2021). Item stability describes how often an item is placed in the same dimension, as identified by the empirical data, across multiple samples (Christensen & Golino, 2021). Specifically, using a newly designed engineering interest measure as an example, we explored the relationship between item stability and the following item characteristics: 1) network loading, 2) item redundancy, 3) item mean, 4) item-total correlation, 5) item discrimination, and 6) item location. These item characteristics were selected as they are often used as item quality indices within different measurement frameworks, including network psychometrics, Classical Test Theory (CTT), and Item Response Theory (IRT). In this current study, we focused on addressing the following research questions:
• RQ1: What is the relationship between item stability and the item quality indices?
• RQ2: Do stable and unstable items differ in the average value of the item quality indices?
• RQ3: How well do the item quality indices perform when classifying stable and unstable items?

Our sample consisted of 476 Latinx engineers who completed all 89 items of the Engineering Interests Scale (EIM). The Engineering Interest Measure was developed to assess individuals’ interests in engineering-related tasks and skills for engineers in the workplace. A sequence of steps was taken to develop the EIM instrument, including reviewing the literature, specifying construct domains, generating items, and refining measurement items. We conducted EGA on the initial item pool (89 items) to assess the dimensionality of the instrument and conducted bootEGA with 500 random samples to verify the item and structural stability of the dimensionality findings from the EGA. The analyses were conducted using the EGAnet package (Hudson & Alexander, 2023) in R. We analyzed the relationship between item stability and item quality indices produced from CTT item analysis and IRT analysis. Specifically, we used 1) Pearson correlation to analyze the association between the item quality indices and raw item stability value (RQ1), 2) independent-sample t-tests to compare mean differences in the item quality indices between stable and unstable items (RQ2), and 3) the receiver operating characteristic (ROC) curve plot and the area under the curve (AUC) to determine the classification accuracy (RQ3).

Our results showed that item stability had a positive and moderate relationship with network loadings, item-total correlation, and IRT item discrimination. Our findings showed that stable items, on average, were characterized by higher network loadings, higher item-total correlation, higher item discrimination, and higher item difficulty (i.e., lower item means and higher IRT item location) compared to unstable items. Finally, in the ROC curve plot, we observed that network loadings, item mean, item-total correlation, and IRT item difficulty perform well in classifying stable and unstable items. Completed results and implications for the application of exploratory graph analysis in education research will be discussed in the full paper.