2023 ASEE Annual Conference & Exposition

Design and Testing of a Quantitative Instrument to Evaluate Engineering Research Center Participation

Presented at Instrument Design and Development

This research paper reports the in-progress validation of a quantitative instrument designed to assess the perceived impact of participating in a National Science Foundation (NSF)-funded Engineering Research Center (ERC). A multi-institutional consortium composed of ERC education directors, researchers, and evaluators from six NSF-funded ERCs designed easily accessible evaluation instruments and tools that specifically help measure anticipated outcomes for ERC participants for all ERCs. The total effort underway by the consortium includes creating a suite of qualitative and quantitative instruments, an evaluator toolkit, and a user-friendly online platform to host the inventory materials.
This paper focuses on the quantitative instrument created to evaluate the experiences of those who engage with a center. It consists of Likert-type questions assessing the impact of the ERC on participants' self-reported: 1) understanding of the ERC, 2) research and communication skills, 3) climate of inclusion, 4) mentorship experiences, and 5) program satisfaction. The instrument also included additional demographic questions and questions to capture STEM-related future plans. The instrument was designed using multiple rounds of design iterations and pilot tests. Separate surveys used by individual ERCs were compiled and categorized to ensure all requirements from the National Science Foundation were met.
The web-based survey was administered to six ERCs during the Summer of 2021, Fall of 2021, and Spring of 2022. A total of 549 responses were collected; 535 were used following data cleaning procedures. Sample sizes for each component of the survey varied because some ERCs chose to only use some parts of the new instrument. Exploratory Factor Analyses (EFA) were performed to identify latent factors and items that needed further revision. The following factors emerged from our analyses: 1) ERC general understanding; 2) development of research skills; 3) development of professional skills; 4) experience in the ERC; 5) feelings toward the ERC; 6) Beliefs about the ERC, 7) mentors performance; and 8) mentorship experience.
The results provide preliminary evidence that the survey can be used across ERCs. This effort is the first that has been undertaken to develop a shared ERC instrument. The data collected was used to continue in-progress validation. The collaborative nature of this effort can provide ways for ERCs to benchmark impacts of their efforts and share effective practices across ERCs and other similarly structured STEM centers going forward.

Authors

Mr. Marcus Vinicius Melo de Lyra http://orcid.org/0000-0003-4309-6826 Arizona State University [biography]

Marcus is a second-year Ph.D. student in the Engineering Education Department (EED) at The Ohio State University. His research interests include teaching faculty development and early-career faculty experiences. Before joining the EED program, Marcus studied two years in the Engineering Education Systems and Design program at Arizona State University, he also earned his BS in Civil Engineering at the Federal University of Rio Grande do Norte and his MS in Civil and Environmental Engineering at the Federal University of Campina Grande in Brazil.
Dr. Adam R. Carberry http://orcid.org/0000-0003-0041-7060 Arizona State University [biography]

Dr. Adam Carberry is an associate professor at Arizona State University in the Fulton Schools of Engineering, The Polytechnic School. He earned a B.S. in Materials Science Engineering from Alfred University, and received his M.S. and Ph.D., both from Tufts University, in Chemistry and Engineering Education respectively. His research investigates the development of new classroom innovations, assessment techniques, and identifying new ways to empirically understand how engineering students and educators learn. He currently serves as the Graduate Program Chair for the Engineering Education Systems and Design Ph.D. program. He is also the immediate past chair of the Research in Engineering Education Network (REEN) and a deputy editor for the Journal of Engineering Education (JEE). Prior to joining ASU he was a graduate research assistant at the Tufts’ Center for Engineering Education and Outreach.
Dr. Jean S. Larson http://orcid.org/0000-0003-4898-2149 Arizona State University [biography]

Jean Larson, Ph.D., is the Educational Director for the NSF-funded Engineering Research Center for Bio-mediated and Bio-inspired Geotechnics (CBBG), and Associate Research Professor in both the School of Sustainable Engineering and the Built Environment and the Division of Educational Leadership and Innovation at Arizona State University. She has a Ph.D. in Educational Technology, postgraduate training in Computer Systems Engineering, and many years of experience teaching and developing curriculum in various learning environments. She has taught technology integration and teacher training to undergraduate and graduate students at Arizona State University, students at the K-12 level locally and abroad, and various workshops and modules in business and industry. Dr. Larson is experienced in the application of instructional design, delivery, evaluation, and specializes in eLearning technologies for training and development. Her research focuses on the efficient and effective transfer of knowledge and learning techniques, innovative and interdisciplinary collaboration, and strengthening the bridge between K-12 learning and higher education in terms of engineering content.
Dr. Zhen Zhao Massachusetts Institute of Technology [biography]

Zhen Zhao is a Postdoctoral Associate at the Massachusetts Institute of Technology. His research interests include engineering student mentorship and leadership development, engineering research center education and diversity impact evaluation, and engine
Dr. Allison Godwin http://orcid.org/0000-0002-0741-3356 Cornell University [biography]

Allison Godwin, Ph.D. is an associate professor in the Robert Frederick Smith School of Chemical and Biomolecular Engineering at Cornell University. Her research focuses on how identity, among other affective factors, influences diverse students to choose engineering and persist in engineering. She also studies how different experiences within the practice and culture of engineering foster or hinder belonging and identity development. Dr. Godwin graduated from Clemson University with a B.S. in Chemical Engineering and Ph.D. in Engineering and Science Education. Her research earned her a 2016 National Science Foundation CAREER Award focused on characterizing latent diversity, which includes diverse attitudes, mindsets, and approaches to learning to understand engineering students’ identity development. She has won several awards for her research including the 2021 Journal of Civil Engineering Education Best Technical Paper, the 2021 Chemical Engineering Education William H. Corcoran Award, the 2022 American Educational Research Association Education in the Professions (Division I) 2021-2022 Outstanding Research Publication Award, and the 2023 American Institute of Chemical Engineers Award for Excellence in Engineering Education Research.
Dr. Wilhelmina C. Savenye Arizona State University [biography]

Dr. Wilhelmina "Willi" C. Savenye is a Professor Emeritus of Learning, Design and Technologies / Educational Technology at Arizona State University. She is a former Education Director, and currently serves as Senior Education Advisor, for the NSF Engineering Research Center for Bio-mediated and Bio-inspired Geotechnics (CBBG). She previously taught at the University of Texas at Austin and San Diego State University. She earned her M.Ed. and Ph.D. in Educational Technology from ASU, and B.A/ in Anthropology from the University of Washington. Dr. Savenye focuses on instructional design and evaluation of technology-based and online learning systems, employing both quantitative and qualitative research methodologies. She has published over 70 articles and book chapters; made over 140 conference presentations and workshops in the U.S., Europe and Asia; been awarded numerous grants, and has produced many digital learning programs. She is Editor Emeritus of the Journal of Applied Instructional Design. She has served on the editorial boards of journals including Educational Technology: Research and Development and the Quarterly Review of Distance Education, and reviews for additional journals. She served on the editorial board for the Encyclopedia of Educational Technology and has held elected leadership positions.
Dr. Savenye’s instructional design and evaluation work has been conducted in such diverse settings as engineering education, school districts, museums, botanical gardens, zoos, universities, corporations, and Army tank maintenance training.
Dr. Christopher Barr [biography]

Rice University Office of Assessment and Evaluation of STEM Programs led by Dr. Barr is the Director of Assessment and Evaluation of STEM Programs at Rice University. He has been an evaluator and psychometric expert on several federally funded projects in

Download paper (851 KB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session

For those interested in:

Academia-Industry Connections
Broadening Participation in Engineering and Engineering Technology