2023 ASEE Annual Conference & Exposition

Work in Progress: Evaluating the Effect of Symbolic Problem Solving on Testing Validity and Reliability

Presented at Will This Be on the Mechanics Test? Concept Inventories and Understanding Exams

Problem-solving is a typical assessment topic in engineering dynamics tests. To solve a problem, students need to set up equations and find a numeric answer. It usually takes ten to thirty minutes to solve a quantitative problem, depending on its difficulty and complexity level. Due to the time constraint of in-class testing, a typical test can only include a limited number of problems with insufficient problem types. It may lower testing validity and reliability, two essential factors which contribute to assessment results.

A test with a high validity should cover proper content. It should be able to distinguish high-performing students from low-performing students and every student in between. A reliable test should have a sufficient number of items to provide consistent information about the student’s mastery of the materials. Both validity and reliability are two important criteria for quality assessment.

In this work-in-progress study, we will investigate whether we can develop a valid and reliable test on students’ solving symbolic engineering problems in engineering dynamics tests. Symbolic problem solving in this study refers to solving problems by setting up a system of equations without finding numeric solutions. It usually takes much less time. As a result, more problems of a variety of types can be included on a test. We will use the Classical Test Theory to evaluate the validity and reliability metrics of a collection of symbolic problems. Examples on rectilinear kinematics and angular motion will be provided to illustrate how symbolic problem solving is used in both homework and testing.

As numerous studies in the literature have shown that symbolic questions impose greater challenges to students because of their difficulties with math, we will also share strategies on how to prepare students to ease the learning curve.

Authors

Dr. Yan Tang http://orcid.org/0000-0002-9089-5746 Embry-Riddle Aeronautical University, Daytona Beach [biography]

Dr. Yan Tang is an associate professor of mechanical engineering at Embry-Riddle Aeronautical University in Daytona Beach, Fla. Her current research in engineering education focuses on cognitive load theory, deliberate practice, and effective pedagogical strategies.
Lin Ding The Ohio State University [biography]

Lin Ding, Ph.D., is an associate professor in the Department of Teaching and Learning at The Ohio State University. Dr. Dingâ€™s scholarly interests lie in discipline-based STEM education research. His work includes theoretical and empirical investigation
Dr. Haiyan Bai University of Central Florida [biography]

Haiyan Bai, PhD., is Professor of Quantitative Research Methodology in the College of Community Innovation and Education at the University of Central Florida. Her interests include resampling method, propensity score analysis, research design, measurement
Dr. Richard Catrambone http://orcid.org/0000-0002-7334-7406 Georgia Institute of Technology [biography]

Richard Catrambone is a Professor in the School of Psychology at the Georgia Institute of Technology. He received his B.A. from Grinnell College and his Ph.D. in Experimental Psychology from the University of Michigan.

The question Catrambone likes to ask--and the thread that runs through the projects he does alone and in collaboration with others--is: What does someone need to know in order to solve novel problems or carry out tasks within a particular domain?

Catrambone’s research interests include problem solving, educational technology, and human-computer interaction. He is particularly interested in how people learn from examples in order to solve problems in domains such as algebra, probability, and physics. He explores how to create instructional materials that help learners understand how to approach problems in a meaningful way rather than simply memorizing a set of steps that cannot easily be transferred to novel problems. He researches the design of teaching and training materials--including software and multimedia environments--based on cognitive principles that help students learn basic tasks quickly and promote transfer to novel problems. He uses task analysis to identify what someone needs to know in order to solve problems or carry out tasks in a domain and then to use the results of the task analysis to guide the construction of teaching and training materials/environments.

Catrambone has served on the Cognitive Science Society governing board from 2011-2016 and was chair of the Society in 2015. He was co-chair of the Cognitive Science Conference in 2010. He has served as a consulting editor for the Journal of Educational Psychology (1/2008 - 12/2011), the Journal of Experimental Psychology: Learning, Memory, and Cognition (6/2000 - 12/2001 and 1/2009 - 12/2009), the Journal of Experimental Psychology: Applied (1/2001 - 12/2007), and the Journal of Experimental Psychology: General (6/2000 - 12/2001). He has published his research in journals such as the Journal of Experimental Psychology: General; Journal of Experimental Psychology: Learning, Memory, and Cognition; Journal of Experimental Psychology: Applied; Memory & Cognition; Journal of Educational Psychology; Human-Computer Interaction; Human Factors; and other basic and applied journals. He has also served on grant review panels for a variety of funding agencies including the National Science Foundation and the Institute of Education Sciences (U.S. Department of Education).

Download paper (1.02 MB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session