2024 ASEE Annual Conference & Exposition

The Justification Effect on Two-Tier Multiple-Choice Exams

Presented at Educational Research and Methods Division (ERM) Technical Session 8

Booming enrollment in computer science has raised the need for efficiently gradable assessments, among which are Multiple-Choice Question (MCQ) assessments. MCQs have drawbacks, among which are: random guessing, limited instructor insight into conceptual misunderstandings, student frustration at not being able to explain for partial credit. Recent years have seen several MCQ refinements that feature a two-tier structure that elicits justifications in addition to a correct-choice answer. These justifications can themselves be structured as choices to enable rapid grading. This paper introduces an additional refinement to the two-tier MCQ to acquire more information about a student's understanding by also eliciting explanations for why the wrong answers are wrong. In addition, this paper introduces a simple new metric, the Justification effect, or J-effect, that is easy to extract and apply in order to detect students that need help and identify questions that have design issues. The use of this approach allows instructors to easily provide automated yet rich feedback to students and to pinpoint issues with test implementations, all while remaining easy to design, implement, answer and grade. The study explores the testing and implementation of three different trials over two semesters.

Authors

Dr. Pablo Frank Bolton http://orcid.org/0000-0002-1095-9245 Smith College [biography]

I am a Lecturer in the Computer Science department at Smith College. I received my PhD. from the George Washington University under the direction of Professor Rahul Simha. I currently teach a variety of undergraduate courses and have taught graduate courses in the past.

My research is currently focused on STEM, especially on the areas of identifying misconceptions, creating scalable and informative assessments, and in the use of active learning techniques such as learning-by-teaching, and peer learning.

In addition, I work on Human-Computer Interaction and how it might allow us to interact with virtual worlds and robots.

I enjoy collaborating with colleagues in other fields where I get to combine CS with Biology or Physics and play with their data.

Topics of interest include:

Flipped Classroom techniques to teach programming
The benefits of games and puzzles in learning
Construction of fair, scalable assessments
Multimodal teaching with an emphasis on getting students to articulate their understanding
3D-Shape reconstruction and analysis
The use of Embedded Systems and Machine Learning to automate (Biology) Laboratory tasks.
Liberty Rose Lehr Smith College
Rahul Simha The George Washington University
Michelle Lawson Smith College

Download paper (2.33 MB)

Are you a researcher? Would you like to cite this paper? Visit the ASEE document repository at peer.asee.org for more tools and easy citations.

» Download paper

« View session

For those interested in:

computer science
undergraduate