Multiple Choice Questions that Test Conceptual Understanding: A Proposal for Qualitative Two-Tier Exam Questions

Dion Timmermann; Christian H Kautz

Download Paper | Permalink

Conference: 2015 ASEE Annual Conference & Exposition
Location: Seattle, Washington
Publication Date: June 14, 2015
Start Date: June 14, 2015
End Date: June 17, 2015
ISBN: 978-0-692-50180-1
ISSN: 2153-5965
Conference Session: Concept Inventories and Assessment of Knowledge
Tagged Division: Educational Research and Methods
Page Count: 15
Page Numbers: 26.1179.1 - 26.1179.15
DOI: 10.18260/p.24516
Permanent URL: https://peer.asee.org/24516
Download Count: 3159

Paper Authors

biography

Dion Timmermann Hamburg University of Technology

visit author page

Dion Timmermann studied electrical engineering at Hamburg University of Technology, Hamburg, Germany. In his master thesis he worked on simulation methods for the signal and power analysis of high speed data links. He currently pursues his Ph.D. in the Engineering Education Research Group at Hamburg University of Technology, where he investigates students understanding in introductory electrical engineering.

visit author page

biography

Christian H Kautz Hamburg University of Technology

visit author page

Christian H. Kautz has a Diplom degree in Physics from University of Hamburg and a Ph.D. in Physics (for work in Physics Education Research) from the University of Washington. Currently, he leads the Engineering Education Research Group at Hamburg University of Technology.

visit author page

Download Paper | Permalink

Abstract

Multiple Choice Questions that Test Conceptual Understanding: A Proposal for Qualitative Two-Tier QuestionsMultiple-choice questions have become a common assessment method for a variety ofreasons, including the easy administration and grading of such tests. Although often only usedto test declarative knowledge, they can also be used to test conceptual understanding, acommon learning goal in engineering courses. However, in many cases it is difficult todevelop valid and reliable questions. This may be due to a small number of possibledistractors resulting in a high probability to guess the correct answer or because of similaritiesbetween tasks or phrases resulting in students memorizing patterns.This problem can be overcome by using two-tier multiple choice questions that satisfy severalcriteria:a) The first tier forces students to apply their conceptual understanding to make a predictionabout a specific situation.b) In the second tier students have to choose one of several statements to justify theirprediction.c) Ideally, there is no obvious relation between the items in both tiers.d) Distractors should be based on known student misconceptions.Multiple-choice questions that follow these criteria have, at least implicitly, been used in thepast. As an example, we note the multiple choice version of the "Classroom test of scientificreasoning" developed by Lawson [1]. However, in several cases, these criteria are not appliedsystematically, indicating a non-structured approach and thus a possible lack of literature onthis type of questions.Using examples from electrical engineering, we are going to explain and illustrate the above-mentioned criteria for two-tier multiple choice questions. We are going to point out thebenefits and limitations of this type of questions and compare them to others found inliterature, including assertion-reason questions and those found in the Chemical ConceptsInventory. Additionally, we are going to present the results of a 500-student exam whichincluded four questions meeting the criteria mentioned above. The answers to these closed-ended questions will be compared to the answers to very similar open-ended questions askedin previous years.We encourage the use of this type of question in cases where the aim is not to identify newstudent misconceptions but instead monitor the prevalence of known difficulties.[1] Lawson, Anton E. "The development and validation of a classroom test of formalreasoning." Journal of Research in Science Teaching 15.1 (1978): 11-24.

Citation
Format

Timmermann, D., & Kautz, C. H. (2015, June), Multiple Choice Questions that Test Conceptual Understanding: A Proposal for Qualitative Two-Tier Exam Questions Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24516

TY  - CPAPER
AB  -        Multiple Choice Questions that Test Conceptual Understanding:               A Proposal for Qualitative Two-Tier QuestionsMultiple-choice questions have become a common assessment method for a variety ofreasons, including the easy administration and grading of such tests. Although often only usedto test declarative knowledge, they can also be used to test conceptual understanding, acommon learning goal in engineering courses. However, in many cases it is difficult todevelop valid and reliable questions. This may be due to a small number of possibledistractors resulting in a high probability to guess the correct answer or because of similaritiesbetween tasks or phrases resulting in students memorizing patterns.This problem can be overcome by using two-tier multiple choice questions that satisfy severalcriteria:a) The first tier forces students to apply their conceptual understanding to make a predictionabout a specific situation.b) In the second tier students have to choose one of several statements to justify theirprediction.c) Ideally, there is no obvious relation between the items in both tiers.d) Distractors should be based on known student misconceptions.Multiple-choice questions that follow these criteria have, at least implicitly, been used in thepast. As an example, we note the multiple choice version of the &quot;Classroom test of scientificreasoning&quot; developed by Lawson [1]. However, in several cases, these criteria are not appliedsystematically, indicating a non-structured approach and thus a possible lack of literature onthis type of questions.Using examples from electrical engineering, we are going to explain and illustrate the above-mentioned criteria for two-tier multiple choice questions. We are going to point out thebenefits and limitations of this type of questions and compare them to others found inliterature, including assertion-reason questions and those found in the Chemical ConceptsInventory. Additionally, we are going to present the results of a 500-student exam whichincluded four questions meeting the criteria mentioned above. The answers to these closed-ended questions will be compared to the answers to very similar open-ended questions askedin previous years.We encourage the use of this type of question in cases where the aim is not to identify newstudent misconceptions but instead monitor the prevalence of known difficulties.[1] Lawson, Anton E. &quot;The development and validation of a classroom test of formalreasoning.&quot; Journal of Research in Science Teaching 15.1 (1978): 11-24.
AU  - Dion Timmermann
AU  - Christian H Kautz
CY  - Seattle, Washington
DA  - 2015/06/14
PB  - ASEE Conferences
TI  - Multiple Choice Questions that Test Conceptual Understanding: A Proposal for Qualitative Two-Tier Exam Questions
UR  - https://peer.asee.org/24516
DO  - 10.18260/p.24516
ER  -