Comparing Expert Predictions to Student Performance on Challenging Conceptual Questions: Towards an Adaptive Learning Module for Materials Science

Nutnicha Nigon; Dana Simionescu; Thomas Ekstedt; Julie Tucker; Milo Koretsky

Download Paper | Permalink

Conference: 2022 ASEE Annual Conference & Exposition
Location: Minneapolis, MN
Publication Date: August 23, 2022
Start Date: June 26, 2022
End Date: June 29, 2022
Conference Session: Materials Division Technical Session 3
Page Count: 13
DOI: 10.18260/1-2--40781
Permanent URL: https://peer.asee.org/40781
Download Count: 380

Paper Authors

biography

Nutnicha Nigon Oregon State University

visit author page

Nutnicha (Kate) Nigon is currently a Ph.D. student in Materials Science in the School of Mechanical, Industrial and Manufacturing Engineering with a minor in Education at Oregon State University, USA. She received B.Eng. and M.Eng. in Metallurgical and Materials Engineering from Chulalongkorn University, Thailand.

visit author page

author page

Dana Simionescu Oregon State University

biography

Thomas Ekstedt Oregon State University

visit author page

Thomas Ekstedt is a software developer in the School of Chemical, Biological and Environmental Engineering at Oregon State University. He is involved in the development of technology-based educational systems, particularly in the areas of concept-based instruction, adaptive learning, and interactive simulation of physical phenomena.

visit author page

biography

Julie Tucker

visit author page

Dr. Tucker earned her B.S. in Nuclear Engineering from the University of Missouri – Rolla. She attended graduate school at the University of Wisconsin – Madison as a Naval Nuclear Propulsion Fellow, where she received her M.S. and Ph.D. in Nuclear Engineering with a minor in Materials Science in 2008. After graduation, Dr. Tucker spent five years as a Principal Scientist at Knolls Atomic Power Laboratory in Schenectady, NY studying the thermal stability of structural alloys for nuclear power systems. She joined the School of Mechanical, Industrial, and Manufacturing Engineering at Oregon State University as an Assistant Professor in 2013 and was promoted to Associate Professor in 2019. In 2019 she was awarded the Dean’s Professorship and was also appointed as the Materials Science Interdisciplinary Graduate Program Director. Dr. Tucker has an active research group focused on degradation of materials in extreme environments and alloy development. Her research efforts leverage both modeling and experimental approaches to gain fundamental understanding of materials performance.

visit author page

biography

Milo Koretsky Tufts University

visit author page

Milo Koretsky (he/him/his) is the McDonnell Family Bridge Professor holding a joint appointment in Chemical and Biological Engineering and Education at Tufts University. He received his BS and MS degrees from UC San Diego and his PhD from UC Berkeley, all in chemical engineering. He is interested in integrating technology into effective educational practices and in promoting the use of higher-level cognitive and social skills in engineering problem solving.

visit author page

Download Paper | Permalink

Abstract

The emphasis on conceptual learning and the development of adaptive instructional design are both emerging areas in science and engineering education. Instructors are writing their own conceptual questions to promote active learning during class and utilizing pools of these questions in assessments. For adaptive assessment strategies, these questions need to be rated based on difficulty level (DL). Historically DL has been determined from the performance of a suitable number of students. The research study reported here investigates whether instructors can save time by predicting DL of newly made conceptual questions without the need for student data. In this paper, we report on the development of one component in an adaptive learning module for materials science – specifically on the topic of crystallography. The summative assessment element consists of five DL scales and 15 conceptual questions This adaptive assessment directs students based on their previous performances and the DL of the questions. Our five expert participants are faculty members who have taught the introductory Materials Science course multiple times. They provided predictions for how many students would answer each question correctly during a two-step process. First, predictions were made individually without an answer key. Second, experts had the opportunity to revise their predictions after being provided an answer key in a group discussion. We compared expert predictions with actual student performance using results from over 400 students spanning multiple courses and terms. We found no clear correlation between expert predictions of the DL and the measured DL from students. Some evidence shows that discussion during the second step made expert predictions closer to student performance. We suggest that, in determining the DL for conceptual questions, using predictions of the DL by experts who have taught the course is not a valid route. The findings in this paper can be applied to assessments in both in-person, hybrid, and online settings and is applicable to subject matter beyond materials science.

Citation
Format

Nigon, N., & Simionescu, D., & Ekstedt, T., & Tucker, J., & Koretsky, M. (2022, August), Comparing Expert Predictions to Student Performance on Challenging Conceptual Questions: Towards an Adaptive Learning Module for Materials Science Paper presented at 2022 ASEE Annual Conference & Exposition, Minneapolis, MN. 10.18260/1-2--40781

TY - CPAPER
AB - The emphasis on conceptual learning and the development of adaptive instructional design are both emerging areas in science and engineering education. Instructors are writing their own conceptual questions to promote active learning during class and utilizing pools of these questions in assessments. For adaptive assessment strategies, these questions need to be rated based on difficulty level (DL). Historically DL has been determined from the performance of a suitable number of students. The research study reported here investigates whether instructors can save time by predicting DL of newly made conceptual questions without the need for student data. In this paper, we report on the development of one component in an adaptive learning module for materials science – specifically on the topic of crystallography. The summative assessment element consists of five DL scales and 15 conceptual questions This adaptive assessment directs students based on their previous performances and the DL of the questions. Our five expert participants are faculty members who have taught the introductory Materials Science course multiple times. They provided predictions for how many students would answer each question correctly during a two-step process. First, predictions were made individually without an answer key. Second, experts had the opportunity to revise their predictions after being provided an answer key in a group discussion. We compared expert predictions with actual student performance using results from over 400 students spanning multiple courses and terms. We found no clear correlation between expert predictions of the DL and the measured DL from students. Some evidence shows that discussion during the second step made expert predictions closer to student performance. We suggest that, in determining the DL for conceptual questions, using predictions of the DL by experts who have taught the course is not a valid route. The findings in this paper can be applied to assessments in both in-person, hybrid, and online settings and is applicable to subject matter beyond materials science.
AU - Nutnicha Nigon
AU - Dana Simionescu
AU - Thomas Ekstedt
AU - Julie Tucker
AU - Milo Koretsky
CY - Minneapolis, MN
DA - 2022/08/23
PB - ASEE Conferences
TI - Comparing Expert Predictions to Student Performance on Challenging Conceptual Questions: Towards an Adaptive Learning Module for Materials Science
UR - https://peer.asee.org/40781
DO - 10.18260/1-2--40781
ER -