Is performance on tests affected by the difficulty of the first question and an informational message about the benefits of the 'testing effect'?

Bruno Korst; Dan Wolczuk; Daniel Smilek

Download Paper | Permalink

Conference: 2022 ASEE Annual Conference & Exposition
Location: Minneapolis, MN
Publication Date: August 23, 2022
Start Date: June 26, 2022
End Date: June 29, 2022
Conference Session: ERM: Let's Talk about Tests! (Tests Part 1)
Page Count: 9
DOI: 10.18260/1-2--41115
Permanent URL: https://peer.asee.org/41115
Download Count: 471

Paper Authors

biography

Bruno Korst University of Toronto

visit author page

Bruno Korst completed his undergraduate studies at the Faculdade de Engenharia Industrial (FEI), in Brazil, where he received the title of Electrical Engineer. In Canada, he completed his Master’s at Carleton University, specializing in signal processing for acoustics, and worked in the industry in this field. He started his work at the University of Toronto as a member of technical staff and served as the Director of Teaching Laboratories for ECE prior to joining the Faculty as a Teaching Stream professor. He has received multiple awards on innovation, and was the first staff member to receive the Gordon R. Slemon Award for excellence in the teaching of design. Motivated by his strong interest in laboratory teaching within engineering education, he is presently completing a PhD in Cognitive Neuroscience at the University of Waterloo, with his research concentrating on prospective attention as applied to video instruction. In addition to his technical training and practice, he also holds a B.A. in Political Science/Int’l Relations (Calgary) and an MBA in Marketing (FGV – Brazil). He is a licensed Professional Engineer in the province of Ontario.

visit author page

biography

Dan Wolczuk University of Waterloo

visit author page

Dan Wolczuk is a Lecturer in the Faculty of Mathematics at the University of Waterloo. He primarily teaches calculus and linear algebra and conducts research in the scholarship of teaching and learning. In 2022 he won the Canadian Mathematical Society's Excellence in Teaching Award and in 2021 he won the University of Waterloo's Distinguished Teacher Award. His website is wolczuk.com.

visit author page

author page

Daniel Smilek University of Waterloo

Download Paper | Permalink

Abstract

This paper presents evidence-based practice applied to course design and delivery, through a study conducted during an in-person undergraduate course exploring several aspects of test delivery. An undergraduate linear algebra course was initially designed to draw on the beneﬁts of the well-documented testing effect, which is characterized by better student learning as a result of frequent testing. A study was conducted over one semester with the goal of assessing objectively whether the addition of a lecture-long informational message about the testing effect delivered by the instructor could enhance overall performance. In addition, the study aimed to investigate an aspect of test design, namely whether the diﬃculty of the ﬁrst question (easy vs. hard) would affect the overall performance on the tests.

The cohort consisted of 119 students of different STEM areas across a number of sections, all taught by the same instructor. The course included a total of 9 quizzes, 8 of which relevant to the study, each consisting of 3 questions that varied in diﬃculty; four quizzes started with a hard question and four started with an easy question. The course also included a midterm test and a ﬁnal exam. The cohort was divided into two counterbalanced groups with one counterbalance receiving the easy question ﬁrst on odd numbered quizzes and hard questions ﬁrst on the even numbered quizzes, and with the other counterbalance group experiencing the reverse. All quizzes and exams were delivered at appropriately scheduled times to all students and the same amount of time was given to all students to solve the questions on the quizzes. Critically, one section of the course was chosen to receive an informational message about the testing effect explaining how frequent testing improves performance and encouraging students to use the quizzes as a learning opportunity. For this one section, the informational message was delivered once, after the ﬁrst quiz (second week in the term). All students received messages of encouragement from the instructor throughout the term.

Results showed signiﬁcantly higher performance on the easy questions than the hard questions indicating the manipulation of question difficulty was successful. However, there was no difference in performance between those participants for whom the quiz started with an easy question than those for whom the quizzes started with a hard question. Notably, grades were higher for the group that received the motivation message than the group that did not receive the message. It is hoped that this promising result can be extended in future experiments, which may include multiple informational messages about the effectiveness of testing throughout the term.

Citation
Format

Korst, B., & Wolczuk, D., & Smilek, D. (2022, August), Is performance on tests affected by the difficulty of the first question and an informational message about the benefits of the 'testing effect'? Paper presented at 2022 ASEE Annual Conference & Exposition, Minneapolis, MN. 10.18260/1-2--41115

TY - CPAPER
AB - This paper presents evidence-based practice applied to course design and delivery, through a study conducted during an in-person undergraduate course exploring several aspects of test delivery. An undergraduate linear algebra course was initially designed to draw on the beneﬁts of the well-documented testing effect, which is characterized by better student learning as a result of frequent testing. A study was conducted over one semester with the goal of assessing objectively whether the addition of a lecture-long informational message about the testing effect delivered by the instructor could enhance overall performance. In addition, the study aimed to investigate an aspect of test design, namely whether the diﬃculty of the ﬁrst question (easy vs. hard) would affect the overall performance on the tests.

The cohort consisted of 119 students of different STEM areas across a number of sections, all taught by the same instructor. The course included a total of 9 quizzes, 8 of which relevant to the study, each consisting of 3 questions that varied in diﬃculty; four quizzes started with a hard question and four started with an easy question. The course also included a midterm test and a ﬁnal exam. The cohort was divided into two counterbalanced groups with one counterbalance receiving the easy question ﬁrst on odd numbered quizzes and hard questions ﬁrst on the even numbered quizzes, and with the other counterbalance group experiencing the reverse. All quizzes and exams were delivered at appropriately scheduled times to all students and the same amount of time was given to all students to solve the questions on the quizzes. Critically, one section of the course was chosen to receive an informational message about the testing effect explaining how frequent testing improves performance and encouraging students to use the quizzes as a learning opportunity. For this one section, the informational message was delivered once, after the ﬁrst quiz (second week in the term). All students received messages of encouragement from the instructor throughout the term.

Results showed signiﬁcantly higher performance on the easy questions than the hard questions indicating the manipulation of question difficulty was successful. However, there was no difference in performance between those participants for whom the quiz started with an easy question than those for whom the quizzes started with a hard question. Notably, grades were higher for the group that received the motivation message than the group that did not receive the message. It is hoped that this promising result can be extended in future experiments, which may
include multiple informational messages about the effectiveness of testing throughout the term.

AU - Bruno Korst
AU - Dan Wolczuk
AU - Daniel Smilek
CY - Minneapolis, MN
DA - 2022/08/23
PB - ASEE Conferences
TI - Is performance on tests affected by the difficulty of the first question and an informational message about the benefits of the 'testing effect'?
UR - https://peer.asee.org/41115
DO - 10.18260/1-2--41115
ER -