Board 170: PADS -- The Performance Assessment of Design Skills (Work in Progress)

Cathy P. Lachapelle; Elizabeth Parry

Download Paper | Permalink

Conference: 2023 ASEE Annual Conference & Exposition
Location: Baltimore , Maryland
Publication Date: June 25, 2023
Start Date: June 25, 2023
End Date: June 28, 2023
Conference Session: Pre-College Engineering Education Division (PCEE) Poster Session
Tagged Division: Pre-College Engineering Education Division (PCEE)
Tagged Topic: Diversity
Page Count: 6
DOI: 10.18260/1-2--42532
Permanent URL: https://peer.asee.org/42532
Download Count: 98

Request a correction

Paper Authors

biography

Cathy P. Lachapelle STEM Education Insights orcid.org/0000-0002-8487-6071

visit author page

Cathy is particularly interested in how collaborative interaction and scaffolded experiences with disciplinary practices help children learn science, math, and engineering. Her work on STEM education research projects includes design, evaluation, and efficacy research. She also teaches the engineering of design for learning (Learning Engineering!) at Boston College.

visit author page

biography

Elizabeth Parry STEM Education Insights

visit author page

Elizabeth (Liz) Parry is a partner in STEM Education Insights, a woman owned business developing programs, performing external evaluation and consulting on research and practice in P12 Engineering Education.

A graduate of the Missouri Institute of Science and Technology (Missouri S&T), Elizabeth Parry is a partner in STEM Education Insights, a woman owned consulting company specializing in external evaluation, grant writing, curriculum development, engineering coaching and professional learning for P12 teachers and research.

Previously, Liz held an appointment in various roles in the Dean’s Office at the College of Engineering at North Carolina State University. For the past twenty five years, she has worked extensively with students from preschool to graduate school, parents, preservice and in- service teachers and administrators to implement engineering and STEM in meaningful and measurable ways. As the Co-PI and project director of a National Science Foundation GK-12 grant, Parry developed a highly effective tiered mentoring model for graduate and undergraduate engineering and education teams as well as a popular Family STEM event offering for both elementary and middle school communities.

Parry has been a co-Pi on two NSF DR-K12 Projects: the Exploring the Efficacy of Elementary Engineering Project led by the Museum of Science Boston studying the efficacy of two elementary curricular programs and Engineering For All, a middle school project led by Hofstra University.

In 2014, Liz was appointed by the ASEE Board of Directors to found and chair the inaugural Board Committee on P12 Engineering Education, now the P12 Engineering Commission. She is a Past Chair of the ASEE K-12 and Precollege Division; served as the Vice President of the executive board of the Triangle Coalition for STEM Education, has been a board member of the STEM Consortium and is a member of the K-12 Advisory Committee for the American Society of Mechanical Engineering. She has authored or co-authored over 80 papers, articles and book chapters on issues relating to P20 integrated STEM, including “Perspectives on Failure in the Classroom by Elementary Teachers New to Teaching Engineering,” (co-author with Dr. Pamela Lottero-Perdue of Towson University) which was awarded best Division (K-12 and Precollege), Best PIC (IV) and Best Overall Conference paper for ASEE in 2014. Liz is a frequent invited keynote speaker both nationally and internationally. Prior to joining NCSU, Liz worked in engineering and management positions at IBM Corporation for ten years. A longtime mentor, in 2015 Liz received the Presidential Award for Excellence in Science, Mathematics and Engineering Mentoring from President Barack Obama.

visit author page

Download Paper | Permalink

Abstract

The Performance Assessment of Design Skills (PADS) is designed to be used with youth ages 9-14, in either in- or out-of-school settings. It assesses a youth’s engineering mindset—how they approach an engineering design problem. PADS accomplishes this by presenting a narrative about an engaging challenge and asking youths to describe ways they might solve it. The particular engineering practices or “habits of mind” that the PADS is designed to assess are:

• Consider a real-world problem in context and engineer a solution. • Use a systematic problem-solving process. • Investigate the properties and uses of materials to solve a problem. • Consider constraints and criteria that require trade-offs. • Envision multiple solutions. • Apply science and math knowledge to problem solving. • Evaluate designs and make improvements.

Performance assessments are a form of contextual assessment where students engage in tasks within a context that affords the use of practices of interest to the assessor. There are many advantages to performance assessment, including face validity, the emphasis on skills and the ability to deal with complexity and relevance. Performance assessment tasks should meet several criteria: they should elicit observable performances, use a standard set of tasks, have high fidelity to “real life” performances, measure a variety of levels of performance, and afford improvement with practice. In engineering in the P-12 setting in particular, there is need for assessment focusing on engineering design performance—not just knowledge about it. Familiarity with the process of designing a product is the aspect of engineering that is most frequently called for and researched (Sneider & Ravel, 2021).

Over the past year, we have worked to further develop, improve, and gather evidence of validity and reliability for the PADS when used to measure improvements in youths’ capabilities to engage in productive problem solving in out-of-school settings. Over the summer, we collected matched pre- and post-PADS from 110 youths in six summer camps in five states. Of these, the majority were entering grades 5-7; the diverse sample includes sixty-five girls; 40% are Black, and 5.5% each are Latinx and Native American. Most youths indicated on the pre-PADS no or little experience with engineering projects (79%). We also conducted online interviews with 15 participating youths as a comparison to the PADS: youths were asked to address a design challenge using materials shipped to them in a kit. We also asked these youths to comment on the PADS as compared to the interviews.

Finally, we had camp educators score approximately half of the sample, to gather feedback on usability and appropriateness of the PADS for camp use. Each PADS was scored independently by two trained scorers. We are working on and will present results of quantitative analysis of the PADS and qualitatively compare to interview data. We will also present evidence of validity and reliability, ease of use by educators, and reception by youths in an out-of-school setting.

Citation
Format

Lachapelle, C. P., & Parry, E. (2023, June), Board 170: PADS -- The Performance Assessment of Design Skills (Work in Progress) Paper presented at 2023 ASEE Annual Conference & Exposition, Baltimore , Maryland. 10.18260/1-2--42532

TY - CPAPER
AB - The Performance Assessment of Design Skills (PADS) is designed to be used with youth ages 9-14, in either in- or out-of-school settings. It assesses a youth’s engineering mindset—how they approach an engineering design problem. PADS accomplishes this by presenting a narrative about an engaging challenge and asking youths to describe ways they might solve it. The particular engineering practices or “habits of mind” that the PADS is designed to assess are:

• Consider a real-world problem in context and engineer a solution.
• Use a systematic problem-solving process.
• Investigate the properties and uses of materials to solve a problem.
• Consider constraints and criteria that require trade-offs.
• Envision multiple solutions.
• Apply science and math knowledge to problem solving.
• Evaluate designs and make improvements.

Performance assessments are a form of contextual assessment where students engage in tasks within a context that affords the use of practices of interest to the assessor. There are many advantages to performance assessment, including face validity, the emphasis on skills and the ability to deal with complexity and relevance. Performance assessment tasks should meet several criteria: they should elicit observable performances, use a standard set of tasks, have high fidelity to “real life” performances, measure a variety of levels of performance, and afford improvement with practice. In engineering in the P-12 setting in particular, there is need for assessment focusing on engineering design performance—not just knowledge about it. Familiarity with the process of designing a product is the aspect of engineering that is most frequently called for and researched (Sneider &amp;amp; Ravel, 2021).

Over the past year, we have worked to further develop, improve, and gather evidence of validity and reliability for the PADS when used to measure improvements in youths’ capabilities to engage in productive problem solving in out-of-school settings. Over the summer, we collected matched pre- and post-PADS from 110 youths in six summer camps in five states. Of these, the majority were entering grades 5-7; the diverse sample includes sixty-five girls; 40% are Black, and 5.5% each are Latinx and Native American. Most youths indicated on the pre-PADS no or little experience with engineering projects (79%). We also conducted online interviews with 15 participating youths as a comparison to the PADS: youths were asked to address a design challenge using materials shipped to them in a kit. We also asked these youths to comment on the PADS as compared to the interviews.

Finally, we had camp educators score approximately half of the sample, to gather feedback on usability and appropriateness of the PADS for camp use. Each PADS was scored independently by two trained scorers. We are working on and will present results of quantitative analysis of the PADS and qualitatively compare to interview data. We will also present evidence of validity and reliability, ease of use by educators, and reception by youths in an out-of-school setting.
AU - Cathy P. Lachapelle
AU - Elizabeth Parry
CY - Baltimore , Maryland
DA - 2023/06/25
PB - ASEE Conferences
TI - Board 170: PADS -- The Performance Assessment of Design Skills (Work in Progress)
UR - https://peer.asee.org/42532
DO - 10.18260/1-2--42532
ER -