Asee peer logo

Using an Instrument Blueprint to Support the Rigorous Development of New Surveys and Assessments in Engineering Education

Download Paper |

Conference

2015 ASEE Annual Conference & Exposition

Location

Seattle, Washington

Publication Date

June 14, 2015

Start Date

June 14, 2015

End Date

June 17, 2015

ISBN

978-0-692-50180-1

ISSN

2153-5965

Conference Session

Survey and Assessment Development

Tagged Division

Educational Research and Methods

Page Count

17

Page Numbers

26.1657.1 - 26.1657.17

DOI

10.18260/p.24993

Permanent URL

https://peer.asee.org/24993

Download Count

187

Request a correction

Paper Authors

biography

Jessica Menold Menold Pennsylvania State University, University Park

visit author page

Jessica Menold is a second year graduate student interested in entrepreneurship, the design process, and innovativeness of engineering graduates and professionals. She is currently working as a student mentor in the Lion Launch Pad program, where she works to support student entrepreneurs. Jessica is currently conducting her graduate research with Dr. Kathryn Jablokow on a project devoted to the development of a psychometric instrument that will measure the skills, behaviors, and traits of an innovative engineer. Her hope is that this awareness of individual innovativeness levels will enhance engineering professionals and student’s innovative skillsets. Jessica is also interested in studying and teaching design thinking methods to students, and is currently working to spread design thinking through mini-workshops across Penn State.

visit author page

biography

Kathryn W. Jablokow Pennsylvania State University, Great Valley

visit author page

Dr. Kathryn Jablokow is an Associate Professor of Mechanical Engineering and Engineering Design at Penn State University. A graduate of Ohio State University (Ph.D., Electrical Engineering), Dr. Jablokow’s teaching and research interests include problem solving, invention, and creativity in science and engineering, as well as robotics and computational dynamics. In addition to her membership in ASEE, she is a Senior Member of IEEE and a Fellow of ASME. Dr. Jablokow is the architect of a unique 4-course module focused on creativity and problem solving leadership and is currently developing a new methodology for cognition-based design. She is one of three instructors for Penn State’s Massive Open Online Course (MOOC) on Creativity, Innovation, and Change, and she is the founding director of the Problem Solving Research Group, whose 50+ collaborating members include faculty and students from several universities, as well as industrial representatives, military leaders, and corporate consultants.

visit author page

biography

Senay Purzer Purdue University, West Lafayette Orcid 16x16 orcid.org/0000-0003-0784-6079

visit author page

Ṣenay Purzer is an Assistant Professor in the School of Engineering Education. She is the recipient of a 2012 NSF CAREER award, which examines how engineering students approach innovation. She serves on the editorial boards of Science Education and the Journal of Pre-College Engineering Education (JPEER). She received a B.S.E with distinction in Engineering in 2009 and a B.S. degree in Physics Education in 1999. Her M.A. and Ph.D. degrees are in Science Education from Arizona State University earned in 2002 and 2008, respectively.

visit author page

biography

Daniel Michael Ferguson Purdue University, West Lafayette

visit author page

Daniel M. Ferguson is the recipient of four NSF awards for research in engineering education and a research associate at Purdue University. Prior to coming to Purdue he was Assistant Professor of Entrepreneurship at Ohio Northern University. Before assuming that position he was Associate Director of the Inter-professional Studies Program and Senior Lecturer at Illinois Institute of Technology and involved in research in service learning, assessment processes and interventions aimed at improving learning objective attainment. Prior to his University assignments he was the Founder and CEO of The EDI Group, Ltd. and The EDI Group Canada, Ltd, independent professional services companies specializing in B2B electronic commerce and electronic data interchange. The EDI Group companies conducted syndicated market research, offered educational seminars and conferences and published The Journal of Electronic Commerce. He was also a Vice President at the First National Bank of Chicago, where he founded and managed the bank’s market leading professional Cash Management Consulting Group, initiated the bank’s non credit service product management organization and profit center profitability programs and was instrumental in the breakthrough EDI/EFT payment system implemented by General Motors. Dr. Ferguson is a graduate of Notre Dame, Stanford and Purdue Universities and a member of Tau Beta Pi.

visit author page

biography

Matthew W. Ohland Purdue University Orcid 16x16 orcid.org/0000-0003-4052-1452

visit author page

Matthew W. Ohland is Professor of Engineering Education at Purdue University. He has degrees from Swarthmore College, Rensselaer Polytechnic Institute, and the University of Florida. His research on the longitudinal study of engineering students, team assignment, peer evaluation, and active and collaborative teaching methods has been supported by over $14.5 million from the National Science Foundation and the Sloan Foundation and his team received Best Paper awards from the Journal of Engineering Education in 2008 and 2011 and from the IEEE Transactions on Education in 2011. Dr. Ohland is Chair of the IEEE Curriculum and Pedagogy Committee and an ABET Program Evaluator for ASEE. He was the 2002–2006 President of Tau Beta Pi and is a Fellow of the ASEE and IEEE.

visit author page

Download Paper |

Abstract

Title: Using an Instrument Blueprint to Support the Rigorous Development ofNew Surveys and Assessments in Engineering EducationAbstract:Many sound methods exist for creating the items or questions that make up educational surveysand assessments. These methods include the use of content experts, reviews of existinginstruments, and lists of behaviors and descriptors commonly associated with the construct(s) wewish to assess. Unfortunately, however, item creation sometimes becomes overly dependentupon a researcher’s personal attitudes about the construct(s) being tested, or on “borrowing”items from other instruments that may or may not be sound measures of the construct(s) ofinterest. These risks are particularly likely for new researchers in engineering education, whomay have little experience with best practices in social science research.One way to support best practices in the development of new surveys and assessments is to usean instrument blueprint to guide the creation of items, as well as the collection of validityevidence. This paper outlines a process for instrument blueprint creation and content validationto help support best practices in educational assessment. Based on Messick’s unified theory ofvalidity, the instrument blueprint includes a process for item construction that incorporatesmultiple resources, including: (1) the views of content experts; (2) research from the relevantdomain of interest; (3) reviews of existing instruments; and (4) the expertise of the researchteam.This paper uses the development of a new instrument to measure engineering innovativeness asan illustrative example of the blueprinting process. Our new instrument will assess 20characteristics of innovative engineers as identified by in-depth studies of expert engineeringinnovators in previous research. This work highlights the positive impact of using a systematicprocess for item construction to transform current methods of assessment in engineeringeducation.

Menold, J., & Jablokow, K. W., & Purzer, S., & Ferguson, D. M., & Ohland, M. W. (2015, June), Using an Instrument Blueprint to Support the Rigorous Development of New Surveys and Assessments in Engineering Education Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24993

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015