Evolution of a Survey for Self-Reported Engineering Design Space Exploration Tendency

Andrew Jeremiah Lance; Gregory Michael Mocko; Venkat Jaya Deep Jakka; Rahul Sharan Renu

Download Paper | Permalink

Conference: 2024 South East Section Meeting
Location: Marietta, Georgia
Publication Date: March 10, 2024
Start Date: March 10, 2024
End Date: March 12, 2024
Page Count: 14
DOI: 10.18260/1-2--45524
Permanent URL: https://peer.asee.org/45524
Download Count: 105

Gregory Mocko is an Associate Professor in the School of Mechanical and Automotive Engineering at Clemson University. His research interests include distributed design and manufacturing of complex systems, and computational approaches in engineering design, systems engineering, and creativity in conceptual design. His research is supported NSF, BMW Manufacturing Corporation, BMW AG, National Institute of Standards and Technology (NIST), Johnson Controls Incorporated, and US Army TACOM / GVSC, Michelin, The Boeing Company, South Carolina Department of Commerce, and Fraunhofer USA Alliance. He serves as the ME Department Capstone Faculty Coordinator, working with industry partners and students to address design and manufacturing challenges. In addition, he leads a multi-university student project focused on distributed design and manufacturing of UAVs. He is the Associate Director of Education and Training at the Product Lifecycle Management (PLM) Center and Director of External Engagement for the Virtual Prototyping of Autonomy-Enabled Ground Systems (VIPR-GS) Center at Clemson University.

visit author page

author page

Venkat Jaya Deep Jakka Clemson University

biography

Rahul Sharan Renu Francis Marion University

visit author page

Dr. Renu is an associate professor of mechanical engineering at Francis Marion University. He also serves as the program coordinator for mechanical engineering. His research interests are in the fields of digital manufacturing, AI in design, and engineering education.

visit author page

Download Paper | Permalink

Abstract

develop a survey consisting of statements that provide insight into an individual’s design space exploration tendency. There are formal exercises to evaluate design space exploration, but these exercises are resource intensive, time consuming, challenging to deploy and difficult to process the results. The survey instrument is intended to address several of these challenges. To develop the survey instrument, the Shah-Vargas (SV) metrics of engineering ideation effectiveness were used as a basis for quantifying engineering Design Space Exploration (DSE). These metrics are 1) Quantity – the number of ideas generated, 2) Quality – the conformance of each idea to engineering requirements, 3) Variety – the dissimilarity of an idea within an individual’s set of generated ideas, and 4) Novelty – the dissimilarity of an idea within the collectively exhaustive set of ideas. With these metrics as a guide, an initial list of statements was developed using two approaches. First, literature was reviewed for statements that have been used to collect self-reported data on the four metrics. Second, the definitions of the four metrics from Shah and colleagues were reviewed and converted to question form. This resulted in four statements per metric, totaling 16 statements. Next, to assess question clarity regarding the four metrics and to ensure survey respondents accurately grasped the metric each statement pertained to, Latent Semantic Analysis (LSA) was employed to evaluate overlap. In addition, the statements were processed by a Large Language Model which was asked to assess overlap. Based on the findings from these analyses, the statements were modified to reduce overlap. A final verification of mutual exclusivity will be where participants are going to be asked to categorize each question into one of the four metrics. The result of this work is a survey with statements which allows an individual to self-report their DSE tendency. In the future, this validity of self-reported data will be assessed by comparing it with direct assessment of DSE tendency. Once validated, the DSE survey is intended for researchers to gain a deeper understanding about DSE tendencies without having the resource-intensive, subjective task of performing direct assessments. Additionally, the survey can be used as a pre-screening if/when design exercises are deployed.

Citation
Format

Lance, A. J., & Mocko, G. M., & Jakka, V. J. D., & Renu, R. S. (2024, March), Evolution of a Survey for Self-Reported Engineering Design Space Exploration Tendency Paper presented at 2024 South East Section Meeting, Marietta, Georgia. 10.18260/1-2--45524

TY - CPAPER
AB - develop a survey consisting of statements that provide insight into an individual’s design space exploration tendency. There are formal exercises to evaluate design space exploration, but these exercises are resource intensive, time consuming, challenging to deploy and difficult to process the results. The survey instrument is intended to address several of these challenges. To develop the survey instrument, the Shah-Vargas (SV) metrics of engineering ideation effectiveness were used as a basis for quantifying engineering Design Space Exploration (DSE). These metrics are 1) Quantity – the number of ideas generated, 2) Quality – the conformance of each idea to engineering requirements, 3) Variety – the dissimilarity of an idea within an individual’s set of generated ideas, and 4) Novelty – the dissimilarity of an idea within the collectively exhaustive set of ideas.
With these metrics as a guide, an initial list of statements was developed using two approaches. First, literature was reviewed for statements that have been used to collect self-reported data on the four metrics. Second, the definitions of the four metrics from Shah and colleagues were reviewed and converted to question form. This resulted in four statements per metric, totaling 16 statements.
Next, to assess question clarity regarding the four metrics and to ensure survey respondents accurately grasped the metric each statement pertained to, Latent Semantic Analysis (LSA) was employed to evaluate overlap. In addition, the statements were processed by a Large Language Model which was asked to assess overlap. Based on the findings from these analyses, the statements were modified to reduce overlap. A final verification of mutual exclusivity will be where participants are going to be asked to categorize each question into one of the four metrics.
The result of this work is a survey with statements which allows an individual to self-report their DSE tendency. In the future, this validity of self-reported data will be assessed by comparing it with direct assessment of DSE tendency. Once validated, the DSE survey is intended for researchers to gain a deeper understanding about DSE tendencies without having the resource-intensive, subjective task of performing direct assessments. Additionally, the survey can be used as a pre-screening if/when design exercises are deployed.

AU - Andrew Jeremiah Lance
AU - Gregory Michael Mocko
AU - Venkat Jaya Deep Jakka
AU - Rahul Sharan Renu
CY - Marietta, Georgia
DA - 2024/03/10
PB - ASEE Conferences
TI - Evolution of a Survey for Self-Reported Engineering Design Space Exploration Tendency
UR - https://peer.asee.org/45524
DO - 10.18260/1-2--45524
ER -