Considering Professional Diversity as a Factor in a Consensus Building Method for Expert Crowdsourcing of Curriculum Topics

Brian Khoa Ngac; Mihai Boicu

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: July 12, 2024
Conference Session: Engineering Technology Division Curriculum Development
Tagged Division: Engineering Technology Division (ETD)
Tagged Topic: Diversity
Permanent URL: https://peer.asee.org/47067

Request a correction

Paper Authors

biography

Brian Khoa Ngac George Mason University

visit author page

Brian K. Ngac is an Instructional Faculty Member and Dean's Teaching Fellow at George Mason University's School of Business. Moreover, he is a PhD Candidate (ABD) at George Mason University’s College of Engineering & Computing. He holds 12 internationally recognized cyber security and management certifications including the C|CISO, CISSP, ISSMP, CISM, and PMP. His areas of expertise are in cyber security, digital engineering (RDT&E), and business process improvement (solving business challenges with technology solutions). His research focus are in cyber executive management, expert crowdsourcing, and decision analytics. Brian is also the Deputy Vice President for Digital Engineering Programs at Parsons Corporation.

visit author page

biography

Mihai Boicu George Mason University orcid.org/0000-0002-6644-059X

visit author page

Mihai Boicu, Ph.D., is Assistant Professor of Information Technology at George Mason University, Associate Director of the Learning Agents Center (http://lac.gmu.edu), Co-Director of IT Entrepreneurship Laboratory (http://lite.gmu.edu) and Co-Director of

visit author page

Download Paper | Permalink

Abstract

State of the art curriculum development efforts today are generally undertaken by solely faculty members of the program. However, in our previous work [anonymized citation], we showed how expert crowdsourcing combined with the application of a consensus building method can be used to perform curriculum development asynchronously with a larger group of experts beyond a program’s faculty. The consensus building method included two operations by the expert crowd: (1) validating the existing list of curriculum topics and their subtopics; and (2) suggesting additional topics and subtopics to be added to the current curriculum. This paper will show results yielded by a finalized experiment utilizing consensus building method against a graduate technology management course’s curriculum development.

This paper will then detail how this research effort incorporated a professional diversity factor into the consensus building method when performing expert crowdsourcing. Professional diversity is important because when building consensus among the experts, we also want to ensure there are enough representatives from various relevant categories of professionals. To calculate professional diversity, we performed a five-step process: (1) Identify the relevant factors, (2) Identify the interactions between factors and relevant categories to consider, (3) Identify the relevance of the identified categories, (4) Identify the ideal recruitment needs, and (5) Compute a global professional diversity measure based on current recruitment. Relevant factors represent statistical variables for the expert crowd population which may include years of domain experience (e.g., junior, senior), the industry they’re in (e.g., industry, government, or education), and the position they hold (e.g., executive, manager, practitioner). Identifying the interaction between the different factors and which combinations will also be considered (if we have independent variables or dependent variables). For instance, the input of a junior engineer in industry versus an assistant professor in academia. Both are juniors but in different industries. One solution is to consider them as diversity categories: junior versus senior, or industry versus academia. Another solution is to consider the categories: junior in industry and junior in academia. If the factors are independent with respect to the subject matter, then keeping simple distinct categories will be easier. However, we may determine that the views of a junior in industry are quite different from the views of a junior in academia, and therefore decide to keep the combined categories (this will lead to more categories but also more accurate results). In this case the factors will be dependent. Such an analysis will be performed both at the beginning using domain expertise but also during the experiment analyzing the answers received and adapting prior decisions.

While this paper is a work in progress, the experiment currently running to test the professional diversity factor will be completed by the end of 2023 – so the results, analysis, and discussions will be available for the draft and final paper submissions.

Citation
Format

Ngac, B. K., & Boicu, M. (2024, June), Considering Professional Diversity as a Factor in a Consensus Building Method for Expert Crowdsourcing of Curriculum Topics Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. https://peer.asee.org/47067

TY - CPAPER
AB - State of the art curriculum development efforts today are generally undertaken by solely faculty members of the program. However, in our previous work [anonymized citation], we showed how expert crowdsourcing combined with the application of a consensus building method can be used to perform curriculum development asynchronously with a larger group of experts beyond a program’s faculty. The consensus building method included two operations by the expert crowd: (1) validating the existing list of curriculum topics and their subtopics; and (2) suggesting additional topics and subtopics to be added to the current curriculum. This paper will show results yielded by a finalized experiment utilizing consensus building method against a graduate technology management course’s curriculum development.

This paper will then detail how this research effort incorporated a professional diversity factor into the consensus building method when performing expert crowdsourcing. Professional diversity is important because when building consensus among the experts, we also want to ensure there are enough representatives from various relevant categories of professionals. To calculate professional diversity, we performed a five-step process: (1) Identify the relevant factors, (2) Identify the interactions between factors and relevant categories to consider, (3) Identify the relevance of the identified categories, (4) Identify the ideal recruitment needs, and (5) Compute a global professional diversity measure based on current recruitment. Relevant factors represent statistical variables for the expert crowd population which may include years of domain experience (e.g., junior, senior), the industry they’re in (e.g., industry, government, or education), and the position they hold (e.g., executive, manager, practitioner). Identifying the interaction between the different factors and which combinations will also be considered (if we have independent variables or dependent variables). For instance, the input of a junior engineer in industry versus an assistant professor in academia. Both are juniors but in different industries. One solution is to consider them as diversity categories: junior versus senior, or industry versus academia. Another solution is to consider the categories: junior in industry and junior in academia. If the factors are independent with respect to the subject matter, then keeping simple distinct categories will be easier. However, we may determine that the views of a junior in industry are quite different from the views of a junior in academia, and therefore decide to keep the combined categories (this will lead to more categories but also more accurate results). In this case the factors will be dependent. Such an analysis will be performed both at the beginning using domain expertise but also during the experiment analyzing the answers received and adapting prior decisions.

While this paper is a work in progress, the experiment currently running to test the professional diversity factor will be completed by the end of 2023 – so the results, analysis, and discussions will be available for the draft and final paper submissions.
AU - Brian Khoa Ngac
AU - Mihai Boicu
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - Considering Professional Diversity as a Factor in a Consensus Building Method for Expert Crowdsourcing of Curriculum Topics
UR - https://peer.asee.org/47067
ER -