BOARD # 89: WIP: Developing an Instrument to Understand Engineering Student uses of Digital External Resources from Solution Manuals to Generative AI

Christopher Allen Calhoun; David Reeping; Siqing Wei; Aarohi Shah

Download Paper | Permalink

Conference: 2025 ASEE Annual Conference & Exposition
Location: Montreal, Quebec, Canada
Publication Date: June 22, 2025
Start Date: June 22, 2025
End Date: August 15, 2025
Conference Session: Computers in Education Division (COED) Poster Session (Track 1.A)
Tagged Division: Computers in Education Division (COED)
Page Count: 10
DOI: 10.18260/1-2--55905
Permanent URL: https://peer.asee.org/55905
Download Count: 4

Dr. David Reeping is an Assistant Professor in the Department of Engineering and Computing Education at the University of Cincinnati. He earned his Ph.D. in Engineering Education from Virginia Tech and was a National Science Foundation Graduate Research Fellow. He received his B.S. in Engineering Education with a Mathematics minor from Ohio Northern University. His main research interests include transfer student information asymmetries, threshold concepts, curricular complexity, and advancing quantitative and fully integrated mixed methods.

visit author page

biography

Siqing Wei University of Cincinnati orcid.org/0000-0002-7086-5953

visit author page

Dr. Siqing Wei received a B.S. and M.S. in Electrical Engineering and a Ph.D. in Engineering Education program at Purdue University as a triple boiler. He is a postdoc fellow at the University of Cincinnati under the supervision of Dr. David Reeping. His research interests span three major research topics, which are teamwork, cultural diversity, and international and Asian/ Asian American student experiences. He utilizes innovative and cutting-edge methods, such as person-centered approaches, NLP, ML, and Social Relation Models. He studies and promotes multicultural teaming experiences to promote an inclusive and welcoming learning space for all to thrive in engineering. Particularly, he aims to help students improve intercultural competency and teamwork competency through interventions, counseling, pedagogy, and mentoring. Siqing received the Outstanding Graduate Student Research Award in 2024 from Purdue College of Engineering, the Bilsland Dissertation fellow in the 2023-24 academic year, and the 2024 FIE New Faculty Fellow Award.

visit author page

biography

Aarohi Shah University of Cincinnati

visit author page

Aarohi Shah is a third-year Computer Science student at the University of Cincinnati. She is an undergraduate research assistant in the Department of Engineering and Computing Education.

visit author page

Download Paper | Permalink

Abstract

This work-in-progress research paper describes an instrument development effort to understand how engineering students use external resources to supplement their learning and the associated metacognitive strategies they employ with those resources. With the increasing prevalence of generative AI tools like ChatGPT, traditional problem-solving platforms (e.g., Chegg) and educational video resources (e.g., YouTube, Khan Academy), engineering students now have unprecedented access to external learning resources. Considering the pace of technological developments lowering barriers to entry for students to get on-demand assistance with their studies, there is a growing need to understand how students engage with these different resources.

Our instrument is being developed to explore two key research questions: (1) How do engineering students use external resources, including generative AI, to assist in problem-solving within their coursework? and (2) What factors drive their preference for certain tools over others? Later in our project, we will explore how students use metacognitive strategies with various external resources – which is a separate research question not directly relevant to the instrument itself. Therefore, we plan to include scales related to metacognitive strategies to enable purposive sampling for student interviews based on the results of the instrument distribution.

Our theoretical framework to ground the instrument development process is the technology acceptance model. The model's premise is that individuals choose to adopt new technologies based on the perceived usefulness and ease of use of that technology, which has been used to explore the adoption of e-learning tools, learning management systems, and video conferencing platforms among faculty and students.

To source items for our instrument, we are reviewing the literature on student adoption of external resources (i.e., “homework help” websites like Chegg, video platforms like YouTube, and generative AI systems like ChatGPT). We are using databases including Education Research Complete, ERIC, and Arxiv to search for papers; this search is currently in progress. After extracting questions, we will align them with the technology acceptance model to construct a draft instrument. Later in Fall 2024, we will pilot the instrument with a sample of 10-30 undergraduate engineering students – members of the intended population – through cognitive interviewing to determine the relevance and comprehensibility of the questions. By the time of the draft, we will have the findings to share from the cognitive interviews.

After our cognitive interviewing phase, the instrument will be administered to at least 200 undergraduate engineering students at a large Midwestern university in Spring 2025. We plan to analyze the instrument using exploratory factor analysis if items are primarily customized and confirmatory factor analysis if more minor changes are made to specific scales. By the time of the conference, we expect to present initial survey findings from the Spring data collection.

We anticipate this research will enable us to better understand how students leverage external resources in engineering education to provide better support for maximizing their utility. We invite suggestions for improving the instrument by the draft submission and seek feedback on refining the instrument for broader use at other institutions in later semesters.

Citation
Format

Calhoun, C. A., & Reeping, D., & Wei, S., & Shah, A. (2025, June), BOARD # 89: WIP: Developing an Instrument to Understand Engineering Student uses of Digital External Resources from Solution Manuals to Generative AI Paper presented at 2025 ASEE Annual Conference & Exposition , Montreal, Quebec, Canada . 10.18260/1-2--55905

TY  - CPAPER
AB  - This work-in-progress research paper describes an instrument development effort to understand how engineering students use external resources to supplement their learning and the associated metacognitive strategies they employ with those resources. With the increasing prevalence of generative AI tools like ChatGPT, traditional problem-solving platforms (e.g., Chegg) and educational video resources (e.g., YouTube, Khan Academy), engineering students now have unprecedented access to external learning resources. Considering the pace of technological developments lowering barriers to entry for students to get on-demand assistance with their studies, there is a growing need to understand how students engage with these different resources.  

Our instrument is being developed to explore two key research questions: (1) How do engineering students use external resources, including generative AI, to assist in problem-solving within their coursework? and (2) What factors drive their preference for certain tools over others? Later in our project, we will explore how students use metacognitive strategies with various external resources – which is a separate research question not directly relevant to the instrument itself. Therefore, we plan to include scales related to metacognitive strategies to enable purposive sampling for student interviews based on the results of the instrument distribution. 

 

Our theoretical framework to ground the instrument development process is the technology acceptance model. The model&#39;s premise is that individuals choose to adopt new technologies based on the perceived usefulness and ease of use of that technology, which has been used to explore the adoption of e-learning tools, learning management systems, and video conferencing platforms among faculty and students.  

To source items for our instrument, we are reviewing the literature on student adoption of external resources (i.e., “homework help” websites like Chegg, video platforms like YouTube, and generative AI systems like ChatGPT). We are using databases including Education Research Complete, ERIC, and Arxiv to search for papers; this search is currently in progress. After extracting questions, we will align them with the technology acceptance model to construct a draft instrument. Later in Fall 2024, we will pilot the instrument with a sample of 10-30 undergraduate engineering students – members of the intended population – through cognitive interviewing to determine the relevance and comprehensibility of the questions. By the time of the draft, we will have the findings to share from the cognitive interviews.  

After our cognitive interviewing phase, the instrument will be administered to at least 200 undergraduate engineering students at a large Midwestern university in Spring 2025. We plan to analyze the instrument using exploratory factor analysis if items are primarily customized and confirmatory factor analysis if more minor changes are made to specific scales. By the time of the conference, we expect to present initial survey findings from the Spring data collection. 

We anticipate this research will enable us to better understand how students leverage external resources in engineering education to provide better support for maximizing their utility. We invite suggestions for improving the instrument by the draft submission and seek feedback on refining the instrument for broader use at other institutions in later semesters. 
AU  - Christopher Allen Calhoun
AU  - David Reeping
AU  - Siqing Wei
AU  - Aarohi Shah
CY  - Montreal, Quebec, Canada 
DA  - 2025/06/22
PB  - ASEE Conferences
TI  - BOARD # 89: WIP: Developing an Instrument to Understand Engineering Student uses of Digital External Resources from Solution Manuals to Generative AI
UR  - https://peer.asee.org/55905
DO  - 10.18260/1-2--55905
ER  -