Asee peer logo

A Student Engagement Evaluation Methodology Inspired from Usability Engineering for Extracting Course Design Requirements

Download Paper |


2020 ASEE Virtual Annual Conference Content Access


Virtual On line

Publication Date

June 22, 2020

Start Date

June 22, 2020

End Date

June 26, 2021

Conference Session

Approaches to Encouraging Student Engagement

Tagged Division

Educational Research and Methods

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Panagiotis Apostolellis University of Virginia

visit author page

Panagiotis Apostolellis is a full-time Assistant Professor in the Computer Science Department at the University of Virginia. He received his PhD in Computer Science & Applications from Virginia Tech in 2017. He also holds a MSc in Computer Science from Virginia Tech and an MSc in Human-Centered Systems from the University of Sussex (UK). Additionally, Panagiotis has a Graduate Certificate in Human-Computer Interaction from Virginia Tech (2015). While a graduate student at Virginia Tech (2011-2017), he worked at the Center for Human-Computer Interaction under the guidance of Dr. Doug Bowman, researching the impact of audience interaction using serious games and VR on young student audiences visiting informal learning spaces. His teaching experience involves being an Adjunct faculty member and a Visiting Assistant Professor at Virginia Tech, serving as a Teaching Assistant in multiple CS courses, and teaching diverse audiences about IT and New Media Technologies. His dissertation received the Outstanding Research award for 2017-2018 from the CS department at Virginia Tech. Panagiotis has also extensive experience as a Senior Interactive Systems Designer and Developer at a cultural institution in his home country, Greece (2000-2011).

visit author page


Sitong Wang University of Cincinnati

visit author page

Sitong Wang is an undergraduate student from Chongqing University – University of Cincinnati Joint Coop Institution, major in Electrical Engineering. Her academic interest is in the Human-Computer Interaction (HCI) field. She had internship experience as a research assistant at Computer Science Department, University of Virginia in Fall 2018. She also took the role of being the research assistant in HCI Initiative group at The Hong Kong University of Science and Technology in Summer 2019. On the industry side, she was once a trainee user interface designer at a big data product development group of a technology company in Chongqing, China.

visit author page

Download Paper |


Measuring student engagement inside the classroom and developing techniques for improving it has been traditionally very challenging for educators. This research paper describes a student engagement evaluation model that combined data from three sources: in- class observations using the Behavioral Engagement Related to Instruction (BERI) protocol, one-to-one student interviews, and anonymous online surveys. We tested this model on a higher-level elective Computer Science class with 134 students, focusing on user experience (UX) design. We used the exact same usability engineering process that students employed in the course to design software products on assessing the course’s effectiveness, with students acting as users and the class being the system under development. We present our exact methodology of data-driven analysis borrowed from UX design, which combines inductive reasoning (similar to narrative analysis or inductive coding) with deductive reasoning (similar to content analysis of qualitative data) used in social science research. Our paper includes a sample of the extracted requirements, similar to software non-functional requirements, which we used to redesign the course for the next semester. The results of this case study showed that the application of this mixed methods type of analysis informed by user-centered design of software systems was effective as a surrogate model of student-centered instructional design. Concluding, we extrapolate the lessons learned from this process and the significant implications we believe our industry-inspired methodology can have for engineering educators, in terms of evaluating student engagement in college classrooms.

Apostolellis, P., & Wang, S. (2020, June), A Student Engagement Evaluation Methodology Inspired from Usability Engineering for Extracting Course Design Requirements Paper presented at 2020 ASEE Virtual Annual Conference Content Access, Virtual On line . 10.18260/1-2--34056

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2020 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015