Asee peer logo

Psychometric Analysis of Residence and MOOC Assessments

Download Paper |


2015 ASEE Annual Conference & Exposition


Seattle, Washington

Publication Date

June 14, 2015

Start Date

June 14, 2015

End Date

June 17, 2015





Conference Session

Examining "Big" Data

Tagged Division

Educational Research and Methods

Page Count


Page Numbers

26.1284.1 - 26.1284.14



Permanent URL

Download Count


Request a correction

Paper Authors


Eric Loken Pennsylvania State University

visit author page

Eric Loken is a Research Associate Professor at Penn State specializing in applied statistical analysis with an emphasis on education and health data. Funding for this work was supported by NSF grant GSE/RES 1036731.

visit author page


Zita Oravecz Pennsylvania State University

visit author page

Zita Oravecz is an assistant professor in the department of Human Development and Family Studies at Penn State University. Her research interests involve hierarchical Bayesian modeling, psychometrics,
intensive longitudinal data, and latent variable modeling. She also focuses on individual differences (e.g., in well-being, cognitive functioning) from a process modeling perspective.

visit author page


Conrad Tucker Pennsylvania State University, University Park

visit author page

Dr. Tucker holds a joint appointment as Assistant Professor in Engineering Design and Industrial Engineering at The Pennsylvania State University. He is also affiliate faculty in Computer Science and Engineering. He teaches Introduction to Engineering Design (EDSGN 100) at the undergraduate level and developed and taught a graduate-level course titled Data Mining–Driven Design (EDSGN 561). As part of the Engineering Design Program’s “Summers by Design” (SBD) program, Dr. Tucker supervises students from Penn State during the summer semester in a two-week engineering design program at the École Centrale de Nantes in Nantes, France.

Dr. Tucker is the director of the Design Analysis Technology Advancement (D.A.T.A) Laboratory. His research interests are in formalizing system design processes under the paradigm of knowledge discovery, optimization, data mining, and informatics. His research interests include applications in complex systems design and operation, product portfolio/family design, and sustainable system design optimization in the areas of engineering education, energy generation systems, consumer electronics, environment, and national security.

visit author page


Fridolin Jakob Linder Pennsylvania State University

visit author page

Fridolin Linder is a graduate student in the Department of Political Science at Pennsylvania State University. His work is supported by Pennsylvania State University and the National Science Foundation under an IGERT award # DGE-1144860, Big Data Social Science

visit author page

Download Paper |


Psychometric Analysis of Residence and MOOC Assessment DataUndergraduate STEM programs are faced with the daunting challenge of managing instructionand assessment for classes that enroll thousands of students per year, and the bulk of studentassessment is often determined by multiple choice tests. Instructors try to monitor the reliabilitymetrics and diagnostics for item quality, but rarely is there a more formal evaluation of thepsychometric properties of these assessments. We see an opportunity to have a major impacton undergraduate science instruction by incorporating more rigorous measurement models fortesting, and using them to assist instructional goals and assessment.We propose to apply item response theory to analyze the tests from recent years inundergraduate STEM classes (physics, chemistry and statistics) involving tens of thousands ofstudents. We will evaluate whether the tests are equally informative across the gradedistribution, and we will assess the dimensionality of the test data to infer separable aspects ofachievement.We will also examine data from two large MOOCs conducted by University X. Our researchquestions here will be whether these assessments exhibit similar properties in terms informationcontent, population heterogeneity, and factors affecting performance. The student populations inMOOCs are an order of magnitude larger than in introductory STEM classes, and they areoperating under a very different educational context.Large undergraduate courses (both residence and MOOC) deliver millions of multiple choicetests each year. These tests determine grades and student progress into further STEM study,and are also the most common outcome measure for evaluation educational interventions.These courses should use the same advanced measurement models used to develop so-called“high stakes” tests, such as college admissions tests. A rigorous assessment program canserve three important functions: (1) to improve assessment by allowing the construction ofpsychometrically sound tests, and by facilitating standardization across different sections, anddifferent years of the same course; (2) to offer enhanced learning opportunities for studentsthrough computer adaptive study materials with feedback; and (3) to open up opportunities forresearch on methods for enhancing student learning and engagement, and for investigatingfairness in assessment.

Loken, E., & Oravecz, Z., & Tucker, C., & Linder, F. J. (2015, June), Psychometric Analysis of Residence and MOOC Assessments Paper presented at 2015 ASEE Annual Conference & Exposition, Seattle, Washington. 10.18260/p.24621

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2015 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015