Asee peer logo

Developing Cognitive Affective Behavioral Work Sampling Methodologies To Assess Student Learning Outcomes

Download Paper |


2007 Annual Conference & Exposition


Honolulu, Hawaii

Publication Date

June 24, 2007

Start Date

June 24, 2007

End Date

June 27, 2007



Conference Session

Cognitive and Motivational Issues in Student Performance II

Tagged Division

Educational Research and Methods

Page Count


Page Numbers

12.491.1 - 12.491.10



Permanent URL

Download Count


Request a correction

Paper Authors


Tuba Pinar Yildirim University of Pittsburgh

visit author page

Tuba Pinar Yildirim is a Ph.D. candidate in Industrial Engineering and Marketing Departments of University of Pittsburgh. Her research interests are engineering assessment, financial optimization, market segmentation and customer portfolio management. She holds an M.S. (University of Pittsburgh) and a B.S. in Industrial Engineering (Middle East Technical University). Address: 1048 Benedum Hall, University of Pittsburgh, Pittsburgh, PA 15261; telephone 412.400.8631; fax: 412.624.9831; e-mail:

visit author page


Joel Townsend University of Pittsburgh

visit author page

Joel Townsend is a graduate student in Industrial Engineering Department of University of Pittsburgh. His research interests are engineering assessment, engineering ethics and public policy. He holds a B.S. in Industrial Engineering (University of Pittsburgh.

visit author page


Mary Besterfield-Sacre University of Pittsburgh

visit author page

Mary Besterfield-Sacre is an Associate Professor of Industrial Engineering and the Fulton C. Noss Faculty Fellow at the University of Pittsburgh. Her research interests are in engineering education evaluation, in empirical mod-eling applications, and K12 district system improvements. In the area of assessment, Dr. Sacre has written numerous conference and journal papers and has given many workshops and pres-entations. Her research in this area has been funded by the NSF, DOE, Sloan Foundation, EiF, and the NCIIA. She is an associate editor of the Journal of Engineering Education.

visit author page


Larry Shuman University of Pittsburgh Orcid 16x16

visit author page

Larry J. Shuman is Associate Dean for Academic Affairs, School of Engineering, University of Pittsburgh and Professor of Industrial Engineering. His areas of interest are improving the engineering education and the study of ethical behavior of engineers. As Associate Dean, he has introduced a many curricula innovations. He has been principle or co-principle investigator on over 20 sponsored projects funded by the NSF, HHS and DoT, the RW Johnson Foundation, and EiF. He is Editor of the new Advances in Engineering Education.

visit author page


Harvey Wolfe University of Pittsburgh

visit author page

Harvey Wolfe Professor Emeritus in the Department of Industrial Engineering at the University of Pittsburgh. He is a Fellow of the Institute of Industrial Engineers (IIE) and served as President. He served as President of the Council of Industrial Engineering Academic Department Heads in 1999-2000. He is a co-author of Engineering Ethics: Balancing Cost Schedule and Risk - Lessons Learned from the Space Shuttle (Cambridge University Press, 1997).

visit author page

Download Paper |

NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract

Developing Cognitive, Affective, Behavioral Work Sampling Methodologies to Assess Student Learning Outcomes Abstract

In this study, we develop and validate a work sampling methodology to assess processes that engineers usually engage in (i.e., working in teams, conducting design work, addressing ethical issues). To obtain in-depth measures for these process oriented student learning outcomes, 100 percent behavioral observation is typically used, but which is time consuming and expensive. Work sampling is a common industry practice used to observe physical activities, as it minimizes time to collect data, yet provides statistically similar results relative to 100% behavioral observation. In our research we have bridged the gap between common practices in industry and assessment in engineering education by extending sampling theories to the observation of intervals that can capture the cognitive, behavioral and affective domains for three student learning processes – teamwork, design, and ethical reasoning.

We designed an experiment to statistically compared 100% behavioral observation with work sampling. Four environments with two examples each were videotaped. Each tape was evaluated by two observer teams: one to conduct 100% behaviorally observation and the other to work sample. ANOVA tests were used to determine inter-rater reliability both within and between teams. Results suggest that work sampling can replace 100% behavioral observation for teamwork. Similar positive results have been obtained for design. For ethical reasoning, although a high reliability could be obtained between observers for 100% behavioral observation, work sampling was not a suitable replacement method. This paper describes the overall study, its overarching results with respect to the three outcomes investigated, and comments on various factors related to each outcome that may permit work sampling to be an effective alternative for some outcomes but not for others.


The engineering criteria has changed the motivation of engineering education accreditation from “what are you [the program] doing?” to “what are your students doing?” As a result, the need for solid, in-depth measurements has become a high priority. At recent engineering education conferences (e.g. Best Assessment Processes in Engineering Education Symposiums, ASEE, FIE) the number of evolving approaches for evaluating engineering programs, as well as methodologies for measuring various student outcomes is growing more rich. Yet, several troublesome issues still remain. First, most of these “assessment” methods had not been fully evaluated. Second, many focus on final products via performance appraisals particular to the outcome(s) using rubrics as the assessment tool. Third, many engineering administrators still voiced concerns about the costs associated with organizing, implementing and maintaining an effective assessment program, given limited resources of time, people (i.e. raters), and money.

Assessing non-sequential outcomes in engineering such as working in teams, development of designs or overcoming ethical dilemmas often require a methodological tool to examine behavior at various levels of the cognitive and affective domains (e.g. analysis, synthesis, evaluation, and valuation). While such a tool has been needed for professional work, the recent movement

Yildirim, T. P., & Townsend, J., & Besterfield-Sacre, M., & Shuman, L., & Wolfe, H. (2007, June), Developing Cognitive Affective Behavioral Work Sampling Methodologies To Assess Student Learning Outcomes Paper presented at 2007 Annual Conference & Exposition, Honolulu, Hawaii. 10.18260/1-2--2799

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2007 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015