Developing Cognitive Affective Behavioral Work Sampling Methodologies To Assess Student Learning Outcomes
- Conference
- Location
-
Honolulu, Hawaii
- Publication Date
-
June 24, 2007
- Start Date
-
June 24, 2007
- End Date
-
June 27, 2007
- ISSN
-
2153-5965
- Conference Session
- Tagged Division
-
Educational Research and Methods
- Page Count
-
10
- Page Numbers
-
12.491.1 - 12.491.10
- DOI
-
10.18260/1-2--2799
- Permanent URL
-
https://strategy.asee.org/2799
- Download Count
-
376
Paper Authors
Tuba Pinar Yildirim University of Pittsburgh
Tuba Pinar Yildirim is a Ph.D. candidate in Industrial Engineering and Marketing Departments of University of Pittsburgh. Her research interests are engineering assessment, financial optimization, market segmentation and customer portfolio management. She holds an M.S. (University of Pittsburgh) and a B.S. in Industrial Engineering (Middle East Technical University). Address: 1048 Benedum Hall, University of Pittsburgh, Pittsburgh, PA 15261; telephone 412.400.8631; fax: 412.624.9831; e-mail: tpy1@pitt.edu.
Joel Townsend University of Pittsburgh
Joel Townsend is a graduate student in Industrial Engineering Department of University of Pittsburgh. His research interests are engineering assessment, engineering ethics and public policy. He holds a B.S. in Industrial Engineering (University of Pittsburgh.
Mary Besterfield-Sacre University of Pittsburgh
Mary Besterfield-Sacre is an Associate Professor of Industrial Engineering and the Fulton C. Noss Faculty Fellow at the University of Pittsburgh. Her research interests are in engineering education evaluation, in empirical mod-eling applications, and K12 district system improvements. In the area of assessment, Dr. Sacre has written numerous conference and journal papers and has given many workshops and pres-entations. Her research in this area has been funded by the NSF, DOE, Sloan Foundation, EiF, and the NCIIA. She is an associate editor of the Journal of Engineering Education.
Larry Shuman
University of Pittsburgh
orcid.org/https://0000-0001-6884-7070
Larry J. Shuman is Associate Dean for Academic Affairs, School of Engineering, University of Pittsburgh and Professor of Industrial Engineering. His areas of interest are improving the engineering education and the study of ethical behavior of engineers. As Associate Dean, he has introduced a many curricula innovations. He has been principle or co-principle investigator on over 20 sponsored projects funded by the NSF, HHS and DoT, the RW Johnson Foundation, and EiF. He is Editor of the new Advances in Engineering Education.
Harvey Wolfe University of Pittsburgh
Harvey Wolfe Professor Emeritus in the Department of Industrial Engineering at the University of Pittsburgh. He is a Fellow of the Institute of Industrial Engineers (IIE) and served as President. He served as President of the Council of Industrial Engineering Academic Department Heads in 1999-2000. He is a co-author of Engineering Ethics: Balancing Cost Schedule and Risk - Lessons Learned from the Space Shuttle (Cambridge University Press, 1997).
Abstract
NOTE: The first page of text has been automatically extracted and included below in lieu of an abstract
Developing Cognitive, Affective, Behavioral Work Sampling Methodologies to Assess Student Learning Outcomes Abstract
In this study, we develop and validate a work sampling methodology to assess processes that engineers usually engage in (i.e., working in teams, conducting design work, addressing ethical issues). To obtain in-depth measures for these process oriented student learning outcomes, 100 percent behavioral observation is typically used, but which is time consuming and expensive. Work sampling is a common industry practice used to observe physical activities, as it minimizes time to collect data, yet provides statistically similar results relative to 100% behavioral observation. In our research we have bridged the gap between common practices in industry and assessment in engineering education by extending sampling theories to the observation of intervals that can capture the cognitive, behavioral and affective domains for three student learning processes – teamwork, design, and ethical reasoning.
We designed an experiment to statistically compared 100% behavioral observation with work sampling. Four environments with two examples each were videotaped. Each tape was evaluated by two observer teams: one to conduct 100% behaviorally observation and the other to work sample. ANOVA tests were used to determine inter-rater reliability both within and between teams. Results suggest that work sampling can replace 100% behavioral observation for teamwork. Similar positive results have been obtained for design. For ethical reasoning, although a high reliability could be obtained between observers for 100% behavioral observation, work sampling was not a suitable replacement method. This paper describes the overall study, its overarching results with respect to the three outcomes investigated, and comments on various factors related to each outcome that may permit work sampling to be an effective alternative for some outcomes but not for others.
Introduction
The engineering criteria has changed the motivation of engineering education accreditation from “what are you [the program] doing?” to “what are your students doing?” As a result, the need for solid, in-depth measurements has become a high priority. At recent engineering education conferences (e.g. Best Assessment Processes in Engineering Education Symposiums, ASEE, FIE) the number of evolving approaches for evaluating engineering programs, as well as methodologies for measuring various student outcomes is growing more rich. Yet, several troublesome issues still remain. First, most of these “assessment” methods had not been fully evaluated. Second, many focus on final products via performance appraisals particular to the outcome(s) using rubrics as the assessment tool. Third, many engineering administrators still voiced concerns about the costs associated with organizing, implementing and maintaining an effective assessment program, given limited resources of time, people (i.e. raters), and money.
Assessing non-sequential outcomes in engineering such as working in teams, development of designs or overcoming ethical dilemmas often require a methodological tool to examine behavior at various levels of the cognitive and affective domains (e.g. analysis, synthesis, evaluation, and valuation). While such a tool has been needed for professional work, the recent movement
Yildirim, T. P., & Townsend, J., & Besterfield-Sacre, M., & Shuman, L., & Wolfe, H. (2007, June), Developing Cognitive Affective Behavioral Work Sampling Methodologies To Assess Student Learning Outcomes Paper presented at 2007 Annual Conference & Exposition, Honolulu, Hawaii. 10.18260/1-2--2799
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2007 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015