June 24, 2007
June 24, 2007
June 27, 2007
Educational Research and Methods
12.491.1 - 12.491.10
Developing Cognitive, Affective, Behavioral Work Sampling Methodologies to Assess Student Learning Outcomes Abstract
In this study, we develop and validate a work sampling methodology to assess processes that engineers usually engage in (i.e., working in teams, conducting design work, addressing ethical issues). To obtain in-depth measures for these process oriented student learning outcomes, 100 percent behavioral observation is typically used, but which is time consuming and expensive. Work sampling is a common industry practice used to observe physical activities, as it minimizes time to collect data, yet provides statistically similar results relative to 100% behavioral observation. In our research we have bridged the gap between common practices in industry and assessment in engineering education by extending sampling theories to the observation of intervals that can capture the cognitive, behavioral and affective domains for three student learning processes – teamwork, design, and ethical reasoning.
We designed an experiment to statistically compared 100% behavioral observation with work sampling. Four environments with two examples each were videotaped. Each tape was evaluated by two observer teams: one to conduct 100% behaviorally observation and the other to work sample. ANOVA tests were used to determine inter-rater reliability both within and between teams. Results suggest that work sampling can replace 100% behavioral observation for teamwork. Similar positive results have been obtained for design. For ethical reasoning, although a high reliability could be obtained between observers for 100% behavioral observation, work sampling was not a suitable replacement method. This paper describes the overall study, its overarching results with respect to the three outcomes investigated, and comments on various factors related to each outcome that may permit work sampling to be an effective alternative for some outcomes but not for others.
The engineering criteria has changed the motivation of engineering education accreditation from “what are you [the program] doing?” to “what are your students doing?” As a result, the need for solid, in-depth measurements has become a high priority. At recent engineering education conferences (e.g. Best Assessment Processes in Engineering Education Symposiums, ASEE, FIE) the number of evolving approaches for evaluating engineering programs, as well as methodologies for measuring various student outcomes is growing more rich. Yet, several troublesome issues still remain. First, most of these “assessment” methods had not been fully evaluated. Second, many focus on final products via performance appraisals particular to the outcome(s) using rubrics as the assessment tool. Third, many engineering administrators still voiced concerns about the costs associated with organizing, implementing and maintaining an effective assessment program, given limited resources of time, people (i.e. raters), and money.
Assessing non-sequential outcomes in engineering such as working in teams, development of designs or overcoming ethical dilemmas often require a methodological tool to examine behavior at various levels of the cognitive and affective domains (e.g. analysis, synthesis, evaluation, and valuation). While such a tool has been needed for professional work, the recent movement
Yildirim, T. P., & Townsend, J., & Besterfield-Sacre, M., & Shuman, L., & Wolfe, H. (2007, June), Developing Cognitive Affective Behavioral Work Sampling Methodologies To Assess Student Learning Outcomes Paper presented at 2007 Annual Conference & Exposition, Honolulu, Hawaii. 10.18260/1-2--2799
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2007 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015