June 15, 2019
June 15, 2019
June 19, 2019
Educational Research and Methods
The purpose for this theoretical paper is to introduce design-based evaluation (DBE) as a novel evaluation approach that can be useful for the engineering education research community. To begin, we define DBE and describe the salient characteristics and elements of the DBE approach. Then, we explain the reasons why DBE is a unique evaluation approach. We link DBE as an evaluation approach to examine design-based research projects and offer a discussion of why current evaluation approaches are not sufficient for design-based research. We also compare and contrast current evaluation approaches with DBE. Finally, we provide a brief description of an effective application of DBE.
The motivation and background for this theoretical paper was grounded within identifying an adequate evaluation approach for a National Science Foundation funded faculty development project (Authors, XXXX). The purpose for this design-based research project was to support and examine undergraduate STEM faculty change processes toward adopting, implementing, examining, and writing about their interactive teaching strategies.
Design-based research is a research methodology that arose from the learning sciences to examine innovative design in educational programs. Kelly et al. (2008) describe the goal of design-based research as the intersection of design processes and research methods to study learning and teaching. Design-based research exists at the juxtaposition of multiple competing processes, such as the dynamic (design) versus the static (research), the creative (design) versus the canon (research), and real-world problem-solving (design) versus theory-building (research). Design-based research is an apt “methodological stance” for research in engineering because of the role of design in solving real world engineering problems (Author, 2008, p. 96).
The evaluation of dynamically designed research poses challenges for evaluators (Author, 2017). Extant evaluation approaches, such as evaluation approaches that examine process (e.g., Stufflebeam, 2005), could have been applied to this project. However, extant evaluation approaches did not adequately capture the elements of this design-based research project. For example, an underlying assumption of this project was that faculty would design their own learning trajectory, and hence faculty processes and outcomes would necessarily vary. The evaluated study was intentionally designed to allow for, and indeed encourage, variation in both the processes evaluated and faculty outcomes. More specifically, there was variation within the program design that encouraged variation within site implementation. The project intention was to study the nature of the variation within the faculty change processes and the unique types of outcomes attained by faculty.
This type of study is not easily addressed by current evaluation approaches. Extant evaluation approaches examine fidelity of implementation, but do not consider that variations in implementation are necessarily acceptable. account for variation in the processes or outcomes. Rather, current evaluation approaches examine implementation and variation within a/the given outcome(s) of a treatment and specifically typically the degree attained of a given outcome. DBE allowed us to address variation across and within processes and outcomes to determine the design that worked best from the different implementations. DBE can be used to evaluate design and changes within engineering education.
Bland, L. C., & Hjalmarson, M., & Samaras, A. P., & Nelson, J. K. (2019, June), Design-based Evaluation: A Novel Evaluation Approach to Examine Designed Programs in Engineering Education Paper presented at 2019 ASEE Annual Conference & Exposition , Tampa, Florida. 10.18260/1-2--32609
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2019 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015