Salt Lake City, Utah
June 23, 2018
June 23, 2018
July 27, 2018
Educational Research and Methods
Motivation and Background: The purpose for this study was to compare two methods for evaluating the design, implementation, and outcomes of an NSF-funded faculty development grant via conjecture mapping1,2 and a logic model3. One grant goal was to support faculty in their learning about and using evidence-based interactive teaching strategies. A second goal was to implement the project across multiple STEM programs. A third goal was to support faculty research about their own teaching. The hypothesis for the grant was that a learning community can support initiation of teaching change and faculty research about the changes. The purpose for using both methods was to determine which method best captures the dynamism in an organically evolving faculty development along a continuum of learning structure (semi-formal and informal).
Methods: Two methods were used to examine the design, implementation, and outcomes of the grant: (a) conjecture mapping1,2, a tool in design-based research4,5; and (b) a logic model3, a tool used to develop program theory and evaluate implementation and outcomes6. The design, implementation, and outcomes were analyzed using both tools. Both tools were also adapted to account for the situated nature of learning within community7 and informally8.
Specifically, the initial design decisions were examined9,10,11 to identify the high-level conjectures for the conjecture map and the situation, external factors, and assumptions for the logic model. The materials and tools, structures for tasks and participant engagement, and discursive practices were examined for the conjecture map12,13. The logic model was used to identify inputs, activities, and strategies to support faculty learning12,13. Mediating process1,2 or outputs3 to map the evolution of project implementation were examined14. The evolution of project implementation and outcomes were also examined.
Results: Initial results revealed that the conjecture mapping revealed multiple stages of learning and enactment of interactive teaching and research about teaching that were not revealed in the development of the logic model. However, the logic model was more useful in describing the initial situation and underlying assumptions related to program development. Conjecture mapping was better able to capture processes and incremental outcomes due to the informal and organic nature of faculty learning.
Implications: Logic models have been historically used to help develop and evaluate implementation of program theory. However, they have limitations when a program evolves organically. Conjecture mapping allows for an examination of organic changes. The advantages and disadvantages of conjecture mapping and logic models to examine faculty development are situated along three continua which can help the STEM faculty development program evaluator identify which model is most useful for research and evaluation: a) a formal-informal design continue; (b) a static-dynamic implementation continua; and c) a process-outcome-impact continua.
1 Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39, 213-223, DOI:10.1207/s15326985ep3904_3
2 Sandoval, W. A. (2014) Conjecture mapping: An approach to systematic educational design Research. Journal of the Learning Sciences, 23,1, 18-36. doi:10.1080/10508406.2013.778204
3 Taylor-Powell, E., & Henert, E. Developing a logic model: Teaching and training guide. Madison, WI: University of Wisconsin-Extension. Retrieved from https://fyi.uwex.edu/programdevelopment/files/2016/03/lmguidecomplete.pdf
4 Kelly, A. E. (2014). Design-based research in engineering education: Current state and next steps. In A. Johri & B. M. Olds (Eds.), Cambridge handbook of engineering education research (pp. 497–418). New York: Cambridge University Press.
5 Kelly, A. E., Lesh, R. A., & Baek, J. Y. (Eds.). (2008). Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching. NY: Routledge.
6 Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass.
7 Johri, A., Olds, B. M., & O’Connor, K. (2014). The social nature of representational engineering knowledge. In A. Johri and B. M. Olds (Eds.), Cambridge Handbook of Engineering Education Research, (pp. 47-66). NY: Cambridge University Press.
8 Bland, L. C. (in press). Assessing Learner-driven Constructs in Informal Learning Environments: Synergies Created by the Nexus of Psychometrics, Learning Analytics, and Educational Data Mining. In: R. Lissitz & J. Hong (Eds.). MARCES Book Series: Data Analytics and Psychometrics: Informing Assessment Practices. Charlotte, NC: Information Age Publishing.
9 Hjalmarson, M. A., & Nelson, J. K. (2014). Creating small interactive teaching groups. In Proceedings of the 121st ASEE Annual Conference. Indianapolis, IN.
10 Nelson, J. K., & Hjalmarson, M. A. (2015). Faculty Development Groups for Interactive Teaching. In Proceedings of the 122nd ASEE Annual Conference. Seattle, WA.
11 Samaras, A. P. (2011). Self-study teacher research: Improving your practice through collaborative inquiry. Thousand Oaks, CA: Sage. [Translated into Korean] http://www.sagepub.com/booksProdDesc.nav?prodId=Book233400&#tabview=title
12 Nelson, J.K., Hjalmarson, M.A., Bland, L., & Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.
13 Hjalmarson, M., Nelson, J., Gerasimova, D., Bland, L., & Samaras, A. (2016, April). Faculty Professional Development through Teaching Development Groups: Principles in Action. Paper presented at the Annual Meeting of American Educational Research Association, Washington, DC.
14 Nelson, J.K., Hjalmarson, M.A., Bland, L., & Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.
Bland, L. C., & Nelson, J. K., & Hjalmarson, M., & Samaras, A. P. (2018, June), To Map or to Model: Evaluating Dynamism in Organically Evolving Faculty Development Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. https://peer.asee.org/31145
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2018 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015