Asee peer logo

To Map or to Model: Evaluating Dynamism in Organically Evolving Faculty Development

Download Paper |


2018 ASEE Annual Conference & Exposition


Salt Lake City, Utah

Publication Date

June 23, 2018

Start Date

June 23, 2018

End Date

July 27, 2018

Conference Session

Career Decisions and Faculty Development

Tagged Division

Educational Research and Methods

Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Lori C. Bland George Mason University Orcid 16x16

visit author page

Lori C. Bland, Ph.D. teaches courses in educational assessment, program evaluation, and data-driven decision-making. Bland received her Ph.D. in Educational Psychology from the University of Virginia. Her current work focuses on evaluating programs in higher education, STEM education, and gifted education, assessing learning and professional outcomes in formal and informal learning environments in higher education and the workforce; with a focus on project- and problem-based learning.

visit author page


Jill K. Nelson George Mason University

visit author page

Jill Nelson is an associate professor in the Department of Electrical and Computer Engineering at George Mason University. She earned a BS in Electrical Engineering and a BA in Economics from Rice University in 1998. She attended the University of Illinois at Urbana-Champaign for graduate study, earning an MS and PhD in Electrical Engineering in 2001 and 2005, respectively. Dr. Nelson's research focus is in statistical signal processing, specifically detection and estimation for applications in target tracking and physical layer communications. Her work on target detection and tracking is funded by the Office of Naval Research. Dr. Nelson is a 2010 recipient of the NSF CAREER Award. She is a member of Phi Beta Kappa, Tau Beta Pi, Eta Kappa Nu, and the IEEE Signal Processing, Communications, and Education Societies.

visit author page


Margret Hjalmarson George Mason University

visit author page

Margret Hjalmarson is a Professor in the Graduate School of Education at George Mason University. Her research interests include engineering education, mathematics education, faculty development and mathematics teacher leadership.

visit author page


Anastasia P. Samaras George Mason University

visit author page

ANASTASIA P. SAMARAS is Professor of Education in the College of Education and Human Development at George Mason University, USA. She is an educational researcher and pedagogical scholar with signature work in self-study research methodology including co-editor of Polyvocal Professional Learning through Self-Study Research (2015) and author of Self-Study Teacher Research (2011) and lead editor of Learning Communities In Practice (2008). She is recipient of the Dissertation Research Award, University of Virginia, the Outstanding Scholar Award, University of Maryland, a Fulbright Scholar, and a Visiting Self-study Scholar. She served as chair of S-STEP from 2013-2015 and is a current Co-PI of two National Science Foundation (NSF) funded grants: Designing Teaching: Scaling up the SIMPLE Design Framework for Interactive Teaching Development and a research initiation grant: Student-directed differentiated learning in college-level engineering education. Her research centers on facilitating and studying her role in faculty development self-study collaboratives.

visit author page

Download Paper |


Motivation and Background: The purpose for this study was to compare two methods for evaluating the design, implementation, and outcomes of an NSF-funded faculty development grant via conjecture mapping1,2 and a logic model3. One grant goal was to support faculty in their learning about and using evidence-based interactive teaching strategies. A second goal was to implement the project across multiple STEM programs. A third goal was to support faculty research about their own teaching. The hypothesis for the grant was that a learning community can support initiation of teaching change and faculty research about the changes. The purpose for using both methods was to determine which method best captures the dynamism in an organically evolving faculty development along a continuum of learning structure (semi-formal and informal).

Methods: Two methods were used to examine the design, implementation, and outcomes of the grant: (a) conjecture mapping1,2, a tool in design-based research4,5; and (b) a logic model3, a tool used to develop program theory and evaluate implementation and outcomes6. The design, implementation, and outcomes were analyzed using both tools. Both tools were also adapted to account for the situated nature of learning within community7 and informally8.

Specifically, the initial design decisions were examined9,10,11 to identify the high-level conjectures for the conjecture map and the situation, external factors, and assumptions for the logic model. The materials and tools, structures for tasks and participant engagement, and discursive practices were examined for the conjecture map12,13. The logic model was used to identify inputs, activities, and strategies to support faculty learning12,13. Mediating process1,2 or outputs3 to map the evolution of project implementation were examined14. The evolution of project implementation and outcomes were also examined.

Results: Initial results revealed that the conjecture mapping revealed multiple stages of learning and enactment of interactive teaching and research about teaching that were not revealed in the development of the logic model. However, the logic model was more useful in describing the initial situation and underlying assumptions related to program development. Conjecture mapping was better able to capture processes and incremental outcomes due to the informal and organic nature of faculty learning.

Implications: Logic models have been historically used to help develop and evaluate implementation of program theory. However, they have limitations when a program evolves organically. Conjecture mapping allows for an examination of organic changes. The advantages and disadvantages of conjecture mapping and logic models to examine faculty development are situated along three continua which can help the STEM faculty development program evaluator identify which model is most useful for research and evaluation: a) a formal-informal design continue; (b) a static-dynamic implementation continua; and c) a process-outcome-impact continua.


1 Sandoval, W. A. (2004). Developing learning theory by refining conjectures embodied in educational designs. Educational Psychologist, 39, 213-223, DOI:10.1207/s15326985ep3904_3

2 Sandoval, W. A. (2014) Conjecture mapping: An approach to systematic educational design Research. Journal of the Learning Sciences, 23,1, 18-36. doi:10.1080/10508406.2013.778204

3 Taylor-Powell, E., & Henert, E. Developing a logic model: Teaching and training guide. Madison, WI: University of Wisconsin-Extension. Retrieved from

4 Kelly, A. E. (2014). Design-based research in engineering education: Current state and next steps. In A. Johri & B. M. Olds (Eds.), Cambridge handbook of engineering education research (pp. 497–418). New York: Cambridge University Press.

5 Kelly, A. E., Lesh, R. A., & Baek, J. Y. (Eds.). (2008). Handbook of Design Research Methods in Education: Innovations in Science, Technology, Engineering, and Mathematics Learning and Teaching. NY: Routledge.

6 Funnell, S. C., & Rogers, P. J. (2011). Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass.

7 Johri, A., Olds, B. M., & O’Connor, K. (2014). The social nature of representational engineering knowledge. In A. Johri and B. M. Olds (Eds.), Cambridge Handbook of Engineering Education Research, (pp. 47-66). NY: Cambridge University Press.

8 Bland, L. C. (in press). Assessing Learner-driven Constructs in Informal Learning Environments: Synergies Created by the Nexus of Psychometrics, Learning Analytics, and Educational Data Mining. In: R. Lissitz & J. Hong (Eds.). MARCES Book Series: Data Analytics and Psychometrics: Informing Assessment Practices. Charlotte, NC: Information Age Publishing.

9 Hjalmarson, M. A., & Nelson, J. K. (2014). Creating small interactive teaching groups. In Proceedings of the 121st ASEE Annual Conference. Indianapolis, IN.

10 Nelson, J. K., & Hjalmarson, M. A. (2015). Faculty Development Groups for Interactive Teaching. In Proceedings of the 122nd ASEE Annual Conference. Seattle, WA.

11 Samaras, A. P. (2011). Self-study teacher research: Improving your practice through collaborative inquiry. Thousand Oaks, CA: Sage. [Translated into Korean]

12 Nelson, J.K., Hjalmarson, M.A., Bland, L., & Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.

13 Hjalmarson, M., Nelson, J., Gerasimova, D., Bland, L., & Samaras, A. (2016, April). Faculty Professional Development through Teaching Development Groups: Principles in Action. Paper presented at the Annual Meeting of American Educational Research Association, Washington, DC.

14 Nelson, J.K., Hjalmarson, M.A., Bland, L., & Samaras, A. (2016). SIMPLE Design Framework for Teaching Development Across STEM. In Proceedings of the 2016 ASEE Annual Conference. New Orleans, LA.

Bland, L. C., & Nelson, J. K., & Hjalmarson, M., & Samaras, A. P. (2018, June), To Map or to Model: Evaluating Dynamism in Organically Evolving Faculty Development Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--31145

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2018 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015