Quantitative Assessment of Students’ Revision Processes

Lisa R. Volpatti; Alex Jordan Hanson; Jennifer M. Schall; Jesse N. Dunietz; Amanda X. Chen; Rohan Chitnis; Eric J. Alm; Alison F. Takemura; Diana M. Chien

Download Paper | Permalink

Conference: 2020 ASEE Virtual Annual Conference Content Access
Location: Virtual On line
Publication Date: June 22, 2020
Start Date: June 22, 2020
End Date: June 26, 2021
Conference Session: Promoting Communication Skills
Tagged Division: Liberal Education/Engineering & Society
Tagged Topic: Diversity
Page Count: 46
DOI: 10.18260/1-2--35117
Permanent URL: https://peer.asee.org/35117
Download Count: 3822

Paper Authors

biography

Lisa R. Volpatti Massachusetts Institute of Technology orcid.org/0000-0001-6197-0703

visit author page

Lisa R. Volpatti is a Ph.D. candidate in the Anderson and Langer Labs at MIT with research interests in the development of responsive materials for biomedical applications. Prior to joining MIT, Lisa received her Masters of Philosophy in the Department of Chemistry at the University of Cambridge, UK and her Bachelor of Science in Chemical Engineering from the University of Pittsburgh. Lisa co-founded the Graduate Women in Chemical Engineering organization at MIT and is a NSF Graduate Research Fellow, a Whitaker International Fellow, and an MIT Chemical Engineering Communication Lab Fellow.

visit author page

author page

Jesse Dunietz is an educational designer for the MIT Communication Lab, an artificial intelligence researcher, and a freelance science writer. He develops training materials for the engineering graduate students who join the Communication Lab to serve as communication coaches for their peers. He holds a bachelor's in computer science from MIT and a Ph.D. in computer science from Carnegie Mellon University.

visit author page

author page

Amanda X. Chen Massachusetts Institute of Technology

author page

Rohan Chitnis Massachusetts Institute of Technology

author page

Eric J. Alm

biography

Alison F. Takemura U.S. Department of Energy Joint Genome Institute

visit author page

Alison loves wading into a good science story. Her first was her MIT doctoral thesis project, unlocking the gastronomical genome of a Vibrio bacterium. For some of the Vibrio’s meals, she collected seaweed from the rocky, Atlantic coastline at low tide. (Occasionally, its waves swept her off her feet.) During grad school, Alison was also a fellow in MIT’s Biological Engineering Communication Lab. Helping students share their science with their instructors and peers, she began to crave the ability to tell the stories of other scientists, and the marvels they discover, to a broader audience. So after graduating in 2015 with a microbiology doctorate, she trekked to the Pacific coast to study science communication at the University of California, Santa Cruz. There, she learned how to interview people, write feature stories, create podcasts, shoot videos, and finally, drive. Her stories were about pesticide residues in children, HIV in South Africa, rainforests in Australia, calorie-burning brown fat, and what hides behind Jupiter’s clouds. Alison graduated in 2016, and like a homing pigeon, migrated back to MIT. There as the EECS Communication Lab manager, she supported the learning and growth of early scientists—eager to share their own stories. She now works at the US Department of Energy Joint Genome Institute, sharing science stories with an international audience.

visit author page

biography

Diana M. Chien Massachusetts Institute of Technology orcid.org/0000-0002-3297-3312

visit author page

Dr. Diana Chien leads the MIT School of Engineering Communication Lab, and holds a PhD in Microbiology from MIT. Since 2013, she has coached, taught, and designed educational resources at multiple levels of the organization, including previous roles as a peer Communication Fellow, as the Biological Engineering Communication Lab manager, and as a Communication Instructor for undergraduate engineering courses. She is the co-founder of the CommKit, the Communication Lab's free online collection of discipline-specific guides to technical and professional communication. She is dedicated to promoting peer-to-peer professional development experiences for scientists and engineers.

visit author page

Download Paper | Permalink

Abstract

Communication is a crucial skillset for engineers, yet graduates [1]–[3] and their employers [4]–[8] continue to report their lack of preparation for effective communication upon completion of their undergraduate or graduate programs. Thus, technical communication training merits deeper investigation and creative solutions. At the 2017 ASEE Meeting, we introduced the MIT School of Engineering Communication Lab, a discipline-specific technical communication service that is akin to a writing center, but embedded within engineering departments [9]. By using the expertise of graduate student and postdoctoral peer coaches within a given discipline, the Communication Lab provides a scalable, content-aware solution with the benefits of just-in-time, one-on-one [10], and peer [11] training. When we first introduced this model, we offered easy-to-record metrics for the Communication Lab’s effectiveness (such as usage statistics and student and faculty opinion surveys), as are commonly used to assess writing centers [12], [13]. Here we present a formal quantitative study of the effectiveness of Communication Lab coaching. We designed a pre-post test study for two related tasks: personal statements for applications to graduate school and graduate fellowships. We designed an analytic rubric with seven categories (strategic alignment, audience awareness, context, evidence, organization/flow, language mechanics, and visual impact) and tested it to ensure inter-rater reliability. Over one semester, we collected and anonymized 119 personal statement drafts from 47 unique Communication Lab clients across four different engineering departments. Peer coaches rubric-scored the drafts, and we developed a statistical model based on maximum likelihood to identify significant score changes in individual rubric categories across trajectories of sequential drafts. In addition, post-session surveys of clients and their peer coaches provided insight into clients’ qualitative experiences during coaching sessions. Taken together, our quantitative and qualitative findings suggest that our peer coaches are most effective in supporting the skills of organization/flow, strategic alignment, and providing appropriate evidence; this aligns with our program’s emphasis on supporting high-level communication skills. Our results also suggest that a major factor in coaching efficacy is coach-client discussion of major takeaways from a session: rubric category scores were more likely to improve across a drafting trajectory when a category had been identified as a takeaway. Hence, we show quantitative evidence that through collaborative conversations, technical peer coaches can guide clients to identify and effectively revise key areas for improvement. Finally, since we have gathered a sizable dataset and developed analytical tools, we have laid the groundwork for future quantitative writing assessments by both our program and others. We argue that although inter-rater variability poses a challenge, statistical methods and skill-based assessments of authentic communication tasks can provide both insights into student writing/revision ability and direction for improvement of communication resources.

Citation
Format

Volpatti, L. R., & Hanson, A. J., & Schall, J. M., & Dunietz, J. N., & Chen, A. X., & Chitnis, R., & Alm, E. J., & Takemura, A. F., & Chien, D. M. (2020, June), Quantitative Assessment of Students’ Revision Processes Paper presented at 2020 ASEE Virtual Annual Conference Content Access, Virtual On line . 10.18260/1-2--35117

TY - CPAPER
AB - Communication is a crucial skillset for engineers, yet graduates [1]–[3] and their employers [4]–[8] continue to report their lack of preparation for effective communication upon completion of their undergraduate or graduate programs. Thus, technical communication training merits deeper investigation and creative solutions. At the 2017 ASEE Meeting, we introduced the MIT School of Engineering Communication Lab, a discipline-specific technical communication service that is akin to a writing center, but embedded within engineering departments [9]. By using the expertise of graduate student and postdoctoral peer coaches within a given discipline, the Communication Lab provides a scalable, content-aware solution with the benefits of just-in-time, one-on-one [10], and peer [11] training. When we first introduced this model, we offered easy-to-record metrics for the Communication Lab’s effectiveness (such as usage statistics and student and faculty opinion surveys), as are commonly used to assess writing centers [12], [13].
Here we present a formal quantitative study of the effectiveness of Communication Lab coaching. We designed a pre-post test study for two related tasks: personal statements for applications to graduate school and graduate fellowships. We designed an analytic rubric with seven categories (strategic alignment, audience awareness, context, evidence, organization/flow, language mechanics, and visual impact) and tested it to ensure inter-rater reliability. Over one semester, we collected and anonymized 119 personal statement drafts from 47 unique Communication Lab clients across four different engineering departments. Peer coaches rubric-scored the drafts, and we developed a statistical model based on maximum likelihood to identify significant score changes in individual rubric categories across trajectories of sequential drafts. In addition, post-session surveys of clients and their peer coaches provided insight into clients’ qualitative experiences during coaching sessions.
Taken together, our quantitative and qualitative findings suggest that our peer coaches are most effective in supporting the skills of organization/flow, strategic alignment, and providing appropriate evidence; this aligns with our program’s emphasis on supporting high-level communication skills. Our results also suggest that a major factor in coaching efficacy is coach-client discussion of major takeaways from a session: rubric category scores were more likely to improve across a drafting trajectory when a category had been identified as a takeaway. Hence, we show quantitative evidence that through collaborative conversations, technical peer coaches can guide clients to identify and effectively revise key areas for improvement.
Finally, since we have gathered a sizable dataset and developed analytical tools, we have laid the groundwork for future quantitative writing assessments by both our program and others. We argue that although inter-rater variability poses a challenge, statistical methods and skill-based assessments of authentic communication tasks can provide both insights into student writing/revision ability and direction for improvement of communication resources.
AU - Lisa R. Volpatti
AU - Alex Jordan Hanson
AU - Jennifer M. Schall
AU - Jesse N. Dunietz
AU - Amanda X. Chen
AU - Rohan Chitnis
AU - Eric J. Alm
AU - Alison F. Takemura
AU - Diana M. Chien
CY - Virtual On line
DA - 2020/06/22
PB - ASEE Conferences
TI - Quantitative Assessment of Students’ Revision Processes
UR - https://peer.asee.org/35117
DO - 10.18260/1-2--35117
ER -