Effectiveness of Inclusive, Reflective Teaching Practices on Problem Solving Proficiency

Casey Jane Ankeny; David P O'Neill; Ken Gentry; Philippa Eshun

Download Paper | Permalink

Conference: 2024 ASEE Annual Conference & Exposition
Location: Portland, Oregon
Publication Date: June 23, 2024
Start Date: June 23, 2024
End Date: July 12, 2024
Conference Session: Biomedical Engineering Division (BED) Technical Session 3
Tagged Division: Biomedical Engineering Division (BED)
Permanent URL: https://strategy.asee.org/47221

Request a correction

Paper Authors

biography

Casey Jane Ankeny Northwestern University

visit author page

Casey J. Ankeny, PhD is an Associate Professor of Instruction and Director of MS Program at Northwestern University.

visit author page

biography

David P O'Neill Northwestern University orcid.org/0000-0002-1609-7666

visit author page

David O'Neill is an Associate Professor of Instruction and the Michael Jaharis Director of Experiential Learning for the Biomedical Engineering Department at Northwestern University. David read Engineering Science at University College, Oxford, receiving his M.Eng. and D.Phil. before undertaking a post-doc in the Department of Physiology, Anatomy and Genetics. During these years, he taught undergraduate tutorials for Keble, New, University, and Harris Manchester Colleges, was College Lecturer for New College and a Senior College Lecturer in Engineering Science for Keble College. He has interests in the formation of engineering identity, and increasing synopticism at a curricular level.

visit author page

biography

Ken Gentry Northwestern University

visit author page

Ken Gentry is a Professor of Instruction and Adviser working mainly with first-year students. He teaches cornerstone design and courses in the biomedical engineering department.

visit author page

author page

Philippa Eshun Northwestern University

Download Paper | Permalink

Abstract

Motivation:

Development and evaluation of inclusive teaching practices is a vibrant research area. For four years, we worked to implement three inclusive practices: standards-based grading with reflection, co-created assessment rubrics, and peer review of written deliverables in a sophomore-level experimental design laboratory and lecture course. This study focuses on peer review effectiveness motivated by the work of Saterbak, et al., which showed a strong positive correlation between instructor review and peer review in a biomedical engineering laboratory and suggested that peer review could provide effective feedback. Further, peer review resulted in the perceived improvement of the students’ ability to critique. We hypothesize that our implementation of peer review leads to increased achievement over time and positive student attitude. More specifically, our research questions are:

1. Do students provide high quality feedback? 2. Is overall class performance improved as assessed by course standards? 3. Is there a favorable student attitude towards the peer review process from the viewpoint of the critic and critiqued?

Methods:

Students completed draft and final two-page abstracts individually and provided reciprocal peer feedback in pairs. A research assistant scored the draft and final versions of the extended abstracts using the course’s student/instructor co-created rubric based on problem solving standards. These scores were detached from grades. Separately, all student work was graded by teaching assistants who underwent grader-calibration.

The research assistant assessed peer review quality on a minimal grading scale (0-2 points) for each of the following: specificity, suggestions, justification, and appropriateness as previously described. Student attitude was assessed using our previously reported survey from the perspective of both the critic and the critiqued.

Then, the following analyses were conducted: - Assessment of peer review quality over the term - Assessment of draft and final scores over the term with one-way ANOVA - Investigation of achievement in weak categories using one-way ANOVA - Investigation of student attitude regarding peer review using 25 Likert-scaled questions where “4” refers to “strongly agree” and “1” refers to “strongly disagree”

Results:

Peer Review Quality:

In terms of peer review quality analysis, students excelled at commenting “appropriate[ly]”; however, there is room for improvement with respect to “specificity”, “justification”, and the ability to provide “suggestions”. The overall peer review quality was 35+/-9 out of 48 points.

Improvement in Achievement over Time:

We saw a significant improvement in achievement between the Abstract #2 draft and the Abstract #3 draft (p=0.046). There was a trend for improvement between the Abstract #1 draft and the Abstract #2 draft (p=0.07, n=16-17). No statistically significant differences existed at the final abstract stage in terms of achievement over time. Additionally, we did not find statistically significant improvements in the two standards with which students typically struggle – “problem identification” and “interpretation”.

Student Attitude regarding Peer Review:

From the perspective of the critic, students rated “enjoying giving feedback” (2.59/4) and “feeling reluctant to give negative non-anonymous feedback” (2.59/4) the lowest. Conversely, students reportedly understood how to assess extended abstracts (3.55/4) and had the confidence to assess others’ extended abstracts (3.55/4).

From the perspective of the critiqued, students provided the highest rating to “believing it is important for students to learn how to implement the feedback that is provided to [them] by peers” (3.50/4). The lowest ratings corresponded to “it is tough to write an extended abstract knowing it would be evaluated by another student without it being anonymous” (1.86/4).

Discussion and Conclusion:

Students may struggle with “specificity”, “justification”, and “providing suggestions” during peer review as they may not have sufficient time and understanding of their peer’s work to provide higher-level comments. Students may benefit from additional instruction of peer review. It is possible we only observe an improvement in achievement in draft scores over time, but not final scores, because students were not willing to dedicate as much time for draft preparation as they did for the final versions (which all rose to a high standard throughout the quarter). Draft scores potentially improved due to increase in skill, efficiency, and familiarity with abstract format. We may not see improvements in “weak” areas because the students were not able to provide high quality review in these areas. The fact that students favorably viewed non-anonymous peer review could be because of the strong classroom community. Further, pair-style peer review may be a best practice for implementation.

In summary, peer review is an inclusive teaching practice that has the potential to improve draft quality, may require additional instruction, and is viewed favorably by students.

Citation
Format

Ankeny, C. J., & O'Neill, D. P., & Gentry, K., & Eshun, P. (2024, June), Effectiveness of Inclusive, Reflective Teaching Practices on Problem Solving Proficiency Paper presented at 2024 ASEE Annual Conference & Exposition, Portland, Oregon. https://strategy.asee.org/47221

TY - CPAPER
AB - Motivation:

Development and evaluation of inclusive teaching practices is a vibrant research area. For four years, we worked to implement three inclusive practices: standards-based grading with reflection, co-created assessment rubrics, and peer review of written deliverables in a sophomore-level experimental design laboratory and lecture course. This study focuses on peer review effectiveness motivated by the work of Saterbak, et al., which showed a strong positive correlation between instructor review and peer review in a biomedical engineering laboratory and suggested that peer review could provide effective feedback. Further, peer review resulted in the perceived improvement of the students’ ability to critique. We hypothesize that our implementation of peer review leads to increased achievement over time and positive student attitude. More specifically, our research questions are:

1. Do students provide high quality feedback?
2. Is overall class performance improved as assessed by course standards?
3. Is there a favorable student attitude towards the peer review process from the viewpoint of the critic and critiqued?

Methods:

Students completed draft and final two-page abstracts individually and provided reciprocal peer feedback in pairs. A research assistant scored the draft and final versions of the extended abstracts using the course’s student/instructor co-created rubric based on problem solving standards. These scores were detached from grades. Separately, all student work was graded by teaching assistants who underwent grader-calibration.

The research assistant assessed peer review quality on a minimal grading scale (0-2 points) for each of the following: specificity, suggestions, justification, and appropriateness as previously described. Student attitude was assessed using our previously reported survey from the perspective of both the critic and the critiqued.

Then, the following analyses were conducted:
- Assessment of peer review quality over the term
- Assessment of draft and final scores over the term with one-way ANOVA
- Investigation of achievement in weak categories using one-way ANOVA
- Investigation of student attitude regarding peer review using 25 Likert-scaled questions where “4” refers to “strongly agree” and “1” refers to “strongly disagree”

Results:

Peer Review Quality:

In terms of peer review quality analysis, students excelled at commenting “appropriate[ly]”; however, there is room for improvement with respect to “specificity”, “justification”, and the ability to provide “suggestions”. The overall peer review quality was 35+/-9 out of 48 points.

Improvement in Achievement over Time:

We saw a significant improvement in achievement between the Abstract #2 draft and the Abstract #3 draft (p=0.046). There was a trend for improvement between the Abstract #1 draft and the Abstract #2 draft (p=0.07, n=16-17). No statistically significant differences existed at the final abstract stage in terms of achievement over time. Additionally, we did not find statistically significant improvements in the two standards with which students typically struggle – “problem identification” and “interpretation”.

Student Attitude regarding Peer Review:

From the perspective of the critic, students rated “enjoying giving feedback” (2.59/4) and “feeling reluctant to give negative non-anonymous feedback” (2.59/4) the lowest. Conversely, students reportedly understood how to assess extended abstracts (3.55/4) and had the confidence to assess others’ extended abstracts (3.55/4).

From the perspective of the critiqued, students provided the highest rating to “believing it is important for students to learn how to implement the feedback that is provided to [them] by peers” (3.50/4). The lowest ratings corresponded to “it is tough to write an extended abstract knowing it would be evaluated by another student without it being anonymous” (1.86/4).

Discussion and Conclusion:

Students may struggle with “specificity”, “justification”, and “providing suggestions” during peer review as they may not have sufficient time and understanding of their peer’s work to provide higher-level comments. Students may benefit from additional instruction of peer review. It is possible we only observe an improvement in achievement in draft scores over time, but not final scores, because students were not willing to dedicate as much time for draft preparation as they did for the final versions (which all rose to a high standard throughout the quarter). Draft scores potentially improved due to increase in skill, efficiency, and familiarity with abstract format. We may not see improvements in “weak” areas because the students were not able to provide high quality review in these areas. The fact that students favorably viewed non-anonymous peer review could be because of the strong classroom community. Further, pair-style peer review may be a best practice for implementation.

In summary, peer review is an inclusive teaching practice that has the potential to improve draft quality, may require additional instruction, and is viewed favorably by students.

AU - Casey Jane Ankeny
AU - David P O'Neill
AU - Ken Gentry
AU - Philippa Eshun
CY - Portland, Oregon
DA - 2024/06/23
PB - ASEE Conferences
TI - Effectiveness of Inclusive, Reflective Teaching Practices on Problem Solving Proficiency
UR - https://strategy.asee.org/47221
ER -