Asee peer logo

Methods for Efficient and Reliable Scoring of Discussion Transcripts

Download Paper |

Conference

2013 ASEE Annual Conference & Exposition

Location

Atlanta, Georgia

Publication Date

June 23, 2013

Start Date

June 23, 2013

End Date

June 26, 2013

ISSN

2153-5965

Conference Session

Educational Research and Methods (ERM) Poster Session

Tagged Division

Educational Research and Methods

Page Count

19

Page Numbers

23.902.1 - 23.902.19

Permanent URL

https://peer.asee.org/22287

Download Count

22

Request a correction

Paper Authors

biography

Jay McCormack University of Idaho

visit author page

Jay McCormack is an assistant professor in the Mechanical Engineering Department at the University of Idaho where he is an instructor for the college's interdisciplinary capstone design course. Dr. McCormack received his Ph.D. in Mechanical Engineering from Carnegie Mellon University in 2003.

visit author page

biography

Ashley Ater Kranov ABET

visit author page

Ashley Ater Kranov, Ph.D., is ABET's managing director of Professional Services. Her department is responsible for partnering with faculty and industry to conduct robust technical education research and providing educational opportunities on sustainable assessment processes for program continual improvement worldwide.

visit author page

biography

Steven W. Beyerlein University of Idaho, Moscow

visit author page

Dr. Beyerlein has a Ph.D. from Washington State University and has taught at the University of Idaho since 1987. For the last fifteen years he has been the college coordinator for an interdisciplinary capstone design course that features industry sponsored projects. In 2012, the faculty team responsible for running this course was recognized by the National Academy of Engineering for creating a capstone course that is an example of real world engineering education.

visit author page

biography

Patrick D. Pedrow P.E. Washington State University

visit author page

Dr. Patrick D. Pedrow received the B.S. in Electrical Engineering from the University of Idaho, Moscow, in 1975. He earned his Master's of Engineering degree in Electric Power Engineering from Rensselaer Polytechnic Institute at Troy, NY, in 1976. He received his M.S. degree in Physics from Marquette University in Milwaukee, WI, in 1981, and his Ph.D. in Electrical Engineering from Cornell University in Ithaca, NY in 1985. From 1976 to 1981, he was with McGraw-Edison Company, where he conducted research and development on electric power circuit breakers. He is currently an associate professor with Washington State University in the School of Electrical Engineering and Computer Science. His research interests are in plasma-assisted materials processing, including the deposition and evaluation of thin plasma-polymerized films deposited at atmospheric pressure using weakly ionized plasma. Dr. Pedrow is a member of the American Physical Society, IEEE, ASEE, Tau Beta Pi and he is a Registered Professional Engineer in the State of Wisconsin.

visit author page

biography

Edwin R. Schmeckpeper Norwich University

visit author page

Edwin Schmeckpeper, P.E. Ph.D., is the chair of the Department of Civil and Environmental Engineering Department at Norwich University. Norwich University was the first private school in the United States to offer engineering courses. In addition, Senator Justin Morrill used Norwich University as the model for the Land-Grant colleges created by the 1862 Morrill Land-Grant Act. Prior to joining the faculty at Norwich University, Dr. Schmeckpeper taught at the University of Idaho, the Land-Grant College for the State of Idaho, and worked as an engineer in design offices and at construction sites.

visit author page

Download Paper |

Abstract

Methods  for  Efficient  and  Reliable  Scoring  of  Discussion  Transcripts  The  Engineering  Professional  Skills  Assessment  (EPSA)  is  a  direct  method  for  both  teaching  and  assessing  professional  skills.  In  this  method,  groups  of  4-­‐6  students  take  part  in  a  45  minute  discussion  prompted  by  a  one  page  scenario  that  frames  an  interdisciplinary,  complex,  societal  problem  related  to  engineering.  Examples  of  scenarios  include  a  discussion  of  the  Fukushima  Daiichi  nuclear  reactor  incident  and  the  use  of  offshore  wind  power.  The  student  discussion  is  then  scored  by  an  instructor  using  the  EPSA  rubric.  The  EPSA  rubric  assesses  student  performance  through  indicators  associated  with  an  ability  to  function  on  multidisciplinary  teams,  understanding  of  professional  and  ethical  responsibility,  ability  to  communicate  effectively,  understanding  of  the  impact  of  engineering  solutions,  recognition  of  and  ability  to  engage  in  life-­‐long  learning,  and  knowledge  of  contemporary  issues.  Collaborators  from  ABET,  Norwich  University,  University  of  Idaho,  and  Washington  State  University  are  currently  validating  the  EPSA  rubric  by  scoring  47  student  discussions  recorded  and  transcribed  during  the  2011-­‐12  academic  year.  This  effort  has  produced  a  number  of  best  practices  for  annotating  transcripts,  summarizing  data  and  justifying  ratings  on  rubric  score  sheets,  arriving  at  consensus  scores  between  multiple  raters,  and  assuring  inter-­‐rater  reliability.    In  this  paper,  we  examine  a  section  from  a  scored  transcript  to  illustrate  the  scoring  methodology  which  includes  rater  practices  and  application  of  decision  rules.    Preliminary  results  are  presented  which  include  inter-­‐rater  statistics  and  initial  validity  studies  with  rubrics  that  have  overlapping  performance  indicators.      ABSTRACT  GUIDELINES  Title  1.  Title:  The  title  must  be  centered  with  at  least  a  one  and  one-­‐half  inch  margin  on  the  left  and  right.      2.  Font:  Times  New  Roman  typeface  is  required,  bold  faced,  14  point.  3.  Author  and  Affiliation:  Submissions  are  done  in  a  double  blind.  No  author  or  affiliation  information  is  to  be  included  on  any  abstracts.  4.  Footers:  Do  not  include  any  in  your  abstract.  Body  of  the  Abstract  (beginning  under  title  information)  1.  Format:  The  document  will  be  in  a  one-­‐column  format  with  left  justification.  There  must  be  a  1  inch  margin  on  the  left,  right,  and  bottom.    2.  Font:  Times  New  Roman  typeface  is  required,  12  point,  skipping  one  line  between  paragraphs.  3.  Length:  Abstracts  should  generally  be  between  250  -­‐  500  words.  Biographical  Information  Biographical  information  is  to  be  saved  in  Monolith  on  the  Author/Co-­‐Author  page.    It  will  be  automatically  added  to  your  paper  at  the  end  of  the  process.  Other  Do  not  include  session  numbers  in  any  part  of  the  document,  unless  specifically  requested  to  do  so  by  a  call  for  papers.  Do  not  include  any  author  or  affiliation  information  in  any  part  of  your  abstract  so  that  your  abstract  is  able  to  receive  a  blind  review.  Additional  Guidelines  and  Suggestions  •  As  appropriate,  include  the  pedagogical  theory  or  approach  being  used;  •  Indicate  the  form  that  your  outcome(s)  will  take  as  appropriate;  •  As  applicable,  methods  of  assessment  should  be  made  clear;  •  A  second  page  may  be  used  to  include  a  graph  or  image  to  clarify  the  nature  of  your  work  or  to  include  limited  references  to  Indicate  a  basis  for  the  work  undertaken.  Peer  review  occurs  for  both  abstracts  and  papers.  Abstract  acceptance  does  not  guarantee  acceptance  of  the  paper.  All  division  have  a  Publish-­‐to-­‐Present  requirement  and  final  papers  must  be  written  and  accepted  in  order  for  the  work  to  be  presented  at  the  2013  ASEE  Annual  Conference  in  Atlanta,  GA.  Submission  of  abstracts  and  final  papers  will  be  via  the  Monolith  system  and  in  accordance  with  ASEE  published  deadlines.      PRELIMINARY  OUTLINE  FOR  PAPER  1. EPSA  overview   1.1. Goals  of  the  methodology   1.2. Specs  for  scenario  creation   1.3. Rubric  overview  2. Testing  implementation  3. Case  study   3.1. Case  study  overview  –  Fukushima  scenario     3.2. scoring  methodology  overview  (read  and  notate,  harvest,  Interpretation)   3.3. show  what  each  of  the  steps  look  like  –  notate,  harvest,  interpret   3.4. work  through  each  part  of  rubric  –  highlight  evidence  for  scoring   3.5. generalizable  tips  –  process  beyond  this  scenario   3.5.1. tips  and  decision  rules  from  scoring  student  work  4. calibration  method  -­‐  inter-­‐rater  reliability  scoring  method   4.1. some  preliminary  results  5. implementing  a  protocol  for  efficiency  of  scoring  student  work    Exemplars  for  study    transcripts  5  &  6  (fukashima)  –  pick  out  key  elements  for  scoring  –  highlight  some  confusion  around  scoring  thoughtfully  scored  exemplar  for  prepping  calibrators  provide  tips  on  what  they  change  to  raise  score  

McCormack, J., & Ater Kranov, A., & Beyerlein, S. W., & Pedrow, P. D., & Schmeckpeper, E. R. (2013, June), Methods for Efficient and Reliable Scoring of Discussion Transcripts Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. https://peer.asee.org/22287

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2013 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015