McCormack, J., & Ater Kranov, A., & Beyerlein, S. W., & Pedrow, P. D., & Schmeckpeper, E. R. (2013, June), Methods for Efficient and Reliable Scoring of Discussion Transcripts  Paper presented at 2013 ASEE Annual Conference & Exposition, Atlanta, Georgia. 10.18260/1-2--22287

\bibitem{asee_peer_22287}
  McCormack, J., \& Ater Kranov, A., \& Beyerlein, S. W., \& Pedrow, P. D., \& Schmeckpeper, E. R. (2013, June), \emph{Methods for Efficient and Reliable Scoring of Discussion Transcripts}
  Paper presented at 2013 ASEE Annual Conference \& Exposition, Atlanta, Georgia.
  10.18260/1-2--22287

Jay McCormack, Ashley Ater Kranov, Steven W. Beyerlein, Patrick D. Pedrow P.E., and Edwin R. Schmeckpeper.  "Methods for Efficient and Reliable Scoring of Discussion Transcripts".  2013 ASEE Annual Conference & Exposition, Atlanta, Georgia, 2013, June.  ASEE Conferences, 2013.  https://peer.asee.org/22287 Internet.  11 Feb, 2025

\bibitem{asee_peer_22287}
  Jay McCormack, Ashley Ater Kranov, Steven W. Beyerlein, Patrick D. Pedrow P.E., and Edwin R. Schmeckpeper.
  "Methods for Efficient and Reliable Scoring of Discussion Transcripts".
  \emph{2013 ASEE Annual Conference \& Exposition, Atlanta, Georgia, 2013, June}.
  ASEE Conferences, 2013.
  https://peer.asee.org/22287 Internet.  11 Feb, 2025

@INPROCEEDINGS{asee_peer_22287
author = "Jay McCormack, Ashley Ater Kranov, Steven W. Beyerlein, Patrick D. Pedrow P.E., and Edwin R. Schmeckpeper"
title = "Methods for Efficient and Reliable Scoring of Discussion Transcripts"
booktitle = "2013 ASEE Annual Conference \& Exposition"
year = "2013"
month = "June"
address = "Atlanta, Georgia"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/22287}
number = {10.18260/1-2--22287}
}

TY  - CPAPER
AB  -         Methods  for  Efficient  and  Reliable  Scoring  of  Discussion  Transcripts  The  Engineering  Professional  Skills  Assessment  (EPSA)  is  a  direct  method  for  both  teaching  and  assessing  professional  skills.  In  this  method,  groups  of  4-­‐6  students  take  part  in  a  45  minute  discussion  prompted  by  a  one  page  scenario  that  frames  an  interdisciplinary,  complex,  societal  problem  related  to  engineering.  Examples  of  scenarios  include  a  discussion  of  the  Fukushima  Daiichi  nuclear  reactor  incident  and  the  use  of  offshore  wind  power.  The  student  discussion  is  then  scored  by  an  instructor  using  the  EPSA  rubric.  The  EPSA  rubric  assesses  student  performance  through  indicators  associated  with  an  ability  to  function  on  multidisciplinary  teams,  understanding  of  professional  and  ethical  responsibility,  ability  to  communicate  effectively,  understanding  of  the  impact  of  engineering  solutions,  recognition  of  and  ability  to  engage  in  life-­‐long  learning,  and  knowledge  of  contemporary  issues.  Collaborators  from  ABET,  Norwich  University,  University  of  Idaho,  and  Washington  State  University  are  currently  validating  the  EPSA  rubric  by  scoring  47  student  discussions  recorded  and  transcribed  during  the  2011-­‐12  academic  year.  This  effort  has  produced  a  number  of  best  practices  for  annotating  transcripts,  summarizing  data  and  justifying  ratings  on  rubric  score  sheets,  arriving  at  consensus  scores  between  multiple  raters,  and  assuring  inter-­‐rater  reliability.    In  this  paper,  we  examine  a  section  from  a  scored  transcript  to  illustrate  the  scoring  methodology  which  includes  rater  practices  and  application  of  decision  rules.    Preliminary  results  are  presented  which  include  inter-­‐rater  statistics  and  initial  validity  studies  with  rubrics  that  have  overlapping  performance  indicators.      ABSTRACT  GUIDELINES  Title  1.  Title:  The  title  must  be  centered  with  at  least  a  one  and  one-­‐half  inch  margin  on  the  left  and  right.      2.  Font:  Times  New  Roman  typeface  is  required,  bold  faced,  14  point.  3.  Author  and  Affiliation:  Submissions  are  done  in  a  double  blind.  No  author  or  affiliation  information  is  to  be  included  on  any  abstracts.  4.  Footers:  Do  not  include  any  in  your  abstract.  Body  of  the  Abstract  (beginning  under  title  information)  1.  Format:  The  document  will  be  in  a  one-­‐column  format  with  left  justification.  There  must  be  a  1  inch  margin  on  the  left,  right,  and  bottom.    2.  Font:  Times  New  Roman  typeface  is  required,  12  point,  skipping  one  line  between  paragraphs.  3.  Length:  Abstracts  should  generally  be  between  250  -­‐  500  words.  Biographical  Information  Biographical  information  is  to  be  saved  in  Monolith  on  the  Author/Co-­‐Author  page.    It  will  be  automatically  added  to  your  paper  at  the  end  of  the  process.  Other  Do  not  include  session  numbers  in  any  part  of  the  document,  unless  specifically  requested  to  do  so  by  a  call  for  papers.  Do  not  include  any  author  or  affiliation  information  in  any  part  of  your  abstract  so  that  your  abstract  is  able  to  receive  a  blind  review.  Additional  Guidelines  and  Suggestions  •  As  appropriate,  include  the  pedagogical  theory  or  approach  being  used;  •  Indicate  the  form  that  your  outcome(s)  will  take  as  appropriate;  •  As  applicable,  methods  of  assessment  should  be  made  clear;  •  A  second  page  may  be  used  to  include  a  graph  or  image  to  clarify  the  nature  of  your  work  or  to  include  limited  references  to  Indicate  a  basis  for  the  work  undertaken.  Peer  review  occurs  for  both  abstracts  and  papers.  Abstract  acceptance  does  not  guarantee  acceptance  of  the  paper.  All  division  have  a  Publish-­‐to-­‐Present  requirement  and  final  papers  must  be  written  and  accepted  in  order  for  the  work  to  be  presented  at  the  2013  ASEE  Annual  Conference  in  Atlanta,  GA.  Submission  of  abstracts  and  final  papers  will  be  via  the  Monolith  system  and  in  accordance  with  ASEE  published  deadlines.      PRELIMINARY  OUTLINE  FOR  PAPER  1. EPSA  overview     1.1. Goals  of  the  methodology     1.2. Specs  for  scenario  creation     1.3. Rubric  overview  2. Testing  implementation  3. Case  study     3.1. Case  study  overview  –  Fukushima  scenario       3.2. scoring  methodology  overview  (read  and  notate,  harvest,  Interpretation)     3.3. show  what  each  of  the  steps  look  like  –  notate,  harvest,  interpret     3.4. work  through  each  part  of  rubric  –  highlight  evidence  for  scoring     3.5. generalizable  tips  –  process  beyond  this  scenario         3.5.1. tips  and  decision  rules  from  scoring  student  work  4. calibration  method  -­‐  inter-­‐rater  reliability  scoring  method     4.1. some  preliminary  results  5. implementing  a  protocol  for  efficiency  of  scoring  student  work    Exemplars  for  study    transcripts  5  &  6  (fukashima)  –  pick  out  key  elements  for  scoring  –  highlight  some  confusion  around  scoring  thoughtfully  scored  exemplar  for  prepping  calibrators  provide  tips  on  what  they  change  to  raise  score  
AU  - Jay McCormack
AU  - Ashley Ater Kranov
AU  - Steven W. Beyerlein
AU  - Patrick D. Pedrow P.E.
AU  - Edwin R. Schmeckpeper
CY  - Atlanta, Georgia
DA  - 2013/06/23
PB  - ASEE Conferences
TI  - Methods for Efficient and Reliable Scoring of Discussion Transcripts
UR  - https://peer.asee.org/22287
DO  - 10.18260/1-2--22287
ER  -