Accuracy in Student Placement Data

Cynthia B. Paschal

Download Paper | Permalink

Conference: 2014 ASEE Annual Conference & Exposition
Location: Indianapolis, Indiana
Publication Date: June 15, 2014
Start Date: June 15, 2014
End Date: June 18, 2014
ISSN: 2153-5965
Conference Session: Student Learning, Problem Solving, and Critical Thinking 1
Tagged Division: Educational Research and Methods
Page Count: 13
Page Numbers: 24.131.1 - 24.131.13
DOI: 10.18260/1-2--20022
Permanent URL: https://peer.asee.org/20022
Download Count: 629

Paper Authors

biography

Cynthia B. Paschal Vanderbilt University

visit author page

Dr. Paschal is associate dean of the Vanderbilt University School of Engineering and is a faculty member in the department of biomedical engineering. She has research experience in medical imaging and engineering education. Paschal earned bachelor's and master's degrees in nuclear engineering from the Massachusetts Institute of Technology and the doctor of philosophy degree in biomedical engineering from Case Western Reserve University.

visit author page

Download Paper | Permalink

Abstract

Accuracy in Student Placement DataPlacement of engineering students at the end of their undergraduate studies is one measure of thesuccess of the educational program. For the measure to be helpful in providing feedback abouteducational programs, the reported data must present an accurate picture of placement. Twosignificant issues affect the accuracy these data: bias in the setting of low placement surveyresponse rates and restrictions on eligibility faced by international students.In a limited survey of ten highly ranked U.S. engineering programs where a Ph.D. is the highestdegree offered, undergraduate placement survey response rates ranged from 44 to 97%, with anaverage of 74%. A modest response rate may not be a serious problem if the responses comefrom a truly random sample of the population. However, when non-response bias exists, a lowresponse rate decreases the likely accuracy of the composite data. Even in the absence ofsampling bias, the probability that the data may not reflect an accurate picture of placement goesup with decreasing response rate.Most engineering programs aim to obtain survey data from an entire graduation cohort ratherthan sampling a subset of the cohort. However, as revealed above, response rates varysignificantly and are often much lower than desired. Failure to complete a survey may resultfrom students not receiving the survey at an opportune time or in a convenient medium, such as along survey administered online during final examinations. More critically to the issue of non-response bias, failure to complete a survey may result from students without placement notwanting to report what they may perceive as failure. If this is the case, then the non-responsebias can be severe, causing a reported placement rate to greatly overestimate the true placementrate. To prevent these problems, several techniques have been implemented at this institution toensure very high response rates (>97%). These techniques have been found to be effective: (1)providing faculty in each major with a list of degree candidates from whom surveys arerequested, (2) administering the survey in the month prior to commencement, while the studentsare still on campus and in contact with their faculty members, (3) limiting the survey to onesingle page, (4) administering the survey on paper in a required senior course or at a specialsocial event for seniors, (5) enlisting staff support to obtain surveys from stragglers, (6)acknowledging degree programs with 100% response rates, and (7) publishing anonymizedsummary data each year in a format useful to students and faculty members.Another issue affecting the accuracy placement results is restrictions on eligibility to work facedby international students. As international students on F1 and J1 visas do not have anunrestricted right to work in the U.S.A., many employers do not wish to consider hiring suchcandidates. Thus, it is helpful to conduct separate placement analyses for domestic andinternational students.If accurate data result from a survey with a high response rate and separate analysis for studentswith and without an unrestricted right to work in the U.S., then the results are more useful andlegitimate comparisons between educational programs can be made. In addition, accurate datacan reveal opportunities for increased synergies with key industrial partners andgraduate/professional programs and for improving educational programs.

Citation
Format

Paschal, C. B. (2014, June), Accuracy in Student Placement Data Paper presented at 2014 ASEE Annual Conference & Exposition, Indianapolis, Indiana. 10.18260/1-2--20022

TY  - CPAPER
AB  -                            Accuracy in Student Placement DataPlacement of engineering students at the end of their undergraduate studies is one measure of thesuccess of the educational program. For the measure to be helpful in providing feedback abouteducational programs, the reported data must present an accurate picture of placement. Twosignificant issues affect the accuracy these data: bias in the setting of low placement surveyresponse rates and restrictions on eligibility faced by international students.In a limited survey of ten highly ranked U.S. engineering programs where a Ph.D. is the highestdegree offered, undergraduate placement survey response rates ranged from 44 to 97%, with anaverage of 74%. A modest response rate may not be a serious problem if the responses comefrom a truly random sample of the population. However, when non-response bias exists, a lowresponse rate decreases the likely accuracy of the composite data. Even in the absence ofsampling bias, the probability that the data may not reflect an accurate picture of placement goesup with decreasing response rate.Most engineering programs aim to obtain survey data from an entire graduation cohort ratherthan sampling a subset of the cohort. However, as revealed above, response rates varysignificantly and are often much lower than desired. Failure to complete a survey may resultfrom students not receiving the survey at an opportune time or in a convenient medium, such as along survey administered online during final examinations. More critically to the issue of non-response bias, failure to complete a survey may result from students without placement notwanting to report what they may perceive as failure. If this is the case, then the non-responsebias can be severe, causing a reported placement rate to greatly overestimate the true placementrate. To prevent these problems, several techniques have been implemented at this institution toensure very high response rates (&gt;97%). These techniques have been found to be effective: (1)providing faculty in each major with a list of degree candidates from whom surveys arerequested, (2) administering the survey in the month prior to commencement, while the studentsare still on campus and in contact with their faculty members, (3) limiting the survey to onesingle page, (4) administering the survey on paper in a required senior course or at a specialsocial event for seniors, (5) enlisting staff support to obtain surveys from stragglers, (6)acknowledging degree programs with 100% response rates, and (7) publishing anonymizedsummary data each year in a format useful to students and faculty members.Another issue affecting the accuracy placement results is restrictions on eligibility to work facedby international students. As international students on F1 and J1 visas do not have anunrestricted right to work in the U.S.A., many employers do not wish to consider hiring suchcandidates. Thus, it is helpful to conduct separate placement analyses for domestic andinternational students.If accurate data result from a survey with a high response rate and separate analysis for studentswith and without an unrestricted right to work in the U.S., then the results are more useful andlegitimate comparisons between educational programs can be made. In addition, accurate datacan reveal opportunities for increased synergies with key industrial partners andgraduate/professional programs and for improving educational programs.
AU  - Cynthia B. Paschal
CY  - Indianapolis, Indiana
DA  - 2014/06/15
PB  - ASEE Conferences
TI  - Accuracy in Student Placement Data
UR  - https://peer.asee.org/20022
DO  - 10.18260/1-2--20022
ER  -