June 15, 2014
June 15, 2014
June 18, 2014
Educational Research and Methods
24.131.1 - 24.131.13
Accuracy in Student Placement DataPlacement of engineering students at the end of their undergraduate studies is one measure of thesuccess of the educational program. For the measure to be helpful in providing feedback abouteducational programs, the reported data must present an accurate picture of placement. Twosignificant issues affect the accuracy these data: bias in the setting of low placement surveyresponse rates and restrictions on eligibility faced by international students.In a limited survey of ten highly ranked U.S. engineering programs where a Ph.D. is the highestdegree offered, undergraduate placement survey response rates ranged from 44 to 97%, with anaverage of 74%. A modest response rate may not be a serious problem if the responses comefrom a truly random sample of the population. However, when non-response bias exists, a lowresponse rate decreases the likely accuracy of the composite data. Even in the absence ofsampling bias, the probability that the data may not reflect an accurate picture of placement goesup with decreasing response rate.Most engineering programs aim to obtain survey data from an entire graduation cohort ratherthan sampling a subset of the cohort. However, as revealed above, response rates varysignificantly and are often much lower than desired. Failure to complete a survey may resultfrom students not receiving the survey at an opportune time or in a convenient medium, such as along survey administered online during final examinations. More critically to the issue of non-response bias, failure to complete a survey may result from students without placement notwanting to report what they may perceive as failure. If this is the case, then the non-responsebias can be severe, causing a reported placement rate to greatly overestimate the true placementrate. To prevent these problems, several techniques have been implemented at this institution toensure very high response rates (>97%). These techniques have been found to be effective: (1)providing faculty in each major with a list of degree candidates from whom surveys arerequested, (2) administering the survey in the month prior to commencement, while the studentsare still on campus and in contact with their faculty members, (3) limiting the survey to onesingle page, (4) administering the survey on paper in a required senior course or at a specialsocial event for seniors, (5) enlisting staff support to obtain surveys from stragglers, (6)acknowledging degree programs with 100% response rates, and (7) publishing anonymizedsummary data each year in a format useful to students and faculty members.Another issue affecting the accuracy placement results is restrictions on eligibility to work facedby international students. As international students on F1 and J1 visas do not have anunrestricted right to work in the U.S.A., many employers do not wish to consider hiring suchcandidates. Thus, it is helpful to conduct separate placement analyses for domestic andinternational students.If accurate data result from a survey with a high response rate and separate analysis for studentswith and without an unrestricted right to work in the U.S., then the results are more useful andlegitimate comparisons between educational programs can be made. In addition, accurate datacan reveal opportunities for increased synergies with key industrial partners andgraduate/professional programs and for improving educational programs.
ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2014 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015