Asee peer logo

RTTD-ID: Tracked Captions with Multiple Speakers for Deaf Students

Download Paper |


2018 ASEE Annual Conference & Exposition


Salt Lake City, Utah

Publication Date

June 23, 2018

Start Date

June 23, 2018

End Date

July 27, 2018

Conference Session

Computing Technology Applications-I

Tagged Division

Computing and Information Technology

Tagged Topic


Page Count




Permanent URL

Download Count


Request a correction

Paper Authors


Raja S. Kushalnagar Gallaudet University Orcid 16x16

visit author page

Raja Kushalnagar is the Director of the Information Technology program in the Department of Science, Technology and Mathematics at Gallaudet University in Washington, DC.
His research interests encompass the fields of accessible computing and accessibility/intellectual property law, with the goal of improving information access for deaf and hard of hearing (deaf) individuals. In the accessible computing field, he investigates information access disparities between hearing and deaf. For example, he investigates how deaf individuals acquire information through speech-to-text (captions/subtitles) or sign language interpreters, compared with their hearing peers who listen directly. He also develops accessible computing solutions to address these disparities in multimodal information access. For example, he has implemented and verified the benefit of enhanced captions that provide visual cues to non-speech information.
In the accessibility/intellectual property law field, he advocates for updates in accessible and intellectual property law, to incorporate accessible computing advances such as automatic captioning/subtitling.
He worked in industry for over five years before returning to academia and disability law policy. Towards that end, he completed a J.D. and LL.M. in disability law, and an M.S. and Ph.D. in Computer Science. He served on the Federal Communications Commission’s Consumer Advocacy Commission. He has published several peer-reviewed publications and received grants in the fields of accessible computing, accessible law and intellectual property law. He can be reached at

visit author page

author page

Gary W. Behm Rochester Institute of Technology

author page

Kevin T. Wolfe

author page

Peter Yeung

author page

Becca Dingman


Shareef Sayel Ali Center on Access Technology

visit author page

Shareef wrote and designed the RTTD software. Earned his BS in Computer Science

visit author page

author page

Abraham Glasser Rochester Institute of Technology

author page

Claire Elizabeth Ryan

Download Paper |


Students who are deaf and hard of hearing cannot hear in class and do not have full access to spoken information. They can use accommodations such as captions that display speech as text. However, compared with their hearing peers, the caption accommodations do not provide equal access, because they are focused on reading captions on their tablet and cannot see who is talking. This viewing isolation contributes to student frustration and risk of doing poorly or withdrawing from introductory engineering courses with lab components. It also contributes to their lack of inclusion and sense of belonging. We report on the evaluation of an extension to our Real-Time Text Display, which displays the location of a speaker in a group (RTTD-ID). RTTD-ID aims to reduce frustration in identifying and following an active speaker when there are multiple speakers, e.g., in a lab. It has three different display schemes to identify the location of the active speaker, which helps deaf students in viewing both the speaker’s words and the speaker’s expression and actions. We evaluated three RTTD speaker identification methods: 1) traditional: captions stay in one place and viewers search for the speaker, 2) pointer: captions stay in one place, and a pointer to the speaker is displayed, and 3) pop-up: captions “pop-up” next to the speaker. We gathered both quantitative and qualitative information through evaluations with deaf and hard of hearing users. The users preferred the pointer identification method over the traditional and pop-up methods.

Kushalnagar, R. S., & Behm, G. W., & Wolfe, K. T., & Yeung, P., & Dingman, B., & Ali, S. S., & Glasser, A., & Ryan, C. E. (2018, June), RTTD-ID: Tracked Captions with Multiple Speakers for Deaf Students Paper presented at 2018 ASEE Annual Conference & Exposition , Salt Lake City, Utah. 10.18260/1-2--30945

ASEE holds the copyright on this document. It may be read by the public free of charge. Authors may archive their work on personal websites or in institutional repositories with the following citation: © 2018 American Society for Engineering Education. Other scholars may excerpt or quote from these materials with the same citation. When excerpting or quoting from Conference Proceedings, authors should, in addition to noting the ASEE copyright, list all the original authors and their institutions and name the host city of the conference. - Last updated April 1, 2015