Wang, Y., & Lualdi, C. P., & Angrave, L., & Purushotam, G. N. (2023, June), Using Deep Learning and Augmented Reality to Improve Accessibility: Inclusive Conversations Using Diarization, Captions, and Visualization  Paper presented at 2023 ASEE Annual Conference & Exposition, Baltimore , Maryland. 10.18260/1-2--44572

\bibitem{asee_peer_44572}
  Wang, Y., \& Lualdi, C. P., \& Angrave, L., \& Purushotam, G. N. (2023, June), \emph{Using Deep Learning and Augmented Reality to Improve Accessibility: Inclusive Conversations Using Diarization, Captions, and Visualization}
  Paper presented at 2023 ASEE Annual Conference \& Exposition, Baltimore , Maryland.
  10.18260/1-2--44572

Yun Wang, Colin P. Lualdi, Lawrence Angrave, and Guru Nanma Purushotam.  "Using Deep Learning and Augmented Reality to Improve Accessibility: Inclusive Conversations Using Diarization, Captions, and Visualization".  2023 ASEE Annual Conference & Exposition, Baltimore , Maryland, 2023, June.  ASEE Conferences, 2023.  https://peer.asee.org/44572 Internet.  30 Apr, 2024

\bibitem{asee_peer_44572}
  Yun Wang, Colin P. Lualdi, Lawrence Angrave, and Guru Nanma Purushotam.
  "Using Deep Learning and Augmented Reality to Improve Accessibility: Inclusive Conversations Using Diarization, Captions, and Visualization".
  \emph{2023 ASEE Annual Conference \& Exposition, Baltimore , Maryland, 2023, June}.
  ASEE Conferences, 2023.
  https://peer.asee.org/44572 Internet.  30 Apr, 2024

@INPROCEEDINGS{asee_peer_44572
author = "Yun Wang, Colin P. Lualdi, Lawrence Angrave, and Guru Nanma Purushotam"
title = "Using Deep Learning and Augmented Reality to Improve Accessibility: Inclusive Conversations Using Diarization, Captions, and Visualization"
booktitle = "2023 ASEE Annual Conference \& Exposition"
year = "2023"
month = "June"
address = "Baltimore , Maryland"
publisher = "ASEE Conferences"
note = {https://peer.asee.org/44572}
number = {10.18260/1-2--44572}
}

TY  - CPAPER
AB  - The problem of diarization - identifying different speakers in a conversation stream - has not been sufficiently addressed for deaf and hard-of-hearing students in learning communities such as student design teams in engineering and related STEM disciplines. Though the accuracy of the latest automated real-time speech-to-text systems is now approaching usable low word error rates, the generated text output is an incomplete representation of a multi-party conversation; In short, it solves the “what” but not the “who.” This creates barriers to our ideal of an inclusive and equitable learning community. Thus students who are deaf or hard of hearing are further marginalized and excluded from multi-party peer discussions with non-deaf participants because it is hard to visually follow who is speaking. To address these communication barriers, we utilized the  Human Centered Engineering Design framework to identify a set of features that overcomes the above barriers. This paper explores computerized diarization techniques that utilize a wide set of algorithms and audio metrics to assist in speaker identification. These techniques include mel-frequency cepstrum coefficients (MFCC), volume, fundamental frequency identification, and deep learning of voice prints. For the goals described in this paper, a subset of existing algorithms that respected privacy and legal constraints was selected and evaluated for the purposes of identifying speakers using a live audio stream. Several visualization methods were also designed and evaluated. These included visualization of embedding mel-frequency cepstrum, speaker identifier, pitch, volume, and other voice characteristics into a live caption stream. Both diarization and visualization were integrated into a live captioning tool, ScribeAR, previously introduced in ASEE regional proceedings, and rendered using a lightweight Augmented Reality display. In order to facilitate captioning services in areas with limited network connectivity, whisper.cpp, a derivative of OpenAI’s Whisper project, was also incorporated into the application. Links to the open source project are included so that other educators may adopt this inclusive practice. Some accessibility-related opportunities that could be used as motivating design projects for engineering students are described.
AU  - Yun Wang
AU  - Colin P. Lualdi
AU  - Lawrence Angrave
AU  - Guru Nanma Purushotam
CY  - Baltimore , Maryland
DA  - 2023/06/25
PB  - ASEE Conferences
TI  - Using Deep Learning and Augmented Reality to Improve Accessibility: Inclusive Conversations Using Diarization, Captions, and Visualization
UR  - https://peer.asee.org/44572
DO  - 10.18260/1-2--44572
ER  -