TY - GEN
T1 - Improving and aligning speech with presentation slides
AU - Swaminathan, Ranjini
AU - Thompson, Michael E.
AU - Fong, Sandiway
AU - Efrat, Alon
AU - Amir, Arnon
AU - Barnard, Kobus
PY - 2010
Y1 - 2010
N2 - We present a novel method to correct automatically generated speech transcripts of talks and lecture videos using text from accompanying presentation slides. The approach finesses the challenges of dealing with technical terms which are often outside the vocabulary of speech recognizers. Further, we align the transcript to the slide word sequence so that we can improve the organization of closed captioning for hearing impaired users, and improve automatic highlighting or magnification for visually impaired users. For each speech segment associated with a slide, we construct a sequential Hidden Markov Model for the observed phonemes that follows slide word order, interspersed with text not on the slide. Incongruence between slide words and mistaken transcript words is accounted for using phoneme confusion probabilities. Hence, transcript words different from aligned high probability slide words can be corrected. Experiments on six talks show improvement in transcript accuracy and alignment with slide words.
AB - We present a novel method to correct automatically generated speech transcripts of talks and lecture videos using text from accompanying presentation slides. The approach finesses the challenges of dealing with technical terms which are often outside the vocabulary of speech recognizers. Further, we align the transcript to the slide word sequence so that we can improve the organization of closed captioning for hearing impaired users, and improve automatic highlighting or magnification for visually impaired users. For each speech segment associated with a slide, we construct a sequential Hidden Markov Model for the observed phonemes that follows slide word order, interspersed with text not on the slide. Incongruence between slide words and mistaken transcript words is accounted for using phoneme confusion probabilities. Hence, transcript words different from aligned high probability slide words can be corrected. Experiments on six talks show improvement in transcript accuracy and alignment with slide words.
UR - http://www.scopus.com/inward/record.url?scp=78149488622&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=78149488622&partnerID=8YFLogxK
U2 - 10.1109/ICPR.2010.802
DO - 10.1109/ICPR.2010.802
M3 - Conference contribution
AN - SCOPUS:78149488622
SN - 9780769541099
T3 - Proceedings - International Conference on Pattern Recognition
SP - 3280
EP - 3283
BT - Proceedings - 2010 20th International Conference on Pattern Recognition, ICPR 2010
T2 - 2010 20th International Conference on Pattern Recognition, ICPR 2010
Y2 - 23 August 2010 through 26 August 2010
ER -