TY - JOUR
T1 - Deep multiple instance learning for foreground speech localization in ambient audio from wearable devices
AU - Hebbar, Rajat
AU - Papadopoulos, Pavlos
AU - Reyes, Ramon
AU - Danvers, Alexander F.
AU - Polsinelli, Angelina J.
AU - Moseley, Suzanne A.
AU - Sbarra, David A.
AU - Mehl, Matthias R.
AU - Narayanan, Shrikanth
N1 - Publisher Copyright:
© 2021, The Author(s).
PY - 2021/12
Y1 - 2021/12
N2 - Over the recent years, machine learning techniques have been employed to produce state-of-the-art results in several audio related tasks. The success of these approaches has been largely due to access to large amounts of open-source datasets and enhancement of computational resources. However, a shortcoming of these methods is that they often fail to generalize well to tasks from real life scenarios, due to domain mismatch. One such task is foreground speech detection from wearable audio devices. Several interfering factors such as dynamically varying environmental conditions, including background speakers, TV, or radio audio, render foreground speech detection to be a challenging task. Moreover, obtaining precise moment-to-moment annotations of audio streams for analysis and model training is also time-consuming and costly. In this work, we use multiple instance learning (MIL) to facilitate development of such models using annotations available at a lower time-resolution (coarsely labeled). We show how MIL can be applied to localize foreground speech in coarsely labeled audio and show both bag-level and instance-level results. We also study different pooling methods and how they can be adapted to densely distributed events as observed in our application. Finally, we show improvements using speech activity detection embeddings as features for foreground detection.
AB - Over the recent years, machine learning techniques have been employed to produce state-of-the-art results in several audio related tasks. The success of these approaches has been largely due to access to large amounts of open-source datasets and enhancement of computational resources. However, a shortcoming of these methods is that they often fail to generalize well to tasks from real life scenarios, due to domain mismatch. One such task is foreground speech detection from wearable audio devices. Several interfering factors such as dynamically varying environmental conditions, including background speakers, TV, or radio audio, render foreground speech detection to be a challenging task. Moreover, obtaining precise moment-to-moment annotations of audio streams for analysis and model training is also time-consuming and costly. In this work, we use multiple instance learning (MIL) to facilitate development of such models using annotations available at a lower time-resolution (coarsely labeled). We show how MIL can be applied to localize foreground speech in coarsely labeled audio and show both bag-level and instance-level results. We also study different pooling methods and how they can be adapted to densely distributed events as observed in our application. Finally, we show improvements using speech activity detection embeddings as features for foreground detection.
KW - Foreground speech detection
KW - Multiple instance learning
KW - Weakly labeled audio
KW - Wearable audio
UR - http://www.scopus.com/inward/record.url?scp=85100417648&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85100417648&partnerID=8YFLogxK
U2 - 10.1186/s13636-020-00194-0
DO - 10.1186/s13636-020-00194-0
M3 - Article
AN - SCOPUS:85100417648
SN - 1687-4714
VL - 2021
JO - Eurasip Journal on Audio, Speech, and Music Processing
JF - Eurasip Journal on Audio, Speech, and Music Processing
IS - 1
M1 - 7
ER -