Modality effects in deception detection and applications in automatic-deception-detection

Modality is an important context factor in deception, which is context-dependent. In order to build a reliable and flexible tool for automatic-deception-detection (ADD), we investigated the characteristics of verbal cues to deceptive behavior in three modalities: text, audio and face-to-face communication. Seven categories of verbal cues (21 cues) were studied: quantity, complexity, diversity, verb nonimmediacy, uncertainty, specificity and affect. After testing the interaction effects between modality and condition (deception or truth), we found significance only with specificity and observed that differences between deception and truth were in general consistent across the three modalities. However, modality had strong effects on verbal cues. For example, messages delivered face-to-face were largest in quantity (number of words, verbs, and sentences), followed by the audio modality. Text had the sparsest examples. These modality effects are an important factor in building baselines in ADD tools, because they make it possible to use them to adjust the baseline for an unknown modality according to a known baseline, thereby simplifying the process of ADD. The paper discusses in detail the implications of these findings on modality effects in three modalities.