Words, Probability, and Segmental Information: Less Probable Words Have More Informative Segments

There is a correlation between the phonological shape of a word and the word’s probability in use. Less probable words tend to be longer and more probable words shorter (see Piantadosi et al., Zipf) . This has been attributed to the lexicon evolving for efficient communication (Zipf) . To identify less probable words, listeners need more information from the segments in the phonological word itself. In this case, longer lengths for less probable words mean a greater amount of information to be used in word identification. However, this does not take into account how listeners actually process words. Research in spoken word recognition has shown that words are processed incrementally and some segments may in fact be more informative (Allopenna et al., Luce and Pisoni, van Son and Pols, Weber and Scharenborg) . Here, we use corpus data from American English to provide evidence that less probable words contain more informative segments. We also show that the distribution of segmental information is correlated with the word’s probability and that less probable words contain more of their total information in the early segments. We discuss these findings and possible evolutionary avenues for language to reach this state. This work provides support for the idea that the words in the lexicon evolve under pressure for efficient communication.