Toward natural language interfaces for robotic agents: Grounding linguistic meaning in sensors

The results of an experiment, in which human subjects generated unrestricted natural language utterances to describe the activities of a Pioneer-1 mobile robot, is presented. A combination of word clustering applied to the utterances and a common subsequence algorithm applied to the time series of sensor values recorded by the robot made it possible for the Pioneer-1 to identify and represent a variety of features of its environment to which the utterances referred. There are three distinct ways that the robot can push objects, each of which is described with the word pushed by at least on subject, and the sensor time series that result from these situations are all identified as frequent subsequences.