GAME-BASED APPROACHES FOR STUDYING CONVERSATIONAL SPEECH


One aspect of language processing we are interested in is how people communicate during conversational speech. However, it is difficult to run a controlled experiment in the lab and, at the same time, have people engage in natural conversation. Participants might not engaged in the task, and as a result, we might not get robust estimates of the acoustic cues they use to convey linguistic information. To overcome this, we've developed a computer game approach using Minecraft.

Screenshot from the game showing the two participants. The player on the left has a color sequence they must give to the player on the right to unlock the door.

Specifically, we have created maps in Minecraft in which two participants work together to solve puzzles. In the puzzles, one participant has information they have to give to their partner. For example, the figure on this page shows the avatars of two participants in different rooms. The participant on the left has a sequence of colors (white, brown, green) that they must give to their partner as a code to unlock the door to the next room. We can then vary whether particular color sequences are new or previously mentioned.

Running the experiment in Minecraft gives us a controlled environment, and it gives participants the ability to have more natural conversations. Using this approach, we've been able to more identify several acoustic features that people use to emphasize words. For example, talkers used word duration to indicate discourse status (new words tend to be longer than previously mentioned words), and these differences are more distinct than they are in less engaging laboratory tasks. Thus, examining conversations in the game gives us a new way of studying spoken language processing and a better understanding of the acoustic cues to prosody.

Visitors at the 2013 Beckman Open House playing a demo of the experiment.

We are also using this approach to study other aspects of conversational speech. In particular, we are interested in measures of phonetic convergence, the degree to which conversational partners adjust their speech patterns to sound more like each other. We hope that game-based paradigms, like this one, will help us better understand the processes that cause talkers to converge.

More information