Collaborative Construction and Communication with Minecraft
Virtual gaming platforms such as Minecraft allow us to study situated natural language generation and understanding tasks for agents that operate in complex 3D environments.
In this talk, I will present work done by my group on defining a collaborative Blocks World construction task in Minecraft. In this task, one player (the Architect) needs to instruct another (the Builder) via a chat interface to construct a given target structure that only the Architect is shown. Although humans easily complete this task (often after lengthy back-and-forth dialogue), creating agents for each of this role poses a number of challenges for current NLP technologies. To understand these challenges, I will describe the dataset we have collected for this task, as well as the models that we have developed for both roles.