As robots become more commonly used in everyday scenarios, a wide range of individuals will need appropriate tools to easily and efficiently create programs that specify what their robot should do or how it should behave. This work seeks to make programming human robot interactions more natural for non-roboticists by employing programming language techniques, such as program synthesis, to translate multimodal user input into a full robot program.
We present a novel, multimodal, and on-the-fly development system, Tabula. Inspired by a formative design study with a prototype, Tabula leverages a combination of spoken language for specifying the core of a robot task and sketching for contextualizing the core. The result is that developers can script partial, sloppy versions of robot programs to be completed and refined by a program synthesizer.
Figaro is a tabletop authoring environment where demonstrators use instrumented figurines to play out scenes of human-robot interactions. The positions, movements, behaviors, and speech of the figurines is recorded for each scene. Figaro then synthesizes a full human-robot interaction program that can be executed on a robot.