Deep Multimodal Embedding: Manipulating Novel Objects with Point-clouds, Language and Trajectories. ICRA, 2017. (finalist for ICRA Best cognitive robotics paper award)

Deep Multimodal Embedding: Manipulating Novel Objects with Point-clouds, Language and Trajectories. ICRA, 2017. (finalist for ICRA Best cognitive robotics paper award)

https://arxiv.org/pdf/1509.07831v1.pdf

A system operating in a real-world environment such as homes needs to perform reasoning with a variety of sensing modalities.

However, manually designing such features is extremely challenging. This work presents an algorithm to learn to embed point-cloud, natural language, and trajectory data into a shared embedding space using a deep neural network.