The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces

Ondra Miksik 2   Vibhav Vineet 1   Morten Lidegaard 2   Raju Selva 1   Matthias Niessner 1   Stuart Golodetz 2   Stephen Hicks 2   Patrick Perez 4   Shahram Izadi 3   Philip H. S. Torr 2  

1 Stanford University    2 University of Oxford    3 Microsoft Research    4 Technicolor Research   

33nd annual ACM conference on Human factors in computing systems (CHI) 2015


We present an augmented reality system for large scale 3D reconstruction and recognition in outdoor scenes. Unlike exist- ing prior work, which tries to reconstruct scenes using active depth cameras, we use a purely passive stereo setup, allowing for outdoor uses, and extended sensing range. Our system not only produces a map of the 3D environment in real-time, it also allows the user to draw (or paint) with a laser pointer directly onto the reconstruction to segment the model into objects and semantic parts. Given these examples our system then learns to segment other parts of the 3D map during online acquisition. Unlike typical object recognition systems, ours therefore very much places the user in the loop to segment particular objects of interest, rather than learning from predefined databases. The laser pointer is additionally triangulated by the stereo camera rig during capture, which provides a strong 3D prior to help clean up the stereo reconstruction and final 3D map, interac- tively . Using our system, within minutes, a user can capture a full 3D map, segment it into objects of interest, and refine parts of the model during capture, all by simply controlling her handheld laser pointer and metaphorically painting or brushing onto the world. We provide full technical details of our system to aid replication, as well as quantitative evaluation of system components. We are particularly interested in appli- cation scenarios that can exploit these large-scale semantic 3D maps. We demonstrate the possibility of using our system for helping the visually impaired navigate through spaces. Beyond this use, these semantic maps can used for playing large-scale augmented reality games, shared online to augment streetview data, and used for more detailed car and person navigation.