Conditional Random Fields as Recurrent Neural Networks

Shuai Zheng 1   Sadeep Jayasumana 1   Bernardino Romear-Paredes 1   Vibhav Vineet 2   Zizhong Su 3   Dalong Du 3   Chang Huang 3   Philip H. S. Torr 2  

1 University of Oxford    2 Stanford University    3 Baidu Research   

arXiv pre-print


Pixel-level labelling tasks, such as semantic segmentation and depth estimation, play a central role in image understanding. Recent approaches have attempted to harness the capabilities of deep learning techniques for image recognition to tackle pixel-level labelling tasks. One central issue in this approach is the limited capacity of deep learning techniques to delineate visual objects. To solve this problem, we introduce a new form of convolutional neural network, called CRF-RNN, which expresses a Conditional Random Field (CRF) as a Recurrent Neural Network (RNN). Our short network can be plugged in as a part of a deep Convolutional Neural Network (CNN) to obtain an end-to-end system that has desirable properties of both CNNs and CRFs. Importantly, our system fully integrates CRF modelling with CNNs, making it possible to train the whole system end-to-end with the usual back-propagation algorithm. We apply this framework to the problem of semantic image segmentation, obtaining competitive results with the state-of-the-art without the need of introducing any post- processing method for object delineation.


paper   demo