HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods

1. HOPS:EfficientRegionLabelingusingHigherOrderProxyNeighborhoodsHOPS:EfficientRegionLabelingusingHigherOrderProxyNeighborhoods Albert Y. C. Chen1,JasonJ.Corso1,andLeWang2 1Dept.ofComputer Science and Engineering 2Dept.ofGeography University at Buffalo, The State University of New York

2. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Image Labeling via Energy Minimization • Image labeling: assigning an object class (e.g. sky, water, tree, grass) to each pixel. • Energy minimization, or equivalently, posterior probability maximization, is the standard approach for solving image labeling problems. • Markov Random Fields or Conditional Random Fields, are the typical models used for energy minimization. 1

3. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Typical Labeling Results versus the more ideal labeling by HOPS Input Image to be labeled Labels using std. 1st order More Idea labels 2

4. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Markov Random Field (MRF) in image labeling • Undirected-Graphs with cond. independent nodes. • The labels of hidden nodes xi are chosen to minimize the global energy. • A hidden nodes xi is dependent only on other hidden nodes within their Markov blanket (in practice, only first-order neighborhood is used). • Typical energy used: • (bias) (local evidence) (Markov blanket) • (E1: unary term) (E2: binary term) 3

5. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Image Labeling via Energy Minimization – Previous Approaches • Simulated Annealing: Can converge to global optimum in theory, but is extremely slow in practice (several hours to several days). • Approximation Algorithms: • Belief Propagation: energy reduction done by updating with the messages passed along the nodes. Performance is to some degree un-expectable on loopy graphs. • Graph-Cuts: takes large leap in energy space, but is not guaranteed to converge on all energy functions. • Graph-Shifts: adaptive hierarchies are used to better represent the underlying data, while “shifts” are used to efficiently minimize the energy. * For a detailed comparison of the performances, please refer to: Richard Szelisk et al., “A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors”, PAMI 2008 4

6. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Graph-Shifts: Its hierarchy and “shifts” 1. Build the Adaptive Hierarchy A “SHIFT” occurs. It can happen at any level of the adaptive hierarchy. 3. Repeat until the overall energy is minimized. 5

7. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Graph-Shifts: Its Effectiveness and Efficiency 6 Top: from J. J. Corso et al., IPMI 2007; Bottom: from J. J. Corso et al., CVPR 2008

8. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Are MRF Blankets using first-order neighborhoods sufficient? • Algorithms with first-order neighborhood can be myopic, thus producing unnecessary label changes and even noisy/incorrect labeling results. • However, first-order neighborhood have been widely used by previous methods because: • Higher-order neighborhood increases the inter-nodal connectivity. • Extending the neighborhood to the nth order increases the comp. time by |N2|.|N3|. … |Ni| • HOPS, inspired by the Belief Propagation algorithm, approximates the energies of higher-order neighbors using the first order neighborhood. 7

9. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering HOPS (Higher Order Proxy NeighborhoodS) • Instead of calculating the binary energies between {μ,τi| τi:μ’s higher order neighbors} directly: • The binary energy between νi (μ’s 1st order neighbors) and their 1st order neighbors τj are passed to μ, to approximate the higher order binary energy between {μ,τi}. 8

10. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering HOPS (Higher Order Proxy NeighborhoodS) – why it works and how it works • When MRF with smoothness based prior is used and higher order neighborhood is requested, it means that: • The label of a node μand its 1st order neighbors νare likely to be equivalent. • Thus, the approximated higher order binary energy ofτ will likely be the same as the direct calculation between {μ, τ}. • First order binary energies are cached at each node n. Thus when its neighboring node m is trying to approximate higher order energies, n would work as a proxy and pass its cached energy to m. 9

11. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Natural Image Labeling Results Graph-Shifts with Higher Order Proxy NeighborhoodS Graph-Shifts using first order neighborhood* 10 * J. J. Corso et al., CVPR 2008

12. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Discussion • Quality/Accuracy of the Labeling • The number of small noisy labels is greatly reduced. • Object boundaries are better followed in the labeled image • The labeled images are much closer to what a human expert would produce. • The computation time for one “shift” have increased only linearly. However, since redundant shifts are effectively avoided: • The number of shifts required until convergence decreases by an average of 60%. • The overall convergence time is reduced by 30%. 11

13. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering What about other types of Energy function and problem sets? • Aerial photo and airborne LiDAR (light detection and ranging) - labeling results: • HOPS gets approximately the same accuracy rate yet constantly converge 30% faster than standard 1st order neighbor ones. • Other spatial constraints that relies on context of a larger neighborhood can be added (such as shape information) into the model to achieve better results. 12

14. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering Conclusion • HOPS produces aesthetically and quantitatively better labeling results compared to those using standard first order neighbors. • HOPS estimates higher order energies in a recursive and cached manner, which induces little additional computational cost without increasing the node connectivity of the graph. • HOPS constantly converges 30% faster while used in the Graph-Shifts algorithm, since more context information is incorporated and redundant / incorrect label changes are more likely to be avoided. 13