Dictionary Learning on a Manifold

Dictionary Learning on a Manifold 1Mingyuan Zhou, 2Hongxia Yang, 3Guillermo Sapiro, 2David Dunson and 1Lawrence Carin 1Department of Electrical & Computer Engineering, Duke University 2Department of Statistical Science,Duke University 3Department of Electrical & Computer Engineering, University of Minnesota Zhou 2011

Outline • Introduction • Dependent Hierarchical Beta Process (dHBP) • Dictionary Learning with dHBP • Image Interpolation and Denoising • Conclusions

Introduction: Background • Dictionary learning and sparse coding • Sparse factor analysis model (Factor/feature/dish/dictionary atom) • Indian Buffet process and beta process

Introduction: Background • Beta process and Bernoulli process (Thibaux & Jordan AISTATS2007) • Indian Buffet process (Griffiths & Ghahramani 2005)

Introduction: Motivation • Exchangeability assumption is not true Image interpolation with BPFA (Zhou et al. 2009) 80% pixels are missing at random

Introduction: Covariate Dependence • Dependent Dirichlet process • MacEachern (1999), Duan et al. (2007), Griffin & Steel (2006) • Non-exchangeable IBP • Phylogenetic IBP (Miller, Griffiths & Jordan 2008) • Dependent IBP (Williamson, Orbanz & Ghahramani 2010) • Bayesian density regression (Dunson & Pillai 2007)

Review of Beta Process (Thibaus & Jordan 2007) • A beta process is a positive Levy process, whose Levy measure lives on and can be expressed as • If the base measure is continuous, then is drawn from a degenerate beta distribution parameterized by • If , then

Dependent Hierarchical Beta Process • Random walk matrix • dHBP • Covariate-dependent correlations

Dictionary Learning with dHBP BP dHBP

Missing Data and Outliers • Missing data • Full data: • Observed: , Missing: • Sparse spiky noise • Recoverd data:

MCMC Inference • Independence chain Metropolis-Hastings • Slice sampling • Gibbs sampling

Experiments • Image interpolation • Missing pixels • Locations of missing pixels are known • Image denoising • WGN + sparse spiky noise • Amplitudes unknown • Locations of spiky noise are unknown • Covariates: patch spatial locations

Image Interpolation: BP vs. dHBP BP: 26.9 dB dHBP: 29.92 dB 80% pixels missing at random

Image Interpolation: BP vs. dHBP dHBP atom usage BP atom usage dHBP recovery BP dictionary dHBP dictionary Dictionary atom activation probability map

Image Interpolation: BP vs. dHBP Observed (20%) BP recovery dHBP recovery dHBP recovery Original

Spiky Noise Removal: BP vs. dHBP dHBP denoised image dHBP dictionary Original image Noisy image (WGN + Sparse Spiky noise) BP denoised image BP dictionary

Future Work • Landmark-dHBP • J landmarks • J << N • Locality constraint for manifold learning • Covariates: cosine distance between samples • Dictionary atoms look like the data • Variational inference, online learning • Other applications • Super-resolution • Deblurring • Video background & foreground modeling

Conclusions • A dependent hierarchical beta process is proposed • Efficient hybrid MCMC inference is presented • Encouraging performance is demonstrated on image-processing applications

Dictionary Learning on a Manifold

Dictionary Learning on a Manifold

Presentation Transcript

Manifold Learning Using Geodesic Entropic Graphs

Manifold learning: Laplacian Eigenmaps

Manifold Topics

Manifold learning: MDS and Isomap

Manifold Learning Via Homology

Use a Dictionary

Manifold learning

Manifold Parameterization

Manifold learning

Manifold learning: Locally Linear Embedding

A Tolkien’s Dictionary

Online Robust Dictionary Learning

Similarities, Distances and Manifold Learning

Online Manifold Regularization: A New Learning Setting and Empirical Study

Topology in Manifold Learning

Manifold learning: MDS and Isomap

Manifold learning: Locally Linear Embedding

Manifold Hydraulics

Manifold learning: Locally Linear Embedding

Manifold Hydraulics

Manifold Learning

Manifold learning