1 / 68

Compressed Sensing Theory Geometric Interpretations

Compressed Sensing Theory Geometric Interpretations. April 7 th 2016, Ilmenau. Prof. Giovanni Del Galdo. Compressed Sensing Theory Geometric Interpretations. April 7 th 2016, Ilmenau. Prof. Giovanni Del Galdo. With contributions from…

pegeen
Download Presentation

Compressed Sensing Theory Geometric Interpretations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Compressed Sensing TheoryGeometric Interpretations April 7th 2016, Ilmenau Prof. Giovanni Del Galdo

  2. Compressed Sensing TheoryGeometric Interpretations April 7th 2016, Ilmenau Prof. Giovanni Del Galdo With contributions from… Dr. Florian RömerAnastasia LavrenkoAlexandra CraciunMagdalena PrusMohamed Gamal IbrahimRoman Alieiev Heavily inspired by… the tutorial of Dr. Dejan E. Lazichgiven at the 22nd meeting of the ITG section on “Applied Information Theory” on Oct. 7th, 2013

  3. Goals of this lecture • Answer the following questions: • Why is the desired sparse solution unique? • Why is the hyperplane of solutions tangent to the L1-ball at the right point? • What is the impact of measurement noise? ?

  4. Complete Data Model Reconstructionstrategy

  5. Solution Set for the Equality Constraint • x must lie on a line

  6. Lp-Norms and Lp-Balls • Definition of Lp-norm: • Lp-Balls for N = 2

  7. Minimization Problem: p=0.5

  8. Minimization Problem: p=0.5 L2 2D animation

  9. Minimization Problem: p=0.5 L2 2D animation • The “arms” of the L0.5 ball reach out making sparse solutions favored wrt non-sparse

  10. Minimization Problem: p=1 • Same effect as L0.5, although not as prominent

  11. Minimization Problem: p=1

  12. Minimization Problem: p=1

  13. Minimization Problem: p=2

  14. Minimization Problem: p=2

  15. Minimization Problem: p=2 • This is the LS solution

  16. Escape Velocities for the L1-ball • Vertices move faster than edges, which move faster than sides • They correspond to 1-sparse, 2-sparse, and 3-sparse respectively

  17. Choice of p-Norm • p>1 lead to non-sparse solutions • p<=1 lead to sparse solutions • p=0 = cardinality • logical choice • gives the sparsest solution • NP-hard • p=1 • identical to p=0 for practical cases • linear and convex problem

  18. Goals of this lecture • Answer the following questions: • Why is the desired sparse solution unique? • Why is the hyperplane of solutions tangent to the L1-ball at the right point? • What is the impact of measurement noise? ?

  19. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

  20. System of Linear Equations (SLE) underdetermined overdetermined fully determined infinite number ofsolutions one unique solution no solutions ! assuming B to be full rank

  21. Solutions to the i-th Linear Equation solution to the homogeneous problem solution to the inhomogeneous problem

  22. Solutions to the Homogeneous Linear Equation • Any point x0 on the plane orthogonalto biis a solution to the homogeneous linearequation

  23. Solutions to the Inhomogeneous Linear Equation • Distance between the planes: • Any point on the affine planeis a solution to the inhomogeneous linearequation

  24. Solutions to the Inhomogeneous SLE • Each equation removes a degreeof freedom • There exists one unique solution

  25. Shooting for Higher Dimensional Spaces • In order to predict the characteristics of the affine subspace of solutions we cannot rely on intuition alone • Therefore, we need to introduce more powerful tools • the definition of affine subspaces • the rules applying to the intersection of affine subspaces ND 3D

  26. K-Flats, Affine Subspaces in N-dimensional Ambient Space • The origin-free generalization of points, lines, planes, and K-dimensional subspaces are termed: • affine subspaces of dimension 0, 1, 2, and K, respectively or • 0-flats, 1-flats, 2-flats, and K-flats, respectively • A K-flat is determined by K+1 linearly independent points • K points are linearly independent iff • taking a subset, no 3 points lie on a line (1-flat) • taking a subset, no 4 points lie on a plane (2-flat) • … • taking a subset, no k points lie on a (k-2)-flat, for 2<=k<=K • Ambient space of dimension N: the N-flat which contains the universeconsidered

  27. K-Flats, Affine Subspaces in N-dimensional Ambient Space • A line is a 1-flat and is determined by 2 linearly independent points • A plane is a 2-flat and is determined by 3 linearly independent points • 3 points are linearly independent if no 2 points overlap and no 3 points lie on a line

  28. Intersection of Affine Subspaces • Assume a p-flat and a q-flat in N-dimensional ambient space • Ignoring special cases, the intersection of the two flats is • (p+q-N)-flat, if p+q-N >= 0 • empty, if p+q-N < 0 • If two flats obey the rules above, they are said to be in • General Position • Randomly generated subspaces are in General Position with probability one

  29. Intersection of Affine Subspaces: Examples Two lines in 3D space p = 1 q = 1 N = 3 Intersection = empty …as 1 + 1 – 3 = -1 The lines are skew

  30. Intersection of Affine Subspaces: Examples Two lines in 2D space p = 1 q = 1 N = 2 Intersection = 0-flat …as 1 +1 – 2 = 0 The lines intersect in a point Degeneratecase: Twooverlappinglines Intersection = 1-flat

  31. Intersection of Affine Subspaces: Examples Two planes in 3D space p = 2 q = 2 N = 3 Intersection = 1-flat …as 2 + 2 – 3 = 1 The planes intersect in a line

  32. Intersection of Affine Subspaces: Examples A special case can be visualized! The rest of the plane is in the fourth dimension Two planes in 4D space p = 2 q = 2 N = 4 Intersection = 0-flat, i.e., a point …as 2 + 2 – 4 = 0 The two planes intersect in a point

  33. Degenerate Cases: Examples • Two 1-flats (lines) overlapping in 2D • The General Position would be an intersection in one point • Two 2-flats (planes) intersecting on a line in 5D • The General Position would be an empty intersection • Affine subspaces generated randomly are degenerate with probability zero

  34. Flats, their Intersection for a SLE • Each linear constraint gives rise to a (N-1)-flat • The subspace of solutions is the intersection of the M (N-1)-flats in N-dimensional space. It is an (N-M)-flat!

  35. Flats, their Intersection for a SLE

  36. Recap: Geometrical Interpretation on the SLE • Given a SLE featuring • an N dimensional unknown vector x • M linear equations • The solutions to each linear equation span an affine subspace of dimensionality N-1, i.e., an (N-1)-flat • The solutions to the whole SLE span the intersection of the M (N-1)-flats • Given an underdetermined system of equations • the affine subspace of solutions has size N-M • we call it: the “Solutions Monster” (N-M)-flat Solutions Monster

  37. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

  38. Affine Subspaces for K-sparse Vectors • The K-sparse vectors span specific affine subspaces in N-dimensional space • 1-sparse vectors span 1-flats

  39. Affine Subspaces for K-sparse Vectors • 2-sparse vectors span 2-flats

  40. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

  41. Affine Subspaces for K-sparse Vectors • In general: • K-sparse vectors span K-flats • Given a system of linear equations in General Position: • What is the probability of having sparse solutions? I.e., when will the solutions monster touch the flats containing sparse vectors? ? 1-flat (N-M)-flat Solutions Monster

  42. Intersection: Solution Monster <-> Sparse Flats • Assuming a SLE of M linear constraints in N-dimensional space • Assuming General Position the results in the table occur with Prob. = 1

  43. Intersection: Solution Monster <-> Sparse Flats 2-flat 2-flat 1-flat 1-flat (N-M)-flat Solutions to the SLE 2-flat 1-flat • For K < M the Solutions Monster does not intersect with any K-flat carrying K-sparse solutions

  44. Recap: Solution Monster vs. Sparse Flats • In the General Position, the Solutions Monster dodges all sparse vectors up to K = M-1 Me too! (N-M)-flat Solutions to the SLE

  45. Outline • General geometrical considerations • affine subspace of solutions for a generalized System of Linear Equations (SLE) • subspaces spanned by sparse vectors • intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i.e., When does an SLE have sparse solutions? • Geometrical considerations specific to Compressed Sensing • solutions for a system originating from a compressed sensing scenario • impact of the noise: designing the sensing matrix • Conclusions

  46. SLE in a Compressed Sensing Scenario • Let us now take an underdetermined system of equations originating from a sparse solution x • The Solution Monster corresponding to the homogeneous system will bein General Position (N-M)-flat Solutions Monster

More Related