Sublinear Algorithms for Big Data. Part 4: Random Topics

Size: px

Start display at page:

Download "Sublinear Algorithms for Big Data. Part 4: Random Topics"

Shona Fletcher
8 years ago
Views:

1 Sublinear Algorithms for Big Data Part 4: Random Topics Qin Zhang 1-1

2 2-1 Topic 1: Compressive sensing

3 Compressive sensing The model (Candes-Romberg-Tao 04; Donoho 04) Applicaitons Medical imaging reconstruction Single-pixel camera Compressive sensor network etc. 3-1

4 Formalization Lp/Lq guarantee: The goal to acquire a signal x = [x 1,..., x n ] (e.g., a digital image). The acquisition proceeds by computing a measurement vector Ax of dimension m n. Then, from Ax, we want to recover a k-sparse approximation x of x so that x x q C min x 0 k x x p ( ) 4-1

The acquisition proceeds by computing a measurement vector Ax of

5 Formalization Lp/Lq guarantee: The goal to acquire a signal x = [x 1,..., x n ] (e.g., a digital image). The acquisition proceeds by computing a measurement vector Ax of dimension m n. Then, from Ax, we want to recover a k-sparse approximation x of x so that x x q C min x 0 k x x p ( ) Err k p (x) 4-2

The acquisition proceeds by computing a measurement vector Ax of dimension

6 Formalization Lp/Lq guarantee: The goal to acquire a signal x = [x 1,..., x n ] (e.g., a digital image). The acquisition proceeds by computing a measurement vector Ax of dimension m n. Then, from Ax, we want to recover a k-sparse approximation x of x so that x x q C min x 0 k x x p Often study: L 1 /L 1, L 1 /L 2 and L 2 /L 2 ( ) Err k p (x) 4-3

Then, from Ax, we want to recover a k-sparse approximation x of x so that x x q C min

7 Formalization Lp/Lq guarantee: The goal to acquire a signal x = [x 1,..., x n ] (e.g., a digital image). The acquisition proceeds by computing a measurement vector Ax of dimension m n. Then, from Ax, we want to recover a k-sparse approximation x of x so that x x q C min x 0 k x x p Often study: L 1 /L 1, L 1 /L 2 and L 2 /L 2 ( ) Err k p (x) For each: Given a (random) matrix A, for each signal x, ( ) holds w.h.p. For all: One matrix A for all signals x. Stronger. 4-4

Then, from Ax, we want to recover a k-sparse approximation x of x so that x x q C min x 0 k x x p Often study: L 1 /L

8 Results 5-1 Up to year copied from Indyk s talk

9 Pre-history: Orthogonal Matching Pursuit Given a signal n-dimentional vector x {0, 1} n with k n non-zero entries. Let y = Ax, where A rand { 1, 0, 1} m n. The following algorithm can recover x exactly using A, y with m = O(k log(n/k)), (i.e., O(k log(n/k)) measurements) 6-1

10 Pre-history: Orthogonal Matching Pursuit Given a signal n-dimentional vector x {0, 1} n with k n non-zero entries. Let y = Ax, where A rand { 1, 0, 1} m n. The following algorithm can recover x exactly using A, y with m = O(k log(n/k)), (i.e., O(k log(n/k)) measurements) Algorithm Orthogonal Matching Pursuit Set r = y. Denote A = (A 1,..., A n ). For i = 1 to t do 1. Set A j = arg max Al {A 1,...,A n } r, A l 2. Set γ j = arg max γ r A j γ 2 3. Set r = r A j γ j Return x where x j = γ j. 6-2

The following algorithm can recover x exactly using A, y with m = O(k log(n/k)), (i.e., O(k log(n/k)) measurements) Algorithm Orthogonal Matching Pursuit Set r = y.

11 Pre-history: Orthogonal Matching Pursuit Given a signal n-dimentional vector x {0, 1} n with k n non-zero entries. Let y = Ax, where A rand { 1, 0, 1} m n. The following algorithm can recover x exactly using A, y with m = O(k log(n/k)), (i.e., O(k log(n/k)) measurements) Algorithm Orthogonal Matching Pursuit Set r = y. Denote A = (A 1,..., A n ). For i = 1 to t do 1. Set A j = arg max Al {A 1,...,A n } r, A l 2. Set γ j = arg max γ r A j γ 2 3. Set r = r A j γ j Return x where x j = γ j. Will converge since in each step r 2 deceases. 6-3

Denote A = (A 1,..., A n ). For i = 1 to t do 1. Set A j = arg max Al {A 1,...,A n } r, A l 2. Set γ j = arg max γ r A j γ 2 3.

12 Pre-history: Orthogonal Matching Pursuit Given a signal n-dimentional vector x {0, 1} n with k n non-zero entries. Let y = Ax, where A rand { 1, 0, 1} m n. The following algorithm can recover x exactly using A, y with m = O(k log(n/k)), (i.e., O(k log(n/k)) measurements) Algorithm Orthogonal Matching Pursuit Set r = y. Denote A = (A 1,..., A n ). For i = 1 to t do May stop when r 2 or γ is very small 1. Set A j = arg max Al {A 1,...,A n } r, A l 2. Set γ j = arg max γ r A j γ 2 3. Set r = r A j γ j Return x where x j = γ j. Will converge since in each step r 2 deceases. 6-4

Denote A = (A 1,..., A n ). For i = 1 to t do May stop when r 2 or γ is very small 1. Set A j = arg max Al {A 1,...,A n } r, A l 2.

13 L 1 point query (recall) Algorithm Count-Min [Cormode and Muthu 05] Pick d (d = log(1/δ)) independent hash functions h 1,..., h d where h i : {1,..., n} {1,..., w} (w = 4/ɛ) from a 2-universal family. Maintain d vectors Z 1,..., Z d where Z t = {Z1, t..., Zw t } such that = i:h t (i)=j x i Z t j Estimator: x i = min t Z t h t (i) Theorem We can solve L 1 point query, with approximation ɛ, and failure probability δ by storing O(1/ɛ log(1/δ)) words. 7-1

Maintain d vectors Z 1,..., Z d where Z t = {Z1, t.

14 For each (L 1 /L 1 ) The algorithm for L 1 point query gives a L 1 /L 1 sparse approximation. 8-1

15 For each (L 1 /L 1 ) The algorithm for L 1 point query gives a L 1 /L 1 sparse approximation. Recall L 1 Point Query Problem: Given ɛ, after reading the whole stream, given i, report x i = x i ± ɛ x 1 8-2

16 For each (L 1 /L 1 ) The algorithm for L 1 point query gives a L 1 /L 1 sparse approximation. Recall L 1 Point Query Problem: Given ɛ, after reading the whole stream, given i, report x i = x i ± ɛ x 1 Set α = kɛ (0, 1) and δ = 1/n 2 in L 1 point query. And then return a vector x consisting of k largest (in magnitude) elements of x. It gives w.p. 1 δ, x x 1 (1 + 3α) Err k 1 Total measurements: m = O(k/α log n) (All analysis on board) 8-3

Set α = kɛ (0, 1) and δ = 1/n 2 in L 1 point query.

17 For all (L 1 /L 2 ) A matrix A satisfies (k, δ)-rip (Restricted Isometry Property) if k-sparse vector x we have (1 δ) x 2 Ax 2 (1 + δ) x

18 For all (L 1 /L 2 ) A matrix A satisfies (k, δ)-rip (Restricted Isometry Property) if k-sparse vector x we have (1 δ) x 2 Ax 2 (1 + δ) x 2. Theorem Johnson-Linderstrauss Lemma x with x 2 = 1, we have 7/8 Ax 2 8/7 w.p. 1 e O(m). 9-2

19 For all (L 1 /L 2 ) A matrix A satisfies (k, δ)-rip (Restricted Isometry Property) if k-sparse vector x we have (1 δ) x 2 Ax 2 (1 + δ) x 2. Theorem Johnson-Linderstrauss Lemma x with x 2 = 1, we have 7/8 Ax 2 8/7 w.p. 1 e O(m). Theorem We If each canentry solveof L 1 Apoint is i.i.d. query, as N with (0, 1), approximation and m = O(klog(n/k)), ɛ, and failure then A satisfies probability (k, δ1/3)-rip by storing w.h.p. O(1/ɛ log(1/δ)) numbers. 9-3

1 e O(m). Theorem We If each canentry solveof L 1 Apoint is i.i.d.

20 For all (L 1 /L 2 ) A matrix A satisfies (k, δ)-rip (Restricted Isometry Property) if k-sparse vector x we have (1 δ) x 2 Ax 2 (1 + δ) x 2. Theorem Johnson-Linderstrauss Lemma x with x 2 = 1, we have 7/8 Ax 2 8/7 w.p. 1 e O(m). Theorem We If each canentry solveof L 1 Apoint is i.i.d. query, as N with (0, 1), approximation and m = O(klog(n/k)), ɛ, and failure then A satisfies probability (k, δ1/3)-rip by storing w.h.p. O(1/ɛ log(1/δ)) numbers. 9-4 Theorem Main Theorem We If Acan has solve (6k, 1/3)-RIP. L 1 point query, Let x with be the approximation solution to the ɛ, and LP: failure minimize x probability 1 subject δ bytostoring Ax = O(1/ɛ Ax (xlog(1/δ)) is k-sparse). numbers. Then x x 2 C/ k Err1 k for any x (All analysis on board)

h.p. O(1/ɛ log(1/δ)) numbers. 9-4 Theorem Main Theorem We If Acan has solve (6k, 1/3)-RIP.

A Negative Result Concerning Explicit Matrices With The Restricted Isometry Property

A Negative Result Concerning Explicit Matrices With The Restricted Isometry Property Venkat Chandar March 1, 2008 Abstract In this note, we prove that matrices whose entries are all 0 or 1 cannot achieve