Approaches to Problem Solving

Approaches to Problem Solving • greedy algorithms • dynamic programming • backtracking • divide-and-conquer

Problem: Huffman Coding Def: binary character code = assignment of binary strings to characters e.g. ASCII code A = 01000001 B = 01000010 C = 01000011 … fixed-length code How to decode: ? 01000001010000100100001101000001

Problem: Huffman Coding Def: binary character code = assignment of binary strings to characters e.g. code A = 0 B = 10 C = 11 … variable-length code How to decode: ? 0101001111

Problem: Huffman Coding Def: binary character code = assignment of binary strings to characters e.g. code A = 0 B = 10 C = 11 … variable-length code Def: A code is prefix-free if no codeword is a prefix of another codeword. How to decode: ? 0101001111

Problem: Huffman Coding Def: binary character code = assignment of binary strings to characters e.g. another code A = 1 B = 10 C = 11 … variable-length code Def: A code is prefix-free if no codeword is a prefix of another codeword. How to decode: ? 10101111

Problem: Huffman Coding Def: Huffman coding is an optimal prefix-free code. • Optimization problems • Input: • Output: • Objective:

Problem: Huffman Coding Def: Huffman coding is an optimal prefix-free code. • Huffman coding • Input: • Output: • Objective: an alphabet with frequencies a prefix-free code minimize expected number of bits per character

Problem: Huffman Coding Example: A 60% B 20% C 10% D 10% Is fixed-width coding optimal ? • Huffman coding • Input: • Output: • Objective: an alphabet with frequencies a prefix-free code minimize expected number of bits per character

Problem: Huffman Coding Example: A 60% B 20% C 10% D 10% Is fixed-width coding optimal ? NO, exists a prefix-free code using 1.6 bits per character ! • Huffman coding • Input: • Output: • Objective: an alphabet with frequencies a prefix-free code minimize expected number of bits per character

Problem: Huffman Coding • Huffman ( [a1,f1],[a2,f2],…,[an,fn]) • if n=1 then • code[a1] “” • else • let fi,fj be the 2 smallest f’s • Huffman ( [ai,fi+fj],[a1,f1],…,[an,fn] ) • omits ai,aj • code[aj]  code[ai] + “0” • code[ai]  code[ai] + “1” • Huffman coding • Input: • Output: • Objective: an alphabet with frequencies a prefix-free code minimize expected number of bits per character

Problem: Huffman Coding Lemma 1: Let x,y be the symbols with frequencies fx > fy. Then in an optimal prefix code length(Cx)  length(Cy).

Problem: Huffman Coding Lemma 1: Let x,y be the symbols with frequencies fx > fy. Then in an optimal prefix code length(Cx)  length(Cy). Lemma 2: If w is a longest codeword in an optimal code then there exists another codeword of the same length.

Problem: Huffman Coding Lemma 1: Let x,y be the symbols with frequencies fx > fy. Then in an optimal prefix code length(Cx)  length(Cy). Lemma 2: If w is a longest codeword in an optimal code then there exists another codeword of the same length. Lemma 3: Let x,y be the symbols with the smallest frequencies. Then there exists an optimal prefix code such that the codewords for x and y differ only in the last bit.

Problem: Huffman Coding Lemma 1: Let x,y be the symbols with frequencies fx > fy. Then in an optimal prefix code length(Cx)  length(Cy). Lemma 2: If w is a longest codeword in an optimal code then there exists another codeword of the same length. Lemma 3: Let x,y be the symbols with the smallest frequencies. Then there exists an optimal prefix code such that the codewords for x and y differ only in the last bit. Theorem: The prefix code output by the Huffman algorithm is optimal.

Greedy Algorithms • make decisions “greedily”, previous decisions are never reconsidered • e.g. Huffman coding • another example: the divisible version of KNAPSACK KNAPSACK Input: Output: Objective: a number W and a set of n items, the i-th item has a weight wi and a cost ci a subset of items with total weight · W maximize cost Version 1: Items are divisible.

KNAPSACK – divisible: a greedy solution • KNAPSACK-DIVISIBLE(n,c,w,W) • sort items in decreasing order of ci/wi • i = 1 • currentW = 0 • while (currentW + wi < W) { • take item of weight wi and cost ci • currentW += wi • i++ • } • take W-currentW portion of item i Correctness: Running time:

KNAPSACK – indivisible Version 2: Items are indivisible. Does previous algorithm work for this version of KNAPSACK?

BackTracking • “brute force”: try all possibilities • running time?

Dynamic Programming • idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, …

Dynamic Programming • idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … A recursive solution: • Fib(n) • if n < 0 then RETURN “undefined” • if n · 1 then RETURN n • RETURN Fib(n-1) + Fib(n-2)

Dynamic Programming • idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … A recursive solution: • Fib(n) • if n < 0 then RETURN “undefined” • if n · 1 then RETURN n • RETURN Fib(n-1) + Fib(n-2) Can do better (faster) ?

Dynamic Programming • idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … A recursive solution: • Fib(n) • if n < 0 then RETURN “undefined” • if n · 1 then RETURN n • RETURN Fib(n-1) + Fib(n-2) Can do better (faster) ? YES ! Just store Fib[k] in an array !

Dynamic Programming • idea: remember values for smaller instances, use it for computing the value for a larger instance Example: Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, … • FibDynProg(n) • Fib[0] = 0 • Fib[1] = 1 • for i=2 to n do • Fib[i] = Fib[i-1] + Fib[i-2] • RETURN Fib[n] The heart of the solution: Fib[k] = the k-th Fibonacci number Can do better (faster) ? YES ! Just store Fib[k] in an array !

KNAPSACK – indivisible: a dyn-prog solution A recursive approach (returns only the best cost, not the set of items): • KNAP-IND-REC(n,c,w,W) • if n·0 • return 0 • if W < wn • withLastItem = -1 // undefined • else • withLastItem =cn+KNAP-IND-REC(n-1,c,w,W-wn) • withoutLastItem = KNAP-IND-REC(n-1,c,w,W) • return max{withLastItem,withoutLastItem}

KNAPSACK – indivisible: a dyn-prog solution The heart of the algorithm: S[k][v] =

KNAPSACK – indivisible: a dyn-prog solution The heart of the algorithm: S[k][v] = maximum cost of a subset of the first k items, where the weight of the subset is at most v

KNAPSACK – indivisible: a dyn-prog solution The heart of the algorithm: S[k][v] = maximum cost of a subset of the first k items, where the weight of the subset is at most v • KNAPSACK-INDIVISIBLE(n,c,w,W) • init S[0][v]=0 for every v=0,…,W • init S[k][0]=0 for every k=0,…,n • 3. for v=1 to W do • for k=1 to n do • S[k][v] = S[k-1][v] • if (wk· v) and • (S[k-1][v-wk]+ck > S[k][v]) • then • S[k][v] = S[k-1][v-wk]+ck • RETURN S[n][W]

KNAPSACK – indivisible: a dyn-prog solution The heart of the algorithm: S[k][v] = maximum cost of a subset of the first k items, where the weight of the subset is at most v • KNAPSACK-INDIVISIBLE(n,c,w,W) • init S[0][v]=0 for every v=0,…,W • init S[k][0]=0 for every k=0,…,n • 3. for v=1 to W do • for k=1 to n do • S[k][v] = S[k-1][v] • if (wk· v) and • (S[k-1][v-wk]+ck > S[k][v]) • then • S[k][v] = S[k-1][v-wk]+ck • RETURN S[n][W] Running time:

KNAPSACK – indivisible: a dyn-prog solution The heart of the algorithm: S[k][v] = maximum cost of a subset of the first k items, where the weight of the subset is at most v • KNAPSACK-INDIVISIBLE(n,c,w,W) • init S[0][v]=0 for every v=0,…,W • init S[k][0]=0 for every k=0,…,n • 3. for v=1 to W do • for k=1 to n do • S[k][v] = S[k-1][v] • if (wk· v) and • (S[k-1][v-wk]+ck > S[k][v]) • then • S[k][v] = S[k-1][v-wk]+ck • RETURN S[n][W] How to output a solution ?

Approaches to Problem Solving