Dynamic Programming

1. Dynamic Programming CS4413 Algorithms Dynamic Programming

2. Dynamic Programming • An Algorithm Design Technique • A framework to solve Optimization problems • Elements of Dynamic Programming • Dynamic programming version of a recursive algorithm. • Developing a Dynamic Programming Algorithm • Example: Multiplying a Sequence of Matrices

3. Why Dynamic Programming? • It sometimes happens that the natural way of dividing an instance suggested by the structure of the problem leads us to consider several overlapping subinstances. • If we solve each of these independently, they will in turn create a large number of identical subinstances. • If we pay no attention to this duplication, it is likely that we will end up with an inefficient algorithm. • If, on the other hand, we take advantage of the duplication and solve each subinstance only once, saving the solution for later use, then a more efficient algorithm will result.

4. Why Dynamic Programming? … • The underlying idea of dynamic programming is thus quite simple: avoid calculating the same thing twice, usually by keeping a table of known results, which we fill up as subinstances are solved. • Dynamic programming is a bottom-up technique. • Examples: 1) Computing a Binomial coefficient. • 2) The World Series.

5. Dynamic Programming • Dynamic Programmingis a general algorithm design • technique. • Invented by American mathematician Richard Bellman in the • 1950s to solve optimization problems. • “Programming” here means “planning”. • Main idea: • solve several smaller (overlapping) subproblems. • record solutions in a table so that each subproblem is • only solved once. • final state of the table will be (or contain) solution.

6. Dynamic Programming • Fibonacci numbers: 0, 1, 1, 2, 3, 5, 8, 13, 21, 24 • Recurrence Relation of Fibonacci numbers ?

7. Example: Fibonacci numbers • Recall definition of Fibonacci numbers: • f(0) = 0 • f(1) = 1 • f(n) = f(n-1) + f(n-2) for n ≥ 2 • Computing the nth Fibonacci number recursively (top-down): • f(n) • f(n-1) + f(n-2) • f(n-2) + f(n-3) f(n-3) + f(n-4) • ...

8. Example: Fibonacci numbers • Computing the nth fibonacci number using bottom-up iteration: • f(0) = 0 • f(1) = 1 • f(2) = 0+1 = 1 • f(3) = 1+1 = 2 • f(4) = 1+2 = 3 • f(5) = 2+3 = 5 • f(n-2) = f(n-3)+f(n-4) • f(n-1) = f(n-2)+f(n-3) • f(n) = f(n-1) + f(n-2)

9. Examples of Dynamic Programming Algorithms • Computing binomial coefficients • Optimal chain matrix multiplication • Constructing an optimal binary search tree • Warshall’s algorithm for transitive closure • Floyd’s algorithms for all-pairs shortest paths • Some instances of difficult discrete optimization problems: • travelling salesman • knapsack

10. A framework to solve Optimization problems • For each current choice: • Determine what subproblem(s) would remain if this choice were made. • Recursively find the optimal costs of those subproblems. • Combine those costs with the cost of the current choice itself to obtain an overall cost for this choice • Select a current choice that produced the minimum overall cost.

11. Elements of Dynamic Programming • Constructing solution to a problem by building it up dynamically from solutions to smaller (or simpler) sub-problems • sub-instances are combined to obtain sub-instances of increasing size, until finally arriving at the solution of the original instance. • make a choice at each step, but the choice may depend on the solutions to sub-problems.

12. Elements of Dynamic Programming … • Principle of optimality • the optimal solution to any nontrivial instance of a problem is a combination of optimal solutions to some of its sub-instances. • Memorization (for overlapping sub-problems) • avoid calculating the same thing twice, • usually by keeping a table of know results that fills up as sub-instances are solved.

13. Development of a dynamic programming algorithm • Characterize the structure of an optimal solution • Breaking a problem into sub-problem • whether principle of optimality apply • Recursively define the value of an optimal solution • define the value of an optimal solution based on value of solutions to sub-problems • Compute the value of an optimal solution in a bottom-up fashion • compute in a bottom-up fashion and save the values along the way • later steps use the save values of pervious steps • Construct an optimal solution from computed information

14. Binomial Coefficient • Binomial coefficient: • Binomial formula:

15. Binomial Coefficient • Record the values in a table of n+1 rows and k+1 columns

16. Binomial Coefficient ALGORITHM Binomial(n,k) //Computes C(n, k) by the dynamic programming algorithm //Input: A pair of nonnegative integers n ≥ k ≥ 0 //Output: The value of C(n ,k) fori 0 to n do forj 0 to min (i ,k) do ifj = 0 orj = k C [i , j]  1 elseC [i , j]  C[i-1, j-1] + C[i-1, j] return C [n, k]

17. 3 3 1 1 4 4 2 2 Warshall’s Algorithm: Transitive Closure • Computes the transitive closure (next slide) of a relation • (Alternatively: all paths in a directed graph) • Example of transitive closure: 0 0 1 0 1 1 1 1 0 0 0 0 1 1 1 1 0 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0

18. Transitive Closure • The Transitive closure of a directed graph with n vertices can be defined as the n-by-n boolean matrix T = {tij}, in which the element in the ith row (1 ≤ i ≤ n) and the jth column (1 ≤ j ≤ n) is 1 if there exists a nontrivial directed path (i.e., a directed path of a positive length) from the ith vertex to the jth vertex; otherwise, tij is 0.

19. 3 3 3 3 3 1 1 1 1 1 4 2 4 4 2 2 4 4 2 2 Warshall’s Algorithm • Main idea: a path exists between two vertices i, j, iff • there is an edge from i to j; or • there is a path from i to j going through vertex 1; or • there is a path from i to j going through vertex 1 and/or 2; or • there is a path from i to j going through vertex 1, 2, and/or 3; or • ... • there is a path from i to j going through any of the other vertices R0 0 0 1 0 1 0 0 1 0 0 0 0 0 1 0 0 R1 0 0 1 0 1 01 1 0 0 0 0 0 1 0 0 R2 0 0 1 0 1 0 1 1 0 0 0 0 1 1 1 1 R3 0 0 1 0 1 0 1 1 0 0 0 0 1 1 1 1 R4 0 0 1 0 1 1 1 1 0 0 0 0 1 1 1 1

20. Warshall’s Algorithm • In the kth stage determine if a path exists between two vertices i, j using just vertices among 1,…,k • R(k-1)[i,j] (path using just 1 ,…,k-1) • R(k)[i,j] = or • (R(k-1)[i,k] and R(k-1)[k,j]) (path from i to k • and from k to i • using just 1 ,…,k-1) { k i kth stage j

21. Warshall Algorithm • ALGORITHM Warshall (A[1…n, 1…n]) • //output: the transitive closure of the graph • R(0) ← A • For k ←1 to n do • For i ← 1 to n do • For j ← 1 to n do • R(k)[i, j] ←R(k-1)[i, j] or R(k-1)[i, k] and R(k-1)[k, j] • Return R(n)

22. Warshall’s algorithm a b c d

23. Example: Transitive Closure • Example: Apply Warshall’s algorithm to find the transitive closure of the digraph defined by the following adjacency matrix • 0 1 0 0 • 0 0 1 0 • 0 0 0 1 • 0 1 0 0

24. 4 3 1 1 6 1 5 4 2 3 Floyd’s Algorithm: All pairs shortest paths • In a weighted graph, find shortest paths between every pair of vertices • Same idea: construct solution through series of matrices D(0), D(1), … using an initial subset of the vertices as intermediaries. • Example:

25. Floyd’s Algorithm: All pairs shortest paths … • ALGORITHM Floyd (W[1 … n, 1… n]) • For k ← 1 to n do • For i ← 1 to n do • For j ← 1 to n do • W[i, j] ← min{W[i,j], W{i, k] + W[k, j]} • Return W

26. Floyd’s Algorithm 2 a b 6 7 3 c d 1

27. Example: All-pairs shortest-path problem • Example: Apply Floyd’s algorithm to find the t All-pairs shortest-path problem of the digraph defined by the following weight matrix • 0 2 ∞ 1 8 • 6 0 3 2 ∞ • ∞ ∞ 0 4 ∞ • ∞ ∞ 2 0 3 • 3 ∞ ∞ ∞0

28. Dynamic programming, e.g. • Problem: Matrix-chain multiplication • a chain of <A1, A2, …, An> of n matrices • find a way that minimizes the number of scalar multiplications to compute the product A1A2…An • Strategy: • Breaking a problem into sub-problem • A1A2...Ak, Ak+1Ak+2…An • Recursively define the value of an optimal solution • m[i,j] = 0 if i = j • m[i,j]= min{i<=k<j} (m[i,k]+m[k+1,j]+pi-1pkpj) • for 1 <= i <= j <= n

29. The World Series example for DP • The teams A and B play in not more than 2n - 1 games. • Assume that • there are no tied games • the probability of winning by team A is p • the team B wins with q = 1 - p • P(i, j) is the probability that the team A will win if it gets i more victories and the team B will win if it gets j more victories.

30. The World Series … • For example, before the first game of the series the probability that team A will be the overall winner is • P(n, n): both teams still need n victories. • If team A has already won all the matches it needs, then it is of course certain that they will win the series: P(0, i) = 1, 1 ≤ i ≤ n. • Similarly, P(i, 0) = 0, 1 ≤ i ≤ n. • Finally, since team A wins any given match with probability p and loses it with probability q, • P(i, j) = ?

31. The World Series … • P(i; j) = pP(i - 1; j) + qP(i; j - 1); i ≥1; j ≥1 • function P(i; j) • if i = 0 then return 1 • else if j = 0 then return 0 • else return pP(i - 1; j) + qP(i; j - 1) • Let T(k) be the time required to compute P(i; j) where k = i + j • T(1) = c • T(k) ≤2T(k ¡ 1) + d; k > 1

32. The World Series … • T(k) ≤4T(k -2) + 2d + d; k > 2 • ... • ≤ 2k-1T(1) + (2k-2 + 2k-3 + … + 2 + 1)d • = 2k-1c + (2k-1- 1)d • = 2k(c/2 + d/2) - d • → T(k) є O(2k)

33. The World Series … • P(i, j) k matches left • calls P(i -1, j) P(i, j-1) k-1 matches left • That calles P(i-2, j) P(i-1, j-1) P(I, j -2) • k-2 matches left • Recursive calls made by a call on P(i, j)

34. The World Series … • Function P(i, j) • if i = 0 then return 1 • else if j = 0 then return 0 • else return pP(i -1, j) + qP(i, j – 1)

35. The World Series … • j • 0 1 2 3 4 5 • 0 0 1 1 1 1 1 • 1 0 1/2 3/4 • i 2 0 1/4 • 3 0 1/8 • 4 0 1/16 • 5 0 1/32