IT 301: Algorithm Analysis Lecture-00

1. JesminAkhter Lecturer,IIT Jahangirnagar University, Savar, DhakaBangladesh IT 301: Algorithm AnalysisLecture-00

2. Course Overview • Algorithmic Basics: • Introduction(L00) • Algorithm analysis(L01) • Techniques for analysis of Algorithm(L02+L03). • Methodes for the design of efficient algorithms: 3 sets • Divide and Conquer (L04) • Greedy Algorithms (L05) • Dynamic Programming (L06) • Back tracking,Branch and bound : (L07) • Binary search and Traversal Techniques: (L8) • Sorting and Selection : • Heapsort : L9 3 sets • Quicksort : L10 • Bucket Sort, Radix Sort: (L11+L12) • Spanning Trees, Shortest paths: (L13) • Lower Bound Theory: (L14) • NP-Completeness : • NP-Completeness: (L15) 2 sets • NP-hard: (L16) • NP-Complete problems: (L17) • Tutorial: (L18+L19+L20)

3. Course Policy • No late homework is acceptable. • Cheating directly affects the reputation of Department and the University and lowers the morale of other students. • Cheating in homework and exam will not be tolerated. • An automatic grade of 0 will be assigned for any student caught cheating. • Presenting another person’s work as your own constitutes cheating.

4. Primary Focus Develop thinking ability. • problem solving skills. (algorithm design and application) • formal thinking. (proof techniques & analysis)

5. What Will We Do? Examine interesting problems. Devise algorithms to solve them. Prove these algorithms correct. Analyze their resource usage (time, memory). Understand data structures & core algorithms. Learn problem-solving techniques. See applications in real-world situations.

6. Goals Be very familiar with a collection of core algorithms. Be fluent in algorithm design paradigms: divide & conquer, greedy algorithms, dynamic programming. Be able toanalyze the correctness and runtime performance of a given algorithm. Be familiar with the inherent complexity (lower bounds & intractability) of some problems. Be intimately familiar with basic data structures. Be able to apply techniques in practical problems.

7. Textbook & References • Introduction to Algorithms, 2nd Ed.by Cormen, Leiserson, Rivest, & Stein (CLRS), McGraw Hill, 2002. OTHER REFERENCES: • Algorithmics: The Spirit of Computing, Harel • How to Solve It, Polya. • The Design and Analysis of Computer Algorithms, Aho, Hopcroft and Ullman. • Algorithms, Sedgewick. • Algorithmics: Theory & Practice, Brassard & Bratley. • Writing Efficient Programs & Programming Pearls, Bentley. • The Science of Programming, by Gries. • The Craft of Programming, by Reynolds.

8. How to Succeed in this Course Start early on all assignments. DON'T procrastinate. Complete all reading before class. Participate in class. Think in class. Review after each class. Be formal and precise on all problem sets and in-class exams.

9. JesminAkhter Lecturer,IIT Jahangirnagar University, Savar, DhakaBangladesh IT 301: Algorithm AnalysisLecture-01

10. What is an Algorithm?(And how do we analyze one?)

11. What is an Algorithm • An algorithm is a finitesequence ofunambiguous instructions for solving a well-specified computational problem. • All algorithm must satisfy the following criteria: • Input. • Output. • Definiteness. • Effectiveness. • Finiteness.

12. Definiteness Effectiveness Finiteness Input Output What is an Algorithm • A technique for solving a well-specified computational problem. • Example: sorting input: A sequence of numbers. output: An ordered permutation of the input.

13. What is an Algorithm • Definiteness: Each instruction is clear and unambiguous. • Effectiveness: Each instruction is executable; in other words, feasibility. • Finiteness: The algorithm terminates after a finite number of steps.

14. Algorithm Analysis • Determining performance characteristics. (Predicting the resource requirements.) • Time, memory, communication bandwidth etc. • Computation time(running time) is of primary concern. • Why analyze algorithms? • Choose the most efficient of several possible algorithms for the same problem. • Is the best possible running time for a problem reasonably finite for practical purposes? • Is the algorithm optimal(best in some sense)? – Is something better possible?

15. Running Time • Run time expression should be machine-independent. • Use a model of computation or “hypothetical” computer. • Our choice – RAM model (most commonly-used). • Model should be • Simple. • Applicable.

16. RAM Model • Generic single-processor model. • Supports simpleconstant-timeinstructions found in real computers. • Arithmetic (+, –, *, /, %, floor, ceiling). • Data Movement (load, store, copy). • Control (conditional and unconditional branch, subroutine call). • Data types: Integer and floating point. • Run time (cost) is uniform (1 time unit) for all simple instructions. • Memory is unlimited. • Flat memory model – no hierarchy. • Access to a word of memory takes 1 time unit. • Sequential execution – no concurrent operations.

17. Running Time – Definition • Call each simple instruction and access to a word of memory a “primitive operation” or “step.” • Running time of an algorithm for a given input is • The number of steps executed by the algorithm on that input. • Often referred to as the complexity of the algorithm.

18. Complexity and Input • Complexity of an algorithm generally depends on • Size of input. • Input size depends on the problem. • Examples: No. of items to be sorted. • No. of vertices and edges in a graph. • Other characteristics of the input data. • Are the items already sorted? • Are there cycles in the graph?

19. Worst, Average, and Best-case Complexity • Worst-case Complexity • Maximum steps the algorithm takes for any possible input. • Most tractable measure. • Average-case Complexity • Average of the running times of all possible inputs. • Demands a definition of probability of each input, which is usually difficult to provide and to analyze. • Best-case Complexity • Minimum number of steps for any possible input. • Not a useful measure. Why?

20. A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for. OUTPUT:true if key occurs in the sequence, false otherwise. • LinearSearch(A, key) cost times • 1 i  1c1 1 • 2 whilei ≤ nand A[i] != key c2x • 3do i++ c3x-1 • ifi  nc4 1 • then return true c5 1 • else return false c6 1 x ranges between 1 and n+1. So, the running time ranges between c1+ c2+ c4+ c5– best case and c1+ c2(n+1)+ c3n + c4+ c6– worst case

21. A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for. OUTPUT: true if key occurs in the sequence, false otherwise. • LinearSearch(A, key) cost times • 1 i  11 1 • 2 whilei ≤ nand A[i] != key 1 x • 3do i++ 1 x-1 • ifi  n11 • then return true 11 • else return false 1 1 Assign a cost of 1 to all statement executions. Now, the running time ranges between 1+ 1+ 1+ 1 = 4 – best case and 1+ (n+1)+ n + 1+ 1 = 2n+4 – worst case

22. A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for. OUTPUT: true if key occurs in the sequence, false otherwise. • LinearSearch(A, key) cost times • 1 i  11 1 • 2 whilei ≤ nand A[i] != key 1 x • 3do i++ 1 x-1 • ifi  n11 • then return true 11 • else return false 1 1 If we assume that we search for a random item in the list, on an average, Statements 2 and 3 will be executed n/2 times. Running times of other statements are independent of input. Hence, average-case complexity is 1+ n/2+ n/2 + 1+ 1 = n+3

23. Order of growth • Principal interest is to determine • how running time grows with input size – Order of growth. • the running time for large inputs – Asymptotic complexity. • In determining the above, • Lower-order terms and coefficient of the highest-order term are insignificant. • Ex: In insertion sort the worst case running time : an2+bn+c, which term dominates the running time for very large n? • Complexity of an algorithm is denoted by the highest-order term in the expression for running time. • Ex: Θ(n2) • Constant complexity when running time is independent of the input size – denoted Ο(1). • Linear Search: Best caseΘ(1), Worst and Average cases: Θ(n).

24. Order of growth • One algorithm to be more efficient than other • if its worst case running time has a lower order of growth. • Due to constant factors and lower order terms • an algorithm whose running time has a higher order of growth might take less time for small inputs than an algorithm whose running time has a lower order of growth. • But for large enough inputs, a Θ(n2) algorithm will run more quickly than a Θ(n3) algorithm .

25. Comparison of Algorithms • Complexity function can be used to compare the performance of algorithms. • Algorithm A is more efficient than Algorithm B for solving a problem, if the complexity function of A is of lower order than that of B. • Examples: • Linear Search – (n) vs. Binary Search – (lg n) • Insertion Sort – (n2) vs. Quick Sort – (n lg n)

26. Comparisons of Algorithms • Sorting • For 106 numbers, • insertion sort: (n2): it took 5.56 hrs on a supercomputer using machine language • merge sort: (n lg n): It took 16.67 min on a PC using C/C++.

27. Efficiency of Algorithms • Computer speed is not infinite and memory is not free so we need efficient algorithms. • Algorithms efficiency can be determined by time and space it need. • For example, insertion sort takes time equal to C1n2 to sort n items while merge sort takes C2nlogn. • Which one would work faster depends on size of n.

28. Example of Efficiency • Computer A is 1000 times faster than B. • A takes 2n2time to sort n items using insertion sort • B takes 50logn time to sort n items using merge sort

29. Correctness Proofs • Proving (beyond “any” doubt) that an algorithm is correct. • Prove that the algorithm produces correct output when it terminates. Partial Correctness. • Prove that the algorithm will necessarily terminate. Total Correctness. • Techniques • Proof by Construction. • Proof by Induction. • Proof by Contradiction.

30. Loop Invariant • Logical expression with the following properties. • Holds true before the first iteration of the loop – Initialization. • If it is true before an iteration of the loop, it remains true before the next iteration – Maintenance. • When the loop terminates, the invariant gives a useful property that helps show that the algorithm is correct – Termination. • Similar to mathematical induction.

31. Correctness Proof of Linear Search • Use Loop Invariant for the while loop: • At the start of each iteration of the while loop, the search key is not in the subarray A[1..i-1]. • LinearSearch(A, key) • 1 i  1 • 2 whilei ≤ nand A[i] != key • 3do i++ • ifi  n • then return true • else return false • If the algm. terminates, then it produces correct result. • Initialization. • Maintenance. • Termination. • Argue that it terminates.

32. Go through correctness proof of insertion sort in the text.(2.1)(18)

33. The problem of sorting Input: sequence áa1, a2, …, anñ of numbers. Output: permutation áa'1, a'2, …, a'nñsuch that a'1£ a'2£ …£ a'n. Example: Input:8 2 4 9 3 6 Output:2 3 4 6 8 9

34. Insertion sort INSERTION-SORT (A, n) ⊳ A[1 . . n] forj ← 2ton dokey ← A[ j] i ← j – 1 while i > 0 and A[i] > key doA[i+1] ← A[i] i ← i – 1 A[i+1] = key “pseudocode” 1 i j n A: key sorted

35. Example of insertion sort 8 2 4 9 3 6

36. Example of insertion sort 8 2 4 9 3 6

37. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6

38. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6

39. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6

40. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6

41. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 4 8 9 3 6

42. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 4 8 9 3 6

43. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 4 8 9 3 6 2 3 4 8 9 6

44. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 4 8 9 3 6 2 3 4 8 9 6

45. Example of insertion sort 8 2 4 9 3 6 2 8 4 9 3 6 2 4 8 9 3 6 2 4 8 9 3 6 2 3 4 8 9 6 2 3 4 6 8 9 done