Discrete Mathematics: Growth of Functions

1. Discrete Mathematics: Growth of Functions

2. Section Summary • Big-O Notation • Big-O Estimates for Important Functions • Big-Omega:  • Big-Theta Notation: 

3. The Growth of Functions • In both computer science and in mathematics, there are many times when we care about how fast a function grows. • In computer science, we want to understand how quickly an algorithm can solve a problem as the size of the input grows. • We can compare the efficiency of two different algorithms for solving the same problem. • We can also determine whether it is practical to use a particularalgorithm as the input grows.

4. Big-O Notation Definition: let f and g be functions from the set of integers or the set of real numbers to the set of real numbers. We say that f(x) is O(g(x)) if there are constants C and k such that whenever x > k. • This is read as “f(x) is big-O of g(x)” or “g asymptotically dominates f.” • The constants C and k are called witnesses to the relationship f (x) O(g(x)). Only one pair of witnesses is needed.

5. Illustration of Big-O Notation f (x) O(g(x))

6. Some Points about Big-O Notation • If one pair of witnesses is found, then there are infinitely many pairs. • You may see “f(x) = O(g(x))” instead of “f(x) O(g(x)).” • But “f(x) = O(g(x))” is an abuse of the equals sign since the meaning is that there is an inequality relating the values of f and g, for sufficiently large values of x. • It is ok to write f(x) ∊O(g(x)), since O(g(x)) represents the set of functions that are O(g(x)). • Usually, we will drop the absolute value sign since we will always deal with functions that take on positive values.

7. Using the Definition of Big-O Notation Example: show that is . Solution: since when x > 1, x < x2 and 1 < x2 • Can take C = 4and k = 1 as witnesses to show that • Alternatively, when x > 2, we have 2x<x2 and 1 < x2: • Can take C = 3and k = 2 as witnesses instead.

8. Illustration of Big-O Notation C g(x) f (x) g(x) 

9. Big-O Notation • Both and are such that we say that the two functions are of the same order. • If f (x)  C (g(x))and g(x) h(x) for all positive real numbers, then • Note that if |f (x)|  C |g(x)| for x > k and if |g(x)|  |h(x)| for all x > k, then |f (x)|  C|h(x)| if x > k. Hence, • For many applications, the goal is to select the function g(x) for O(g(x))as small as possible (up to multiplication by a constant, of course).

10. Using the Definition of Big-O Notation Example: show that 7x2O(x3). Solution: when x > 7, 7x2 < x3. TakeC=1andk = 7as witnesses to establish that 7x2O(x3). Example: show that x2O(x). Solution: suppose there are constants C and k for which x2≤C x, whenever x > k. We divide both sides of x2≤C x by x when x> 0, then x≤Cmust hold for all x > k. A contradiction!

11. Big-O Estimates for Polynomials Example: let f (x) = anxn+ an-1 xn-1 + ∙∙∙+ a1x1 + a0 where an,an-1, …, a0 are real numbers with an ≠0. Then f(x) is O (xn). Proof: |f(x)| = |anxn+ an-1 xn-1 + ∙∙∙+ a1x1 + a0| ≤ |an|xn + |an-1| xn-1 + ∙∙∙+ |a1|x1+ |a0| = xn(|an| + |an-1| /x+ ∙∙∙+ |a1|/xn-1+ |a0|/xn) ≤ xn(|an| + |an-1| + ∙∙∙+ |a1|+ |a0|) • Take C = |an| + |an-1| + ∙∙∙+ |a1|+ |a0| and k = 1. Then f (x) is O (xn). In fact, the leading term anxn of a polynomial dominates its growth. Assuming x > 1

12. Big-O Estimates for Major Functions Example-1: use big-O notation to estimate the sum of the first n positive integers Solution: Example-2: use big-O notation to estimate the factorial function Solution: Example-3: use big-O notation to estimate log(n!) Solution: given that then Hence, log(n!) O (n ∙ log (n )) taking C = 1 and k = 1.

13. Display of Growth of Functions Note the difference in behavior of functions as n gets larger

14. Useful Big-O Estimates • If d > c > 1, then ncO(nd), but ndO(nc). • If b > 1 and c and d are positive, then (we know log n  O(n)) (log b n)cO(nd), but nd O((logb n)c). • If b > 1 and d is positive, then (e.g., consider n2 and 2n for n>4) ndO(bn), but bnO(nd). • If c > b > 1, then (bases are different) bnO(cn), but cnO(bn).

15. Combinations of Functions • If f1 (x) O(g1(x)) and f2 (x) O(g2(x)) then ( f1 + f2 )(x) O(max(|g1(x) |,|g2(x) |)). • If f1 (x) O(g(x)) and f2 (x) O(g(x)) then ( f1 + f2 )(x) O(g(x)). • If f1 (x) O(g1(x)) and f2 (x) O(g2(x)) then ( f1f2 )(x) O(g1(x)g2(x)).

16. Ordering Functions by Order of Growth • Put the functions below in order so that each function is big-O of the next function on the list. • f1(n) = (1.5)n • f2(n) = 8n3+17n2 +111 • f3(n) = (log n )2 • f4(n) = 2n • f5(n) = log (log n) • f6(n) = n2 (log n)3 • f7(n) = 2n(n2 +1) • f8(n) = n3+ n(log n)2 • f9(n) = 10000 • f10(n) = n! • f9(n) = 10000 (constant, does not increase with n) • f5(n) = log (log n) (grows slowest of all the others) • f3(n) = (log n)2(grows next slowest) • f6(n) = n2 (log n)3((log n)3 factor smaller than any power of n) • f2(n) = 8n3+17n2 +111 (tied with the one below) • f8(n) = n3+ n(log n)2 (tied with the one above) • f1(n) = (1.5)n (an exponential function) • f4(n) = 2n (grows faster than one above since 2 > 1.5) • f7(n) = 2n(n2 +1) (faster than above because of the n2 +1 factor) • f10(n) = 3n ( n! grows faster than cnfor every c)

17. Big-Omega Notation:  Definition: let f and g be functions from the set of integers or the set of real numbers to the set of real numbers. We say that if there are constants C and k such that (when x > k): • We say that “f(x) is big-Omega of g(x).” Big-Omega tells us that a function grows at least as fast as another. • Big-O gives an upper bound on the growth of a function, while Big-Omega gives a lower bound. • f (x)  Ω(g(x))if and only if g(x) O(f (x)).

18. Big-Omega Notation Example: show that is where . Solution: for all positive real numbers x. • Is it also the case that is ?

19. Big-Theta Notation:  Definition: let f and g be functions from the set of integers or the set of real numbers to the set of real numbers. The function if and • We say that “f is big-Theta of g(x)” and also that “f (x) is of orderg(x)” and also that “f (x) and g(x) are of the same order.” • if and only if there exists constants C1,C2 and k such that C1g(x)<f (x) <C2 g(x) if x > k. This follows from the definitions of big-O and big-Omega.

20. Big Theta Notation Example: the sum of the first n positive integers is Θ(n2). Solution: let f (n) = 1 + 2 + ∙∙∙ + n. • We have already shown that f (n) O(n2); see slide 12. • To show that f (n) is Ω(n2), we need a positive constant C such that f (n) > Cn2 for sufficiently large n. Summing only the terms greater thann/2 we obtain the inequality 1 + 2 + ∙∙∙ + n≥ ⌈ n/2⌉ + (⌈ n/2⌉ + 1) + ∙∙∙ + n ≥ ⌈ n/2⌉ + ⌈ n/2⌉ + ∙∙∙ + ⌈ n/2⌉ = (n − ⌈ n/2⌉ + 1 ) ⌈ n/2⌉ ≥ (n/2)(n/2) = n2/4 • Taking C = ¼, f(n) > Cn2 for all positive integers n. Hence, f(n) Ω(n2), and we can conclude that f (n) Θ(n2). Lower bound

21. Big-Theta Notation Example: sh0w that f (x) = 3x2+ 8x(log x)is Θ(x2). Solution: • 3x2+ 8x(log x)≤ 11x2 for x > 1, since 0 ≤8xlog x ≤ 8x2 • Hence, 3x2+ 8x(log x)is O(x2). • x2 is clearly(3x2 + 8x(log x)) • Hence, 3x2+ 8x(log x)is Θ(x2).

22. Big-Theta Notation • When f (x)  Θ(g (x)) it must also be the case that g (x)  Θ(f (x)) • Note that f (x)  Θ(g (x)) if and only if it is the case that f (x)  O(g (x)) and g(x)  O(f (x)) • Theorem: let where are real numbers with an≠0. Then f (x)  Θ(xn)