Orders of Growth

1. Orders of Growth Rosen 5th ed., §2.2

2. Orders of Growth (§2.2) • For functions over numbers, we often need to know a rough measure of how fast a function grows. • If f(x) is faster growing than g(x), then f(x) always eventually becomes larger than g(x) in the limit (for large enough values of x). • Useful in engineering for showing that one design scales better or worse than another. First of all, what is order of growth? It is a gross measure of the resources required by an algorithm as the input size becomes larger. Suppose algorithm A to solve a problem requires a complexity in the form of function f(x) where x is the input size. Likewise, another algorithm B for the same problem will require the complexity of g(x).

3. Orders of Growth - Motivation • Suppose you are designing a web site to process user data (e.g., financial records). • Suppose database program A takes fA(n)=30n+8 microseconds to process any n records, while program B takes fB(n)=n2+1 microseconds to process the n records. • Which program do you choose, knowing you’ll want to support millions of users? There are two kinds of programs or algorithms. The answer is obvious. Everybody (who can do the math) will choose the program A.

4. Visualizing Orders of Growth fA(n)=30n+8 • On a graph, asyou go to theright, a fastergrowingfunctioneventuallybecomeslarger... Value of function  fB(n)=n2+1 Increasing n  This plotting shows how fast each algorithm’s complexity increases. I guess two functions will meet around 30 or 31.

5. Concept of order of growth • We say fA(n)=30n+8is order n, or O(n). It is, at most, roughly proportional to n. • fB(n)=n2+1 is order n2, or O(n2). It is roughly proportional to n2. • Any O(n2) function is faster-growing than any O(n) function. • For large numbers of user records, the O(n2) function will always take more time. Why do we use this notation O(n)? The exact expression of complexity of algorithm is somewhat long and perplexing. So we want to grasp the essence of the complexity. That’s why we use O(n) notation and let’s see how we can derive the O() notation from the original complexity equation.

6. Definition: O(g), at mostorder g Let g be any function RR. • Define “at most order g”, written O(g), to be: {f:RR | c,k: x>k: f(x) cg(x)}. • “Beyond some point k, function f is at most a constant c times g (i.e., proportional to g).” • “f is at most order g”, or “f is O(g)”, or “f=O(g)” all just mean that fO(g). • Sometimes the phrase “at most” is omitted.

7. Points about the definition • Note that f is O(g) so long as any values of c and k exist that satisfy the definition. • But: The particular c, k, values that make the statement true are not unique: Any larger value of c and/or k will also work. • You are not required to find the smallest c and k values that work. You should prove that the values you choose do work.

8. “Big-O” Proof Examples • Show that 30n+8 is O(n). • Show c,k: n>k:30n+8  cn. • Let c=31, k=8. Assume n>k=8. Thencn = 31n = 30n + n > 30n+8, so 30n+8 < cn. • Show that n2+1 is O(n2). • Show c,k: n>k: n2+1  cn2. • Let c=2, k=1. Assume n>1. Then cn2 = 2n2 = n2+n2 > n2+1, or n2+1< cn2. When we simplify the complexity by the big-o notation, we normally focus on the highest-order term. Why? Because it will signify how fast the function will change In the first case, the highest-order term is linear term In the 2nd case, the highest-order term is quadratic term

9. cn =31n n>k=8  Big-O example, graphically • Note 30n+8 isn’tless than nanywhere (n>0). • It isn’t evenless than 31neverywhere. • But it is less than31neverywhere tothe right of n=8. 30n+8 30n+8O(n) Value of function  n Increasing n 

10. In general • an xn + ... + a1 x + a0 is O( xn ) for any real numbers an , ..., a0 and any nonnegative number n . A polynomial function of a variable x of a degree n will have this form in general.

11. Useful Facts about Big O • Big O, as a relation (ch. 7), is transitive: fO(g)  gO(h)  fO(h) • Sums of functions:If gO(f) and hO(f), then g+hO(f).

12. More Big-O facts • c>0, O(cf)=O(f+c)=O(fc)=O(f) • f1O(g1)  f2O(g2)  • f1 f2 O(g1g2) • f1+f2 O(g1+g2) = O(max(g1,g2)) = O(g1) if g2O(g1)

13. Useful Facts about Big O •  f,g & constantsa,bR, with b0, • af = O(f); (e.g. 3x2 = O(x2)) • f+O(f) = O(f); (e.g. f = x2+x = O(x2)) • Also, if f=(1) (at least order 1), then: • |f|1-b = O(f); (e.g.x1 = O(x)) • (logb |f|)a= O(f). (e.g. log x = O(x)) • g=O(fg) (e.g.x = O(x log x)) • fg O(g) (e.g.x log x  O(x)) • a=O(f) (e.g. 3 = O(x))

14. Definition: (g), exactly order g • If fO(g) and gO(f) then we say “g and f are of the same order” or “f is (exactly) order g” and write f(g). • Another equivalent definition:(g)  {f:RR | c1c2kR+ x>k: |c1g(x)||f(x)||c2g(x)| } • “Everywhere beyond some point k, f(x)lies in between two multiples of g(x).”

15. Rules for  • Mostly like rules for O( ), except: •  f,g>0 & constantsa,bR, with b>0,af (f), but  Same as with O.f  (fg) unless g=(1)  Unlike O.|f|1-b (f), and  Unlike with O.(logb |f|)c (f).  Unlike with O. • The functions in the latter two cases we say are strictly of lower order than (f).

16. Other Order-of-Growth Relations • (g) = {f | gO(f)}“The functions that are at least orderg.” • o(g) = {f | c>0 k x>k : |f(x)| < |cg(x)|}“The functions that are strictly lower order than g.” o(g)  O(g)  (g). • (g) = {f | c>0 k x>k : |cg(x)| < |f(x)|}“The functions that are strictly higher order than g.” (g)  (g)  (g).

17. O:Big-Oh (≤) • f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n) • :big-Omega (≥) • f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n) • :big-Theta (=) • f(n) is (g(n)) if f(n) is asymptotically equal to g(n) • o:little-oh (<) • f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n) • :little-omega (>) • f(n) is (g(n)) if is asymptotically strictly greater than g(n)

18. Ordering functions by growth rate f (n) is o (g (n)) if lim n→∞(f (n) / g (n)) = 0 log log n log n log2 n sqrt n n n log n n2 nk 2n n! nn Hint: Can use L’Hopital’s rule O(nk): tractable O(2n ), O(n!): intractable

19. Strict Ordering of Functions • let’s write fg to mean fo(g) f~g to mean f(g) • Note that • Let k>1. Then the following are true:1  log log n log n ~ logkn logknn1/k n n log nnkkn n! nn…

20. Relations Between the Relations • Subset relations between order-of-growth sets. RR ( f ) O( f ) • f o( f ) ( f ) ( f )

21. Why o(f)O(x)(x) • A function that is O(x), but neither o(x) nor (x):

22. Review: Orders of Growth (§1.8) Definitions of order-of-growth sets, g:RR • O(g)  {f|c>0 k x>k |f(x)|  |cg(x)|} • o(g)  {f | c>0 k x>k |f(x)| < |cg(x)|} • (g)  {f | gO(f)} • (g)  {f | go(f)} • (g)  O(g)  (g)