Provider and Peer Selection in the Evolving Internet Ecosystem

1. Provider and Peer Selection in the Evolving Internet Ecosystem Amogh Dhamdhere Committee: Dr. Constantine Dovrolis (advisor) Dr. Mostafa Ammar Dr. Nick Feamster Dr. Ellen Zegura Dr. Walter Willinger (AT&T Labs-Research)

2. The Internet Ecosystem • 27,000 autonomous networks independently operated and managed • The “Internet Ecosystem” • Different types of networks • Interact with each other and with “environment” • Network interactions • Localized, in the form of interdomain links • Competitive (customer-provider), symbiotic (peering) • Distributed optimizations by each network • Select providers and peers to optimize utility function • The Internet ecosystem evolves

3. High Level Questions • How does the Internet ecosystem evolve? • What is the Internet heading towards? • Topology • Economics • Performance • How do the strategies used by networks affect their utility (profits/costs/performance)? • How do these individual strategies affect the global Internet?

4. Previous Work • Static graph properties • No focus on how the graph evolves • “Descriptive” modeling • Match graph properties e.g. degree distribution • Homogeneity • Nodes and links all the same • Game theoretic, computational • Restrictive assumptions • Dynamics of the evolving graph • Birth/death • Rewiring • “Bottom-up” • Model the actions of individual networks • Heterogeneity • Networks with different incentives • Semantics of interdomain links

5. Our Approach “Measure – Model – Predict” Measure the evolution of the Internet Ecosystem Topological, but focus on business types and rewiring Model strategies and incentives of different network types Predict the effects of provider and peer selection strategies

6. Outline • Measuring the Evolution of the Internet Ecosystem • The Core of the Internet: Provider and Peer Selection for Transit Providers • The Edge of the Internet: ISP and Egress Path Selection for Stub Networks • ISP Profitability and Network Neutrality [IMC ’08] [to be submitted] [Infocom ‘06] [Netecon ’08]

7. Motivation • How did the Internet ecosystem evolve during the last decade? • Is growth more important than rewiring? • Is the population of transit providers increasing or decreasing? • Diversification or consolidation of transit market? • Given that the Internet grows in size, does the average path length also increase? • Where is the Internet heading?

8. Approach • Focus on Autonomous Systems (ASes) • As opposed to networks without AS numbers • Start from BGP routes from RouteViews and RIPE monitors during 1997-2007 • Focus on primary links • Classify ASes based on their business function • Enterprise ASes, small transit providers, large transit providers, access providers, content providers • Classify inter-AS relations as “transit” and “peering” • Transit link – Customer pays provider • Peering link – No money exchanged Visibility Issue

9. Internet growth • Number of CP links and ASes showed initial exponential growth until mid-2001 • Followed by linear growth until today • Change in trajectory followed stock market crash in North America in mid-2001

10. Path lengths stay constant • Number of ASes has grown from 5000 in 1998 to 27000 in 2007 • Average path length remains almost constant at 4 hops

11. Rewiring more important than growth • Most new links due to internal rewiring and not birth (75%) • Most dead links are due to internal rewiring and not death (almost 90%)

12. Classification of ASes based on business function • Four AS types: • Enterprise customers (EC) • Small Transit Providers (STP) • Large Transit Providers (LTP) • Content, Access and Hosting Providers (CAHP) • Based on customer and peer degrees • Classification based on decision-trees • 80-85% accurate

13. Evolution of AS types • LTPs: constant population (top-30 ASes in terms of customers) • Slow growth of STPs (30% increase since 2001) • EC and CAHP populations produce most growth • Since 2001: EC growth factor 2.5, CAHP growth factor 1.5

14. Multihoming by AS types • CAHPs have increased their multihoming degree significantly • On the average, 8 providers for CAHPs today • Multihoming degree of ECs almost constant (average < 2) • Densification of the Internet occurs at the core

15. Conjectures on the Evolution of peering • Peering by CAHPs has increased significantly • CAHPs try to get close to sources/destinations of content

16. Conclusions Where is the Internet heading? Initial exponential growth up to mid-2001, followed by linear growth phase Average path length practically constant Rewiring more important than growth Need to classify ASes according to business type ECs contribute most of the overall growth Increasing multihoming degree for STPs, LTPs and CAHPs Densification at the core CAHPs are most active in terms of rewiring, while ECs are least active

17. Outline • Measuring the Evolution of the Internet Ecosystem • The Core of the Internet: Provider and Peer Selection for Transit Providers • The Edge of the Internet: ISP and Egress Path Selection for Stub Networks • ISP Profitability and Network Neutrality

18. Modeling the Internet Ecosystem • From measurements: Significant rewiring activity • Especially by transit providers • Networks rewire connectivity to optimize a certain objective function • Distributed • Localized spatially and temporally • Rewiring by changing the set of providers and peers • What are the global, long-term effects of these distributed optimizations? • Topology and traffic flow • Economics • Performance (path lengths)

19. The Feedback Loop Routing Cost/price parameters Interdomain TM Interdomain topology Traffic flow Per-AS profit Provider selection • When does it converge? • When no network has the incentive to change its connectivity – “steady-state” Peer selection

20. Impact of provider/peer selection strategies P1 P1 open to peering with CPs C P2 and P3 peer No peering C P2 P3 S S C S C S

21. Our Approach • What is the outcome when networks use certain provider and peer selection strategies? • Model the feedback loop in the Internet ecosystem • Real-world economics of transit, peering, operational costs • Realistic routing policies • Geographical constraints • Provider and peer selection strategies • Computationally find a “steady-state” • No network has further incentive to change connectivity • Measure properties of the steady-state • Topology, traffic flow, economics

22. Network Types • Enterprise Customers (EC) • Stub networks at the edge (e.g. Georgia Tech) • Either sources or sinks • Small Transit Providers (STP) • Provide Internet transit • Mostly regional in presence (e.g. France Telecom) • Large Transit Providers (LTP) • Transit providers with global presence (e.g. AT&T) • Content Providers (CP) • Major sources of content (e.g. Google) Provider and peer selection for STPs and LTPs

23. What would happen if..? • The traffic matrix consists of mostly P2P traffic? • P2P traffic benefits STPs, can make LTPs unprofitable • LTPs peer with content providers? • LTPs could harm STP profitability, at the expense of longer end-to-end paths • Edge networks choose providers using path lengths? • LTPs would be profitable and end-to-end paths shorter

24. Provider and Peer Selection • Provider selection strategies • Minimize monetary cost (PR) • Minimize AS path lengths weighted by traffic (PF) • Avoid selecting competitors as providers (SEL) • Peer selection strategies • Peer only if necessary to maintain reachability (NC) • Peer if traffic ratios are balanced (TR) • Peer by cost-benefit analysis (CB) • Peer and provider selection are related

25. A ? X C B A B B A C U U C Provider and Peer Selection are Related • Restrictive peering • Peering by necessity • Level3-Cogent peering dispute

26. Economics, Routing and Traffic Matrix • Realistic transit, peering and operational costs • Transit prices based on data from Norton • Economies of scale • BGP-like routing policies • No-valley, prefer customer, prefer peer routing policy • Traffic matrix • Heavy-tailed content popularity and consumption by sinks • Predominantly client-server: Traffic from CPs to ECs • Predominantly peer-to-peer: Traffic between ECs

27. Algorithm for network actions • Networks perform their actions sequentially • Can observe the actions of previous networks • And the effects of those actions • Network actions in each move • Pick set of preferred providers • Attempt to convert provider links to peering links “due to necessity” • Evaluate each existing peering link • Evaluate new peering links • Networks make at most one change to their set of peers in a single move

28. Solving the Model • Determine the outcome as each network selects providers and peers according to its strategy • Too complex to solve analytically: Solve computationally • Typical computation • Proceeds iteratively, networks act in a predefined sequence • Pick next node n to “play” its possible moves • Compute routing, traffic flow, AS fitness • Repeat until no player has incentive to move • “steady-state” or equilibrium

29. Properties of the steady-state • Is steady-state always reached? • Yes, in most cases • Is steady-state unique? • No, can depend on playing sequence • Different steady-states have qualitatively similar properties • Multiple runs with different playing sequence • Average over different runs • Confidence intervals are narrow

30. Canonical Model • Parameterization of the model that resembles real world • Traffic matrix is predominantly client-server (80%) • Impact of streaming video, centralized file sharing services • 20% of ECs are content sources, 80% sinks • Heavy tailed popularity of traffic sources • Edge networks choose providers based on price • 5 geographical regions • STPs cheaper than LTPs

31. Model Validation • Reproduces almost constant average path length • Activity frequency: How often do networks change their connectivity? • ECs less active than providers – Qualitatively similar to measurement results

32. Results – Canonical Model LTP • Hierarchy of STPs • Traffic can bypass LTPs – LTPs unprofitable • STPs should not peer with CPs • Resist the temptation! S1 S2 CP EC EC CP CP

33. Results – Canonical Model • What-if: LTPs peer with CPs • Generate revenue from downstream traffic • Can harm STP fitness • Long paths CP LTP CP CP S1 S2 EC EC

34. Deviation 1: P2P Traffic matrix CP LTP CP • P2P traffic helps STPs • Smaller traffic volume from CPs to Ecs • More EC-EC traffic => balanced traffic ratios • More opportunities for STPs to peer • Peering by “traffic ratios” makes sense LTP CP S1 S1 S2 S2 S3 EC EC EC EC EC EC

35. Conclusions • A model that captures the feedback loop between topology, traffic and fitness in the Internet • Considers effects of • Economics • Geography • Heterogeneity in network types • Predict the effects of provider and peer selection strategies • Topology, traffic flow, economics, and performance

36. Outline • Measuring the Evolution of the Internet Ecosystem • The Core of the Internet: Peer and Provider Selection for Transit Providers • The Edge of the Internet: ISP and Egress Path Selection for Stub Networks • ISP Profitability and Network Neutrality

37. The Edge of the Internet • Sources and sinks of content • Content Providers (CP): sources • Enterprise Customers (EC): sinks • From measurements: • ECs connect increasingly to STPs • Cost conscious ? • CPs connect increasingly to LTPs • Performance ? • Increasing multihoming (about 60% of stubs) • Redundancy, load balancing, cost effectiveness • How should stub networks choose their providers?

38. Major Questions • How to select the set of upstream ISPs ? • Low monetary cost • Short AS paths to major destinations • Path diversity to major traffic destinations – robustness to network failures • How to allocate egress traffic to the set of selected ISPs ? • Objective: Avoid congestion on the upstream paths • Also maintain low cost

39. ISP Selection • Select k ISPs out of N • Let C be a subset of k ISPs out of N • Total cost of a selection of ISPs C: Weighted sum of monetary, path length and path diversity costs • Select combination C with minimum total cost • Feasible to enumerate all combinations

40. Monetary and Path Length Cost • For set of ISPs C, what is the monetary and path length cost of routing egress flows? • Find the minimum cost mapping G* of flows to ISPs (Bin Packing) • Flows = items • ISPs = bins • NP hard ! • Use First Fit Decreasing (FFD) heuristic • Mapping G* very close to optimal • Monetary and path length costs of C are calculated using the mapping G*

41. Path Diversity • selection C gives K paths to each destination d • K-shared link to d: link shared by all K paths to d • If a K-shared link fails, destination d is unreachable • Minimize the number of K-shared links • Path diversity metric: The number of k-shared links to destination d averaged over all destinations • Gives the best resiliency to single-link failures

42. Summary • Algorithms for ISP selection • Choosing best set of upstream ISPs • Objectives are minimum monetary cost, short AS paths and high path diversity • ISP selection for monetary and performance constraints • Formulated as a bin-packing problem • Heuristic gives solution very close to optimal • ISP selection for path diversity • Returns set of ISPs with best path diversity to the set of major destinations

43. Outline • Measuring the Evolution of the Internet Ecosystem • The Edge of the Internet: ISP and Egress Path Selection for Stub Networks • The Core of the Internet: Peer and Provider Selection for Transit Providers • ISP Profitability and Network Neutrality

44. The debate • Recent evolution trend: Large amounts of video and peer-to-peer traffic • Content providers (CP) generate the content • Provide content and services “over the top” of the basic connectivity provided by ISPs • Profitable (think Google) • Access Providers (AP) deliver content to users • Recent trend: Not profitable • Commoditization of basic Internet access • Want a share of the pie • Tension between AP and CPs: “Network neutrality”

45. A Technical View • Previous work • Mostly non-technical • Highly emotional debates in the press • Legislation/policy aspects: Do we need network neutrality legislation? • But what about the underlying problem: Non-profitability of Access Providers? • Our approach: A quantitative look at AP profitability • Investigate reasons for non-profitability • Evaluate strategies for remaining profitable

46. Modeling AP Profitability • Three AS types: AP, CP and transit provider (TP) • Focus on the AP • AS links • customer-provider (customer pays provider) • peering (no payments) • AP and CPs can transfer traffic either through customer-provider or peering links

47. AP Profitability • Reasons why APs can be unprofitable • AP users • The impact of video traffic • AP strategies: Pricing • Heavy hitter charging • Heavy hitter blocking • Non-neutral charging • AP strategies: Connection • Caching CP content • Peering selectively with CPs

48. Major Findings • Variability in AP users can cause large variability in costs • Video traffic: Increases costs for AP • AP strategies based on differential/non-neutral pricing may not succeed • Have to account for user departure due to competition • AP strategies based on connection are promising • Caching content from CPs • Peering selectively with large CPs

49. Contributions of this Thesis • A measurement study of the evolution of the Internet ecosystem • Modeling the evolution of the Internet ecosystem • “what-if” questions about possible evolution paths • Optimizations at the edge of the Internet • Algorithms for provider selection and egress routing • A technical view of the network neutrality debate • Strategies for ISP profitability

50. Future Directions • Measurements: Investigate the evolution of the connectivity for monitor ASes • We can observe all links for such ASes • Focus on transitions between peering and customer-provider links • Measurements: What does the interdomain traffic matrix really look like? • Can we use measurements from a large Tier-1 provider? • Can we augment that data with information about the interdomain topology?