1 / 28

Data-Driven Network Analysis:  Do You Really Know Your Data?

Data-Driven Network Analysis:  Do You Really Know Your Data?. Walter Willinger AT&T Labs-Research walter@research.att.com. Heard about “Network Science”?. Recent “hot topic” area in science Thousands of papers, many in high-impact journals such as Science or Nature

halla-moss
Download Presentation

Data-Driven Network Analysis:  Do You Really Know Your Data?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data-Driven Network Analysis:  Do You Really Know Your Data? Walter Willinger AT&T Labs-Research walter@research.att.com

  2. Heard about “Network Science”? • Recent “hot topic” area in science • Thousands of papers, many in high-impact journals such as Science or Nature • Interdisciplinary flavor: (Stat.) Physics, Math, CS • Main apps: Internet, social science, biology, … • Offers an alluring new recipe for doing network analysis • Largely measurement-driven • Main focus is on universal properties • Exploiting the predictive power of simple models • small world networks: clustering and path lengths • scale free networks: power law degree distributions • Emphasis on self-organization and emergence

  3. NETWORK SCIENCE January, 2006 • “First, networks lie at the core of the economic, political, and social fabric of the 21st century.” • “Second, the current state of knowledge about the structure, dynamics, and behaviors of both large infrastructure networks and vital social networks at all scales is primitive.” • “Third, the United States is not on track to consolidate the information that already exists about the science of large, complex networks, much less to develop the knowledge that will be needed to design the networks envisaged…” http://www.nap.edu/catalog/11516.html

  4. Network Science • What? “The study of network representations of physical, biological, and social phenomena leading to predictive models of these phenomena.” (National Research Council Report, 2006) • Why? “To develop a body of rigorous results that will improve the predictability of the engineering design of complex networks and also speed up basic research in a variety of applications areas.” (National Research Council Report, 2006) • Who? • Physicists (statistical physics), mathematicians (graph theory), computer scientists (algorithm design), etc.

  5. As Internet researchers, why should we care? • The teaching of “Network Science”

  6. The “New Science of Networks”

  7. Why should we care? • The teaching of “Network Science” • The claims “Network Science” makes about the Internet • High-degree nodes form a hub-like core • Fragile/vulnerable to targeted node removal • Achilles’ heel • Zero epidemic threshold • Network Science and the Internet • Lies, damned lies, statistics … • Rich source for wrong/bad models/theories • The published claims about the Internet are not “controversial” – they aresimply wrong!

  8. What is wrong with “Network Science”? • No critical assessment of available data • Ignores all networking-related “details” • Overarching desire to reproduce observed properties of the data even though the quality of the data is insufficient to say anything about those properties with sufficient confidence • Reduces model validation to the ability to reproduce an observed statistics of the data (e.g., node degree distribution)

  9. How to fix “Network Science”? • Know your data! • Importance of data hygiene • Take model validation more serious! • Model validation ≠ data fitting • Apply an engineering perspective to engineered systems! • Design principles vs. random coin tosses

  10. Some illustrative Examples • Example 1 • Data: Traceroute measurements • Objective: Inferring Internet topology at the router-level • Example 2 • Data: Traceroute measurements • Objective: Inferring Internet topology at the level of Autonomous Systems (ASes) • Example 3 • Data: BGP measurements • Objective: Inferring Internet topology at the level of Autonomous Systems (ASes)

  11. Measurement tool: traceroute • traceroute www.duke.edu • traceroute to www.duke.edu (152.3.189.3), 30 hops max, 60 byte packets • 1 fp-core.research.att.com (135.207.16.1) 2 ms 1 ms 1 ms • 2 ngx19.research.att.com (135.207.1.19) 1 ms 0 ms 0 ms • 3 12.106.32.1 1 ms 1 ms 1 ms • 4 12.119.12.73 2 ms 2 ms 2 ms • 5 tbr1.n54ny.ip.att.net (12.123.219.129) 4 ms 5 ms 3 ms • 6 ggr7.n54ny.ip.att.net (12.122.88.21) 3 ms 3 ms 3 ms • 7 192.205.35.98 4 ms 4 ms 8 ms • 8 jfk-core-02.inet.qwest.net (205.171.30.5) 3 ms 3 ms 4 ms • 9 dca-core-01.inet.qwest.net (67.14.6.201) 11 ms 11 ms 11 ms • 10 dca-edge-04.inet.qwest.net (205.171.9.98) 11 ms 15 ms 11 ms • 11 gw-dc-mcnc.ncren.net (63.148.128.122) 18 ms 18 ms 18 ms • 12 rlgh7600-gw-to-rlgh1-gw.ncren.net (128.109.70.38) 18 ms 18 ms 18 ms • 13 roti-gw-to-rlgh7600-gw.ncren.net (128.109.70.18) 20 ms 20 ms 20 ms • 14 art1sp-tel1sp.netcom.duke.edu (152.3.219.118) 23 ms 20 ms 20 ms • 15 webhost-lb-01.oit.duke.edu (152.3.189.3) 21 ms 38 ms 20 ms • 1 traceroute measurement: about 1KB

  12. Large-scale traceroute experiments 1 million x 1 million traceroutes: 1PB

  13. Two Examples of inferred ISP topology http://www.isi.edu/scan/mercator/mercator.html

  14. About the Traceroute tool (1) • traceroute is strictly about IP-level connectivity • Originally developed by Van Jacobson (1988) • Designed to trace out the route to a host • Using traceroute to map the router-level topology • Engineering hack • Example of what we can measure, not what we want to measure! • Basic problem #1: IP alias resolution problem • How to map interface IP addresses to IP routers • Largely ignored or badly dealt with in the past • New efforts in 2008 for better heuristics …

  15. Interfaces 1 and 2 belong to the same router

  16. IP Alias Resolution Problem for Abilene (thanks to Adam Bender)

  17. About the Traceroute tool (2) • traceroute is strictly about IP-level connectivity • Basic problem #2: Layer-2 technologies (e.g., MPLS, ATM) • MPLS is an example of a circuit technology that hides the network’s physical infrastructure from IP • Sending traceroutes through an opaque Layer-2 cloud results in the “discovery” of high-degree nodes, which are simply an artifact of an imperfect measurement technique. • This problem has been largely ignored in all large-scale traceroute experiments to date.

  18. (a) (b)

  19. About the Traceroute tool (3) • The irony of traceroute measurements • The high-degree nodes in the middle of the network that traceroute reveals are not for real … • If there are high-degree nodes in the network, they can only exist at the edge of the network where they will never be revealed by generic traceroute-based experiments … • Additional irony • Bias in (mathematical abstraction of) traceroute • Has been a major focus within CS/Networking literature • Non-issue in the presence of above-mentioned problems

  20. Example 1: Lessons learned • Know your measurement technique! • Question: Can you trust the data obtained by your tool? • Know your data! • Critical role of Data Hygiene in the Petabyte Age • Corollary: Petabytes of garbage = garbage • Data hygiene is often viewed as “dirty/unglamorous” work • Question: Can the data be used for the purpose at hand? • Regarding Example 1: • (Current) traceroute measurements are of (very) limited use for inferring router-level connectivity • It is unlikely that future traceroute measurements will be more useful for the purpose of router-level inference

  21. A textbook example for what can go wrong … • J.-J. Pansiot and D. Grad, “On routes and multicast trees in the Internet,” ACM Computer Communication Review 28(1), 1998. • Original traceroute data -- purpose for using the data is explicitly stated • Most of the issues with traceroute are listed! • M. Faloutsos, P. Faloutsos, and C. Faloutsos, “On the power-law relationships of the Internet topology”, Proc. ACM SIGCOMM’99, 1999. • Rely on the Pansiot-Grad data, but use it for a very different purpose • Take the available data at face value, even though Pansiot/Grad list most of the problems • There is no scientific basis for the reported power-law findings! • R. Albert, H. Jeong, and A.-L. Barabasi, “Error and attack tolerance of complex networks”, Nature, 2000. • Do not even cite original data source (i.e., Pansiot/Grad) • Take the results of FFF’99 at face value • The reported results are all wrong!

  22. Applying lessons to Example 2 • Example 2: Use of traceroute measurements to infer Internet topology at the level of Autonomous Systems (ASes) • Know your measurement technique! • traceroute (see Example 1) • Know your data! • Main source of errors: IP address sharing between BGP neighbors makes mapping traceroute paths to AS paths very difficult • Up to 50% of traceroute-derived AS adjacencies appear to be bogus

  23. Applying lessons to Example 2 (cont.) • Regarding Example 2 • (Current) traceroute measurements are of (very) limited use for inferring AS-level connectivity • Obtaining the “ground truth” is very challenging • It is possiblethat in the future, more targeted traceroute measurements in conjunction with BGP data will be more useful for the purpose of inferring AS-level connectivity

  24. Applying lessons to Example 3 • Example 3: Use of BGP data to infer Internet topology at the level of Autonomous Systems (ASes) • Know your measurement technique! • BGP -- de facto inter-domain routing protocol • BGP -- designed to propagate reachability information among ASes, not connectivity information • Engineering hack – not designed to obtain connectivity information • Example of what we can measure, not what we want to measure! • Collect BGP routing information base (RIB) information from as many routers as possible

  25. Applying lessons to Example 3 (cont.) • Know your data! • Examining the hygiene of BGP measurements requires significant commitment and domain knowledge • Parts of the available data seem accurate and solid (i.e., customer-provider links, nodes) • Parts of the available data are highly problematic and incomplete (i.e., peer-to-peer links) • “Ground truth” is hard to come by • Regarding Example 3 • (Current) BGP-based measurements are of questionable quality for inferring AS-level connectivity • Obtaining the “ground truth” is very challenging • It is possiblethat in the future, more targeted traceroute measurements in conjunction with BGP data will be more useful for the purpose of inferring AS-level connectivity

  26. A Reminder • Data-driven network analysis in the presence of high-quality data that can be taken at face value • “All models are wrong … but some are useful” (G.E.P. Box) • Data-driven network analysis in the presence of highly ambiguous data that should not be taken at face value • “When exactitude is elusive, it is better to be approximately right than certifiably wrong.” (B.B. Mandelbrot)

  27. SOME RELATED REFERENCES • L. Li, D. Alderson, W. Willinger, and J. Doyle, A first-principles approach to understanding the Internet’s router-level topology,Proc. ACM SIGCOMM 2004. • J.C. Doyle, D. Alderson, L. Li, S. Low, M. Roughan, S. Shalunov, R. Tanaka, and W. Willinger. The "robust yet fragile" nature of the Internet.PNAS 102(41), 2005. • D. Alderson, L. Li, W. Willinger, J.C. Doyle. Understanding Internet Topology: Principles, Models, and Validation.ACM/IEEE Trans. on Networking 13(6), 2005. • L. Li, D. Alderson, J.C. Doyle, W. Willinger. Toward a Theory of Scale-Free Networks: Definition, Properties, and Implications. Internet Mathematics 2(4), 2006. • R. Oliveira, D. Pei, W. Willinger, B. Zhang, L. Zhang. In Search of the elusive Ground Truth: The Internet's AS-level Connectivity Structure.Proc. ACM SIGMETRICS 2008. • B. Krishnamurthy and W. Willinger.What are our standards for validation of measurement-based networking research? Proc. ACM HotMetrics Workshop 2008. • W. Willinger, D. Alderson, and J.C. Doyle. Mathematics and the Internet: A Source of Enormous Confusion and Great Potential. Notices of the AMS, Vol. 56, No. 2, 2009.

More Related