End-to-end Bandwidth Estimation in the Wide Internet

1. End-to-end Bandwidth Estimationin the Wide Internet Daniele Croce PhD dissertation, April 16, 2010

2. Internet is wonderful • “Breakfast Can Wait. The Day’s First Stop Is Online.” [NYTimes‘09] • but is our connection performing well?

3. The Internet Net3 Net2 Net4 Net1 Objective: characterize the E2E performance • Inter-connected networks • Different technologies, many operators • No global view 3

4. Performance metrics • Simple metrics • Packet loss • Delay (One-Way, RTT), jitter • (TCP) throughput • Advanced metrics • End-to-end capacity C=min(Ci) • End-to-end available bandwidth (AB) • i.e., the unused capacity A=min(Ai)

5. Available Bandwidth On a generic link i :

6. Narrow link and tight link Available bandwidth Capacity C = AB = 100 Mbps 90 Mbps 1000 Mbps 400 Mbps 155 Mbps 20 Mbps Narrow Link Tight Link An example:

7. Contribution 1 Net3 Net2 Net4 Net1 • Tools require access to both end hosts • Impossible between different organizations! • Three single-ended tools

8. Contribution 2 Net3 Net2 Net4 Net1 • Large-scale deployments of active AB tools • Routing, P2P optimization, improve TCP • Performance evaluation of AB techniques in large-scale measurement systems 8

9. Contribution 3 NEW!!! • Three AB measurement paradigms exist: • PRM (Probe Rate Model) • “Is rate higher than the AB?” • PGM (Probe Gap Model) • “Has the Inter-Packet Gap increased?” • PDM (Probe Delay Model) • “Has the packet queued?” • Only analytical or simulative studies • Better than PRM or PGM? • Real implementation and comparison with other classic PRM and PGM tools 9

10. SINGLE-ENDED TECHNIQUES

11. ACK probes DSLAM Sender TCP RSTs Receiver ACKs RSTs Receiver Sender Sender Non-cooperative estimation • RTT = OWDf + OWDr Can we separate the effects of the two paths? 11

12. ACKs Sender RSTs Sender Where is the tight link? • RSTs are always 40 Bytes • No matter the size of the ACK probes • By varying the ACK size • We can load the two paths equally (SACK = SRST) • We can load the downlink more than the uplink (SACK > SRST) • We can NOT load the uplink more than the downlink (SACK < SRST)

13. ABw-Probe (ABP) • Measuring the downlink (no uplink traffic) cooperative non-coop. • Impact of “cross”-traffic on the uplink

14. Uplink cross-traffic 14

15. Filtering uplink cross-traffic • Cross-traffic is not just MTU packets • Use DT to remove large packets • Then use RR for refining 15

16. FAB-probe (large-scale) Do we really need a 40 kbps precision?

17. Real-world experience Tested on 1244 ADSL hosts, 10 different ISPs Participating in Kademlia DHT (eMule) Used KAD crawler (ACM IMC 2007) Selected ADSL using Maxmind Capacity of the ADSL link A snapshot of the available bandwidth Average AB on over 10 days 82 hosts online for over one month Static IP address Measured every 5 minutes On average 6 seconds per measurement 17

18. Capacity estimation Comparison of 2 large ISPs The policy used by Free is quite uncommon (see IMC07) Downlink Uplink 2.5Mbps 0.3Mbps 0.7Mbps 1Mbps 18

19. Available bandwidth (I) Snapshot of 1244 (eMule) hosts Hosts are divided in congested or idle 19

20. Available bandwidth (II) 82 hosts, 10 days average Each point is an average of one user over 10 days 30% congested, 30-40% frequently idle 20

21. ANALYSIS OF LARGE-SCALEAB MEASUREMENT SYSTEMS

22. Motivation 22 • We have a dream: measure AB everywhere • Route selection, server selection • Overlay performance optimization • Improve TCP • ... • Naïve approach: • pick one of the existing techniques! • BUT what if we all do the same simultaneously? • Interference between measurements

23. In brief • Existing techniques • Pathload, Spruce, pathChirp • Experimental testbed • All toolssuffer from mutual interference • But not in the same way!!! • High intrusiveness and overhead • Analytical models • Probability of interference • Measurement bias • What can we do?

24. Pathload – Packet Trains • Probing strategy: • Iteratively send N trains at different rates • Binary search to converge to the AB • Inference: • Detect One-Way Delayincrease (rate > AB) 24

25. Spruce – Packet Pairs ∆in ∆in Bottleneck ∆out ∆out • Probing strategy: • Two packets with specific inter-packet gap • Inference: • Measure dispersion (gapincrease) of the pair • Accuracy is debated, out of our scope 25

26. Interference in Spruce ∆in ∆out • 100% error! 0 • One pair interfering… • What is the probability that this happens? • Hint: similar to ALOHA protocol 26

27. pathChirp – Packet “chirps” ABw Limit • Probing strategy: • One train with exponentially increasing rate • Inference: • Detect One-Way Delay increase 27

28. Testbed results • 62 hosts running linux • Half are senders, half receivers • Single bottleneck (10 Mbps), CBR traffic • Ideal conditions for ABw tools • Errors are due to mutual interference only

29. Pathload 29

30. Spruce Results are biased True?? How much OVERHEAD? 30

31. pathChirp Results seem better True?? High OVERHEAD!

32. Intrusiveness x10 x100

33. Possible Solutions • Mutual interference • Direct probing more promising • Simple, Spruce-like algorithms. No binary search • Identify interference (and correct it)?? • Overhead • “In-band” measurements (piggy-backing) • Best, no overhead at all • Complex! (SIGCOMM09) + delay constraints • “Out-of-band” measurements • At least, make the overhead scale with the ABw! • Lets help each other! Network Tomography

34. Conclusions • Non-cooperative estimation • Three highly optimized tools • No need to install software or buy new equipment • An Italian ISP already interested!  • Analysis of large-scale AB measurements • Tools can not be used off-the-shelf • Mutual interference, Intrusiveness, Overhead • Interference can be predicted and modeled • Discussed possible solutions • Future work includes • Technologies different from ADSL (cable, FTTH) • New, lightweight techniques (passive?), tomography

35. BACKUP

36. Collision with ON-OFF meas. Few hosts cause > 10% collisions! 36

37. Non-cooperative estimation Who is answering to what (Monarch, IMC’06)

38. Measurement bias: Spruce Measurement error in Spruce Depends on the # of interfering pairs n : The average number of interf. pairs is This explains why Spruce bias is proportional to 38

39. Pathload interference, two trains 39

40. PathChirp, two chirps With only two trains, errors up to 80%!!! 40

41. Measurement Overhead • Spruce • Overhead = min(240kbps, 5% of Bneck Capacity) • Few hosts can consume a LOT of Bw! • Pathload • Overhead ≈ ABw • Cons: measurements consume all the ABw • Pro: overhead “scales” with the ABw • pathChirp • Overhead = 300kbps (tunable parameter) • What if 10 hosts are measuring? If 100? 41

42. With traffic load 20 hosts running, ABw=6 Mbps 42

43. All tools together 9 hosts per type (27 senders) 43

44. Delay-based tools • Consider a single server queue • The utilization can be computed as • p0 is the probability of the queue being empty • Probe-Delay-Model (PDM) tools estimate p0 • PDM tools • Make no assumptions on cross-traffic • Inject very little overhead • no need for high probing rates 44

45. Forecaster Model The AB is estimated by “projecting” the utilization 45

46. Threshold problem In our experiments, must allow ~100us for inaccuracies! 46