Bandwidth estimation in computer networks: measurement techniques & applications

1. Bandwidth estimation in computer networks: measurement techniques &applications Constantine Dovrolis College of Computing Georgia Institute of Technology

2. The Internet as a “black box” • Several network properties are important for applications and transport protocols: • Delay, loss rate, jitter, capacity, congestion, load, etc • But, routers and switches do not provide direct feedback to end-hosts (except ICMP, also of limited use) • Mostly due to scalability, policy, and simplicity factors

3. probe packets Can we guess what is in the black box? • End-systems can infer network state through end-to-end (e2e) measurements • Without any feedback from routers • Objectives:accuracy, speed, non-intrusiveness

4. Bandwidth estimation in packet networks • Bandwidth estimation (bwest): • Inference of various throughput-related metrics with end-to-end network measurements • Early works: • Keshav’s packet pair method for congestion control (‘89) • Bolot’s capacity measurements (’93) • Carter-Crovella’s bprobe and cprobe tools (’96) • Jacobson’s pathchar – per hop capacity estimation (‘97) • Melander’s TOPP avail-bw estimation (’00) • In last ± 5 years: • Several new estimation techniques • Many research papers at prestigious conferences • More than a dozen of new measurement tools • Several applications of bwest methods • Significant commercial interest in bwest technology

5. Overview • This talk is an overview of the most important developments in bwest area over the last 10 years • A personal bias could not be avoided.. • Overview • Bandwidth-related metrics • Capacity estimation • Packet pairs and CapProbe technique • Available bandwidth estimation • Iterative probing: Pathload and PathChirp • Direct probing: Spruce • Comparison of tools • Percentile estimation: Pathvar • Applications • Overlay-based Video Streaming • SOcket Buffer Auto-Sizing (SOBAS)

6. Network bandwidth metrics Ravi S.Prasad, Marg Murray, K.C. Claffy, Constantine Dovrolis, “Bandwidth Estimation: Metrics, Measurement Techniques, and Tools,” IEEE Network, November/December 2003.

7. Capacity definition • Maximum possible end-to-end throughput at IP layer • In the absence of any cross traffic • Achievable with maximum-sized packets • If Ci is capacity of link i, end-to-end capacity C defined as: • Capacity determined by narrow link

8. Available bandwidth definition • Per-hop average avail-bw: • Ai = Ci (1-ui) • ui: average utilization • A.k.a. residual capacity • End-to-end avg avail-bw A: • Determined by tight link • ISPs measure per-hop avail-bw passively (router counters, MRTG graphs)

9. The avail-bw as a random process • Instantaneous utilization ui(t): either 0 or 1 • Link utilization in (t, t+t) • Averaging timescale: t • Available bandwidth in (t, t+t) • End-to-end available bandwidth in (t, t+t)

10. Available bandwidth distribution • Avail-bw has significant variability • Need to estimate second-order moments, or even better, the avail-bw distribution • Variability depends on averaging timescale t • Larger timescale, lower variance • Distribution is Gaussian-like, if t >100-200 msec and with sufficient flow multiplexing

11. E2E Capacity estimation (a’):Packet pair technique Constantine Dovrolis, Parmesh Ramanathan, David Moore, “Packet Dispersion Techniques and Capacity Estimation,” In IEEE/ACM Transactions on Networking, Dec 2004.

12. Packet pair dispersion • Packet Pair (P-P) technique • Originally, due to Jacobson & Keshav • Send two equal-sized packets back-to-back • Packet size: L • Packet trx time at link i: L/Ci • P-P dispersion: time interval between last bit of two packets • Without any cross traffic, the dispersion at receiver is determined by narrow link:

13. Cross traffic interference • Cross traffic packets can affect P-P dispersion • P-P expansion: capacity underestimation • P-P compression: capacity overestimation • Noise in P-P distribution depends on cross traffic load • Example: Internet path with 1Mbps capacity

14. Multimodal packet pair distribution • Typically, P-P distribution includes several local modes • One of these modes (not always the strongest) is located at L/C • Sub-Capacity Dispersion Range (SCDR) modes: • P-P expansion due to common cross traffic packet sizes (e.g., 40B, 1500B) • Post-Narrow Capacity Modes (PNCMs): • P-P compression at links that follow narrow link

15. E2E Capacity estimation (b’):CapProbe Rohit Kapoor, Ling-Jyh Chen, Li Lao, Mario Gerla, M. Y. Sanadidi, "CapProbe: A Simple and Accurate Capacity Estimation Technique," ACM SIGCOMM 2004

16. Compression of p-p dispersion • First packet queueing => compressed dispersion • Capacity overestimation

17. Expansion of p-p dispersion • Second packet queueing => expansion of dispersion • Capacity underestimation

18. CapProbe • Both expansion and compression result from queuing delays • Key insight: packet pair with zero queueing delay would yield correct estimate • measure RTT for each probing packet of packet pair • estimate capacity from pair with minimum RTT-sum Capacity

19. Measurements - Internet, Internet2 • CapProbe implemented using PING packets, sent in pairs

20. Avail-bw estimation (a’):Pathload Manish Jain, Constantine Dovrolis, “End-to-End Available Bandwidth: Measurement Methodology, Dynamics, and Relation with TCP Throughput,” IEEE/ACM Transactions on Networking, Aug 2003.

21. Probing methodology • Sender transmits periodic packet stream of rate R • K packets, packet size L, interarrival T = L/R • Receiver measures One-Way Delay (OWD) for each packet • D(k) = tarv(k) - tsnd(k) • OWD variations: Δ(k) = D(k+1) – D(k) • Independent of clock offset between sender/receiver • With stationary & fluid-modeled cross traffic: • If R > A, then Δ(k) > 0 for all k • Else, Δ(k) = 0 for all k

22. Self-loading periodic streams • Increasing OWDs means R>A • Almost constant OWDs means R<A

23. Example of OWD variations • 12-hop path from U-Delaware to U-Oregon • K=100 packets, A=74Mbps, T=100μsec • Rleft = 97Mbps, Rright=34Mbps • Ri

24. Iterative probing and Pathload Avail-bw time series from NLANR trace • Iterative probing in Pathload: • Send multiple probing streams (duration τ) at various rates • Each stream samples path at different time interval • Outcome of each stream is either Ri > A or Ri < A • Estimate upper and lower bound for avail-bw variation range

25. Avail-bw estimation (b’):PathChirp Vinay Ribeiro, Rolf Riedi, Jiri Navratil, Rich Baraniuk, Les Cottrell, “PathChirp: Efficient Available Bandwidth Estimation,” PAM 2003

26. Chirp Packet Trains • Exponentially decrease packet spacing within packet train • Wide range of probing rates • Efficient: few packets

27. Chirps vs. CBR Trains • Multiple rates in each chirping train • Allows one estimate per-chirp • Potentially more efficient estimation

28. Comparison with Pathload • 100Mbps links • pathChirp uses 10 times fewer bytes for comparable accuracy

29. Avail-bw estimation (c’):Direct probing & Spruce Jacob Strauss, Dina Katabi, and Frans Kaashoek “A Measurement Study of Available Bandwidth Estimation Tools,” IMC 2003

30. Direct probing • Iterative methods are relatively slow and they need to probe at multiple rates • Direct probing methods • Probe only once at a rate Ri and determine avail-bw from output rate Ro • Spruce (IMC’03) followed this approach • With a single-link fluid model (capacity Ct): • Sender sends periodic stream with input rate Ri • Receiver measures output rate Ro

31. What’s wrong with Direct Probing? • Two major issues: • Tight link capacity Ct is assumed to be known • What if tight link is different than narrow link? • Input rate Ri equal to source rate (set equal to Ct) • But what would happen in a multi-hop path? • Probing stream can arrive at tight link with lower rate than source rate(due to a link with lower avail-bw than Ct before the tight link) • Results in avail-bw underestimation • For details, see CCR paper, Oct 2006, by Lao, Dovrolis and Sanadidi, • “The Probe Gap Model can Underestimate the Available Bandwidth of Multihop Paths”

32. Avail-bw estimation (d’): percentile estimation Manish Jain, Constantine Dovrolis, “End-to-End Estimation of the Available Bandwidth Variation Range,” ACM SIGMETRICS 2005

33. Avail-bw distribution and percentiles • Avail-bw random process, in timescale t: At(t) • Assume stationarity • Marginal distribution of At: • Ft(R) = Prob [At ≤ R] • Ap :pth percentile of At, such that p = Ft(Ap) • Objective: Estimate variation range [AL, AH] for given averaging timescale t • ALand AH are pL and pH percentiles of At • Typically, pL =0.10 and pH =0.90

34. Percentile sampling • Question : Which percentile of At corresponds to rate R? • Given R and t, estimate Ft(R) • Assume that Ft(R) is inversible • Sender transmits periodic packet stream of rate R • Length of stream = measurement timescale t • Receiver classifies stream as • Type-G if At ≤ R: I(R)= 1 with probability Ft(R) • Type-L otherwise: I(R)= 0 with probability 1-Ft(R) • Collect N samples of I(R) by sending N streams of rate R • Number of type-G streams: I(R,N)= ΣiIi(R) • E[ I(R,N) ] = Ft(R) * N • Percentile rank of probing rate R: Ft(R) = E[I(R,N)] / N • Use I(R,N)/N as estimator of Ft(R)

35. Non-parametric estimation • Does not assume specific avail-bw distribution • Iterative algorithm • Stationarity requirement across iterations • N-th iteration: probing rate Rn • Use percentile sampling to estimate percentile rank of Rn • To estimate the upper percentile AH with pH = Ft(AH): • fn = I(Rn,N)/N • If fn is between pH±r, report AH = Rn • Otherwise, • If fn > pH +r, set Rn+1 < Rn • If fn < pH -r, set Rn+1 > Rn • Similarly, estimate the lower percentile AL

36. Validation example • Verification using real Internet traffic traces b=0.05 b=0.15 • Non-parametric estimator tracks variation range within 10-20% • Selection of b depends on traffic • Traffic spikes/dips may not be detected if b is too small • But larger b causes larger MSRE

37. Parametric estimation • Assume Gaussian avail-bw distribution • Justified assumption for large degree of traffic multiplexing • And/or for long averaging timescale (>200msec) • Gaussian distribution completely specified by • Mean m and standard deviation st • pth percentile of Gaussian distribution • Ap = m + st f-1(p) • Sender transmits N probing streams of rates R1 and R2 • Receiver determines percentiles ranks corresponding to R1 and R2 • m and st can be then estimated by solving • R1 = m + st f-1(p1) • R2 = m + st f-1(p2) • Variation range is then calculated from: • AH = m + st f-1(pH) • AL = m + st f-1(pL)

38. Validation example • Verification using real Internet traffic traces • Evaluated with both Gaussian and non-Gaussian traffic Gaussian traffic non-Gaussian traffic • Parametric algorithm is more accurate than non-parametric algorithm, when • traffic is good match to Gaussian model • in non-stationary conditions

39. Experimental comparison of avail-bw estimation tools Alok Shriram, Marg Murray, Young Hyun, Nevil Brownlee, Andre Broido, k claffy, ”Comparison of Public End-to-End Bandwidth Estimation Tools on High-Speed Links,” PAM 2005

40. Testbed topology

41. Accuracy comparison - SmartBits Direction 1, Measured AB Direction 2, Measured AB Actual AB

42. Accuracy comparison - TCPreplay Actual Available Bandwidth Measured Available Bandwidth

43. Measurement latency comparison • Abing: 1.3 to 1.4 s • Spruce: 10.9 to 11.2 s • Pathload: 7.2 to 22.3 s • Patchchirp: 5.4 s • Iperf: 10.0 to 10.2 s

44. Probing traffic comparison

45. Applications of bandwidth estimation

46. Applications of bandwidth estimation • Large TCP transfers and congestion control • Bandwidth-delay product estimation • Socket buffer sizing • Streaming multimedia • Adjust encoding rate based on avail-bw • Intelligent routing systems • Overlay networks and multihoming • Select best path based on capacity or avail-bw • Content Distribution Networks (CDNs) • Choose server based on least-loaded path • SLA and QoS verification • Monitor path load and allocated capacity • End-to-end admission control • Network “spectroscopy” • Several more..

47. Overlay-based Video Streaming M. Jain and C. Dovrolis, “Path Selection using Available Bandwidth Estimation in Overlay-based Video Streaming,” published at IFIP Networking 2007.

48. Motivation • Video/IPTV next “killer-application” in the Internet • IP networks present several challenges • Network impairments such as losses and jitter • Lack of QoS guarantees • Previous approaches • Adapt encoding rate • Video quality not consistent • Use proactive error correction techniques such as forward error correction (FEC) or retransmission • Bandwidth overhead for FEC • Potentially larger playback delay • Error concealment techniques • Limited applicability & effectiveness

49. Exploiting Path Diversity • Multihoming and overlay networks make multiple paths available between sender and receiver • Applications can dynamically switch from one path to another • Based on observed (or predicted) performance • Serves as an additional adaptation mechanism • Schemes using path diversity based on network measurement have been explored • Path selection techniques: Tao et al. [ACM Multimedia ‘04] and Amir et al. [NOSSDAV ‘05] • Multi-description coding techniques: Begen et al. [Signal Processing ‘05] and Apostolopoulos et al. [INFOCOM ‘02] • They use only loss and jitter measurements • Employ dummy packets for measurement

50. Our Approach • We consider use of overlay network for video streaming • To maximize perceived video quality • Novel features in our approach: • Use avail-bw to drive dynamic path selection • Use VQM technique instead of PSNR to measure video quality • Described in ITU-T recommendation J.144 • Use data packets to substitute “dummy” packets for network measurements • Objective: Evaluate the performance of different path selection schemes in terms of perceived video quality