Towards a Meaningful MRA for Traffic Matrices

1. Towards a Meaningful MRA for Traffic Matrices D. Rincón, M. Roughan, W. Willinger IMC 2008

2. Outline • Seeking a sparse model for TMs • Multi-Resolution Analysis on graphs with Diffusion Wavelets • MRA of TMs: preliminary results • Open issues

3. Context: Abilene 12 nodes (2004) STTL NYCM CHIN DNVR WASH SNVA IPLS KSCY ATLA LOSA ATLA-M5 HSTN Abilene topology (2004)

4. Example: Abilene traffic matrix

5. Traffic matrices • Open problems • Good TM models • Synthesis of TMs for planning / design of networks • Traffic prediction – anomaly detection • Traffic engineering algorithms • Traffic and topology are intertwined • Hierarchical scales in the global Internet apply also to traffic • Time evolution of TMs • How to reduce the dimensionality catch of the inference problem? • Our goals • Can we find a general model for TMs? • Can we develop Multi-Resolution machinery for jointly analyzing topology and traffic, in spatial and time scales?

6. Can we find a general model for TMs? • Our criterion: the TM model should be sparse • Sparsity: energy concentrates in few coefficients (M << N2) • Tradeoff between predictive power and model fidelity • Easier to attach physical meaning • Could help with the underconstrained inference problem • Multiresolution analysis (MRA) • “Classical MRA”: wavelet transforms observe the data at different time / space resolutions • Wavelets (approximately) decorrelate input signals • Energy concentrates in few coefficients • Threshold the transform coefficients  sparse representation (denoising, compression) • Successfully applied in time series (1D) and images (2D)

7. How to perform MRA on TMs? • Traffic matrices are 2D functions defined on a graph • 2D Discrete Wavelet Transform of TM as images • Uniform sampling in R2 • TMs are NOT images! – the intrinsic geometry is lost • Graph wavelets (Crovella & Kolaczyk, 2003) • Spatial analysis of differences between link loads -anomaly detection • Drawbacks of the graph wavelets approach • Non-orthogonal transform - overcomplete representation • Lack of fast computation algorithm

8. How to perform MRA on TMs? Operator T • Diffusion Wavelets (Coifman & Maggioni. 2004) • MRA on manifolds and graphs • Diffusion operator “learns” the underlying geometry as powers increase – random walk steps • Amount of “important” eigenvalues -vectors decreases with powers of T • Those under certain precision are related to high-frequency details, while those over  are related to low-frequency approximations W1 V1 W2 V2 W3 V3

9. Cv2 5 CW2 3 CW1 2 How to perform MRA on TMs? Operator T • Diffusion Wavelets (Coifman & Maggioni. 2004) W1 V1 W2 V2  Eigenvalues (low to high frequency)

10. Diffusion Wavelets and our goals • Unidimensional functions of the vertices F(v1) can be projected onto the multi-resolution spaces defined by the DW. • Network topology can be studied by defining the right operator and representing the coarsened versions of the graph. • But Traffic Matrices are 2D functions of the origin and destination vertices, and can also be functions of time: TM(V1,V2,t)

11. 2D Diffusion wavelets Operator T • Extension of DW to 2D functions defined on a graph • F(v1,v2) • Construction of separable 2D bases by “projecting twice” into both “directions” • Tensor product • Similar to 2D DWT • Orthonormal, invertible, energy conserving transform WW1 VW1 WV1 VV1 WW2 VW2 WV2 VV2 WW3 VW3 WV3 VV3

12. 2D Diffusion wavelets Operator T • Extension of DW to 2D functions defined on a graph WW1 VW1 WV1 VV1 WW2 VW2 WV2 VV2

13. MRA of Traffic Matrices • More than 20000 TMs from operational networks • Abilene (2004), granularity 5 mins • GÉANT (2005), granularity 15 mins • Acknowledgments: Yin Zhang (UTexas), S. Uhlig (Delft), • Diffusion operator: • A: unweighted adjacency matrix • “Symmetrised” version of the random walk – same eigenvalues • Double stochastic (!) • Precision ε = 10-7

14. 1 1 2 1 4 2 0 0 1 12 12 10 3 5 2 6 2D Diffusion wavelets – Abilene example V0 12 V4 6 W1 V1 W5 V5 W2 V2 W6 V6 W3 V3 W7 V7 W4 V4 W8 V8 # eigenvalues at each subspace Wj = WVj + VWj + WWj

15. 2D Diffusion wavelets – Abilene example STTL SNVA DNVR LOSA KSCY HSTN IPLS ATLA CHIN NYCM WASH ATLA-M5

16. 2D Diffusion wavelets – Abilene example

17. 2D Diffusion wavelets – Abilene example DW coefficients Abilene 14th July 2004 (24 hours) Time (5 min intervals) Coefficient index (high to low freq)

18. 2D Diffusion wavelets – Abilene example • How concentrated is the energy of the TM? • Wavelet coefficients for the Abilene TM • 12 x 12 = 144 coefficients, low- to high-frequency Coefficients – high to low frequency

19. Compressibility of TMs

20. Stability of DW coefficients

21. Coefficient rank – Abilene March 2004 Time (5 min intervals) Coefficient index Rank signature

22. Rank signature – anomaly detection?

23. Conclusions and open issues • Representation of TMs in the DW domain • TMs seem to be sparse in the DW domain • Consistency across time and different networks • Ongoing work • Develop a sparse model for TMs • How the sparse representation relates to previous models (e.g. Gravity) ? • Exploit DW’s dimensionality reduction in the inference problem • Exploring weighted / routing-related diffusion operators • Exploring bandwidth-related diffusion operators • Introducing time correlations in the diffusion operator • Diffusion wavelet packets – best basis algorithms for compression • DW analysis of network topologies

24. Thank you ! Questions?

25. Extra slides

26. Géant 23 nodes (2005)

27. AS1 AS2 AS3 Network/AS PoPs Access Networks Context: topology • Spatial hierarchy

28. Multi-Resolution Analysis • Intuition: “to observe at different scales”

29. Multi-Resolution Analysis • Approximations: coarse representations of the original data

30. V0 W1 V1 V2 W2 W3 V3 Multi-Resolution Analysis • Mathematical formalism • Set of nested scaling subspaces (low-frequency approximations) generated by the scaling functions • The orthogonal complement of Vi inside Vi+1 are called detail (high-frequency) or wavelet subspaces Wi, generated by waveletfunctions

31. Multi-Resolution Analysis • Scaling functions: averaging, low-frequency functions • Wavelet functions: differencing, high-frequency functions

32. Multi-Resolution Analysis (2D) • Separable bases: horizontal x vertical • Example: 2D scaling function

33. Wavelet transform example • 2D wavelet decomposition of the image for j=2 levels • Vertical/horizontal high/low frequency subbands

34. Our approach • Can we develop Multi-Resolution machinery for analyzing topology and traffic, in spatial and time scales? • Classical 1D or 2D wavelet transforms are not an option • We need a new graph-based wavelet transform! • Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) • Diffusion wavelets (M. Maggioni et al, 06)

35. The tools: Graph wavelets • Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) • Exploit spatial correlation of traffic data • Sampled 2D wavelets

36. The tools: Graph wavelets • Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) • Link analysis • Definition of scale j: j-hop neighbours

37. traffic j=1 j=3 j=5 The tools: Graph wavelets • Graph wavelets for spatial traffic analysis (Crovella & Kolaczyk 03) • Anomaly detection in Abilene