Distributed, Self-stabilizing Placement of Replicated Resources in Emerging Networks

1. Distributed, Self-stabilizing Placement of Replicated Resources in Emerging Networks Bong-Jun Ko, Dan Rubenstein Presented by Jason Waddle

2. Overview • Problem Resource allocation in a network • Solution Asynchronous distributed coloring protocol • Analysis • Convergence • Quality of approximation • Performance

3. Resource Allocation • Distribute resources in a network fairly, homogenously – all resource types nearby each node • File replica distribution • Distributed computation • Channel allocation in wireless networks

4. Requirements • Scalable • Dynamic • Asynchronous • Don’t assume coordinated clocks • Symmetric • Every node runs same algorithm, no special roles • Local • Nodes use only local information • Avoid long-distance communication

5. Resource Allocation Model • Network: undirected connected graph G = (V,E) , |V| = n • Edge weights w(e) for e 2 E • Model latency • Colors G = {c1, c2, …, ck} • Model resources • Vertex color function C : V !G • Assignment of resources to nodes

6. Example Network 10 5 2 10 17 8

7. Multiple Resources? 10 5 2 10 17 8

8. Yes. 0 0 10 0 5 2 10 17 8

9. Notation (1/2) • Distance d(x,y) = sum of weights on shortest path from x to y • symmetric • color of node x at time t: Ct(x) • distance to color at time t: dt(x,ci) • dt(x,ci) = minv {d(x,v) | Ct(v) = ci} • dt(x,Ct(x)) = 0 • dt(x,ci) = 1 if no node has color ci

10. Notation (2/2) • Distance to closest node of same color at time t: dt(x) • dt(x) = minv {dt(x,v) | Ct(x) = Ct(v) and v ¹ x}

11. Optimizated Coloring (1/2) • Want to assign colors to optimize one of: • Minimize maxx maxi ¹ Ct(x) dt(x,ci) • So nobody has it bad • Minimize åxåi ¹ Ct(x) dt(x,ci) • Good on average • Maximize åxdt(x) • Separate resources of same type

12. Optimized Coloring (2/2) • These problems are NP-Hard even with global information • so we’ll approximate

13. Asynchronous Distributed Coloring Protocol (ADC) • Decentralized asynchronous protocol • Each node • maintains information about colors of nearby nodes • changes its own color to contribute to optimization • without causing deadlock or livelock

14. Local Information • Nodes are working with old, possibly faulty information • d’t(x,ci) is the distance to the closest node with color ci that node x knows of • Similarly, d’t(x) is the nearest known node of the same color

15. Stability • Node x is locally stable at time t if d’t(x,ci) ·d’t(x) for all 1 · i · k • x perceives that all other colors closer to x than any other node of x’s color • Node x is stable at time t if dt(x,ci) ·dt(x) for all 1 · i · k • all other colors actually are closer to x than any other node of x’s color • The graph coloring is stable when all nodes are stable

16. Color Change Rule (1/2) • Locally stable nodes don’t change color • a locally unstable node x will try to change to color ci that satisfies d’t(x,ci) ¸ d’t(x,cj) for all 1 · j · k • x changes to furthest color • increases d’t(x), decreases åi d’t(x,ci) • does not increase maxi dt(x,ci)

17. Color Change Rule (2/2) y x

18. Color Change Rule (2/2) y x

19. Distance Propagation • Use modified Bellman-Ford, Colored Bellman-Ford (CBF) • maintain distances for closest two nodes of each of the k colors • updates sent to neighbors when local distance estimates change • CBF messages quickly stop after graph becomes stable • Side-benefit: how to route to each color • How does it handle distance increases?

20. CBF Lag d’(x,c1) = 5 d’(x,c2) = 1 y 5 x

21. CBF Lag y 5 d’(y,c1) = 5 x d’(y,c2) = 1

22. CBF Lag y 5 x

23. CBF Lag y 5 CBF x

24. CBF Lag d’(x,c1) = 1 d’(x,c2) = 5 y 5 x

25. Asynchronous Color Change • Need to prevent livelock and deadlock • Need to guarantee progress • want dt’(x) > dt(x) when x changes color between times t and t’

26. Monotonicity • Can show that dt’(x) > dt(x) when x changes from c1 to c2 if we have: • there is a node other than x at distance d’t(x) from x that is color c1 at time t’ • ensuring dt(x) ·d’t(x) • there is no node of color c2 within distance d’t(x) of x at time t’ • ensuring dt(x,c2) > d’t(x) • Protocol should provide these guarantees

27. Asynchronous Color Change • 3-way handshake to change color • Four types of messages • request – x: I want to be color c2 • accept – y: x change to c2 is OK • reject – z: x change to c2 is not OK • decision – x: I did not change to c2

28. Asynchronous Color Change • When x wants to change from c1 to c2i.e., when d’t(x) < d’t(x,c2) • x sends a request to all nodes at distance dt(x) or less: x’s time disk • contains x, c1, c2, z of color c1 • asks one c1 node, z, to stay c1 • asks nobody to change to c2

29. Asynchronous Color Change • When y receives request(x,c1,c2,z) • Reject if: • y is color c2 • y=z (i.e., x thinks y is color c1) but y is not color c1 • Accept otherwise, until y receives decision: • y prohibited from changing to c2 • if y = z, y prohibited form changing from c1

30. Asynchronous Color Change • if any node rejects, x aborts color change • if all nodes accept, c changes color • in both cases, x sends a decision message indicating the choice

31. Livelock / Deadlock • x and y are both color c1, both want to change color • just reject each other? prevent each other from improving: livelock • hold others’ request until they get their own accept/reject? deadlock • cannot just break ties • d’t(x) < d(x,y) < d’t(y)

32. Livelock / Deadlock • Impose a strict ordering on nodes • Keep outstanding requests in three sets: • stalling: requests that don’t conflict • master: conflicting requests with higher priority • slave: conflicting requests less priority

33. Node State Diagram Stalled Moving

34. Node State Diagram request(y): if bad, reject else accept, y !stalling Stalled Moving

35. Node State Diagram Stalled Unstable, no conflict REQUEST Moving

36. Node State Diagram Stalled Moving Request(y) & Conflict if y > x, accept and y !master else y !slave

37. Node State Diagram Stalled All accepted & master empty: reject all slaves, send decision If rejected: accept all slaves, send decision Moving

38. Convergence (1/3) • Map network state to vectors • V(t) = < dt(v1), dt(v2), …, dt(vn) > • lexicographically ordered: V(t) = <v1, v2, …, vn> V(t’) = <w1, w2, …, wn> V(t) < V(t’) means • 9 j, vj < wj and vi = wi for 1 · i < j

39. Convergence (2/3) • if x is the only node that changes color between times t and t’ then V(t) < V(t’) • protocol guarantees monotonicity: • dt(x) < dt’(x) • if another node y has dt(y) > dt’(y) then dt(y) > dt(x) • only values “right” of dt(x) in V(t) decrease • hence V(t) < V(t’)

40. Convergence (3/3) y x

41. Convergence (3/3) y x

42. Quality of Approximation (1/3) • Define the super-optimal distance • dopt(x) = the distance to the kth closest node to x (including x) • if those k nodes have different colors, x can reach all colors in distance at most dopt(x) • if the coloring is stable, then dt(x,ci) · 3dopt(x) for all x and ci • this bound is tight

43. Quality of Approximation (2/3) y dopt(y) = 1 d(y,c3) = 3

44. Quality of Approximation (3/3) • Let Dk(x) be the minimum distance from x to a node that is k hops from x • then dt(x,ci) · Dk(x) for all ci¹ C(x) • dt(x) ¸ dopt(x) in a stable coloring

45. Performance • Measured by simulation • Graph built by adding edges uniformly • extra random edges added until graph is connected • Edge weights uniform from [1,10] • Average degree between 4.5 and 5.5 • For given parameters, test 100 random graphs and take the average

46. Convergence Time

47. Color Changes / Node

48. Messages / Node

49. Max d(x,ci) / dopt

50. Max d(x,ci) / dopt CD