Load Sharing for Cluster-Based Network Service

1. Load Sharing for Cluster-Based Network Service Jiani Guo and Laxmi Bhuyan Architecture Lab Department of Computer Science and Engineering University of California, Riverside

2. Courtesy: “Cluster-Based Scalable Network Services”, Armando Fox, Steven D. Gribble, Yatin Chawathe, Eric A. Brewer and Paul Gauthier.

3. Internet Transcoding Service Video on Demand System a large number of clients heterogeneity in clients’ inbound network bandwidth, CPU/MEM capacity or display resolution Storing multiple copies in server will give rise to server overload and scalability problem

4. Cluster-Based Transcoding Service Transcoding Service • process the stream on-the-fly according to the client’s requirements => make some money • wide range of needs in video rates, sizes, and bandwidths can be met by real-time transcoding service – Need parallel processing

5. Existing Load Balancing Schemes • Plethora of research in the field of load balancing, but most of them only did simulations • Random or Round-robin implemented in practice • Adaptive load balancing is desirable, but the overhead in collecting statistics is very high – we found no real implementation • How does one maintain QoS while doing load balancing? Ex: To reduce out-of-order departures of multimedia units, the GOPs must be assigned to one processor, when a good load balancing needs distribution of the workload

6. EX: Round Robin High communication protocol (UDP) overhead Worker 1 Worker 2 Unit Buffer . . . Receiver Dispatcher Manager Worker N Find an available Worker fetch a unit Send the unit

7. Round Robin – A Multithreaded Model to Reduce Communication Cost Worker 1 Worker 2 Dispatcher 1 Unit Buffer . . . . . . Receiver Dispatcher M Manager Worker N Find an available Worker fetch a unit Send the unit

8. Load Sharing SchemesRound Robin - First Fit • Methods • Searches for an available Worker in round robin way • The first available Worker is chosen to be dispatched a GOP • How a manager detects if a Worker is available is implementation-dependent. • Properties • Load is naturally balanced among all the Workers. • Fast processing rate because no extra load analyzer is needed to guide scheduling. • May incur severe delay jitter for each stream because the GOPs of the same stream are most likely to be distributed to different Workers.

9. Worker 1 Worker 2 Worker N Round Robin – First Fit dispatch queues Manager Node Dispatcher 1 Dispatcher 2 scheduler . . . . . . Receiver . . . GOP Queue Dispatcher N Is the Worker available? = Is there a vacancy in the dispatch queue? Depends on power of the worker!

10. Stream-based Mapping • Methods • The media unit is mapped to a Worker according to the following function: f ( C ) = C mod N where C is the stream number to which the unit belongs; N is the total number of Workers in the cluster. • All media units belonging to one stream are sent to the same Worker. • Properties • Preserves the order of computation among media units. • Simple algorithm. • Most efficient for some specific input patterns in a homogeneous cluster. • Specific patterns : M is multiple of N, where M is the total number of streams • What if M < N?

11. Adaptive Load Balancing - Least Load First • Feedback-based Scheme — Least Load First • Efficient load test mechanism is needed for the Manager to monitor load distribution in the cluster. • Workers periodically report their load statistics information to the Manager. • The Worker with the least load is chosen to dispatch the job. • May incur substantial overhead to implement the feedback mechanism. • Each Worker reports to the Manager its load information during each epoch ∆t. • Load information reported by each Worker • CPU utilization AUi(t) • Maximal possible throughput Ai(t) • Actual throughput: Ai(t) – N • N is the number of outstanding requests, i.e., the number of GOPs already dispatched to it but not yet completed • Manager chooses the least loaded Worker: Worker with the maximal actual throughput

12. Adaptive Load Sharing • Unit-to-ComputingPC Mapping ( Done by the dispatcher) • Robust Hashing Mapping — The unit identifier ( such as the stream number of the unit in our experiment) and the Computing PC number together are used to assign a random value to each Computing PC. The unit is mapped to the Computing PC with the highest random value. If the Computing PCs have unequal capacity, the random value assigned to each Computing PC may be scaled by a weight which guarantees that the Computing PC with higher capacity can receive a proportionately higher portion of the load. • Thus, the mapping is calculated base on three values: the stream number of the unit C (1,2,…S), the Computing PC number J (1,2,…, N) and the weight vector (x1 , x2 , x3 , …, xN). • Minimize the probability of units belonging to the same stream being dispatched to different nodes. And this goal is achieved without keeping state information per stream. • Dynamic Weight Adaptation (Done by the manager) • The workload on the Computing PCs (ρ1(t), ρ2(t), …, ρN(t)) is collected periodically and the weight vector (x1 , x2 , x3 , …, xN) is adapted in a specific way such that the amount of stream re-mappings is minimized as well as load balancing is achieved. • The adapted weight vector is fed to the dispatchers.

13. Adaptive Load Sharing Manager (x1 , x2 , x3 , …, xN) (x1 , x2 , x3 , …, xN) Computing PC 1 ρ1(t) Dispatcher 1 Unit Buffer . . . . Computing PC 2 ρ2(t) . . . Receiver Dispatcher M Computing PC N Routing PC ρN(t) Yes End Start Fetch a unit F( C ) = J J available? Send to node J No

14. Experimental Set-Up Computing PC Computing PC Computing PC Computing PC Gigabit Ethernet Manager Media Server 100M Ethernet 100M Ethernet Processed packets Un-nrocessed packets

15. Transcoding Service • What is transoding? • Transforming video/audio streams such as changing the bit-rate, resizing video frames, and adjusting the frame resolution and so on. • How to transcode? MPEG Encoder MPEG Decoder Video/Audio Frame Manipulator MPEG Stream Raw Stream Manipulated Stream MPEG Stream

16. Transcoding Workload A media unit is a Group Of Pictures(GOP) of MPEG stream • A media unit can be transcoded independently by any Worker in the cluster. Transcoding one media unit is considered an independent job. • No communication is required among jobs. • Each job consumes similar amount of processing time. • Consecutive media units in a stream are preferred to be processed in order.

17. Design Goals of the Load Sharing Schemes • Balance the transcoding workload among all Workers • High system throughput • Low overhead taken by the load balancing algorithm itself • Good tradeoff between computation and communication • Provide good Quality of Service - NEW • In-order departure of media units • Even output time interval among successive media units of a media stream

18. Computation Model of the Transcoding Cluster

19. Manager Node • Receiver Thread • Accepts incoming media units into the GOP Queue • Scheduler Thread • Fetches GOPs from the GOP queue and puts them into an appropriate dispatch queue according to the specific load sharing scheme • Dispatcher Thread per Worker • Each Dispatcher maintains a dispatch queue • Once requested by the corresponding Worker, dispatches one GOP to the Worker • Manager Thread — Only for Least Load First Scheme • Collects the load statistics information from the Workers during each epoch • Feeds the load information to the scheduler • Collector Thread • Collects processed video units from Workers and sends them out

20. Worker Node • Reciever Thread • Receives packets from the Manager Node and assembles them into a complete GOP. • Once a complete GOP is received, gives it to the Transcoder thread, and then requests for another GOP from the Manager Node. • Transcoder Thread • Transcode a GOP. • Sender Thread • Delivers the transcoded GOPs to the clients. • Monitor Thread • Collects the load statistics information on the Worker node and reports to the Manager Node periodically.

21. Scalability of the System 5 media streams

22. Scalability of the System Throughput • System throughput scales well with First Fit and Least Load First. • Load test overhead in Least Load First scheme doesn’t affect the system throughput a lot, because the overhead is relatively small compared with the time taken to transcode one GOP. • Stream-based Mapping cannot disperse media units of the same stream among different Workers even if a Worker is free. • Waste of resources. • Occasional imbalance in load distribution. • Reduced throughput.

23. Out-of-Order Rate per Stream

24. Out-of-Order Rate per Stream • Out-of-order departure of media units Occurs when consecutive GOPs of a stream are transcoded on different Workers • The worklaod on different Workes is different • Different media units consume different amount of computation time • Stream-based Mapping eliminates out-of-order departure of media units. • Largest OFO rate for First Fit. • Least Load First improves 50% over First Fit.

25. Output Time Interval (OTI) per Stream

26. Output Time Interval(OTI) per Stream • Experiment setting • 4 homogeneous Workers, 5 media streams • First Fit achieves the best performance. • Least Load First approaches First Fit. • Longer delay for Stream-based Mapping because of the limitation that one stream can only be processed by one Worker.

27. Load Sharing Overhead with LLF • Load Test Overhead • Average time consumed by the Manager node to poll through all Workers to collect the load statistics information. • Load Remapping Overhead • Time used to set the current loads for each Worker.

28. Load Sharing Overhead with LLF • Load test overhead increases roughly proportional to the cluster size. • Load re-mapping overhead is much smaller than the load test overhead, almost negligible. • The operation overhead involved in load re-mapping is much less than the network communication overhead involved in load test.

30. Find an available Computing PC fetch a unit Send the unit Load Sharing Schemes Transcoding PC 1 Transcoding PC 2 • How to take QoS into consideration? SchedulePC Unit Buffer . . . Receiver Scheduler Transcoding PC N

31. Differentiated Service(Fair Scheduling) • A system is said to be capable of affording differentiated service among service classes if • The system permits its resources to be proportioned among the service classes • Given sufficient request load, a service class receives at least as much resources as were assigned to it irrespective of the load on other service classes • Resources not used by some service class may be distributed among other service classes.

32. Framework of Fair Scheduling

33. Fair Scheduleing • Fairly distribute resource among streams • Streams make reservations • Received service is proportional to the reservations • UnitScheduler - Weighted Round Robin ( WRR ) • Provide differentiated service rate to multiple streams • Weights in each round-robin cycle are dynamically adapted to achieve the best performance • Weight of stream i Wi(t): the number of GOPs scheduled for stream i during one round robin cycle