1 / 48

Adaptive CPU Allocation for Software based Router Systems

Adaptive CPU Allocation for Software based Router Systems. Puneet Zaroo. Software based routers. Implement packet forwarding/processing in software. E.g a PC with multiple NICs. Provide value added services like encryption, network address translation esp. at the network edge. Issues

raghnall
Download Presentation

Adaptive CPU Allocation for Software based Router Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Adaptive CPU Allocation forSoftware basedRouter Systems Puneet Zaroo

  2. Software based routers • Implement packet forwarding/processing in software. • E.g a PC with multiple NICs. • Provide value added services like encryption, network address translation esp. at the network edge. • Issues • Software architecture. • Per flow threads / per-packet threads • Division of input, forwarding and output functions • CPU scheduling. • How to determine CPU shares • How to enforce CPU shares.

  3. Objective • Leverage the advantages of a component based software router system. • Flexibility in designing routers • Reusability of software components • Dynamic addition of element modules • Overlay a QoS provisioning mechanism on top of the component based system. • Develop an adaptive QoS system • Adaptive to varying input rate and per-packet processing costs.

  4. Some Software Router Systems • Router Plugins : ETH Zurich, Uwash St. Louis • Per flow code modules or plugins. • Implemented in the NetBSD kernel. • Click Modular router : MIT • Routers made of elements composed into a flow graph. • ANTS • Programmable and customizable networks. • Customizable applications acting on packets / packets carrying code as well as data. • X-kernel : University of Arizona • Object oriented interface to protocols. • Can be used on end systems as well as routers. • Scout : University if Arizona, Princeton University • Communication oriented OS based on x-kernel. • Path based abstraction. • Advanced CPU scheduling.

  5. OS support for CPU scheduling • Scout • Proportional scheduling. • CPU balance (extension of work on livelock) • Resource Containers : Rice University • Decoupling of protection domain/resource domain. • Proper accounting of resources to processes. • Resources include threads as well as kernel data structures and memory,bound to containers. • E.g a web server serving multiple connections. • Processor Capacity reserves : CMU • Provides support for both time-sharing and real-time systems. • The OS enforces the reservations (cpu share, time period). • Applications free to change their reservations subject to admission control. • Nemesis : Cambridge • OS does low level resource multiplexing. • Avoiding QoS cross-talk • Support for I/O in user level libraries.

  6. Click • Composable flow-graphs from router elements • Packets travel along graph edges • Element based processing (push/pull). • Element based scheduling. • Multithreaded SMP Click • Issues in flow level QoS on top of an element based architecture • Flow level accounting and scheduling. • CPU balance b/w input, output and processing. • CPU conservation of idle elements.

  7. CROSS/Linux – Resource reservation with containers • Containers • Group of related elements • Elements doing per flow processing. • Container – CPU resource reservation unit. • Why use containers and not flows ? • Types of Containers • Input • Output • Forwarding • Best Effort • QoS - Packet rate reservations

  8. Example Router Configuration

  9. CROSS/Linux - CPU scheduling • Three level scheduler • Linux schedules CROSS • Linux process scheduler • CROSS schedules Containers • Proportional (Dynamic stride scheduling) • Containers schedule Elements • Simple Round Robin scheduling

  10. CROSS/Linux – Architectural Enhancements to Click • CPU conservation through sleep/wakeup • Elements tested for scheduling eligibility • Containers tested for scheduling eligibilty • Notifier Queues - wake up elements (make eligible for scheduling) • Delayed wakeup • Network interface Input Element • Switching between polling and interrupt • Based on a threshold packet input rate to reduce programmed I/O overhead • Topology discovery • Discovering input/output queues for a container

  11. CROSS/Linux – Enhancements to Click • virtual Interface queues – especially for interface statistics gathering • Linux /proc interface – • One directory for each container • Directory provides information about • Container tickets • CPU cycles consumed • Packet rate/drop rate • Elements • Input/Output queues • Set container tickets

  12. CROSS/Linux – Share adaptation • Why ? • Inability to do a-priori CPU share calculation • Variations in packet input rate • Variations in per-packet processing cost • How ? • Scheduler for each container keeps track of • Packet input rate. • Packet drop rate. • CPU cycles used. • Recomputes container shares to remove packet drops.

  13. CROSS/Linux – Share adaptation • Statistics maintained by Queues • Packet rates • Packet drop rates • Queues used to connect containers • Packet pass/drop rates at Queues indicate the difference between the required and the actual CPU shares for the container

  14. Share adaptation Algorithm • Invoked every 1 second • Notation used • T – Ticket share • C – Current CPU share • p – Input packet rate • d – packet drop rate • m – maximum input rate • General idea • Increase ticket share of a container so that the drop rate is removed at all the containers

  15. Input Container share adaptation (Issues) • Pass as many packets as possible upto a maximum. • How to arrive at this maximum? • Forwarding more than the maximum adversely affects the effective router throughput. • Reduce share on observing over allocation.

  16. Input Container – Share adaptation(Algorithm) if p > m /* Input rate too high */ /* reduce share */ T = C * (m/p) else if d > 0 /* Increase share to */ /*remove packet drops */ drate = min(d + p,m) T = C * (drate/p) else if (T – C) >= delta /* Over allocation */ /* reduce share */ T = T – eps

  17. QoS container – Share adaptation(Issues) • Always forward till reserved rate. • Target a forwarding rate range. • Reduce share in case of over allocation

  18. QoS container – Share adaptation(Algorithm) If p ε[ R – Dt, R + Dt] /* No change */ return if p > R + Dt /* Reduce share */ T = C * (R/p) else if d > 0 /* Increase share */ drate = min(p + d,R) T = C * (drate/p) else if (T-C) >= delta /* Reduce share */ T = T – eps

  19. Output Container – Share adaptation (Issues) • Try to forward all that is received • Throttling if any has happened upstream • Reduce share in case of over allocation

  20. Output Container – Share adaptation (Algorithm) if d > 0 /*Increase share */ T = C * (1 + d/p) else if (T – C) >= delta / * Reduce Share */ T = T - eps

  21. Best Effort Container – Share adaptation • No action taken • System makes no guarantees

  22. Discussion • Packet rate based reservation • Reservations based on packet rates more intuitive • CPU shares may vary for the same packet rates • C (Actual share) - How is it calculated? • Input container • Only include CPU cycles used in packet processing as opposed to idle polling. • Other containers • Easy to calculate since no idle polling. • m (Maximum forwarding rate) • Constant determined at router initialization • Evaluated at each iteration

  23. Evaluation • Using a simulator • Calculates the forwarding rate , drop rate based on the CPU shares. • Mimics the actions of the adaptive algorithm • Eases loading the “router” and testing of diverse workloads • Using a real implementation • CROSS/Linux on 866 Mhz Pentium III CPU.

  24. Adaptive vs. Non Adaptive(Experimental setup) • Input (2 µs), Output (2 µs) , Best Effort Container (6 µs). • Router – 1 MHz CPU => max forwarding = 100,000 packets/s • Static ticket assignment = 1:1:1 • Input varied for 0 to 110,000 packets/s in increments of 10,000 packet/s every 10s.

  25. Adaptive vs. Non Adaptive(Variation with time)

  26. Adaptive vs. Non Adaptive(Maximum loss free forwarding rate)

  27. Variable packet processing time(Experimental Setup) • Input (2µs), Best Effort/QoS (6µs), Output Container (2µs) • Observe different convergence behavior for QoS / Best Effort • Router – 1 MHz CPU => max forwarding rate initially = 100,000 packets/s • Constant input = 50,000 packets/s • Per packet processing cost increased by 2 µs every 10 secs. • Max. forwarding rate = 50,000 packets/s at t=50s.

  28. Variable packet processing time(Adaptive vs. Non Adaptive)

  29. Variable packet processing time-(Best Effort vs. QoS)

  30. Adaptation in m • Hard to determine m at router initialization • May vary with variations in per packet processing costs. m = maxi (TOTAL_CPU_CPS/cpu_cpp(ci)) where ciε C • TOTAL_CPU_CPS - Total CPU cycles per second available to the router • cpu_cpp(ci) - cycles/packet being used by the flow serviced by container ci cpu_cpp(ci) = cpu_cpi() + cpu_cycles(ci)/num_packets(ci) + cpu_cpo() • C - The set of containers servicing active flows

  31. Fixed vs adaptive m - (Experimental setup) • Input (8µs), Best Effort/QoS (1µs), Output Container (1µs) • Router – 1 MHz CPU => max forwarding rate, initially = 100,000 packets/s • Constant input = 50,000 packets/s • Per packet processing cost increased by 2 µs every 5 secs • Max forwarding rate = 50,000 packets/s at t=30 s.

  32. Fixed vs adaptive m - (Effective Best Effort Forwarding)

  33. Fixed vs. adaptive m(Effective QoS forwarding)

  34. Fixed vs. Adaptive m(Best Effort, QoS , Theoretical maximum)

  35. Advanced Adaptation in m • Previous algorithm gives too much stress to the least expensive flow. • Fine if all packets destined for that flow. • The packet rate to different flows can be variable. • m =(TOTAL_CPU_CPS/weighted_cpu_cpp) • weighted_cpu_cpp = Σ (cpu_cpp(ci) * rate(ci))/ (Σ rate(ci)) where ciεC

  36. Adaptive m vs. advanced adaptive m(Experimental Setup) • Input container (5 µs), Output Container(5 µs) • Router (1 MHz CPU) • 2 flows • QoS container (50,000 p/s,30 µs) => max forwarding rate achievable = 25,000 packets/s • Best Effort container (3 µs) => max forwarding rate achievable = 77,000 packets/s • Input rate to best effort container = 500 packets/s • Input rate to QoS container varied from 15,000 packets/s to 50,000 packets/s in increments of 5,000 packets/s every 5 s.

  37. Adaptive m vs. advanced adaptive m(Forwarding rate vs. time)

  38. Evaluation on a Router • CROSS/Linux software router platform • P III 866 MHZ pc. • 3 network interface cards.

  39. QoS Forwarding (Experimental setup) • 866 MHz , PIII router • Input Container(4.5 µs) , Best Effort Container(3 µs),QoS container (32,000 packets/s), Output Container (4.9 µs) • 3 different per – packet processing costsfor the QoS container • 3, 9.7 and 15.2 µs • Input to QoS => 32,000 packets/ • Input to Best Effort => 27,000 packets/s

  40. QoS Forwarding (Forwarding rate)

  41. QoS Forwarding (Ticket Share)

  42. QoS forwarding (Ticket Shares)

  43. QoS forwarding (CPU Shares)

  44. Effective Forwarding rate(Experimental setup) • Input (4.5 µs), best effort (8.3 µs) and output (4.9 µs) • Maximum forwarding rate = 57,000 p/s • 3 different scenarios • No Adaptation • CPU share Adaptation and m = 65000 packets/s • CPU share Adaptation and m = 110000 packets/s

  45. Effective Forwarding rate

  46. Future Work • Conjoint CPU – Buffer Allocation • Insufficient CPU share => always packet drops • Once sufficient CPU shares, more buffering => more efficiency • More buffering => higher packet delays and packets getting dropped at line cards. • Share adaptation between Linux/CROSS • Can use the SFQ scheduler already implemented

  47. Conclusion • Provide a QoS provisioning layer on top of a component based system. • Adaptive in response to variable packet input and processing costs.

  48. THANK YOU

More Related