Globus Advance Reservation and Network Environment Testbed GARNET

1. Globus Advance Reservation and Network Environment TestbedGARNET Volker Sander, Alain Roy sander@mcs.anl.gov Argonne National Laboratory 10/07/1999

2. Outline • The Basic concept of GARA • GARNET and Experiences • How to join the Effort • Demonstration

3. Goals • Provide end-to-end Quality of Service to applications. This requires: • Discovery and selection of resources • Allocation of resources • Advance reservation of resources Workstation Workstation Supercomputer Supercomputer Workstation Workstation

4. Difficulties/Solutions • Lack of support for advanced reservations • NGI Bandwidth Broker effort is based on immediate reservations • Heterogeneous resources • More than just Networking QoS • We provide uniform interfaces • Need to work with complex sets of resources • Multi-domain, Multi-Resource-type • We use co-reservation agents

5. Solution: GARAGlobus Architecture for Reservation and Allocation • Three important contributions: • Support for Advance Reservations (AG) • Uniform treatment of underlying resources • Network, disk, CPU etc… • Currently supported: Differentiated Services, DPSS, and DSRT • Layered architecture enables generic co-reservation agents • User is able to select API best suitable for his need

6. GARA Basics—Reservations • There is a generic “reservation”, which has several properties: • Start Time (“now” or future) and Duration • Resource type/Underlying resource identifier • Resource-specific (bandwidth, % CPU…) • All reservations are treated uniformly: • Create/Modify (Given properties) => Returns Reservation Handle • Destroy • Monitor (Callbacks or Polling)

7. Gatekeeper (Authenticate and Authorize) User Program or Agent Gatekeeper Service Reservation Handle Local Resource Manager API (LRAM) Resource Manager Resource (Router...) A picture of what happens(creating a reservation) Create

8. Resource Manager Setup reservation User Program bind(handle) LRAM API Resource Enforce Reservation A picture of what happens(claiming a reservation) Note: GARA supports third party binding

9. Example: Network reservation • In advance, make a reservation: • RSL: & (reservation-type=network) (start-time=now) (duration=3600) (endpoint-a=128.135.11.4) (endpoint-b=128.135.11.1) (bandwidth=1000) • Handle: Opaque string H1 • At run-time: • Bind(H1)

10. 100% % of available resource allocated 0% Time The Resource Manager • To track advanced reservations, we use a simple “slot table” • We assume exclusive access to resource through the resource manager • We aren’t tied to this particular resource manager

11. Co-Reservation Agents • When multiple resources are needed, an agent: • Discovers applicable resources via the MDS • Reserves resources • Informs user when resources are ready. • Meta-RM support for integrating other RMs • The uniform interface enables these agents to be created easily. • Given this ease, it is convenient to express different strategies such as: • Depth-first vs. best-first • Coping with failures

12. MDS Co-Reservation Agent Gatekeeper Gatekeeper Gatekeeper Gatekeeper GRIO LRAM Scheduler LRAM Diffserv LRAM DSRT LRAM Workstation Supercomputer Router Workstation Router The Big Picture

13. End-to-End Co-Reservation Agent MDS Resource Manager Resource Manager Workstation User process End-to-End Network Reservations • Algorithm for a single network reservation • Discover which gatekeepers/resource managers need to be contacted (MDS) • Make reservation at each one.

14. The GARA API • globus_gara_reservation_create() • Gatekeeper contact • RSL reservation specification • Reservation Handle or Error • globus_gara_reservation_modify() • Old Reservation Handle • RSL reservation specification • New Reservation Handle or Error

15. The GARA API (continued) • globus_gara_reservation_cancel() • Reservation Handle • Error • globus_gara_reservation_status() • Reservation Handle • Status/Error • globus_gara_reservation_callback_register() • Reservation Handle • Callback Function/User Parameter • Error

16. The End-to-End Agent API • For making network reservations • Deals with co-reservation along path. • Very similar to GARA API: • Ex: globus_gara_end2end_create() • Handles MDS lookups. • Handles making multiple reservations, but provides user with single reservation handle. • Probably the API you want to use.

17. GARNET goals • Provide dedicated testbed for evaluating current and future mechanisms used for implementing GARA • QoS-Type Network: similar to IETF’s diffserv model • QoS-Type CPU: DSRT-Scheduler • QoS-Type Disk: Distributed Parallel Storage System (DPSS) • Demonstrate validity of concept • End-to-End Reservation • Application level QoS

18. Quality of Service for IP Traffic • The ability to define or predict the performance of traffic on a network • Needed in case of congestion • Leave the current best-effort model • End-User is able to specify his requirements Guaranteed bandwidth major concept • Building blocks • Link Layer QoS (ATM QoS) Create VC with specific parameters (ABR, CBR, UBR, …) • IP-QoS Flow Based QoS: Integrated Services (RSVP) Aggregate Based QoS: Differentiated Services (Premium service)

19. Aggregate based scheduling Flow based Set IP precedence for non exceeding flows with bound GARA Reservation Set IP precedence for non exceeding flows with bound GARA Reservation The Basic Concept

20. Server Client GARA API Diffserv Resource Manager The Extension

21. The Multidomain Extension Server Client GARA API Diffserv Resource Manager Diffserv Resource Manager

22. Building Blocks for Diffserv • Traffic marking based on a specific policy • Admission control • Congestion handling • Congestion avoidance • Congestion management • Traffic shaping Service Level Agreement (SLA)

23. Actual Mechanisms (Cisco) • Committed Access Rate (CAR) • Per flow based at End-Systems • Mark packets if they are not exceeding a rate limit • Drop packets beyond the limit or transmit them via best effort • Per Precedence between domains • Corresponds to a static SLS • RM will configure the limits on demand in later releases • Allows short term bursts • Weighted Random Early Detection (WRED) • Weighted Fair Queuing • Generic Traffic Shaping or PVC attributes

24. GARnet- Globus Advance Reservation and Network Env Testbed Linux Linux SUN Ultra 60 SUN Ultra 60 WFQ WFQ CAR CAR WFQ WFQ Cisco 7507 Cisco 7507 Cisco 7507 SUN Ultra2 SUN Ultra2 FastEthernet link OC-3 link

26. Evaluation Tools • Several Modified versions of ttcp (premium) • GARA-enabled (wait for reservation) • Support for a desired application rate • long term control • short term control • Consecutive bandwidth reporting • Bulk transfer ttcp in work • UDP traffic generator (competetive traffic) • Andy Adamson, University of Michigan • Added support for automated testing • Modified version of DPSS (premium) • GARA enabled • Nice example on how to use GARA

27. Overhead for packet headers (40 Bytes)

28. BUFFER

29. Ack Ack Ack The TCP “Problem” Receiver Transmitter TCP’s sliding window introduces burstiness of Traffic Socket rcv buffer Socket send buffer MSS-sized (media specific) TCP segments 40 Byte Header overhead Maximum outstanding of 1 actual window LastByteSent - LastByteAcked <= RcvWindow.

30. The TCP “Problem: Reno-TCP 45 Slow Start Exponential Growth 40 35 30 25 Average window size TCP Congestion Window Size 20 15 Congestion Avoidance Phase Linear Growth Half Window Size in case of duplicate Acks 10 5 0 0 time

31. The TCP “Problem” • TCP’s flow control • Traffic becomes bursty if the actual window size is large • Bandwidth*Latency Product as ideal window size • TCP’s congestion control • “Slow Start Phase” • shrink Window to 1 • increase size with every Ack • Congestion Avoidance • Half Window size • Increase size by one for every full window roundtrip

35. WRED impacts on RTT

36. WRED impacts on the Window size

37. … the Throughput

38. … some updates

39. … increasing tcp_cwnd_max

40. … new throughput

42. … new throughput

43. Using WFQ

44. Using WFQ

45. Using WFQ

46. Current Status • An early working prototype (alpha!) of GARA exists • Differentiated Services-like QoS • Real-Time CPU Scheduling (DSRT) • DPSS Disk Access • An early prototype of the end-to-end reservation agent exists. • Successful demonstration on GARNET and over ESnet’s NGI testbed • Work with bleeding-edge adopters has already started.

47. The GARA Source Tree • Subdirectories represent different layers • end2end • gara • lram • resource_manager • slot_manager • (dsrt) • We are providing examples for every layer • docs directory contains lots of useful information

48. Doing Tests • Select your favorite application • Determine port numbers • edit (set endpoint systems) and run script: gara/gara/utilities/demo_make_reservation • Create congestion • Try demo_modify_reservation for getting different premium bandwidth • Try demo_cancel_reservation

49. Install GARA • Get GARA (currently through E-Mail from sander/roy @mcs.anl.gov) • Get SSLeay 0.9.0 and install it • Get Globus 1.1 configure and compile it deploy is not necessary for doing tests Run grid-cert-request to apply for a Grid X509 Certificate • Read gara/docs/* • Edit gara/build • Run gara/build • Customize GARA • gara/resource_manager/programs/diffserv_manager.conf • gara/resource_manager/programs/setup_flow.cfg • (Samples are in the documentation directory) • Experiment!

50. Router Requirements • Cisco 75xx VIP2-50 • Get new IOS XE beta release Experimental Version 12.0(19990728:032821) [rolsen-conn_4xe_fix 243] Available at ftp://ftpeng.cisco.com/rolsen/rsp-pv-mz.091099 • ip cef distributed • Define an empty policy-map set-precedence on the edge router • Define a consecutive set of access-lists to be used by GARA (100-199) • Define class-map for precedence 3 or 5 on every router • Define a policy-map containing the class assigned with the supported premium BW • Assign this as service policy on each output interface • Define CAR per precedence on the ingress site