Dynamic Circuit Network Hands-On Workshop

1. Dynamic Circuit Network Hands-On Workshop University of Nebraska-Lincoln Nebraska Student Union Lincoln, NE July 19th and 20th, 2008

2. Welcome! • Wireless • cannot access workshop system from Joint Techs Wireless • Wired connections also available

3. Welcome! • This is the 7th DCN Workshop • Nysernet • MAX • NASA Ames • University of Houston • University of Hawaii (double header) • University of Nebraska - Lincoln • Introductions

4. Welcome! • Key objectives of this workshop are: • Disseminate information to the R&E community regarding the emerging class of Hybrid Network and the associated techniques for Dynamic provisioning and configuration • Review in detail and provide instruction on how to use the control plane software currently in service on the Internet2 Dynamic Circuit Network (DCN), ESnet Science Data Network (SDN), and several regional networks. • Obtain feedback directly from the community on how to improve the technologies…Hopefully, to help guide future development and deployment priorities and speed adoption • Review the state of implementation and deployment of these types of dynamic networks throughout the R&E community.

5. Instructors • Tom Lehman (USC/ISI) • Chris Tracy (MAX) • Andy Lake (Internet2) • These people are involved in numerous projects related to deploying dynamic control planes: • Internet2 Dynamic Circuit Network • ESnet OSCARS Project • NSF DRAGON • Internet2 HOPI Testbed • DICE (Dante, Internet2, Canarie, Esnet) – International development activities

6. Why do a workshop? • Dynamic Hybrid Networks are new… • The service concepts are still unfamiliar to many networker experts and users… What does one gain with DCN? • The software and hardware implementations are still evolving… • Even the standards are still evolving… • The networks that support these capabilities are few but growing. • The user base is small [for now]…. But will grow as the capabilities mature and become more ubiquitous, persistent, robust, and the utility of both connection oriented services and dynamic provisioning becomes more widely recognized and accepted. • Providing hands-on experience to design and deploy these architectures is one way to broaden and promote adoption.

7. Agenda • Day 1 • 9:00 am Overview of GMPLS and DRAGON • 10:00 am Exercise #1: Designing a GMPLS Control Plane for Ethernet Data Planes • 10:15-10:45 am Break • 12noon Lunch • 1:00pm Continue working on Exercise #1 • 2:00pm Overview of Web Services and OSCARS • 2:30-3:00pm Break • 3:00pm Exercise #2: IntraDomain provisioning with OSCARS • 5:00pm Adjourn Day 2 • 9:00am Overview of Inter-Domain implementation in OSCARS • 10:00am Exercise #3: Inter-domain Provisioning with OSCARS • 10:15-10:30am Break • 12noon Lunch • 1:00pm Continue working with Exercise #3 • 2:30-3:00pm Break • 3:00pm Use of Internet2 DCN and peering dynamic networks • 4pm Adjourn

8. Office of Science DOE OSCARS Workshop Perspective • In this workshop we focus on implementation • We will design and build a multi-domain GMPLS controlled ethernet network • We have a mobile GMPLS test and evaluation lab consisting of 24 PCs and 12 switches • We will be focused on the GMPLS intra-domain control plane issues • Specifically, OSPF and RSVP protocols and Path Computation • We will do a very brief and cursory review of RSVP and OSPF. • For detailed information on the protocols themselves see the IETF RFCs. • We will not deal with ISIS or CR/LDP or LMP • We will focus on the “DICE” Inter-domain architecture • Web Services based topology distribution and provisioning • We use open source software developed by the NSF DRAGON Project, the DOE OSCARS Project • Intra-domain: Adapted versions of KOM-RSVP and Zebra OSPF plus the NARB for path computing • This software is the only GMPLS software available to support dynamic ethernet services • Uses OSCARS (Dept of Energy) for book-ahead scheduling and AAA • Additional software and interfaces have been developed under auspices of the DICE effort (DANTE, Internet2, Canarie, ESnet) • The code has been adapted to support a wide variety of vendor equipment (e.g. Force10, Extreme, Dell, Ciena, Cisco, Raptor) DRAGON

9. Control Plane Data Plane DCN Workshop Architecture Internet2 Core Dynamic Circuit Network Green Pod ASN4 Red Pod ASN1 Yellow Pod ASN3 Blue Pod ASN2

10. Pod Network Elements Control and Data Planes Control Plane PC (VLSR#-PC, NARB, IDC) Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) Network Aware Resource Broker- “NARB” Inter-Domain Controller – “IDC” NARB / IDC gre6 Virtual Label Switching Router- “VLSR” VLSR2 gre2 gre4 VLSR3 VLSR1 VLSR3-PC gre3 VLSR1-PC D2 D4 VLSR3-SW VLSR1-SW D3 D5 D1 ES1 ES2

11. Dynamic NetworksOverview and Status • Objectives and of Dynamic Hybrid Networks • Hybrid Networking and the Global R&E Community • Standardization Efforts • Internet2 Dynamic Circuit Network (DCN) • Control Plane Software • Network Architecture

12. Hybrid Networking • There has been interest from many communities for the development of network architectures and mechanisms that utilize lower layers of the protocol stack along with IP at layer 3 • This has become known as “hybrid networking” • It is motivated by applications from the research and education community that require greater capabilities • High bandwidth flows (for example, flows that come close to saturating links in the shared IP backbone) • Flows with special requirements related to quality of service, for example jitter requirements • Network and Application Virtualization

13. Hybrid Networks - Motivating Factors • Hybrid networks are intended to provide a flexible mix of IP routed service and “lower layer services” • “flexible” means the network can respond quickly to user/application/connector requirements and requests to access both the IP Routed and/or lower layer services • “lower layer services” means access to layer 2 and below paths which can be utilized in a multitude of ways by creative users. • Typical user requirements for these lower layer services are based on: • critical, large bandwidth flows which may require one of more of the following: deterministic network performance, dedicated network resources, guaranteed network capacity, freedom to use protocols other than (congestion control friendly) TCP, privacy/security requirements, scheduled services • User/application communities which desire to build entire topologies which integrate domain specific resources along with dedicated network resources (which have one or more of the above mentioned characteristics)

14. Hybrid NetworksHeterogeneous By Nature Hybrid networks are extremely heterogeneous at several levels DataPlane can be constructed from router based Multiprotocol Label Switching (MPLS) tunnels Ethernet VLAN based Circuits Synchronous Optical Network / Synchronous Digital Hierarchy (SONET/SDH) circuits Wavelength Division Multiplexing (WDM) connections Combinations of the above

15. Hybrid NetworksHeterogeneous By Nature Control Planes can be based on Multiprotocol Label Switching (MPLS) Generalized Multiprotocol Label Switching (GMPLS) Web Services Management Systems Combinations of the above Client (user) services or attachment points could be Ethernet SONET IP Router InfiniBand

16. Multi-Domain, Multi-Layer Control Planes Key Requirements The “Multi-Layer” is meant to identify several items regarding how hybrid networks may be built. In this context it includes the following: Multi-Technology - MPLS, Ethernet, Ethernet PBB-TE, SONET, NG-SONET, T-MPLS, WDM Multi-Level - domains or network regions may operate in different routing areas/regions, and maybe be presented in an abstracted manner across area/region boundaries Multi-Domain indicates that we want to allow hybrid network service instantiation across multiple domains And of course all this implies that this will be a Multi-Vendor environment. Multi-Control – mpls, gmpls, management, vendor proprietary

17. Dynamic Network ServicesIntraDomain • Source Address • Destination Address • Bandwidth • VLAN TAG (untagged | any | tagged | tunnel) • User Identification (certificate) • Schedule Circuit Request Dynamically Provisioned Dedicated Resource Path (“Circuit”) DRAGON Enabled Control Plane Internet2 IDC Client B XML Ethernet Mapped SONET or SONET Circuits USER API Client A Internet2 DCN Service • api can run on the client, or in a separate machine, or from a web browser Actual Network Path

18. Dynamic Network Services InterDomain No difference from a client (user) perspective for InterDomain vs IntraDomain 2 2 USER API A A 1 XML RON Dynamic Infrastructure Ethernet VLAN RON Dynamic Infrastructure Ethernet VLAN Internet2 DCN Ethernet Mapped SONET A. Abstracted topology exchange 1. Client Service Request 2. Resource Scheduling 5. Service Instantiation (as a result of Signaling) Multi-Domain Dynamically Provisioned Circuit

19. DCN Control Plane

20. DCN Control Plane Software • OSCARS (Web Service) • Started by ESnet, merged with Internet2’s BRUW project in 2006 • Web service architecture, interfaces to lower level network specific provisioning systems • Vendor based MPLS L2VPN (Martini Draft) • Internet2 DCS/HOPI • DRAGON (NSF funded project in development by USC/ISI EAST and MAX) • Uses GMPLS protocols to build layer 2 circuits

21. I2 DCN Software Suite • OSCARS (IDC) • Web service layer, InterDomain messaging, AAA, Scheduling • DRAGON (DC) • Control of domain network elements (Core Directors and/or Ethernet Switches) • Intra and Inter Domain Path Computation • RSVP based signaling • Version 0.3.1 of DCNSS released April, 2008 • https://wiki.internet2.edu/confluence/display/DCNSS

22. OSCARS-DRAGON Integration

23. DRAGON • Virtual Label Switched Router(VLSR) • PC based control plane software • Manages and provisions various network equipment such as ethernet switches, SDH/SONET • Signaling with RSVP packets • Network Aware Resource Broker (NARB) • Stores topology in OSPF-TE database • Performs inter/intradomain path calculation • Exchanges interdomain topology

24. IDC - Web Service Based Definition • Four Primary Web Services Areas: • Topology Exchange, Resource Scheduling, Signaling, User Request

25. Other AAA Models Possible MetaScheduler Topology Topology Scheduling Scheduling Signaling Signaling • Meta-Scheduler Approach • Same set of Web Services used for linear instantiation model can be used by a high level process to build services: • Topology Exchange, Resource Scheduling, Signaling, User Request • A key issue is that this requires a trust relationship between the “meta-scheduler” and all the domains with which it needs to talk

26. InterDomain Controller (IDC) Protocol (IDCP) Developed via collaboration with multiple organizations Internet2, ESnet, GEANT2, Nortel, University of Amsterdam, others The following organizations have implemented/deployed systems which are compatible with this IDCP Internet2 Dynamic Circuit Network (DCN) ESNet Science Data Network (SDN) GÉANT2 AutoBahn System Nortel (via a wrapper on top of their commercial DRAC System) Surfnet (via use of above Nortel solution) LHCNet (use of I2 DCN Software Suite) Nysernet (use of I2 DCN Software Suite) University of Amsterdam (use of I2 DCN Software Suite) DRAGON Network The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP: LambdaStation (FermiLab) TeraPaths (Brookhaven) Phoebus

27. DCN – Global NetworkInteroperation via IDCP

28. InterDomain Controller Protocol Standardization Activities Standardization process and increasing community involvement continues Optical Grid Forum (OGF) Network Markup Language (NML) Working Group Standardizing topology schemas (perfsonar and control plane) Network Services Interface (NIS-WG) Grid High Performance Networking (GHPN) Research Group Network Measurement (NM-WG) Network Measurement Control (NMC-WG) Information Services (IS-WG) GLIF Control Plane Subgroup working on normalizing between various interdomain protocols (IDCP, G-Lambda GNS-WSI, Phosphorus API) Also other GLIF subgroups in this and related space (global id format, PerfSonar)

29. Internet2 DCN Working Group • DCN WG has been formed under NTAC • Chair: Linda Winkler (Argonne National Laboratory) • DCN WG will drive directions and set agenda in this area • Mailing list and Wiki available • dcn-wg@internet2.edu • https://spaces.internet2.edu/display/DCN/Home • DCN WG BOF on Monday, July 21, 12:30 PM 1:50 PM

30. Internet2 DCN Infrastructure

31. Internet2 DCN Services 1-A-5-1-1 1-A-6-1-1 1-A-6-1-1

32. DCN Services - circuits Physical Connection: 1 or 10 Gigabit Ethernet SONET (Future) Circuit Service: Point to Point Ethernet (VLAN) Framed SONET Circuit Point to Point SONET Circuit (future) Bandwidth provisioning in 100 Mbps increments How do Clients Request? Client must specify [VLAN ID | ANY ID | Untagged | Tunnel], SRC Address, DST Address, Bandwidth Request mechanism options are Web Service API, Web Page, phone call, email What is the definition of a Client? Anyone who connects to an ethernet or SONET port on an Ciena Core Director; could be RON, other wide area networks, domain specific applications

33. DCN Services - topologies Individual circuits are the “atomic” service provided by the DCN and control plane These circuits could be intra or inter domain It is envisioned that higher level “services” may be developed which coordinate the instantiation of multiple individual circuits to develop entire “topologies” co-scheduling/allocation of other resources (compute, data storage) may also be desired Probably a task for individual science/application domains or someone developing middleware on their behalf

34. Workshop Details

35. Control Plane Data Plane DCN Workshop Architecture Internet2 Core Dynamic Circuit Network Green Pod ASN4 Red Pod ASN1 Yellow Pod ASN3 Blue Pod ASN2

36. Pod Network Elements Control Plane PC (VLSR#-PC, NARB, IDC) Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) Inter-Domain Controller – “IDC” Network Aware Resource Broker- “NARB” NARB / IDC Virtual Label Switching Router- “VLSR” VLSR2 VLSR3 VLSR1 VLSR3-PC VLSR1-PC VLSR3-SW VLSR1-SW ES1 ES2

37. Basic Pod Data Plane Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) VLSR2-SW Ethernet Switch D2 D4 VLSR3-SW VLSR1-SW D3 D5 D1 ES1 ES2 End System Data Plane via Cat5 Patch Cable Data Plane (D#)

38. Control Plane PC (VLSR#-PC, NARB, IDC) End System (ES#) Basic Pod Control Plane Network Aware Resource Broker- “NARB” Inter-Domain Controller – “IDC” NARB / IDC Virtual Label Switching Router- “VLSR” gre6 gre2 gre4 gre3 VLSR3-PC VLSR1-PC

39. Pod Network Elements Control and Data Planes Control Plane PC (VLSR#-PC, NARB, IDC) Data Plane Ethernet Switch (VLSR#-SW) End System (ES#) Network Aware Resource Broker- “NARB” Inter-Domain Controller – “IDC” NARB / IDC gre6 Virtual Label Switching Router- “VLSR” VLSR2 gre2 gre4 VLSR3 VLSR1 VLSR3-PC gre3 VLSR1-PC D2 D4 VLSR3-SW VLSR1-SW D3 D5 D1 ES1 ES2

40. Pod Management Addressing “Red” pod: ASN=1 “Blue” pod: ASN=2 “Yellow” pod: ASN=3 “Green” pod: ASN=4 Workshop Gateway Router 192.168.1.1 Management VLAN 192.168.<asn>.n/16 .10 NARB / IDC VLSR2 .6 VLSR1 VLSR3 .5 eth0 .4 .8 .7 .3 .9 eth0 eth1 eth1 eth0 .2 ES1 ES2 eth0 - Management Plane Interface and Control Channel (PCs) eth1 - Data Plane Interfaces (PCs)

41. Rack Layout GW2 GW1 SW2 SW1 NARB NARB VLSR1-PC VLSR1-SW VLSR1-SW VLSR1-PC VLSR2-PC VLSR2-SW VLSR2-SW VLSR2-PC VLSR3-PC VLSR3-SW VLSR3-SW VLSR3-PC ES1 ES1 ES2 ES2 NARB NARB VLSR1-PC VLSR1-SW VLSR1-SW VLSR1-PC VLSR2-PC VLSR2-SW VLSR2-SW VLSR2-PC VLSR3-PC VLSR3-SW . VLSR3-SW VLSR3-PC ES1 ES1 ES2 ES2 123456789012345678901234567890 123456789012345678901234567890 Rack 2 Rack 1

42. Workshop Pods

43. Red Pod

44. Green Pod

45. Yellow Pod

46. Blue Pod

47. Exercise #1 Intra-Domain Detail(Answer Sheet) “Red” pod: ASN=1 “Blue” pod: ASN=2 “Yellow” pod: ASN=3 “Green” pod: ASN=4 Workshop Gateway Router 192.168.1.1 Management VLAN 192.168.<asn>.n/16 .10 NARB / IDC GRE6 10.a.6.2 VLSR2-PC 10.a.6.1 GRE2 10.a.2.2 GRE4 .6 10.a.4.1 VLSR2-SW VLSR1-PC VLSR3-PC 10.a.4.2 .5 10.a.2.1 eth0 .4 3 10.a.3.2 .8 1 4 11.a.4.1 D2 11.a.2.2 10.a.3.1 VLSR3-SW VLSR1-SW D4 4 GRE3 .7 4 11.a.2.1 11.a.4.2 .3 3 1 11.a.3.1 1 3 5 5 11.a.3.2 D1 D5 D3 .9 eth0 eth1 eth1 eth0 .2 ES1 ES2 Management VLAN 192.168.<asn>.n/16 Dynamic Data plane port group = g3-g24 Dynamic VLAN range = 100…200 GRE<x> = 10.<asn>.<x>.n / 30 GRE7= 10.1.7.0 / 30 TEaddr = 11.<asn>.<x>.n / 30

48. Exercise #1 Data and Control links NARB / IDC “Red” pod: N=1 “Blue” pod: ASN=2 “Yellow” pod: ASN=3 “Green” pod: ASN=4 GRE6 VLSR2 GRE4 GRE2 4 VLSR1 VLSR3 3 D4 D2 4 GRE3 4 3 5 3 5 D5 D1 D3 eth1 eth1 ES1 ES2

49. Login information • Wireless Network: • SSID: DCNworkshop • WPA Personal Key: Workshop! • Login to all VLSR, ES and NARB • ssh port 22 • username: user[1-16]; password: Workshop! • username: root; password: rootme • Login to all switches • telnet port 23 • username: admin; password: admin • OSCARS configuration; login to the NARB/IDC machine • ssh port 22 • username: tomcat55; password: dragon • OSCARS axis2 login • https://idc.<color>.pod.lan:8443/axis2/axis2-admin/ • username: admin; password: axis2 • OSCARS web user interface; • https://idc.<color>.pod.lan:8443/OSCARS/ • username: oscars-admin; password: oscars

50. Command Line Interface ports dragond 2611 ospfd 2604 (intra-domain) narb 2626 rce 2688 > telnet localhost 2611 > password: dragon Login information