“Internet2 Presentation”

1. “Internet2 Presentation” Brian Stengel, Director of Operations, Kinber Peter J. Heverin, Project Manager, Kinber Mike Carey, Network Engineer, Kinber Jon Paul Herron, Director of Engineering, GlobalNOC Luke Fowler, Senior Manager, Systems Engineering, GlobalNOC Marianne Chitwood, Director of Operations, GlobalNOC September 20th, 2012

2. What is PennREN a high-speed, state-wide, research & education serving healthcare, K-20 and the public good • Capital Budget - $128,958,031 • Federal Stimulus Funds - $99,660,678 • Matching Funds - $29,297,353 • Outside Plant Infrastructure Constructed for PennREN • 48 Stands of NZD Fiber optic Cable • 1700± Route Miles • Outsourced Fiber Maintenance • 13 Optical Regeneration Service Nodes • 56 Service Distribution Access Nodes

3. Project Route Total PennREN Fiber route is estimated at 1,613 miles. The route consists of the following; • 1,086miles of new aerial construction. • 486 miles of aerial overlash. • 14 miles of new underground construction. • 27 miles of leased underground conduit. Network Backbone Engineering is 95% complete overall. 3

4. Project Route 4

5. PennREN Locations

6. Project Teams • Plan/Design • Consulting • Engineering • Support • Training • Plan/Design • Optical engineering • Field services • Professional services • Fiber OSP • Building entrances • Cabinets, Installation • Splicing • Testing • Last mile solutions • Hardware procurement • Design • Integration • Project Management • Professional Services • Staging • Shipping • Configuration • Testing • Turn-up • Field Services • Service Desk, Tier 1, Tier 2 • Systems and Network Tools • Network Engineering • Net, Perf, Change Management • Operations support • Earthlink sourcing for OOB MPLS Network • Last mile solutions

7. Node Installation Status Under Construction Segment in Production Equipment Installed

8. Co-Location at Member Sites Construction Diagrams

9. Optical Core/DWDM System

10. Packet Core – MPLS

11. Plan – Build – Integrate

12. Peering Points (Initial) Internet Internet R&E R&E

13. Network Management PS1 Performance Measurement Servers at every Service Node PS2 PS3 In-band management Out-of-band access External Networks DB External Networks

14. Operations - Support • GRNOC provides to KINBER: • Service Desk – 24x7x365 call center support, ticket management, technical support coordination, and workflow support • Network Engineering – Expert network engineers work with the Service Desk to ensure fast problem resolution, provisioning, and strategic engineering and planning • Software and Systems – Provides support through a fully integrated system of network management, measurement, and visualization tools • The GlobalNOC at Indiana University provides carrier-grade operations, tools, and network expertise while placing a singular focus on the unique requirements of the research and education (R&E) Community • GRNOC supports 20+ R&E networks across the country

15. Operations - Maintenance • Warranty • Hardware replacement (pre-ship) • Software maintenance • Fiber Maintenance • Emergency Restoral • Routine Maintenance • OSP Records • Warranty • Hardware replacement (pre-ship) • Software maintenance • Host IT/Site teams provide remote eyes and hands support upon request • KINBER Network Engineers

16. Foundation for Services Multi-Degree ROADMs MPLS PE Switches 3rd Party Providers West DWDM Ring East DWDM Ring ASBR Router External Networks External Networks

17. On-Net: Delivery AccessNode AccessNode AccessNode AccessNode Access Nodes Internet Internet R&E R&E

18. Off-Net: Delivery MemberOff-Net Carrier CX First/Last Mile Commercial Co-Lo Private NNI Internet Internet R&E R&E

19. EPC – Ethernet Port Connection Service Customer Router PennREN R&E networks VPLS (pt – pt, multipoint) KMEX Service Node: PE Switch Commodity Internet Customer Subscribes to 1/10GE Ethernet Port(s)

20. Services • Services available to a customer with an EPC Member B Member A KINBER Member Exchange R&E Transit VPLS – Pt-Pt, Multipoint Commodity Internet Member C Member A

21. Services • Community-wide, distributed service for members to exchange traffic across a common network. Similar to an Internet Exchange • Member-to-Member peering, ad-hoc R&E activities • Best-effort traffic exchange within the community KINBER Member Exchange R&E Transit • Transit service to major R&E networks such as Internet 2, ESNet, NLR… • Provided by KINBER affiliates • Virtual private networks with committed bandwidth can be established using VPLS instances in the PennREN network • VPWS – Virtual Private Wire Service – Point to Point • VPLS – Virtual Private LAN Service - Multipoint VPLS – Virtual Private LAN Service(Point-to-PointMulti-Point) Commodity Internet • Access to commodity Internet service is available over the PennREN network

22. Optical – Wave/Lambda Services • Optical Waves 10G can be provisioned across the network Member A Member A Member B Member A Member B

23. Performance Measurement • PerfSONAR Measurement Archives allow exchange of data with other network operators • Regularly scheduled testing across the backbone • User-initiated testing for applications such as problem diagnosis • Multiple routing tables allow us to support both 1G and 10G testpoints on a single host

24. Performance Measurement Deploying performance measurement servers at each service node 3 servers per site: • Active throughput measurement (1G and 10G) • Active latency measurement • local data collection / ad hoc performance measurement Specific performance measurement tools include: BWCTL OWAMP MaDDash 24

25. Network Measurement • GlobalNOC tool-set updated to support all PennREN devices, including: • SNAPP – High-resolution SNMP-based network utilization data • LLAMA – DWDM layer performance measurement data • Central storage of other passive data like syslog, configuration, flow, etc. for regular and ad-hoc processing & analysis.

26. PennREN Service Desk • GlobalNOC Specialized Support Technician • Footprints PennREN project created – integrated to TickMon, Operations Calendars, Trouble Tickets • Telephone number for PennREN customers integrated into shared GlobalNOC phone queue • Email established noc@pennren.net to receive customer inquiries and or communications from vendors….this is monitored 24x7x365

27. PennREN Service Desk • Network and Member impact guidelines defined • Web form for customers/vendors to submit trouble tickets into Footprints • Change management process/form created • Internal documentation developed/published for staff training and reference

28. PennREN Service Desk • Pro-active network monitoring • Support for scheduled maintenances and changes • Vendor coordination • Customer install process • Reporting • Security • Tools and Communications

29. Weighing network needs

30. A new network is more than a construction project • It’s a system: • Goals • Infrastructure • People • Services • Operations • Business • Make decisions and plans early (also, there will be more decisions than you think, so watch for bottlenecks) • The work on all of these areas starts right away

31. Service Definition • Unrealistic to imagine services will be completely defined from the beginning of the design • But, enough understanding is needed to guide the design and build plan • Early “anchor” users help a lot!

32. Communications • Communication needs: • to be high-bandwidth • Early is good • Changing staff can be disruptive, do it carefully. • Informal is good • Multi-channel is good • reliable technologies are good • Face to face is good

33. Documentation • Documentation: • Have a place for documents • Keep the place for documents clean • Keep the purpose of each document clear/distinct

34. Operations • Operations Preparedness: • Start early, there are a LOT of things to think about, especially: • Turn-up/acceptance process • Expectations for facilities • How and where to keep network data

35. Build-outs make for lots of information • Need to be ready to put it somewhere where it’ll be usable later • Data entry is cheap • Entering/documenting is best when the information and its context are fresh

36. The lowly management network • Sometimes doesn’t get enough attention • It can get crazy complicated or crazy expensive • NOTE: in an SDN world, this becomes even MORE important!