1 / 43

Gopher GigaNet A Next Generation Campus Network

Gopher GigaNet A Next Generation Campus Network. David Farmer (farmer@umn.edu) Winter 2005 Joint Techs February 14 th 2005. Alternate Titles. How I spent my summer Without any Vacation Firewalls every where But not a Policy to implement Why MPLS Policy, Policy, Policy

erika
Download Presentation

Gopher GigaNet A Next Generation Campus Network

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Gopher GigaNetA Next Generation Campus Network David Farmer (farmer@umn.edu) Winter 2005 Joint Techs February 14th 2005

  2. Alternate Titles • How I spent my summer • Without any Vacation • Firewalls every where • But not a Policy to implement • Why MPLS • Policy, Policy, Policy • Or, I want to build a broken network,But still manage it

  3. Agenda • About UMN • The Old Network • Design Goals • Key Technologies We Picked • Architecture Components • The “Big Picture”

  4. Twin Cities CampusVital Statistics • 897 surface acres • East Bank, West Bank, St. Paul • 251 Buildings • 20 story Office Towers to Garden Sheds • Nearly 13M Assignable ft2 • Nearly 22M Gross ft2 • 50,954 Student Enrollment – Fall 2004 • Second Largest Nationally (first only 41 more) • Ranked 10th in total research

  5. Twin Cities CampusNetwork Statistics • More than 200 on-net Buildings • 1730 Wire Centers (Closets or Terminal Panels) • 842 With Network Electronics • 2774 Edge Access Switches (3750G-24TS) • 312 Aggregation Switches (3750G-12S) • 29 Core Switches (6509-NEB-A) • 5000 Virtual Firewall Instances

  6. The Old Network • Originally installed Sept ’97 – Dec ’99 • Took way too long • 10Mb Switched Ethernet to desktop • Small amount of 100Mb for high-end desktops and servers • Typically multiple 100Mb building links • Partial-Mesh OC3 ATM backbone

  7. The Old Network • Cisco 1924 Closet Switches • 4 switches per 100Mb uplink • Cisco 2924M-XL Closet Switches • Used for small amounts of 100Mb for servers and desktops • single switch with two 100Mb uplinks • Cisco 5500 Core Switches • With RSMs for routing • 25 Core Nodes • FORE ASX-200 and ASX-1000 ATM switches for Core network

  8. The Old Network – Midlife Upgrade • Installed Aug ’00 • Added GigE Backbone • Cisco 5500 Core Switches • Upgraded to Sup3s with GigE uplinks & MLS • Foundry BigIron • Center of Star Topology GigE Backbone

  9. Design Goals • Divorce Logical and Physical Topologies • Provide more than 4096 VLANs network wide • “Advanced” Services • Routed (L3) Core, Switched (L2) Aggregation and Edge • Network Policy – AKA Security • Network Intercept • Other Stuff

  10. Design Goals • Divorce Logical and Physical Topologies • Administrative Topology • Policy Topology • Security or Firewalls • Bandwidth shaping or Usage • QOS • Functional or Workgroup Topology

  11. Design Goals • Provide more than 4096 VLANs network wide • More than 1000 VLANs now • Micro segmentation for Security and other Policy could easily require 4X over the next 5 years • Even if we don’t exceed 4096 VLANs, the VLAN number space will be very full

  12. Design Goals • “Advanced” Services • Native IPv4 Multicast • PIM Sparse Mode, MSDP, BGP for Routing • IGMP v3 (SSM support) for L2 switching • IPv6 • Unicast for sure • Multicast best shot • Jumbo Frame • 9000 Clean

  13. Design Goals • Routed (L3) Core, Switched (L2) Aggregation and Edge • How many L3 control points do you want to configure • Limit scope of Spanning Tree • If possible eliminate Spanning Tree • Minimally, limit it to protecting against mistakes, NOT an active part of the Network Design

  14. Design Goals • Network Policy – AKA Security • Security is, at least partly, the network’s problem • Let’s design it in to the network, rather than add it in as an after thought • The network needs to enforce Policies • Only some of these are actually truly related to Security • Rate Shaping, COS/QOS, AAA, just to name a few • Firewalls with state-full inspection are necessary in some locations • Network Authentication (802.1x)

  15. Design Goals • Network Intercept • Intrusion Detection and Prevention • Trouble shooting • Measurement and Analysis • Legal Intercept and Evidence collection • Sinkhole Routing

  16. Design Goals • Other Stuff • Core Services • DNS • DHCP • NTP • Measurement • Localized Logging • Syslog • Netflow

  17. Design Goals • Other Stuff • Data Centers • Intend to support 6 – 12 Data Centers on campus • Create Separate Infrastructure • Allows different maintenance windows • Provide Higher SLA/SLE • Provide things that can’t scale to the rest of campus • Server load balancing • Dual fiber entrances • Single L2 Domain • Redundant Routers

  18. Design Goals • Other Stuff • Management Network • Console Servers • Remote Power Control • Redundant GigE network • Allow access to critical Core Network equipment at all times • Dial-up Modem on Console Server for Emergency Backup

  19. Key Technologies We Picked • MPLS VPNs • Cisco StackWise Bus on 3750s • Cross Stack EtherChannel provides redundancy without creating loops in the Spanning Tree topology • Cisco FWSM with Transparent Virtual Firewalls • Policy as L2 bumps on the wire • Let the Routers Route

  20. How to Scale • A network with those numbers doesn’t fit in your head • My mind is to small to hold it all • How about yours • “consistency is the hobgoblin of little minds” • Emerson • Consistency is the answer to Scaling

  21. MPLS VPNs – Short Tutorial • RFC 2547 defines layer 3 routed MPLS VPNs • Uses BGP for routing of VPNs • Routers create a VRF (VPN Routing & Forwarding) Instance • VRFs are to Routers as VLANs are to Ethernet Switches

  22. MPLS VPNs – Short Tutorial • P – “Provider” Router • No knowledge of customer VPNs • Strictly routes MPLS tagged packets • PE – “Provider Edge” Router • Knowledge of customer VPNs & provider network • Routes packets from customer network across the provider network by adding VPN MPLS tag and tag for the remote PE

  23. MPLS VPNs – Short Tutorial • CE – “Customer Edge” Router • No knowledge of provider network • Strictly routes IP packets to PE • Only PE routers are necessary in the MPLS VPN Architecture • This is important in a Campus Network

  24. Example Campus MPLS VPN Architecture

  25. Architecture Components • Campus Border • Core Network • Aggregation Networks • Edge Nodes

  26. Campus Border • Border Routers • Redundant routers in diverse locations • Act as CE routers for all VRFs that need Internet Access • Cisco 6509 • Dual SUP720-3BXL • Dual Power Supplies and Fans • All 6700 Series Interface Cards

  27. Campus Border • Border Policy Enforcement • Layer 2 bumps on the wire • Cisco FWSM • Packeteer 9500 • Home grown ResNet Authentication Control & Scanner (RACS) • Attach to or contained within Border Router • Packets get a little dizzy passing through Border Router L2 or L3 switching fabric several times

  28. Core Network • Backbone Nodes • 2 Backbone Nodes producing a Dual-Star Topology • Collocated with the Border Routers • 10Gb interconnection between Backbone Nodes. • 10Gb connection to each Core Node • Cisco 6509

  29. Core Network • Core Nodes • Located at 16 Fiber aggregation sites around campus • 10Gb connection to each Backbone Node • 2 or 3Gb to Aggregators or Edge Nodes • Cisco 6509-NEB-A

  30. Core Network • Core Nodes • Layer 3 routing provide for End User Subnets • Layer 3 MPLS VPNs provide separate Routing Domains • Virtual Firewalls provided per Subnets as needed • Root of a VLAN Domain • 802.1q tags have local significance only • VLANs connected between Core Nodes using Layer 2 MPLS VPNs as needed

  31. Aggregation Networks • Layer 2 only • Aggregates Edge Nodes & connects them to a Core Node • Cisco 3750G-12S

  32. Aggregation Networks • Regional Aggregator • 3Gb Connection to Core Node • Area Aggregator • 3Gb Connection to Regional Distribution Node • Building Aggregator • 2 or 3Gb Connection to Regional or Area Dist. Node or directly to Core Node

  33. Edge Nodes • Connects users and servers to the Network • Connects to a Building Aggregator • If more than one closet in a building • Otherwise connects to • Core Node • Regional Aggregator • Area Aggregator • Cisco 3750G-24TS

  34. Typical Building

  35. Data Center Networks • Data Center Core Nodes • Redundant Routers servicing all Data Centers on Campus • Collocated with the Border Routers and Backbone Nodes • 10Gb interconnection between Data Center Core Nodes. • 10Gb connection to each Backbone Node • 2Gb up to 10G connection to each Data Center • Cisco 6509-NEB-A

  36. Data Center Networks • Data Center Aggregator • Connected to both Data Center Core Nodes • Two 3750G-12S or two 3750G-16TD • Feeds Data Center Edge Nodes within a single Data Center

  37. Data Center Networks • Data Center Edge Nodes • Min Stack of two 3750G-24TS • Connects to Data Center Aggregator • Or directly to Data Center Core Node if a single stack serves the Data Center • Want hosts to EtherChannel to separate switches in the Stack for redundancy

  38. Management Network • Management Node • 3750G-24TS collocated with each Core Node • Routed as part of Control Plane & Management network • Cyclades Console server and Remote Power Control • Management Aggregator • Connects all the Mgmt Nodes

  39. Management Network • Measurement Server collocated with each Core Node • Log Server Collocated with each Core Node • DNS, DHCP, NTP Server Collocated with each Core Node • Using Anycast for DNS Redundancy

  40. Analysis Network • Analysis Node • All switches collocated in single location • Provides access to every Core Node for testing and Analysis • Provides for remote packet sniffing of any traffic on campus • Provides Sinkhole Drains for each Core Node

  41. A Closer Look

  42. That’s enough • That’s enough rambling for now! • I real want to do more, but! • Find me and lets talk more! • I’ll even argue if you want • Email me (farmer@umn.edu)

More Related