NGI: Quick Internet Technology Tour

1. A Quick Internet Technology tourwith special emphasis on NGI,the next generation Internet Lecture # 2 presented at the 26th International Nathiagali Summer College on Physics and Contemporary Needs, 25th June – 14th July, Nathiagali, Pakistan Olivier H. Martin CERN - IT Division June 2001 Olivier.Martin@cern.ch Internet Technology

2. Presentation Outline • Internet, what is it? • Internet Backbone Technologies (ATM, POS, PoWDM, MPLS) • Internet Routers • Internet circuits • Wave Division Multiplex (DWDM, CWDM) • IAB Workshop & State of the Internet • Next Generation Internet • Challenges ahead: • QoS • Gigabit/second file transfer • Security architecture • IPv4 to IPv6 transition & coexistence Internet Technology

3. Internet, what is it? • A network of networks with IP as the thin Inter-network layer, also serving as the insulation layer between layer2 and layer 4 and above. • There is a wealth of layer 2 access technolgies ranging from: • Ethernet (10/100/1000 BaseT) • FDDI • ATM • Packet over Sonet • HDLC • Wireless • Analog/Digital telephones • GSM • Satellite • ADSL, Cable TV • …………………. Internet Technology

4. Internet Technology

5. Internet Backbone Technologies (ATM) • ATM still ubiquitous in many large Internet backbones, especially in Europe: • Back in 1996, the fastest router and switch interfaces available were ATM based • ATM switch based core versus IP router based core. • There is nothing wrong with “cell based” switching, however there is a problem with the availability of very high speed ATM router interfaces (Segmentation And Re-assembly (SAR)). • Will stay as an access technology and for building Virtual Private Networks (VPN). • Offers unparalleled granularity and class of services. • MPLS (see later) can be seen as a form of “frame” ATM. Internet Technology

6. Internet Technology

7. Internet Backbone Technologies (POS) • Packet over SONET (POS) is definitely well ahead of ATM. • Very high speed interfaces available (i.e. 2.5Gbps (OC-48c) and more recently 10Gbps (OC-192c)) • Packet over WDM is becoming the norm (the idea is to bypass the Sonet/SDH layer, which is generally felt to be too heavy and expensive, and to perform the SONET APS (Automatic Protection Switching) function at layer 3, using MPLS). • Note that SONET frame format is still used. Internet Technology

8. IP IP ATM IP SONET/SDH SONET/SDH Optical Optical Optical IP Over Optical IP Over ATM IP Over SONET/SDH High Speed IP Network Transport Multiplexing, protection and management at every layer IP Signalling ATM SONET/SDH Optical B-ISDN Higher Speed, Lower cost, complexity and overhead Internet Technology

9. Internet Technology

10. Internet Backbone Technologies (MPLS/1) • MPLS (Multi-Protocol Label Switching) is an emerging IETF standard that is gaining impressive acceptance, especially with the traditional Telecom Operators and the large Internet Tier 1. • Recursive encapsulation mechanism that can be mapped over any layer 2 technology (e.g. ATM, but also POS). • Departure from destination based routing that has been plaguing the Internet since the beginning. • Fast packet switching performed on source, destination labels, as well as ToS. Like ATM VP/VC, MPLS labels only have local significance. • Better integration of layer 2 and 3 than in an IP over ATM network through the use of RSVP or LDP (Label Distribution Protocol). • Ideal for traffic engineering, QoS routing, VPN, IPv6 even. Internet Technology

11. Internet Backbone Technologies (MPLS/2) • MPLS provides 2 levels of VPNs: • Layer 3 (i.e.conventional VPNs) • Layer 2 (i.e encapsulation of various layer2 frame formats), e.g. • Ethernet • ATM • PPP • MPLS can also be used for circuit and/or wavelength channel restoration. • MPlS (MP”Lambda”S), GMPLS (Generalized MPLS) Internet Technology

12. Emerging Terabit Internet routers (1) • A number of startups are successfully challenging Cisco’s dominant position, e.g. • Juniper (M160), Avici (TSR), Nexabit/Lucent(NX64000), Pluris (TNR20000), Unisphere/Argon • C&W, Qwest,/KPNQwest, UUnet (MCI/Wordlcom) are using Juniper M160. • Some layer 2/3 switch vendors are also trying to enter the WAN market but with mixed success, so far (e.g. Foundry, Cabletron/Interasys): • less functionality • less performance Internet Technology

13. Emerging Terabit Internet routers (2) • Fastest routers are still relatively slow (i.e. less than 300Gbps),but things improved very recently Juniper M160, Cisco GSR 12416 (15*OC-192c). • Density (space) still a problem, e.g. GSR12016 can scale to 5Tbps (i.e.2*2.5 Tbps, but 17 racks needed! • ASICs are problematic: • Juniper is said to have packet re-ordering problems at very high speed • Every bug entails 6 month delay, this is reportedly what caused Cisco to be 12 months behind Juniper for 10Gbps interfaces. • New Cisco’s 7600 OSR family with PXF (Parallel Express Forwarding) technology. • OC-768c (40Gbps) coming, but…. • recent Avici annoucement at SuperComm 2001, BUT over “composite link” (i.e. 16*2.5 Gbps)! Internet Technology

14. Internet circuits • Internet Backbone Circuits • 622Mbps (OC-12c) common, 2.5Gbps (OC-48c) (almost) standard in large backbones, • 10Gbps (OC-192c) coming very fast, even in Europe (GEANT)! • But, partly because of the way the Internet is now organized (e.g. CDNs), reported usage is still very low, i.e. many backbones are grossly over-dimensioned. • Internet Access circuits • 34/45Mbps (common), 155Mbps (rare), 622Mbps (exceptional) • Unlike backbone circuits, the cost of local loops can be quite high. Internet Technology

15. Internet Backbone Speeds MBPS IP/ OC12c OC3c ATM-VCs T3 lines T1 Lines Internet Technology

16. DWDM, CWDM • Dense Wave Division Multiplex (DWDM) • Fiber optic technology has been making gigantic progress • This was needed in order to support the explosive growth of the Internet • and to remove bottlenecks on trans-oceanic routes, in particular. • The technology is evolving very fast in terms of: • number of channels, • capacity per channel, • distance without repeaters. • Coarse Wave Division Multiplex (CWDM) • Cheap form of WDM suitable for Metropolitan Area Networks (MAN) and/or substitute for local SONET/SDH local loops. Internet Technology

17. Internet Technology

18. Internet Technology

19. Internet Technology

20. Internet Technology

21. Internet Technology

22. Internet Technology

23. E M U X Transmission Systems of The Recent Past Low-rate Data Low-rate Data 30-50 km E D M U X XMTR Regen. Repeater Regen. Repeater RCVR Regenerative Receiver Transmitter (DFB Laser) Opto-Electronic Regenerative Repeaters Electronic Multiplexer Electronic Demuliplexer • Single channel operation • Opto-electronic regenerative repeaters - one per 50 km per fiber • 30-50 km repeater spacing • Capacity upgrades: increased speed Still Found In Legacy Network Systems Internet Technology

24. Today’s Transmission System l1 80-140 km XMTR RCVR l1 O M U X O D M U X XMTR RCVR l2 Regen. Repeater l2 ln XMTR RCVR ln Optical Demultiplexer Optical Multiplexer Optical Amplifiers • Multi-channel WDM operation • One amplifier supports many channels • 80-140km amplifier (repeater) spacing; regeneration required every 200-300 km • Capacity upgrades: adding wavelengths (channels) & increasing speeds However, regeneration is still very expensive and fixes the optical line rate Internet Technology

25. Next Generation…The Now Generation l1 80-140 km XMTR l1 O M U X O D M U X RCVR XMTR l2 RCVR l2 ln XMTR RCVR ln 1600 km Optical Demultiplexer Optical Multiplexer • Multi-channel WDM operation • One amplifier supports many channels • 80-140km amplifier (repeater) spacing; regeneration required only every 1600 km • Capacity upgrades: adding wavelengths (channels) & increasing speeds Over 1000 Km optically transparent research network tested on the Qwest network Internet Technology

26. Dare to extrapolate for the next 5 years(Yves Poppe/TeleGlobe)? • Will Moore’s law and related laws for growth of fiber transmission capacity and internet growth continue to apply? • Probably • The laws of gravity still apply, even in the New Economy. Progress alternates between periods of exponential growth and plateaus were the progress is absorbed. • Progress continues unabated: • Alcatel tested 10Tb over single fiber with 256 channels at 40Gb and demonstrated 3TB over 7300km using wide band EDFA • Intel announced chipsets for OC192 and 10GbE • Ciena announces 160 channels at 25GHz spacing • Although traditionally spacing in GHz=2.5x channel capacity in Gb Ciena claims to have 10Gbps using 12.5GHz spacing in lab Internet Technology

27. Internet Technology

28. Internet Technology

29. IAB Workshop • The Internet Architecture Board (IAB) held a workshop on the state of the Internet Network Layer in July 1999, a number of problem areas and possible solutions were identified: • Network/Port Address Translators (NAT/PAT), • Application Level Gateways (ALG) and their impact on existing and future Internet applications. • End to end transport & security requirements (IPSEC) • Transparency (e.g. H.323) • Realm Specific IP (RSIP). • Mobility (completely different set of protocol requirements) • IPv6 • Routing (growth of routing table, route convergence) • DNS (renumbering) Internet Technology

30. Internet Technology

31. Recent evolution of the Internet • The original Internet was: • unregulated, flat charge, simple protocols, few but open protocols, end to end transparency. • Today’s Internet is: • trends toward more regulations, flat charge, sometimes no charge even, but increasing number of paying services! • no longer simple, large number of plug-ins & proprietary protocols in use, end to end principle seem to have more or less disappeared. • Restricted client server model. • What about the peer to peer model? • Increasingly fragmented, • In such a restricted environment, what about the next killer application? • the threat is that everything may be layered on top of the Web (HTTP). • Napster, Gnutella, distributed games,…... Internet Technology

32. For web-based transactions: Sufficient to allow clients in private address spaces to access servers in global address space For telephones and I-Msg You need to use an address when you call them, and are therefore servers in private realm Client/Server Architecture is breaking down Private Address Realm Global Addressing Realm Private Address Realm Internet Technology

33. Loss of End to end transparency • Loss of end to end transparency due to: • proliferation of Firewalls, NATs, PATs • Web caches, Content Engines, Content Distribution Networks (CDN), • Application Level gateways, Proxies, etc. • Cons: • violation of end to end transport principle, • possible alteration of the data, • only partially fits the client-server model (i.e. server must be outside) • Pros: • better performance, service differentiation, SLA, • cheaper to deliver services to large number of recipients, etc. Internet Technology

34. But they cannot be relied on forever Projected routing table growth without CIDR Moore’s Law and CIDR made it work for a while Deployment Period of CIDR Growth in BGP Route Table Internet Technology Source: http//www.telstra.net/ops/bgptable.html

35. Routing and Addressing inthe Billion Node Network • Address Efficiency and Route Aggregation • Using addresses more efficiently • Adopt hierarchies within addresses allow for remote abstraction of routing information • Private Addressing .. Maybe! • Using less public addresses when we can • Network Address Translation (NAT) and Realm-Specific IP (RSIP) • Address extension • Getting more addressesby changing protocol platforms • IPv6 and the next address pool Internet Technology

36. Next generation Internet, what is it? • A natural evolution from what the Internet is today (or rather was yesterday)? • A completely new model following some technological revolution (e.g. all optical networks), or increased regulations, new economic/charging model (e.g. portals). • Being part of a Global community, we need to make sure that new technological developments properly take into consideration the constraints of each region, e.g. • limitations of transoceanic cables (i.e. lmited number of fiber pairs) • bandwidth in less networked advanced countries Internet Technology

37. Several major issues • Quality of Service (QoS) • High performance (i.e. wire speed) file transfer « end to end » • Will CDN technology help? • Is the evolution towards edge services likely to affect global GRID services? • Impact of security • Internet Fragmentation, one vs several Internets • e.g. GPRS top level domain • Transition to IPv6 and long term coexistence between IPv4 & IPv6 Internet Technology

38. Quality of Service (QoS) • Two approaches proposed by the IETF: • integrated services (intserv), • intserv is an end-to-end architecture based on RSVP that has poor scaling properties. • differentiated services (diffserv). • diffserv is a newer and simpler proposal that has much better chances to get deployed in some real Internet Service Providers environments, at least. • even though diffserv has good scaling properties and takes the right approach that most of the complexity must be pushed at the edges of the network, there are considerable problems with large diffserv deployments. • ATM is far from dead, but has serious scaling difficulties (e.g. TEN-155, Qwest/ATM). • MPLS is extremely promising, today it looks like it is where the future lies (including ATM AAL5 emulation!) Internet Technology

39. Quality of Service (QoS) • QoS is an increasing nightmare as the understanding of the implications are growing: • Delivering QoS at the edge and only at the edge is not sufficient to guarantee low jitter, delay bound communications, • Therefore complex functionality must also be introduced in Internet core routers, • is it compatible with ASICs, • is it worthwhile? • Is MPLS an adequate and scalable answer? • Is circuit oriented technology (e.g. dynamic wavelength) appropriate? • If so, for which scenarios? Internet Technology

40. Gigabit/second networking • The start of a new era: • Very rapid progress towards 10Gbps networking in both the Local (LAN) and Wide area (WAN) networking environments are being made. • 40Gbps is in sight on WANs, but what after? • The success of the LHC computing Grid critically depends on the availability of Gbps links between CERN and LHC regional centers. • What does it mean? • In theory: • 1GB file transferred in 11 seconds over a 1Gbps circuit (*) • 1TB file transfer would still require 3 hours • and 1PB file transfer would require 4 months • In practice: • major transmission protocol issues will need to be addressed (*) according to the 75% empirical rule Internet Technology

41. Internet Technology

42. Very high speed file transfer (1) • High performance switched LAN assumed: • requires time & money. • High performance WAN also assumed: • also requires money but is becoming possible. • very careful engineering mandatory. • Will remain very problematic especially over high bandwidth*delay paths: • Might force the use Jumbo Frames because of interactions between TCP/IP and link error rates. • Could possibly conflict with strong security requirements Internet Technology

43. Very high speed file transfer (2) • Following formula proposed by Matt Mathis/PSC (“The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm”) to approximate the maximum TCP throughput under periodic packet loss: (MSS/RTT)*(1/sqrt(p)) • where MSS is the maximum segment size, 1460 bytes, in practice,and “p” is the packet loss rate. • Are TCP's "congestion avoidance" algorithms compatible with high speed, long distance networks. • The "cut transmit rate in half on single packet loss and then increase the rate additively (1 MSS by RTT)" algorithm may simply not work. • New TCP/IP adaptations may be needed in order to better cope with “lfn”, e.g. TCP Vegas Internet Technology

44. Very high speed file transfer (3) • The Mathis formula shows the extreme variability of achievable TCP throughputs in the presence of, • even small, packet loss rates (i.e. less than 1%), • Small packets vs large packets (e.g. Jumbo frames), • Delay (RTT), also called long fat networks (lfn), i.e. with large bandwidth*delay products, hence the need for very large windows: • 3.3MB over 155Mbps link to Caltech and 170ms RTT. • and 53MB over 2.5Gbps to Caltech! • Consider a 10Gbs link with a RTT of 100ms and a TCP connection operating at 10Gbps: • the effect of a packet drop (due to link error) will drop the rate to 5Gbs. It will take 4 *MINUTES* for TCP to ramp back up to 10Gbps. • In order to stay in the regime of the TCP equation, 10 Gbit/s for a single stream of 1460 byte segments, a packet loss rate of about 1E-10 is required • i.e. you should lose packets about once every five hours. Internet Technology

45. Acceptable link error rates Internet Technology

46. Very high speed file transfer (tentative conclusions) • Tcp/ip fairness only exist between similar flows, i.e. • similar duration, • similar RTTs. • Tcp/ip congestion avoidance algorithms need to be revisited (e.g. Vegas rather then Reno/NewReno). • Current ways of circumventing the problem, e.g. • Multi-stream & parallel socket • just bandages or the practical solution to the problem? • Web100, a 3MUSD NSF project, might help enormously! • better TCP/IP instrumentation (MIB) • self-tuning • tools for measuring performance • improved FTP implementation • Non-Tcp/ip based transport solution, use of Forward Error Corrections (FEC), Early Congestion Notifications (ECN) rather than active queue management techniques (RED/WRED)? Internet Technology

47. CERN’s new firewall: technology and topology Gbit Ethernet Cabletron SSR Gbit Ethernet Fast Ethernet FastEthernet DxmonFE and FDDI+bridge CiscoPIX Cisco RSP7000 FastEthernet 100/1000 Ethernet FastEthernet Cabletron SSR Securitymonitor Internet Technology Gbit Ethernet

48. CERN’s New firewall: routing/recovery Cernh3 Cernh6 Cernh9 Fullrouting TEN-155 Static Backupdefault Default Cernh2 iBGP Policyrouting PIX Cernh8 iBGP Rca80 (BGP RR) OSPF(RIP2) CERN Gb backbone Internet Technology