The International Grid Testbed: a 10 Gigabit Ethernet success story

1. The International Grid Testbed: a 10 Gigabit Ethernet success story in memoriam Bob Dobinson Catalin Meirosu on behalf of the IGT collaboration GNEW 2004, Geneva

2. Contents • www: who, why, what is IGT • Native Ethernet over lightpaths: from 1 Gbps to 10 Gbps • First demonstration of transatlantic native 10 GE over legacy OC-192 infrastructure • Future plans • Conclusions GNEW 2004

3. Who’s who in IGT • Universities and Research Institutes • Canada • Carleton University, Ottawa • University of Alberta • University of Victoria • TRIUMF, Vancouver • Université de Montreal • McGill University, Montreal • University of Toronto • The Netherlands • University of Amsterdam • Switzerland • CERN, Geneva • Connectivity provided by CANARIE SURFnet ORION Netera BCnet RISQ • Sources of funding: IGT (Canarie Directed Research Grant), ESTA (IST-2001-33182) • Collaboration: CERN Openlab, the EU DataTAG project GNEW 2004

4. Current IGT topology GNEW 2004

5. Motivation • IGT is a research network for next-generation e-science • Illustration of the application empowered network concept (see Bill St. Arnaud’s panel at SC2003) • Investigate emerging technologies in a demanding real life environment (see Wade Hong’s presentation at the CA*Net 4 Design Meeting May 26-27, 2003 ) • Real Time Farms for the ATLAS experiment at CERN (see Bryan Caron’s presentation at RISQ2003) GNEW 2004

6. The ATLAS Experiment at CERN • Large collaboration of about 2000 scientists for an experiment operational from 2007 • Main goals: the discovery of new particles and exploring physics beyond the Standard Model • Challenging data collection and analysis systems • Online: 20 Gbps to the event filter • Offline: 4 TB/day to storage GNEW 2004

7. Ethernet: from local to global • Ethernet, a brand name for LAN technology • Original Ethernet: shared media, half-duplex, distance limited by protocol • Modern Ethernet: point-to-point, full-duplex, switched, distance limited by the optical components • Cost effective ! GNEW 2004

8. Why native Ethernet long haul? • More than 90% of the Internet traffic originates on an Ethernet LAN • Data traffic on the LAN increases due to new applications • Ethernet services with incremental bandwidth offer new business opportunities to carriers • See IEEE Communications Magazine, Vol. 42, No. 3, March 2004, on additional benefits for both the enterprise and the service providers • Why not native Ethernet ? • Scalability, reliability, service guarantees … • All of the above are active research areas • Native Ethernet long haul connections can be used today as a complement to the routed networks, not as a replacement GNEW 2004

9. 10 GE: a new Ethernet • 10 GE – designed from the beginning for access to long haul networks • 40 km maximum distance specified by the standard … • 1550nm lasers: optical amplifiers can be used to increase distance over dark fibre • State of the art: 250 km demonstrated in Denmark by the EU ESTA project [see Mikkel Olesen’s presentation at NORDunet2003] • what happens when you have to regenerate the signal ? • no signal-agnostic regenerators deployed GNEW 2004

10. LTE LTE The 10 GE WAN PHY traditional • 10GE introduces a gateway from LAN to the WAN by means of the WAN PHY • Compatible with existing WAN infrastructure • Transmission rate • Encapsulation • Partial use of the management bits of the SONET/SDH frame • Today’s WAN PHY modules use SONET-compliant optical components Router OC192 LTE Router WAN LTE OC192 3R 3R 3R WAN PHY WAN PHY novel 10GE switch/router 10GE switch/router GNEW 2004

11. HP Itanium-2 10GE LAN PHY 10GE WAN PHY OC192c Intel Itanium-2 HP Itanium-2 Intel Xeon Ixia 400T Cisco ONS 15454 Cisco ONS 15454 Cisco ONS 15454 Cisco ONS 15454 Cisco ONS 15454 Force10E 600 Force10 E 600 Ixia 400T Geneva Ottawa Toronto Chicago Amsterdam Demo during ITU Telecom World '03 10 GE WAN PHY over an OC-192c circuit using lightpaths provided by SURFnet and CANARIE 9.24 Gbps using traffic generators 6 Gbps using UDP on PCs 5.65 Gbps using TCP on PCs GNEW 2004

12. Results on the transatlantic 10 GE Single stream UDP throughput Single stream TCP throughput • Data rates are limited by the PC, even for our memory-to-memory tests • UDP uses less resources than TCP on high bandwidth-delay product networks GNEW 2004

13. WAN PHY over DWDM Force10 E600 HP Itanium-2 DWDM Ixia 400T 10 GE WAN 10GE LAN Intel Xeon Force10 E600 HP Itanium-2 Ixia 400T 10 GE WAN 10GE LAN DWDM Amsterdam Geneva • Direct lambda access from the provider is required • The DWDM transceiver card as “LTE” HP Itanium-2 GNEW 2004

14. What next ? • More 10 Gigabit Ethernet experiments • Extend the reach of our WAN PHY connection • Adds more latency into the testbed, hence increased pressure on protocols • Might allow us to identify technical showstoppers • Disk-to-disk transfers • Comparative study of data transfer protocols for 10 Gbps networks GNEW 2004

15. What next ? • Support for real-time access to remote computer farms during the ATLAS testbeam run in 2004 • 1 Gbps connectivity is sufficient as a proof of concept • Sites in Copenhagen, Cracow and Edmonton will process in real time data acquired at the CERN testbeam • This is part of the feasibility study for remote real time processing in ATLAS GNEW 2004

16. Conclusions • IGT demonstrated native 10 Gigabit Ethernet over lightpaths • 10GE WAN PHY is the technology that enables inter-continental native Ethernet • IGT will support remote real time applications, with emphasis on data collection and analysis GNEW 2004

17. References • Bill St. Arnaud – contribution to the “Strategies for Application-Empowered Networks” panel at SC2003 • Bryan Caron – “The GARDEN Project: A Testbed for High Bandwidth Real-Time Applications”, http://www.risq.qc.ca/risq2003-canw2003/ppt/23PM_Bryan_Caron.pdf • Wade Hong – “CA*net 4 International Grid Testbed ”, http://lightpath.physics.carleton.ca/C4DesignIGTMay27.ppt • Mikkel Olesen – “The use of dark fibre in Forskningsnttet”, the 21st NORUnet Conference, http://www.nordunet2003.is/smasidur/presentations/MikkelOlesen.ppt GNEW 2004