Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
ESnet Joint Techs, Feb. 2005 William E. Johnston, ESnet Dept. Head and Senior Scientist R. P. Singh, Federal Project Manager Michael S. Collins, Stan Kluz, Joseph Burrescia, and James V. Gagliardi, ESnet Leads Gizella Kapus, Resource Manager and the ESnet Team Lawrence Berkeley National Laboratory 1 ESnet’s Mission Support the large-scale, collaborative science of DOE’s Office of Science Provide high reliability networking to support the operational traffic of the DOE Labs • Provide network services to other DOE facilities Provide leading-edge network and Grid services to support collaboration • ESnet is a component of the Office of Science infrastructure critical to the success of its research programs (program funded through Office of Advanced Scientific Computing Research / MICS; managed and operated by ESnet staff at LBNL) 2 ESnet Science Data Network (SDN) core ESnet Physical Network – mid 2005 High-Speed Interconnection of DOE Facilities and Major Science Collaborators Australia CA*net4 Taiwan (TANet2) Singaren CA*net4 France GLORIAD Kreonet2 MREN Netherlands StarTap TANet2 Taiwan (ASCC) SInet (Japan) Japan – Russia(BINP) CERN (DOE link) GEANT - Germany, France, Italy, UK, etc LIGO PNNL ESnet IP core MIT BNL JGI LBNL NERSC SLAC TWC QWEST ATM LLNL SNLL AMES FNAL INEEL-DC ORAU-DC ANL LLNL/LANL-DC PPPL MAE-E 4xLAB-DC KCP JLAB ORNL YUCCA MT GA Equinix GTN&NNSA PAIX-PA Equinix, etc. OSTI LANL ALB HUB ARM SNLA 42 end user sites Allied Signal Office Of Science Sponsored (22) NNSA Sponsored (12) Joint Sponsored (3) Other Sponsored (NSF LIGO, NOAA) Laboratory Sponsored (6) peering points SND core hubs IP core hubs high-speed peering points ORAU NOAA SRS ESnet IP core: Packet over SONET Optical Ring and Hubs International (high speed) 10 Gb/s SDN core 10G/s IP core 2.5 Gb/s IP core MAN rings (> 10 G/s) OC12 ATM (622 Mb/s) OC12 / GigEthernet OC3 (155 Mb/s) 45 Mb/s and less ESnet Logical Network: Peering and Routing Infrastructure ESnet peering points (connections to other networks) Australia CA*net4 Taiwan (TANet2) Singaren PNW-GPOP University GEANT - Germany - France - Italy - UK - etc SInet (Japan) KEK Japan – Russia (BINP) International Commercial SEA HUB CA*net4 France Kreonet2 Netherlands Taiwan (ASCC) CERN GLORIAD MREN StarTap TANet2 2 PEERS Distributed 6TAP 18 Peers Abilene 1 PEER LBNL CalREN2 1 PEER Abilene + 6 Universities Abilene 2 PEERS PAIX-W 36 PEERS 10 PEERS 16 PEERS MAX GPOP 13 PEERS 14 PEERS 2 PEERS GA MAE-E EQX-ASH 28 PEERS TECHnet 2 PEERS CENIC SDSC NYC HUBS LANL Abilene ATL HUB ESnet supports collaboration by providing full Internet access • manages the full complement of Global Internet routes (about 150,000 IPv4 from 180 peers) at 40 general/commercial peering points • high-speed peerings w/ Abilene and the international R&E networks. This is a lot of work, and is very visible, but provides full Internet access for DOE. Drivers for the Evolution of ESnet August, 2002 Workshop Organized by Office of Science Mary Anne Scott, Chair, Dave Bader, Steve Eckstrand. Marvin Frazier, Dale Koelling, Vicky White Workshop Panel Chairs Ray Bair, Deb Agarwal, Bill Johnston, Mike Wilde, Rick Stevens, Ian Foster, Dennis Gannon, Linda Winkler, Brian Tierney, Sandy Merola, and Charlie Catlett •The network and middleware requirements to support DOE science were developed by the OSC science community representing major DOE science disciplines o o o o Climate simulation Spallation Neutron Source facility Macromolecular Crystallography High Energy Physics experiments •The network is essential for: o o o o Magnetic Fusion Energy Sciences Chemical Sciences Bioinformatics (Nuclear Physics) long term (final stage) data analysis o “control loop” data analysis (influence an experiment in progress) o distributed, multidisciplinary simulation Available at www.es.net/#research 5 o Evolving Quantitative Science Requirements for Networks Science Areas Today End2End Throughput 5 years End2End Throughput 5-10 Years End2End Throughput Remarks High Energy Physics 0.5 Gb/s 100 Gb/s 1000 Gb/s high bulk throughput Climate (Data & Computation) 0.5 Gb/s 160-200 Gb/s N x 1000 Gb/s high bulk throughput SNS NanoScience Not yet started 1 Gb/s 1000 Gb/s + QoS for control channel remote control and time critical throughput Fusion Energy 0.066 Gb/s (500 MB/s burst) 0.198 Gb/s (500MB/ 20 sec. burst) N x 1000 Gb/s time critical throughput Astrophysics 0.013 Gb/s (1 TBy/week) N*N multicast 1000 Gb/s computational steering and collaborations Genomics Data & Computation 0.091 Gb/s (1 TBy/day) 100s of users 1000 Gb/s + QoS for control channel high throughput and steering 6 0 Aug, 04 Mar, 04 Oct, 03 May,03 Dec, 02 Jul, 02 Feb, 02 Sep, 01 Apr, 01 Nov, 00 Jun, 00 Jan, 00 Aug, 99 Mar, 99 Oct, 98 May, 98 Dec, 97 Jul, 97 Feb, 97 Sep, 96 Apr, 96 Nov, 95 Jun, 95 Jan, 95 Aug, 94 Mar, 94 Oct,93 May,93 Dec, 92 Jul, 92 Feb, 92 350 Sep, 91 400 Apr, 91 Nov, 90 Jun, 90 Jan, 90 TBytes/Month TByte/Month ESnet is Currently Transporting About 350 terabytes/mo. ESnet Monthly Accepted Traffic Traffic Through ESnet Monthly Accepted Jan., 1990 Dec. 2004 Dec,–2004 Annual growth in the past five years about 2.0x annually. 300 250 200 150 100 50 7 A Small Number of Science Users Account for a Significant Fraction of all ESnet Traffic 16 Top Flows - ESnet Host-to-Host, 2 Mo., 30 Day Averaged DOE LabInternational R&E 14 Total ESnet traffic (Dec, 2004) = 330 TBy TBytes/Month 12 Lab-U.S. R&E 10 Domestic Lab-Lab 8 International 6 4 1 3 2 2 Top 100 host-host flows = 99 TBy 0 37 34 31 28 25 22 19 16 13 10 7 4 1 Note that this data does not include intra-Lab traffic. ESnet ends at the Lab border routers, so science traffic on the Lab LANs is invisible to ESnet. TBytes/Month 8 6 4 2 1 2 3 4 5 6 7 8 9 LBNL U. Wisc. NERSC LBNL FNAL Karlsruhe (DE) NERSC NASA Ames NERSC LBNL SLAC (US) IN2P3 (FR) NERSC NASA Ames NERSC LBNL BNL (US) IN2P3 (FR) FNAL Johns Hopkins NERSC LBNL SLAC (US) IN2P3 (FR) FNAL SDSC NERSC LBNL SLAC (US) RAL (UK) FNAL MIT NERSC LBNL Fermilab (US) WestGrid (CA) Fermilab (US) WestGrid (CA) 10 ?? LBNL BNL LLNL FNAL MIT BNL LLNL 12 SLAC (US) RAL (UK) SLAC (US) INFN CNAF (IT) 14 LLNL NCAR BNL LLNL LIGO Caltech BNL LLNL 16 Fermilab (US) IN2P3 (FR) Top Flows - ESnet Host-to-Host, 2 Mo., 30 Day Averaged int dom intra Lab 0 10 9 ESnet Traffic • Since BaBar (SLAC high energy physics experiment) production started, the top 100 ESnet flows have consistently accounted for 30% - 50% of ESnet’s monthly total traffic • As LHC (CERN high energy physics accelerator) data starts to move, this will increase a lot (200-2000 times) o • Both LHC tier 1 (primary U.S. experiment data centers) are at DOE Labs – Fermilab and Brookhaven U.S. tier 2 (experiment data analysis) centers will be at universities – when they start pulling data from the tier 1 centers the traffic distribution will change a lot 10 Monitoring DOE Lab ↔ University Connectivity • Current monitor infrastructure (red&green) and target infrastructure • Uniform distribution around ESnet and around Abilene AsiaPac SEA CERN CERN Europe Europe LBNL Abilene FNAL ESnet OSU Japan Japan CHI NYC DEN SNV DC KC BNL IND Japan LA NCS SDG SDSC ALB ELP HOU DOE Labs w/ monitors Universities w/ monitors Initial site monitors network hubs high-speed cross connects: ESnet ↔ Internet2/Abilene ATL ESnet Abilene ORNL 11 ESnet Evolution • With the current architecture ESnet cannot address o the increasing reliability requirements - Labs and science experiments are insisting on network redundancy o the long-term bandwidth needs - LHC will need dedicated 10/20/30/40 Gb/s into and out of FNAL and BNL - Specific planning drivers include HEP, climate, SNS, ITER and SNAP, et al • The current core ring cannot handle the anticipated large science data flows at affordable cost • The current point-to-point tail circuits are neither reliable nor scalable to the required bandwidth New York (AOA) DOE sites ESnet Core Washington, DC (DC) Sunnyvale (SNV) El Paso (ELP) Atlanta (ATL) 12 ESnet Strategy – A New Architecture • Goals derived from science needs o o o • Fully redundant connectivity for every site High-speed access to the core for every site (at least 20 Gb/s) 100 Gbps national bandwidth by 2008 Three part strategy 1) Metropolitan Area Network (MAN) rings to provide dual site connectivity and much higher site-to-core bandwidth 2) A Science Data Network core for - large, high-speed science data flows multiply connecting MAN rings for protection against hub failure a platform for provisioned, guaranteed bandwidth circuits alternate path for production IP traffic 3) A High-reliability IP core (e.g. the current ESnet core) to address Lab operational requirements 13 ESnet MAN Architecture core router R&E peerings International peerings T320 ESnet SDN core ESnet production IP core core router switches managing multiple lambdas ESnet managed λ / circuit services ESnet production IP service ESnet management and monitoring 2-4 x 10 Gbps channels Lab Lab monitor site equip. ESnet managed λ / circuit services tunneled through the IP backbone Site gateway router Site LAN monitor Site gateway router Site LAN site equip. 14 New ESnet Strategy: Science Data Network + IP Core + MANs CERN AsiaPacific Seattle (SEA) Sunnyvale (SNV) GEANT (Europe) ESnet Science Data Network (2nd Core) Metropolitan Area Rings New York (AOA) Core loops ESnet IP Core Albuquerque (ALB) Existing IP core hubs SDN hubs New hubs Primary DOE Labs Possible new hubs El Paso (ELP) Washington, DC (DC) Atlanta (ATL) Tactics for Meeting Science Requirements – 2007/2008 AsiaPac SEA • 10 Gbps enterprise IP traffic • 40-60 Gbps circuit based transport CERN Aus. Europe Europe ESnet Science Data Network (2nd Core – 30-50 Gbps, National Lambda Rail) SNV Japan Japan CHI NYC DEN DC Metropolitan Area Rings Aus. ALB SDG ESnet IP Core (>10 Gbps ??) ATL ESnet hubs ESnet hubs ELP Metropolitan Area Rings Major DOE Office of Science Sites High-speed cross connects with Internet2/Abilene Production IP ESnet core High-impact science core 2.5 Gbs Lab supplied 10 Gbs Major international 10Gb/s 30Bg/s Future phases 40Gb/s 16 ESnet Services Supporting Science Collaboration • In addition to the high-bandwidth network connectivity for DOE Labs, ESnet provides several other services critical for collaboration • That is ESnet provides several “science services” – services that support the practice of science o Access to collaborators (“peering”) o Federated trust - identity authentication – PKI certificates – crypto tokens o Human collaboration – video, audio, and data conferencing 17 5250 5000 4750 4500 4250 4000 3750 3500 3250 3000 2750 2500 2250 2000 1750 1500 1250 1000 750 500 250 0 User Certificates Service Certificates Expired(+revoked) Certificates Total Certificates Issued Total Cert Requests Ja n Fe -0 3 b M - 03 ar Ap - 03 M r- 03 ay Ju -03 nJu 0 3 Au l-03 g Se - 03 p O - 03 ct No 03 v De -0 3 cJa 0 3 n Fe -0 4 b M - 04 ar Ap - 04 r M - 04 ay Ju -04 nJu 0 4 Au l-04 g Se - 04 p O - 04 ct No 04 v De -0 4 cJa 0 4 n05 No.of certificates or requests DOEGrids CA Usage Statistics Production service began in June 2003 User Certificates 1386 Total No. of Certificates 3569 Service Certificates 2168 Total No. of Requests 4776 Host/Other Certificates * FusionGRID CA certificates not included here. 15 Internal PKI SSL Server certificates 36 * Report as of Jan 11,200518 DOEGrids CA Usage - Virtual Organization Breakdown DOEGrids CA Statistics (Total Certs 3569) ANL 4.3% DOESG 0.5% ESG 1.0% ESnet 0.6% FusionGRID 7.4% *Others 38.9% * iVDGL 17.9% LBNL 1.8% NERSC 4.0% LCG 0.3% NCC-EPA 0.1% FNAL 8.6% PNNL PPDG 0.6% 13.4% ORNL 0.7% *DOE-NSF collab. 19 ESnet Collaboration Services: Production Services ISDN Audio, Data H.323 6-T1's 6-T1's 1- PRI .3.86 Production RADVISION ECS-500 Gatekeeper (DELL) .3.171 Production RADVISION ViaIP MCU .3.166 Production Web Latitude Server (DELL) .3.167 Production Latitude M3 AudioBridge Eastern Research .4.185 RADVISION Gateway H.323 Router .3.175 Production RADVISION ECS-500 Gatekeeper (DELL) .3.172 Production Codian MCU ESnet • • • • Web-based registration and audio/data bridge scheduling • Over 1000 registered users worldwide Ad-Hoc H.323 and H.320 videoconferencing Streaming on the Codian MCU using Quicktime or REAL “Guest” access to the Codian MCU via the worldwide Global Dialing System (GDS) 20 ESnet Collaboration Services: H.323 Video Conferencing • Radvision and Codian 70 ports on Radvision available at 384 kbps o 40 ports on Codian at 2 Mbps plus streaming o Usage leveled, but, expect increase in early 2005 (new groups joining ESnet Collaboration) o Radvision increase to 200 ports at 384 kbps by mid-2005 o H.323 MCU Port Hours 4500 4000 3500 3000 2500 2000 1500 1000 500 0 Sep-04 Oct-04 Nov-04 Dec-04 Jan-05 21 Conclusions • ESnet is an infrastructure that is critical to DOE’s science mission and that serves all of DOE • ESnet is working on providing the DOE mission science networking requirements with several new initiatives and a new architecture • ESnet is very different today in both planning and business approach and in goals than in the past 22