Download ESnet 2009 Excellence.gov Awards Winners’ Panel Discussion

Energy Sciences Network
Enabling Virtual Science
June 9, 2009
Supporting Advanced Scientific Computing
Research • Basic Energy Sciences • Biological
and Environmental Research • Fusion Energy
Sciences • High Energy Physics • Nuclear Physics
Steve Cotter [email protected]
Dept. Head, Energy Sciences Network
Lawrence Berkeley National Lab
The Energy Sciences Network
The Department of Energy’s Office of Science is one of the largest supporters of
basic research in the physical sciences in the U.S.
• Directly supports the research of some 15,000 scientists, postdocs and graduate
students at DOE laboratories, universities, other Federal agencies, and industry
worldwide
• Operates major scientific facilities at DOE laboratories that that have participation by the
US and international research and education (R&E) community
Established in 1985, ESnet is the Department of Energy’s science networking
program whose responsibility is to provide the network infrastructure supporting
the missions of the Office of Science
• Enabling a new era in scientific discovery as we tackle global issues like climate change,
alternative energy/fuels and understanding the origins of the universe.
ESnet: Driven by Science
Networking needs of researchers are far different than commercial users.
Therefore, ESnet regularly explores the plans and processes of major
stakeholders to understand:
• The extreme data characteristics of instruments and facilities
–
How much data will be generated by instruments coming on-line over the next 5-10 years?
• The future process of science
–
How and where will the new data be analyzed and used – that is, how will the process of
doing science change over 5-10 years?
• SC Networking Requirements Workshops
–
–
2 workshops a year, rotating thru BES, BER, FES, NP, HEP, ASCR communities
Workshop reports: http://www.es.net/hypertext/requirements.html
Observing current and historical network traffic trends
• What do the trends in network patterns predict for future network needs?
Science: Driven by Data
Scientific data sets are growing exponentially
• Ability to generate data is exceeding our ability to store
and analyze
• Simulation systems and some observational devices grow
in capability with Moore’s Law
Petabyte (PB) data sets will soon be common:
• Climate modeling: estimates of the next IPCC data is in 10s
of petabytes
• Genome: JGI alone will have 0.5 petabyte of data this year
and double each year
• Particle physics: LHC is projected to produce 16 petabytes
of data per year
• Astrophysics: LSST and others will produce 5
petabytes/year
4
ESnet Challenges
ESnet is facing several challenges:
• Needed to scale economically to meet the demands of exponential growth in
scientific data and network traffic
– Avg. 72% YoY since 1990
– Growth is accelerating, rather than moderating
• Scientific collaborations were becoming more international, e.g. CERN, IPCC,
EAST Tokamak, etc.
• Fundamental change in traffic patterns:
– Pre-2005: Similar to commercial traffic - millions of small flows: email, web
browsing, video conferencing
– Post-2005: Large scientific instruments like the Large Hadron Collider in Cern or
supercomputers with petaflops of computing power are generating a smaller
number of very large data flows on the scale of 10’s of Gigabytes per second
•
Today the top 1000 flows are responsible for 90% of network traffic
Solution: ESnet4
ESnet4: a unique hybrid packet- & circuit-switched network
infrastructure specifically designed to handle massive amounts of
data
• Combines the flexibility and resiliency of IP routed networks with the
deterministic, high-speed capability of a circuit-switched infrastructure
Used commercially available technologies to create two logically
separate networks over which traffic seamlessly switches
• IP Network: One network for IP traffic using a single 10 Gbps circuit over which
ESnet provides audio/videoconferencing and data collaboration tools
• Science Data Network: Circuit-switched core network consisting of multiple 10
Gbps circuits connecting directly with other high-speed R&E networks and
utilizing Layer 2/3 switches
ESnet4 – June 2009
Japan (SINet)
Australia (AARNet)
Canada (CA*net4
Taiwan (TANet2)
Singaren
Transpac2
CUDI
KAREN/REANNZ
ODN Japan Telecom
America
NLR-Packetnet
Internet2
Korea (Kreonet2)
CA*net4
France
GLORIAD
(Russia, China)
Korea (Kreonet2
MREN
StarTap
Taiwan (TANet2,
ASCGNet)
SINet (Japan)
Russia (BINP)
PNNL
KAREN / REANNZ Transpac2
Internet2
Korea (kreonet2)
SINGAREN
Japan (SINet)
ODN Japan Telecom
GÉANT
- France, Germany,
Italy, UK, etc
CA*net4
INL
Ames
DOE
NREL
LLNL
FNAL
KCP
IARC
Bechtel-NV
GA
NETL
NNSA
ANL
LANL
ORNL
SNLA
Allied Signal
DOE-ALB
MAN LAN
(32 A of A)
PPPL
JLAB
NOAA
ARM
Yucca
NSTec
BNL
StarLight
FRGP
LBL
SLAC
GÉANT in Vienna
(via USLHCNet circuit)
CERN/LHCOPN
(USLHCnet:
DOE+CERN funded)
AMPATH
CLARA
(S. America)
OSTI
ORAU
SOX
SRS
Pantex
CUDI
(S. America)
IP router
SDN router
Optical node
DOE Lab
10G IP
10G SDN
20G SDN
NLR 10G
MAN
Lab Link
Peering Link
Solution: ESnet4
Key components tying the two networks together:
• ESnet developed and implemented the On-Demand Secure
Circuits and Advance Reservation System (OSCARS) protocol which
spans both networks.
– Allows scientists to request dedicated bandwidth to move large amounts of
data – up to terabytes at a time – across multiple network domains.
• Active participant in the perfSONAR consortium developing an
open, modular infrastructure of services and applications that
enables the gathering and sharing of network performance
information, and facilitates troubleshooting of problems across
network domains.
OSCARS: Multi-Domain Virtual Circuit Service
OSCARS Services
• Guaranteed bandwidth with resiliency: User specified bandwidth for primary
and backup paths - requested and managed in a Web Services framework
• Traffic isolation: Allows for high-performance, non-standard transport
mechanisms that cannot co-exist with commodity TCP-based transport
• Traffic engineering (for ESnet operations): Enables the engineering of explicit
paths to meet specific requirements
– e.g. bypass congested links; using higher bandwidth, lower latency paths; etc.
• Secure connections: Circuits are “secure” to the edges of the network (the site
boundary) because they are managed by the control plane of the network
which is highly secure and isolated from general traffic
• End-to-end, cross-domain connections between Labs and collaborating
institutions
OSCARS 0.5 Architecture (1Q09)
Tomcat
Web Service
Interface
Web Browser
User Interface
OSCARS
Web Service
Interface
Notification Broker
AAA
RMI
Core
• Resource Scheduler
• Path Computation Eng
• Path Setup Modules
BSS DB
RMI
Core
• Manage Subscriptions
• Forward Notifications
AAA DB
RMI
Core
• Authentication
• Authorization
• Auditing
Notify DB
OSCARS 0.6 Architecture (Target 3Q09)
Tomcat
Web Service
Interface
Web Browser
User Interface
Web Service
Interface
OSCARS
AAA
RMI
RMI
RMI
Core
• Resource Scheduler
PCE(s)
BSS DB
Notification
Broker
Core
• Manage Subscriptions
• Forward Notifications
Core
• Authentication
• Authorization
• Auditing
Path Setup
RMI
RMI
RMI
Core
Core
• Constrained
Path
Core
•
Constrained
Computations Path
• Constrained Path
Computations
Computations
RMI
AAA DB
Core
• Network Element
Interface
Notify DB
Modular PCE Function
Tomcat
R(S, D, Ts, Te, B, U)
Web Service
Interface
Web Browser
User Interface
Web Service
Interface
OSCARS
AAA
Path
Setup
RMI
Core
• Resource Scheduler
BSS DB
RMI
RMI
RMI
Core
• Manage Subscriptions
• Forward Notifications
Core
• Network Element
Interface
Notification
Broker
Core
• Authentication
• Authorization
• Auditing
AAA DB
Notify DB
Reservation: R(S, D, Ts, Te, B, U)
PCE List:
PCE(1, 2, …n)
Topology:
G0(Ts, Te)
PCE1
Reservation:
PCE List:
Topology:
R(S, D, Ts, Te, B, U)
PCE(1, 2, …, n)
G1(Ts, Te)
PCEn
PCE2
RMI
Core
• Constrained Path
Computations
Reservation: R(S, D, Ts, Te, B, U)
PCE List:
PCE(1, 2, …, n)
Topology:
Gn-1(Ts, Te)
RMI
Core
• Constrained Path
Computations
RMI
Core
• Constrained Path
Computations
Reservation: R(S, D, Ts, Te, B, U)
PCE List:
PCE(1, 2, …, n)
Topology:
Gn(Ts, Te)
ESnet’s Future Plans
ESnet recently designated to received ~$70M in ARRA
funds for an Advanced Networking Initiative
• Build a prototype wide area network to address our growing data
needs while accelerating the development of 100 Gbps networking
technologies
• Build a network testbed facility for researchers and industry
• Fund $5M in network research with the goal of near term
technology transfer to the production network
100 Gbps Prototype Network & Testbed
West Chicago MAN
600 W.
Chicago
PNNL
Starlight
ANL
USLHCNet
Ames
FNAL
LBL
BNL
StarLight
MAN LAN
PPPL
FNAL
SLAC
(32 A of A)
ANL
San Francisco
Bay Area MAN
JLAB
ORNL
LBNL
SLAC
JGI
NERSC
Atlanta MAN
ORNL
Chicago
IP router
SUNN
SDN router
Optical node
SNLL
LLNL
56 Marietta
SOX
Lab
SNV1
Houston
Nashville
Wash., DC
180 Peachtree
10G IP
10/20G SDN
NLR 10G
MAN
Lab Link
Experimental Optical Testbed
• Will consist of advanced network devices and components assembled to give
network and middleware researchers the capabilities to prototype ESnet
capabilities anticipated in the next decade.
• A community network R&D resource – the experimental facility will be open to
researchers and industry to conduct research activities
• Multi-layer dynamic network technologies - that can support advanced services
such as secure end-to-end on-demand bandwidth and circuits over Ethernet,
SONET, and optical transport network technologies
• Ability to test the automatic classification of large bulk data flows and move them
to a dedicated virtual circuit
• Network-aware application testing – provide opportunities for network
researchers and application developers such as Grid-based middleware, cyber
security services, and so on, to exploit advanced network capabilities in order to
enhance end-to-end performance and security
• Technology transfer to production networks – ESnet, as host of the facility, will
develop strategies to move mature technologies from testing mode to production
service
In Summary
The deployment of 100 Gbps technologies is causing us to rethink
our ‘two network’ strategy and the role for OSCARS
• Still believe that advanced reservations and end-to-end quality of service
guarantees have a role
With these next generation networks, two opportunities exist:
• Ability to carry out distributed science at an unprecedented scale and with
world-wide participation
• Unforeseen commercial applications that will develop using an innovative and
reliable infrastructure that allows people around the world across multiple
disciplines to exchange large datasets and analyses in an efficient way
Extra Slide - OSCARS Status
Community approach to supporting end-to-end virtual circuits in the R&E environment is
coordinated by the DICE (Dante, Internet2, Caltech, ESnet) working group
• Each organization potentially has their own InterDomain Controller approach (though the ESnet/Internet2
OSCARS code base is used by several organizations - flagged OSCARS/DCN)
• The DICE group has developed a standardized InterDomain Control Protocol (IDCP) for specifying the set up
of segments of end-to-end VCs
– While there are several very different InterDomain Controller implementations, they all speak IDCP and
support compatible data plane connections
• The following organizations have implemented/deployed systems which are compatible with the DICE IDCP:
– Internet2 Dynamic Circuit Network (OSCARS/DCN)
− LHCNet (OSCARS/DCN)
– ESnet Science Data Network (OSCARS/SDN)
− LEARN (Texas RON) (OSCARS/DCN)
– GÉANT2 AutoBahn System
− LONI (OSCARS/DCN)
– Nortel (via a wrapper on top of their commercial DRAC System)
− Northrop Grumman (OSCARS/DCN)
– Surfnet (via use of above Nortel solution)
− Nysernet (New York RON) (OSCARS/DCN)
– University of Amsterdam (OSCARS/DCN)
− DRAGON (U. Maryland/MAX) Network
• The following "higher level service applications" have adapted their existing systems to communicate via the
user request side of the IDCP:
– LambdaStation (FermiLab)
– TeraPaths (Brookhaven)
– Phoebus (UMd)
Extra Slide - Production OSCARS
Modifications required by FNAL and BNL
• Changed the reservation workflow, added a notification callback system, and added some parameters
to the OSCARS API to improve interoperability with automated provisioning agents such as
LambdaStation, Terapaths and Phoebus.
Operational VC support
• As of 12/2/08, there were 16 long-term production VCs instantiated, all of which support HEP
– 4 VCs terminate at BNL
– 2 VCs support LHC T0-T1 (primary and backup)
– 12 VCs terminate at FNAL
– 2 VCs support LHC T0-T1 (primary and backup)
– For BNL and FNAL LHC T0-T1 VCs, except for the ESnet PE router at BNL (bnl-mr1.es.net) and FNAL
(fnal-mr1-es.net), there are no other common nodes (router), ports (interfaces), or links between the
primary and backup VC.
• Short-term dynamic VCs
– Between 1/1/08 and 12/2/08, there were roughly 3650 successful HEP centric VCs reservations
• 1950 reservations initiated by BNL using Terapaths
• 1700 reservations initiated by FNAL using LambdaStation