Download cewit 2008 - TeraPaths

Center of Excellence
Wireless and Information Technology
TeraPaths: Managing Flow-Based
End-to-End QoS Paths
Experience and Lessons Learned
Dimitrios Katramatos, Dantong Yu, Kunal Shroff
Brookhaven National Laboratory
Thomas Robertazzi
Stony Brook University
Shawn McKee
University of Michigan
CEWIT 2008
Center of Excellence
Wireless and Information Technology
Abstract
• TeraPaths is a Department of Energy funded network
research project to support efficient, predicable, and
prioritized peta-scale data replication in modern highspeed networks
• The TeraPaths network management framework
establishes on-demand and manages true end-to-end,
QoS-aware, virtual network paths across multiple
administrative network domains
• TeraPaths dedicates network resources to data flows
specifically authorized to use such network paths, in a
transparent and scalable manner. This ensures that
only selected flows receive a pre-determined,
guaranteed level of QoS in terms of bandwidth, jitter,
delay, etc.
CEWIT 2008
2
Center of Excellence
Wireless and Information Technology
Speaker’s Biography
• Dantong Yu
Brookhaven National Laboratory
• Dantong Yu received the Ph.D. degree in Computer
Science from State University of New York at Buffalo,
USA, in 2001. His research interests include highspeed network performance, network Quality of
Service, cluster/grid computing, information retrieval,
data mining, databases, and data warehouses. He
leads the large volume WAN data transfer between
CERN, BNL, ATLAS and RHIC collaboration institutes
over high-speed networks with Grid middleware
CEWIT 2008
3
Center of Excellence
Wireless and Information Technology
Outline
•
•
•
•
•
•
Background: the TeraPaths project
Establishing flow-based end-to-end QoS paths
Domain interoperation
Encountered issues and proposed solutions
Project status and future work
Conclusions
CEWIT 2008
4
Center of Excellence
Wireless and Information Technology
Background
• Provide QoS guarantees at the individual data flow level, all
the way to the end hosts, transparently
– Data flows have varying priority/importance
• Video streams
• Critical data
• Long duration transfers
– Default “best effort” network behavior treats all data flows as equal
– Capacity is not unlimited
• Congestion causes bandwidth and latency variations
• Performance and service disruption problems, unpredictability
• Dynamic flow-based SLAs = schedule network utilization
– Regulate and classify (prioritize) traffic
CEWIT 2008
5
Center of Excellence
Wireless and Information Technology
End-to-End Setup
Site B
Site C
host
router
site
host / border
router
host b1
site
border
router
ACLs:
b1  a1
10.100.1.y2
10.100.1.x2
regional
provider
router
regional
provider
router
host c1
ACLs:
c1  a2
WAN domains
VLAN X
VLAN Y
10.100.1.x1
site
border
router
virtual
border
router
10.100.1.y1
ACLs:
a1  b1
a2  c1
host
router
host a2
host a1
Site A
CEWIT 2008
6
Center of Excellence
Wireless and Information Technology
Establishing End-to-End QoS Paths
• Multiple administrative domains
– Cooperation, trust, but each maintains full
control
– Heterogeneous environment
– Domain controller coordination through web
services
• Coordination models
…
– Star
• Requires extensive information for all domains
…
– Daisy chain
• Requires common flexible protocol across all
domains
– Hybrid (end-sites first)
…
• Independent protocols
• Direct end site negotiation
CEWIT 2008
7
Center of Excellence
Wireless and Information Technology
Path Setup (2)
• End site subnets are configured by TeraPaths software
instances (TeraPaths Domain Controllers or TDCs)
– TDCs configure end site LANs to prioritize and regulate authorized
flows via the DiffServ framework at the network device level
– Source site polices/marks authorized flow packets
– Destination site admits/re-polices/re-marks packets
– End site LANs tx/rx marked packets to/from the WAN
• WAN provides MPLS tunnels or dynamic circuits
– Initiating TDC requests MPLS tunnel or dynamic circuit with
matching bandwidth and lifetime, or…
– TDC groups flows with common src/dst into MPLS tunnel or
dynamic circuit with aggregate bandwidth and lifetime
– WAN preserves packet markings
CEWIT 2008
8
Center of Excellence
Wireless and Information Technology
Path Setup (3)
• WAN domains interoperate
– Each end site’s TDC has a single point of contact for WAN services
– TDCs have no knowledge of WAN internals other than what is exposed
by the WAN services
• End sites have no direct control over the WAN
• Either tunnel or circuit through WAN
– TeraPaths does not mix and match the layer 2 and layer 3 technology.
• TeraPaths “proxy” servers
– Implement interface required by TeraPaths core
– Hide WAN service differences
– Clients to WAN web services (currently OSCARS / DRAGON)
• Close cooperation with ESnet and I2 development teams
– Submit reservations for MPLS tunnels or dynamic circuits
– Handle security requirements
– Handle errors
CEWIT 2008
9
Center of Excellence
Wireless and Information Technology
Addressing L2-Specific Issues
• Limitations with VLANs
– Tag range (tentatively selected 50 VLANs – 3550 to 3599)
• Each site may have its own range
– Tag conflicts
• Rely on WAN service
• Eliminate by synchronizing site databases
• VLAN renaming (if/when possible)
• Scalability issues
– Limited number of VLAN tags/Circuits:
• Flow grouping / circuit consolidation
– Forward flows through same virtual WAN circuit
» Create circuit with new parameters / switch current flows / cancel old circuit
» Modify WAN reservations (if/when possible)
– PBR overhead
• Virtual border router
• Sensitive/3rd party network segments
– VLAN pass-thru
CEWIT 2008
10
Center of Excellence
Wireless and Information Technology
•Flows between same src and dst
sites can share circuit, policing
maintains bandwidth guarantee
•Multiple TeraPaths reservations
associate with the same circuit
reservation
– Easy when requirements are
known in advance
bandwidth
Flow Grouping/Circuit Consolidation
Δbw
4
2
1
CEWIT 2008
5
current time
– Modification of reservations
required otherwise
• Selection/optimization to
minimize resource waste
• Trade-off based on Δbw
(bandwidth difference), Δtb,
Δta (time period before and
after a reservation)
Δt
3
4
2
5
3
1
time
11
Center of Excellence
Wireless and Information Technology
Flow Grouping/Circuit Consolidation (2)
• Similar approach to disk buffering (read ahead / write behind)
– Bring up ahead / teardown behind
– Reuse existing active circuits
– Reserve circuits with more bandwidth and longer duration depending on
differences in start time, duration, bandwidth of reservations
Δtb
bandwidth
bandwidth
– Delay teardown, modify circuit duration and/or bandwidth if possible
Δta
2
2
4
5
3
1
Δta
current time
2
1
2
4
3
5
1
time
CEWIT 2008
5
3
1
current time
4
4
3
5
time
12
Center of Excellence
Wireless and Information Technology
Limitation of Dynamic Circuits
• A recent incident in BNL’s LHCOPN subnet:
– Cisco’s PBR implementation only uses the status of an
interface to decide whether or not to forward packets
– A network circuit breaks somewhere along the path, but the
involved interfaces on both ends are still up
– No probes and/or heartbeat exist to check the “health” of
circuits
– Fail-over to the backup link does not work since primary
interfaces are up even when such a problem exists
• End site monitoring is the most effective way to
detect such a problem
CEWIT 2008
13
Center of Excellence
Wireless and Information Technology
Active Circuit Probing
Each TeraPaths site instance periodically verifies “well
being” of reservations:
– Selects active reservations initiated by site (site
responsibility)
– Finds circuit/VLAN associated with each reservation
– Performs a circuit check with a quick pinging of other site’s
router (private ip address space)
– Less than 100% success triggers a recheck with longer
duration pings in both directions (to and from other site)
– Low success % triggers reservation cancellation reverting
traffic to best effort network
– Optionally, the system adapts reservation data and attempts
to setup a new end-to-end path (for given time
period/number of attempts)
CEWIT 2008
14
Center of Excellence
Wireless and Information Technology
Prioritizing Traffic
TeraPaths QoS test 1 (prioritize traffic)
Bandwidth (Mbits/sec)
1200
1000
800
priority
background
total
600
QoS / circuit
reservation
active
400
200
0
0
CEWIT 2008
competing
traffic
causes
dramatic
drop in
bandwidth
200
400
600
800
1000
time (sec)
15
Center of Excellence
Wireless and Information Technology
Recovering from Circuit Failure
TeraPaths QoS test 2 (prioritize/fallback to best effort)
Bandwidth (Mbits/sec)
1200
1000
800
priority
background
total
600
recovery to
best effort
400
200
circuit
interruption
0
0
200
600
400
800
1000
time (sec)
CEWIT 2008
16
Center of Excellence
Wireless and Information Technology
Competing against BE traffic
remote EF against remote and local BE
remote EF against local BE
10000
10000
9000
9000
8000
8000
7000
7000
6000
6000
remote EF
local BE
remote BE
5000
4000
4000
3000
3000
2000
2000
1000
1000
0
remote EF
local BE
local BE
5000
0
0
100
CEWIT 2008
200
300
400
500
0
100
200
300
400
500
600
17
Center of Excellence
Wireless and Information Technology
Status
• BNL, UMich, BU, all with 10Gbps connections, multiple pass-thru
configurations (BNL, UMich, NoX, Merit, MiLR)
• Utilization of L3 paths (MPLS tunnels, ESnet only), L2 paths
(dynamic circuits, ESnet and Internet2)
• Multiple QoS reservations through same circuit (support for
circuit consolidation)
• Multiple circuits per site subject to per-site VLAN availability
(flow grouping/circuit consolidation)
• Active circuit probing for failures with fallback to best effort
network/attempt to reconfigure e2e path (in testing phase)
• Dynamic bandwidth allocation within service classes (in testing
phase)
• New command line client
CEWIT 2008
18
Center of Excellence
Wireless and Information Technology
Future Work
• Continue working on automatic flow grouping / circuit
consolidation.
• Configurable reservation negotiation
• Grid-style AAA (GUMS/VOMS)
• Plug-ins: SRM (dCache), others
• Compatibility with Lambda Station
• Support for different hardware as needed
• ATLAS Production:
– Replicate ATLAS Physics data from BU and UMich with the
existing ATLAS DDM stack, and with end-to-end QoS circuits
– Tier 1 (BNL) and Tier 2 data replication
• http://www.terapaths.org
CEWIT 2008
19
Center of Excellence
Wireless and Information Technology
Conclusions
• Demonstrated the effective prioritization and protection
from interference of selected data transfers between
three LHC experiment institutes – Brookhaven National
Laboratory, the University of Michigan, and Boston
University – through guaranteed bandwidth virtual
paths, at the presence of intensive best-effort IP traffic
sharing the same network resources
• A practical and economical end-to-end network resource
reservation system, extending new capabilities to
users/applications of end sites without requiring
additional, expensive network infrastructure components
CEWIT 2008
20