Download Toward Millisecond IGP Convergence

Toward Millisecond IGP Convergence
Cengiz Alaettinoglu
Van Jacobson
Haobo Yu
{cengiz,van,haoboy}@packetdesign.com
Packet Design, Inc.
October 24, 2000
NANOG 20
October 22–24, 2000
Washington, D.C.
c 2000 Packet Design
Sub-second re-route times give:
➤ increased network reliability
➤ support for multi-service traffic (e.g., VoIP)
➤ lower cost/complexity compared to layer 2 protection schemes like
SONET.
c 2000 Packet Design
1
Where are we today?
Current IP re-route times are typically tens of seconds.
We need to do better. There are two choices:
➤ Replace IP routing with something else, e.g., MPLS Fast Failure Recovery.
➤ Figure out what’s wrong with IP routing and fix it.
Since improvement usually requires understanding, we think the second
choice is prudent.
c 2000 Packet Design
2
Where does the time go?
There are three steps to IS–IS or OSPF re-routing:
(1) Detect a local topology change (link up/down or peer reachability).
(2) Flood a Link State Packet (LSP) to tell adjacent peers about the change.
(3) Compute new routes by running a Dijkstra Shortest Path First (SPF)
algorithm on the changed topology.
We did experiments to characterize each of these steps for IS–IS running on
Cisco 7200s running 12.0S and 12.1P and Juniper M40s running 4.1.
c 2000 Packet Design
3
Detection
The IS–IS spec allows for two link state change detection mechanisms:
link-level notification and peer–peer Hello packets.
Link level detection should be fastest but it’s not always possible (e.g., switched
ethernet) and seems to be inconsistently implemented by vendors.
Spec says that adjacent routers should send Hello packets to each other at a
fixed interval (default 10 sec., minimum 1 sec.) and declare the adjacency lost
if no Hellos received for three intervals. This works for any interconnect but
constrains the repair time to be at least 3 seconds (3 times the Hello interval).
c 2000 Packet Design
4
Detection — experiment
SPF delay
Hello timeout
0
10
20
30
40
Time (seconds)
c 2000 Packet Design
5
Hellos dropped
Detection — constraints on the Hello interval
70000
60000
50000
40000
30000
20000
10000
0
0.9
0.8
0.7
Oth
er t 0.6 0.5
raff
ic b
0.4
and
0.3
wid
0.2
th
0.1
100
10
%)
dth (
ndwi
0.01
c 2000 Packet Design
1
ge ba
0.1
a
s
es
m
o
l
l
He
6
Detection — what’s possible
Since the protocol’s ultimate limit on the Hello interval is set by the bandwidth
used by Hello packets, extending the spec to allow sub-second intervals would
allow sub-second detection on almost all links.
This doesn’t mean that the detection time can be made arbitrarily small, only
that detection should be limited by the physical constraints of links, not by
arbitrary clock granularity choices made by protocol designers.
In general, detection time should be limited by the transient error spectrum on
a particular link. For example, a link that takes 30ms noise hits should have at
least a 30ms Hello interval.
c 2000 Packet Design
7
Detection — stability and damping
For either event triggered or hello driven detection, there are network-wide
stability issues if routing tries to follow rapid link transients (i.e., a link that goes
down and up several times a second).
The usual way of dealing with this is to treat “bad news” differently from “good
news” so routing is quick to find an alternate path on any failure but slow to
switch back when the link comes up.
The current IS–IS spec treats bad news and good news the same but it should
be possible to change the spec to allow different filtering constants for “down”
and “up” state changes.
c 2000 Packet Design
8
LSP Propagation
A link state packet is generated at the point of detection then flooded,
unmodified, through the network.
It should propagate at near the speed of light plus one store-and-forward delay
per hop.
So in theory LSP propagation should make a negligible contribution to the
re-route time.
Theory doesn’t often resemble reality...
c 2000 Packet Design
9
LSP propagation experiment setup
c 2000 Packet Design
10
Effect of SPF delay on LSP propagation
SPF delay
Hello timeout
0
10
20
30
40
Time (seconds)
c 2000 Packet Design
11
1.0
LSP propagation experiment with zero SPF delay
y
log
o
p
0.8
o
et
00
d
no
0.6
10
po
e to
y
nod
0.4
800
log
y
0.0
0.2
olog
de top
o
n
0
0
5
0
1
2
c 2000 Packet Design
3
12
LSP propagation explanation
Since the SPF calculation can take a significant amount of time (see next
section), commercial router implementations impose a limit on how frequently
the calcularion can be done. In some implementations this limit is fixed (5
seconds) and in some it is changable but with a granularity of 1 second. This
limit essentially adds to the propagation time.
On at least one implementation, if the SPF limit is set to zero the SPF
calculation is done before the changed LSPs are flooded which degrades the
propagation time from O(speed o f light ) to O(diameter SPFtime).
To prevent this, the spec might be amended to explicitly state that LSP flooding
is “higer priority” than SPF calculation or the point might become moot if the
SPF calculation time is improved (next section).
c 2000 Packet Design
13
SPF calculation experiment setup
c 2000 Packet Design
14
200
150
100
50
0
SPF calculation time (ms)
250
300
SPF measurements — random topologies
0
200
400
600
800
1000
Number of nodes
c 2000 Packet Design
15
1e+01
1e+00
1e−01
1e−02
SPF calculation time (ms)
1e+02
SPF measurements—random topologies (log scale)
0
200
400
600
800
1000
Number of nodes
c 2000 Packet Design
16
600
400
200
0
SPF calculation (ms)
800
1000
SPF measurements — full meshes
50
100
150
200
250
300
Number of edges per node (full mesh)
c 2000 Packet Design
17
SPF calculation discussion
After any link state change, each node has to do an SPF calculation to
compute the new topology. Even with high-end platforms (Cisco 7200, Juniper
M40) this calculation can take a lot of time (seconds) and has poor scaling
properties (nlog n to n2).
This has a serious impact:
➤ on convergence (because the SPF calculation is in series with the LSP
propagation).
➤ on overall network stability (because of router CPU saturation).
c 2000 Packet Design
18
Fixing the SPF calculation
1 ms
100 us
SPF calculation (ms)
100 ms
The Dijkstra SPF algorithm is almost 40 years old. More recent algorithms can
compute changes to SPF trees in time proportional to log n rather than n log n.
This allows a net to scale up to virtually any size while bringing the calculation
time down from seconds to microseconds.
50
100
150
200
250
300
Number of edges per node (full mesh)
c 2000 Packet Design
19
Final note — what we didn’t see
During our experiments we injected topologies with 50,000+ edges, drove all
the routers to 100% CPU saturation, then randomly unplugged links and
powered off boxes.
Although we were looking for it, we saw no evidence of instability and we
observed several subtle things done expressly to avoid getting into an unstable
operating regime.
It appears that the two vendors we looked at have learned a lot from a
decade’s worth of routing disasters and meltdowns and are currently shipping
some pretty robust routing code.
c 2000 Packet Design
20
Summary
Stable, robust IP re-routing that works at the network’s propagation rate (the
theoretical maximum for any re-routing scheme) is both possible and
achievable.
To get there, we have to make some minor changes to the IS–IS spec then
customers have to let the router vendors know (if) it’s worth implementing
these changes.
The changes are (in rough priority order):
(1) switch to a modern algorithm for the SPF calculation.
(2) make the granularity of the Hello timer milliseconds rather than seconds.
(3) allow different detection filter constants for link up and down events.
c 2000 Packet Design
21
Acknowledgments and a request
The authors are grateful to the Global Crossing Production Testing Lab, Cisco
Systems and Juniper Networks for allowing us to use their facilities to perform
some of these tests.
We are interested in measuring IS-IS and OSPF behavior on real, operational
ISP networks. If you’d like to work with us on this, please contact
[email protected].
c 2000 Packet Design
22