Download 1 Mapping the Internet Topology Traceroute A Basic Tool for

Mapping the Internet Topology
Traceroute
A Basic Tool for Topology
p gy Discovery
y
Yuval Shavitt
School of Electrical Engineering
http://www.eng.tau.ac.il/~shavitt
http://www.netDimes.org
Simple delay/loss probing with ping
ICMP
C:\>ping www.fer.hr
ICMP is the IP error diagnosis protocol.
Pinging www.fer.hr [161.53.72.111] with 32 bytes of data:
IP header
Type
Reply from 161.53.72.111: bytes
bytes=32
32 time
time=113ms
113ms TTL=49
TTL 49
Reply from 161.53.72.111: bytes=32 time=111ms TTL=49
Reply from 161.53.72.111: bytes=32 time=113ms TTL=49
Reply from 161.53.72.111: bytes=32 time=118ms TTL=49
Code
Checksum
Ping statistics for 161.53.72.111:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 111ms, Maximum = 118ms, Average = 113ms
Sequence number
Any ICMP data
ICMP Message Types
ping
• Can be used to
– Check host is alive
– Measure RTD to a server
• Application layer “ping”
– One can generate application layer messages to
test application reaction time
– Most common:
• TCP SYN message to port 80
1
Type No.
Meaning
0
Echo reply
3
Destination unreachable
4
Source quench
5
Redirect
8
Echo
9
R t advertisement
Router
d ti
t
10
Router solicitation
11
Time exceeded
12
Parameter problem
13
Timestamp
14
Timestamp reply
15
Information requeste
16
Information reply
PING
IP datagram format
IP protocol version
number
header length
(bytes)
“type” of data
32 bits
length
ver
len service
fragment
16-bit identifier flgs
offset
time to upper
Internet
layer
live
checksum
max number
remaining hops
(decremented at
each router)
traceroute
total datagram
length (bytes)
head. type of
for
fragmentation/
reassembly
• Useful to learn the route characteristics
between two hosts.
• Sends a series of probes to successive nodes
along a route to an intended destination and
records the source address and time delay of
the message returned by each.
• Based on ICMP “TTL expired” message
32 bit
bi source IP address
dd
32 bit destination IP address
upper layer protocol
to deliver payload to
E.g. timestamp,
record route
taken, pecify
list of routers
to visit.
Options (if any)
data
(variable length,
typically a TCP
or UDP segment)
ICMP Message Types
traceroute
time
A
B
Regular UDP packets
•
successive TTLs
ICMP “TTL expired”
message
ICMP “port unreachable”
message
C
D
E
Type No.
Meaning
0
Echo reply
3
Destination unreachable
4
Source quench
5
Redirect
8
Echo
9
R t advertisement
Router
d ti
t
10
Router solicitation
11
Time exceeded
12
Parameter problem
13
Timestamp
14
Timestamp reply
15
Information requeste
16
Information reply
Tracing route to www.fer.hr [161.53.72.111]
over a maximum of 30 hops:
<1
19
17
21
19
20
69
82
101
105
117
113
120
114
120
114
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
Trace complete.
<1
20
22
19
23
20
69
82
98
105
112
115
122
112
119
114
2
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
<1
19
20
19
18
20
69
82
98
105
113
115
123
119
119
113
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
192.168.200.254
vxr.tau.ac.il [132.66.8.10]
c6509.tau.ac.il [132.66.8.20]
tel-aviv.tau.ac.il [132.66.4.1]
gp1-tau-fe.ilan.net.il [128.139.191.70]
iucc.il1.il.geant.net [62.40.103.69]
i i
il.it1.it.geant.net
i
[[62.40.96.154]]
it.ch1.ch.geant.net [62.40.96.33]
ch.at1.at.geant.net [62.40.96.1]
at.hu1.hu.geant.net [62.40.96.178]
hu.hr1.hr.geant.net [62.40.96.145]
carnet-gw.hr1.hr.geant.net [62.40.103.218]
193.198.228.6
193.198.229.10
161.53.16.14
duality.cc.fer.hr [161.53.72.111]
Code
0
1
2
3
6
7
description
dest. network unreachable
dest host unreachable
dest protocol unreachable
dest port unreachable
dest network unknown
dest host unknown
traceroute
traceroute versions
C:₩>tracert www.fer.hr
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Type
3
3
3
3
3
3
• UNIX:
– default send UDP packets
• Start at port 33435, and increment port per packet!
Whyy thiss iss not
o advisable?
dv s b e?
• W
– traceroute –l sends ICMP “ECHO request”
– tcptraceroute uses TCP SYN messages
• If port is close gets RST reply
• If port is open gets SYN ACK and reply with RST
• Best to overcome firewalls
• Windows
– ICMP “ECHO request”
From Traceroute to IP Maps
Alias Resolution
Ping (echo request) X
• Problem 1:
or UDP X (high port number
– TR gives interface IP not router ID
• Problem 2:
M
– TR can result in false links
• Problem 3:
echo reply Y
– Non responding routers
ICMP (port unreachable) Y
⇒ IP X = IP Y
Traceroute and Load-Balancing
Actual topology:
Src
A
C
TTL = 2
L
B
E
Dst
D
TTL = 3
Inferred topology:
A
Src
C
False link
E
L
B
Aliasing Based on IP Identifier
• x<y<z, z-x small Æ
likely aliases
• If |x-y|>200
Æ Aliases are
disqualified, third
packet is not sent
Dst
D
Missing nodes and links
1
6
Anomalies: false loops and cycles
Load Balancing can lead to ‘Loops’
TTL = 3
Actual topology:
Src
A
D
L
B
Dst
C
TTL = 2
TTL = 4
TTL = 2
Inferred topology:
TTL = 4
TTL = 3
Src
L
D
B
real topology
traceroute possible outcome
17
3
Dst
Reducing
The Load Balancing Problem
Anomalies: false diamonds
A
Actual topology:
• Load balancing can be done in two ways:
Src
Dst
E
L
1. Packet based
2. Flow based (S IP, D IP, S port, D port, prot)
• Sending all probes as a single flow will
force the same route selection in flow based
Load balancer
• How to do this?
C
B
D
A
C
B
D
Inferred topology:
Src
Dst
E
L
19
Measurement artifacts are common
From LIP6 vantage point:
• Diamonds appear in 30% of the destinations
ƒ Paris traceroute removes 10,662 from 19,159 (56%)
• Loops appear in 4.5% of the measured routes
ƒ Paris traceroute removes 5,047
5 047 from 5,795
5 795 (87%)
Paris traceroute
• Solves the problem with per-flow load balancing
ƒ Probes to a destination belong to same flow
• How to identify probes?
ƒ Use the UDP checksum
• Does not address per-packet
per packet load balancing
• Cycles appear in 0.25% of the measured routes
Checksum 2
Port 1
Checksum 1
A
Port 1
ƒ Paris traceroute removes 3,886 from 5,674 (68%)
• Other causes
ƒ
ƒ
ƒ
ƒ
Routing changes
NAT boxes
Buggy routers
Per-packet load balancing
Src
TTL = 3
E
Dst
D
TTL = 1
21
Anomalies:
Loops caused by NAT boxes
Response TTL = 254
IP Identifier = 12375
Src
A
Response TTL = 252
IP Identifier = 9356
Dst
(NAT)
Dt
Dst
B
B
TTL = 2
TTL = 3
2
Src
4
TTL = 2
L
B
22
24
Checksum 3
Port 1
C
A
Dst
C
TTL = 3
Anomalies:
Loops caused by buggy routers
Src
A
X
Rejects the probe
with a TTL of
0 and
= 1
Forwards theTTL
probe
sends it back to the
with TTL equal to
source
0
Response TTL = 253
IP Identifier = 5286
Src
23
B
Dst
Forwards the probe
probe
with TTL equal to
with a TTL of 0 and
0
sends it back to the
source
Rejects
TTL
TTL == 12the
B
Dst
-bash$ traceroute Dst
traceroute to Dst
1 B
0.289 ms
2 B
0.278 ms
3 Dst 0.578 ms
-bash$ traceroute-paris Dst
traceroute to Dst
1 B
0.289 ms !T0
2 B
0.278 ms
3 Dst 0.578 ms
Non-Responding Routers
True network
C:₩>tracert www.colbud.hu
2
Tracing route to www.colbud.hu [81.182.250.153]
over a maximum of 30 hops:
11
444
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
33
7
5
66
88
9
11
<1
19
20
21
20
26
91
97
95
96
110
*
112
112
114
120
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
<1
21
21
20
22
22
92
97
96
96
112
*
110
114
112
122
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
<1
18
21
19
19
21
92
97
93
150
114
*
111
110
114
124
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
192.168.200.254
vxr.tau.ac.il [132.66.8.10]
c6509.tau.ac.il [132.66.8.20]
tel-aviv.tau.ac.il [132.66.4.1]
gp1-tau-fe.ilan.net.il [128.139.191.70]
iucc.il1.il.geant.net
iucc il1 il geant net [62
[62.40.103.69]
40 103 69]
il.nl1.nl.geant.net [62.40.96.117]
nl.de1.de.geant.net [62.40.96.101]
ffm-b2-pos2-3.telia.net [213.248.77.89]
ffm-bb2-pos2-3-0.telia.net [213.248.64.177]
bpt-b1-pos2-0.telia.net [213.248.64.26]
Request timed out.
10ge-0-0.core0-ip2.net.telekom.hu [145.236.85.2]
tenge1-2.core0.adatpark.hu [145.236.89.10]
fixip-lns2.adatpark.hu [195.228.253.58]
153-250-182-81.adsl-fixip.axelero.hu [81.182.250.153]
Trace complete.
10
10
26
True network revisited
Measured network
Suppose one router is unresponsive:
1
2
4
111
3
44
4
4
33
7
7
7
5
1
7
66
??
8
7
9
11
6
10
10
10
8
28
27
Unresponsive Routers - Example
Measured network
1
4
3
4
7
7
10 ? 7
10-?-7
1
10
7
True network with 3
unresponsive routers
Measured network with 51
unknown nodes
The non-blue nodes are unresponsive.
The non-blue nodes are unknowns (gaps).
30
5
6
6-?-7
29
Unifying Unknowns
Is it possible to identify a group of unknown
nodes that represent the same unresponsive
router?
AS Topology Discovery
31
The Internet Structure
The Internet Structure
routers
The AS graph
from IP to AS routes
IP to AS
C:₩>tracert www.fer.hr
Tracing route to www.fer.hr [161.53.72.111]
over a maximum of 30 hops:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<1
19
17
21
19
20
69
82
101
105
117
113
120
114
120
114
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
<1
20
22
19
23
20
69
82
98
105
112
115
122
112
119
114
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
<1
19
20
19
18
20
69
82
98
105
113
115
123
119
119
113
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
ms
192.168.200.254
vxr.tau.ac.il [132.66.8.10]
c6509.tau.ac.il [132.66.8.20]
tel-aviv.tau.ac.il [132.66.4.1]
gp1-tau-fe.ilan.net.il [128.139.191.70]
iucc.il1.il.geant.net [62.40.103.69]
i i
il.it1.it.geant.net
i
[[62.40.96.154]
]
it.ch1.ch.geant.net [62.40.96.33]
ch.at1.at.geant.net [62.40.96.1]
at.hu1.hu.geant.net [62.40.96.178]
hu.hr1.hr.geant.net [62.40.96.145]
carnet-gw.hr1.hr.geant.net [62.40.103.218]
193.198.228.6
HR-ZZ
193.198.229.10
161.53.16.14
duality.cc.fer.hr [161.53.72.111]
Trace complete.
378
6
20965
2108
private network
Tel Aviv Uni.
AS378
ILAN
MACHBA
DANTE
AS20965
GEANT
CARnet
CARnet
AS2108
How map IP to AS?
Rifining IP-to-AS Mapping
• Start with initial IP-to-AS mapping
• Routing address registry
– Mapping from BGP tables is usually correct
– Good starting point for computing the mapping
– Voluntary public registry such as whois.radb.net
– Used by prtraceroute and “NANOG traceroute”
– Incomplete and quite out-of-date
• Collect
C ll t many BGP andd traceroute
t
t paths
th
– Signaling and forwarding AS path usually match
– Good way to identify mistakes in IP-to-AS map
• Mergers, acquisitions, delegation to customers
• Origin AS in BGP paths
• Successively refine the IP-to-AS mapping
– Public BGP routing tables such as RouteViews
– Incomplete and inaccurate… but usually right
– Find add/change/delete that makes big difference
– Base these “edits” on operational realities
• Multiple Origin ASes (MOAS), no mapping, wrong
mapping
Extra AS due to Sibling ASes
• Sibling: organizations with multiple ASes:
– E.g., Sprint AS 1239 and AS 1791
– AS numbers equipment with addresses of
another
A
B
H
D
E
A
F
B
G
C
D
C
Traceroute AS path
Extra AS due to Internet
eXchange Points
• IXP: shared place where providers meet
– E.g., MAE-East, MAE-West, LINX, AMS-IX
– Large number of fan-in
fan in and fan-out
fan out ASes
E
F
A
E
B
F
B
F
G
C
G
C
G
D
A
Traceroute AS path
BGP AS path
Sibling ASes “belong together” as if they were one AS.
Interfaces in multiple ASes
E
BGP AS path
Physical topology and BGP session graph do not always match.
Weird Paths Due to Unannounced Addresses
12.0.0.0/8
A
A-B-C
A
B
C
C-B-A
A
B
C
B
C does not announce part of
its address space in BGP
(e.g., 12.1.2.0/24)
AC A C
C
Are A and C neighbor ASes?
What AS does the middle router belong to, B or C?
AC
BAC
BC
Fix the IP-to-AS map to associate 12.1.2.0/24 with C
7
Traceroute Inherent Problems
Same problems as with router level maps:
• Load balancing
• Routing change during traceroute
• Effects:
Making AS Maps Larger
– Wrong links
– Loops
– Missing links
Revealing the Internet Structure
Revealing the Internet Structure
Revealing the Internet Structure
Revealing the Internet Structure
Diminishing return!
⇓
Deploying more
boxes does not
pay-off
7 new links
30 new links
NO new links
8
Revealing the Internet Structure
Number of VPs
To obtain the ‘horizontal’
links we need strong
presence in the edge
• Large instrumentation boxes (Ark)
– Cannot reach more than 10s of VPs
• Community base software agents (DIMES)
– Reached 100s VP
– Can grow much more
DIMES
• Very light agents (Ono)
– Reached 1000s VP
– Hard to clean noise
• Small hardware devices (RIPE ATLAS)
– Reached 100s VP
TAU’s DIMES
CAIDA’s Archipelago (Ark)
• ~1000 active agents daily
DIMES
• 54 boxes
May 2011
May 2011
So how many is enough?
RIPE’s ATLAS
• 400 active (600 distributed)
• [Chen et al. 02], [Bradford et al. 01]: when
you combine more and more points of view
the return diminishes veryy fast
• A few VPs are enough
– The mass of the tail is significant
contribution
No. of views
9
May 2011
Data Set
How to Measure Contribution?
• Data is obtained from DIMES
– Community-based infrastructure, using almost
1000 active measuringg software agents
g
– Agents follow a script and perform ~2 probes
per minute (ICMP/UDP traceroute, ping)
– Most agents measure from a single AS (vp)
• Graph size is not the only factor.
• We want our Internet view to share the
same properties as the real Internet?
– Remove measurement bias
• How to quantify this?
• But some (appear to) measure from more…
• Data need to be filtered to remove artifacts
– Traceroute data collected during March 2008
Measurements Per Agent
Filtering the data
• For each agent and each week, classify how
many networks it measured the Internet from
Typical
yp
cases:
– ASi:15300, ASj:8
– ASi:10000, ASj:3178
– ASi:10000, ASj:412 , ASk:201
– 18000, 12, 11, 9, 9, 3, 3, 2, 2, 1, 1, 1, 1, 1, ….
Week 4,2008
Agents per Network
Measurements per Network
500
10
Diminishing Returns?
• Barford et. al. – the utility of adding many
vps quickly diminishes
– In terms of ASes and AS
AS-links
links
• Shavitt and Shir – utility indeed diminishes
but the tail is long and significant
Filtering Results
• 96% of the agents have less than 4
different vps
• High degree ASs tend to have more agents
• High number of measurements for all vps
degrees
– Tail is biased towards horizontal links
• We wish to quantify how different aspects
of AS-level topology are affected by adding
more vps
Creating topologies per VP
Topology Size
• The return (especially for AS links) does
not diminishes fast!
sort by
VP with small local
topology can contribute
many new links!
Convergence of Properties
Direction of Detected Links
• Taking several common AS-level graph
properties, and analyze their convergence as
local topologies
p g are added
• For each link: Plot max adjacent AS degree
and max adjacent ASes degree difference
– Keeping the sort order by number of links
• Slow convergence indicates the need to
have broad and diverse set of vps
11
High degree
difference –
indicates radial
links towards the
core
Low degree
g
difference – indicates
tangential links and
links between smallsize ASes
Power-law and Max Degree
Density and Average Degree
Slow convergence of density and average degree – easy
to detect ASes but difficult to find all links
Fast convergence of maximal
degree – core links are easily
detects
Fair convergence of
power-law exponent
Revisitng Sampling Bias
Betweenness and Clustering
• Lakhina et al. – AS degrees inferred from
traceroute sampling are biased
– ASes in vicinity to vps have higher degrees
– Power-law might be an artifact of this!
• Dall’asta et al. – no…it is quite possible to
have unbiased degrees with traceroutes
• Cohen et al. – when exponent is larger than
2, resulting bias is neglible
Dataset VPs and Distances
Low degree ASes are
seen from less vps than
high-degree Ases…this
makes sense!
Fast convergence of max bc – Level3
(AS3356), a tier-1 AS is immediately
detected as having max bc
Radial links
decrease cc
Tangential links
increase cc
Evaluating Sampling Bias
• For each AS find:
– All the vps that have it in their local topology
– The Valley-Free
Valley Free distance in hops
In our dataset, most ASes
have a vp that is only 1-2
hops away!
Up-hill to the core (c2p), side-ways in
the core (p2p) and down-hill from
the core (p2c)
12
Revisiting Diversity Bias
• What is the effect of diversity in vps geolocation and network type?
– Some infrastructures rely on academic
networks for vp distribution – does it have an
effect on the resulting topology?
Average Distance per Degree
Low degree ASes are
seen from farther
vps sampling bias?
vps…sampling
• We compare iPlane and DIMES
– Classify AS into types: t1,t2, edu, comp, ix, nic
using Dimitropoulos et al.
In Search of Ground Truth
No real bias!
More VPs are located in high-degree ASes•
There are high-degree ASes that are seen from “far” vps•
Broad distribution – all ASes are pretty close-by to a vp!•
Diversity Bias Evaluation
• One week is not sufficient for active
measurements
iPlane uses many PlanetLab nodes (edu), while DIMES resides
mostly at homes (tier
(tier-2)
2)
• Both iPlane and DIMES have lower
average degrees than RouteViews
– Except iPlane’s edu and ix!
– Diversity bias exists – need diverse vp types!
Conclusion
• VP distribution is important
– Number, AS type, geo-location
Indeed DIMES have higher t2 and comp degrees and iPlane have
higher edu degrees – results are slightly biased to vps’ types!
Measuring Within a Network
• Comparing vp average degrees to quantify
the effect of measuring within a network
• AS-level graph properties are affected
– Some converge very fast
– Other converge slowly
• Community based projects have practically
unlimited growth potential!
13
Indeed, the average degree when measuring within a network is mostly
higher (hmm…tier-1 doesn’t count cause most vps are the same!)
BGP Listeners
• Passive listening to BGP messages can
easily give AS level topology information
PoP Level Maps
– U. of Oregon: RouteViews
– RIPE
• Problems:
– Aggregation
– limited
The Internet Structure
What the PoP is ?
• PoP – Point of Presence of the ISP
routers
The Internet Structure
The AS graph
14
The PoP level graph
The Internet Structure
The AS graph
iPlane PoP Discovery Approach
monitor
monitor
PoP Discovery (RocketFuel)
• Group interfaces to routers
• use DNS names
– S1-bb11-nyc-3-0.sprintlink.net is a Sprint router in
New York City
• use connectivity information
– if a router connects only to routers in Seattle, it is in
Seattle
Delay measurements
• Use the router role in the topology?
monitor
– only backbone routers connect to other cities
– use DNS names
• s1-gw2-sea-3-1.sprintlink.net is a Sprint gateway router
[Madhyastha et al. IMC 2006]
Bad Routing Example
Tel Aviv University
[Spring et al. SIGCOMM 2002]
Problems with this Approch
• Delay measurements are noisy
– Change of time
– Routing anomalies: permanent stretch
< 6 km
Azriely Center
TAU to Azriely
Azriely to TAU
BEZEK-NET, AS6810
15
Can we measure link delay?
PoP Discovery (DIMES)
C:₩>tracert www.fer.hr
Tracing route to www.fer.hr [161.53.72.111]
over a maximum of 30 hops:
Link
Min.
delay
0
1
<1 ms
<1 ms
<1 ms 192.168.200.254
19
19
2
19 ms
20 ms
19 ms vxr.tau.ac.il
[132.66.8.10]
-2
17
3
17 ms
22 ms
20 ms c6509.tau.ac.il
[132.66.8.20]
2
19
4
21 ms
19 ms
19 ms tel-aviv.tau.ac.il
[132.66.4.1]
-1
18
5
19 ms
23 ms
18 ms gp1-tau-fe.ilan.net.il
[128.139.191.70] 2
20
6
20 ms
20 ms
20 ms iucc.il1.il.geant.net
[62.40.103.69]
49
7
69 ms
69 ms
69 ms il.it1.it.geant.net
i i
i69
[[62.40.96.154]]
13
82
8
82 ms
82 ms
82 ms it.ch1.ch.geant.net
[62.40.96.33]
16
98
9 101 ms
98 ms
98 ms ch.at1.at.geant.net
[62.40.96.1]
7
105
10 105 ms 105 ms 105 ms at.hu1.hu.geant.net
[62.40.96.178]
7
112
11 117 ms 112 ms 113 ms hu.hr1.hr.geant.net
[62.40.96.145]
1
113
12 113 ms 115 ms 115 ms carnet-gw.hr1.hr.geant.net
[62.40.103.218]7
120
13 120 ms 122 ms 123 ms 193.198.228.6
2
112
14 114 ms 112 ms 119 ms 193.198.229.10
7
119
15 120 ms 119 ms 119 ms 161.53.16.14
-6
113
16 114 ms 114 ms 113 ms duality.cc.fer.hr
[161.53.72.111]
Negative delays
• Most delay measurements are OK
• But some have noise
• Use structure to clean ‘bad’ measurements
Trace complete.
What is the graph obtained from
running traceroutes thru a POP?
A delay of a link inside TAU
Link Delay Measurements Histogram
5000
4500
4000
Distribution of the delay
among 1 ms bins
s
3500
3000
2500
2000
1500
1000
500
0
-150
-100
-50
0
50
Link delay [ms]
100
150
200
250
negative delay
Motifs in IP interface Graphs
Network Motifs and Significance
• Z-Score in IP level Graphs
• “Network Motifs: Simple Building Blocks of
Complex Networks”
R. Milo, S. Shen-Orr, S. Itzkovitz, N. Kashtan, D. Chklovskii, and U. Alon. Network motifs: simple
building blocks of complex networks. Science, 2002.
• Qualitative measure of statistical significance
significance,
the Z score:
• We should look for motifs with small weights
of the edges – representing small physical
delays.
16
Z=
X −μ
σ
POP Extraction Flow
Example
Remove all edges with:
•delay higher than 5 ms
•small number of
measurements.
Unify adjacent parent/child
groups to POPs cores, then unify
adjacent POPs
Build an induced sub-graph
of G for each connected
component.
Run Localization
Algorithm (on subgraphs)
Divide the sub
sub-graph
graph to
parent/child groups (on
sub-graphs)
Connect unassociated
nodes to the nearest POP
cores.
[Feldman & Shavitt, Globecom ‘08]
Localization Algorithm
• Checks whether all nodes that belong to the same parent/child group are
located at the same physical location (belongs to the same POP).
• Are nodes A and B collocated? (multi-dimensions)
Parent-Child Classification
„
„
A
I
A
B
II
B
• If we assume Gaussian noise – in case I nodes are closer
• If we assume Impulse noise – most likely we should select case II
Localization Algorithm (cont.)
• For bipartite graph G(V,E) with weight function
• For each pair of parent nodes
• Define common children group CC with members {cc1,cc2,…}
„
Parent pair - both of them
points to two or more nodes.
Child pair - both of them are
pointed by two or more nodes.
Groups of parent/child nodes
are union of parent/child
couples.
17
C2
P1
P2
blue nodes - parent , red nodes - child,
blue and red nodes - both parent and child,
dashed nodes - neither of two.
Localization example
• Is 5 co-located with 1,2,3?
– No: by node 10
– Yes: by nodes 9 & 11
– Maybe: by node 6
• Clearly {6,7,8} are not
collocated with {9,10,11}
• Error Ratio Vector
C1
POP Extraction
connect nodes and core elements.
• Repeat Localization algorithm for child nodes, too
• Unify adjacent parent/child groups to formulate
POP cores
• Unify adjacent POP cores
• Optional: Connect “lonely nodes” (singletons) to
POPs.
PoP Discovery
–
–
–
–
Log function brings small errors towards zero.
Using log function on x and 1/x has equal impact
Median is used to disregard small number of outliers
Note: It is also possible to take an absolute difference into
account too. However no significant benefit was observed.
• Using Traceroute measurements
– Increased number of discovered PoPs compared to 1 week
period.
– Less sensitive to topology
p gy changes
g than 4 weeks p
period.
No. of PoPs
• Compare | median(log( ER (u , v)) | to a threshold
PoP Discovery
• Running on bi-weekly basis
Time Frames
Localization Algorithm (cont.)
IPs in PoPs
Distinct Edges
1 week to 1 week
<1%
<1%
±20%
1 week to 2 weeks
+58%
+79%
+43%
2 weeks to 4 weeks
+10%
+15%
+59%
PoP Discovery
• The number of discovered PoPs directly relates to the number
of discovered IP edges.
– 30M-40M measurements per week.
– 5.5M-6.5M distinct edges discovered.
– ~1000
1000 agents in over 200 ASes are used for the
measurements.
– 2.5M IP addresses in over 26,000 ASes are being targeted.
– Using Median algorithm to estimate distance between
nodes.
PoP Discovery
• Discovered PoPs
– ~4400 discovered PoPs.
– Over 50,000 IPs within discovered PoPs.
• Over 100,000 IPs with singletons
g
• Discovered mostly large PoPs and not access PoPs.
• Enhancements
– Targeting iPlanes’s PoP’s IP addresses – increased the
number of discovered PoPs by less than 20%.
– Targeted measurements to specific AS doubled the number
of discovered PoPs in small ASes.
• Had some effect in large ISPs but not to that extent.
18
The Internet Structure
PoP Discovery
• Sensitivity to delay threshold:
Number of PoP IPs
The PoP level graph
Number of PoPs
• Sensitivity to number of measurements threshold:
Number of PoP IPs
Number of PoPs
The AS graph
Embedding PoPs in Geography
Evaluating GeoLocation databases
GeoLocation: Possible Approaches
• Use available GeoIP databases
– Better first validate their accuracy
• Use triangulation from known locations
– We know where PlanetLab nodes are
– Can we clean the noise?
Verizon/MCI/UUNET (ASN 703)
10-nodes PoP (w/Singletons)
Evaluation of Geolocation
Databases
How to Evaluate GeoIP DBs
• Seven databases were used for the evaluation.
–
–
–
–
–
–
–
NetAcuity (Digital Element) – High end
GeoBytes
GeoIP (MaxMind)
IPligence Max
IP2Location
HostIP.info – Free service
Spotter – Research tool
• Dataset: DIMES measurements, March 2010
– 52K IP addresses (+ 52K singletons IP addresses)
– 3800 PoPs
19
• Main problem: NO ground truth!
• Idea: compare several databases
– Problem: some databases may get their data
from the same source
• Better idea:
– Use aggregation into PoPs to check DB
coherency
Evaluation methods
•
•
•
•
Vendor Reported Accuracy
Null Replies
Agreement within a database - coherency
“Ground Truth” location
Comparison Between databases
– Similarity
– By majority Vote
• Database anomalies
†US
state accuracy
115
PoP Location
Null Replies
• For each IP in the PoP (N IPs), each
database (M) get a vote on the geo-location
– Number of votes N•M
• Using the votes we define the PoP location
and convergence radius
118
Stage 1
Alternative Method
117
PoP Location and ‘Convergence’
• CAIDA use GeoIP DB voting for IP
location:
– Select a threshold. They use 80km
– Cluster locations within the threshold
– Node can belong to different clusters
– The largest cluster is selected as winner
– Use the centroid of the winning cluster
119
20
Stage 1
PoP spread radius
PoP spread radius
CDF of Location
Votes Percentage
Within 500km
from PoP Center
Correlation among Databases
CDF of Range of
Convergence
within Databases
Ground Truth evaluation
• Using CAIDA’s 25K “Ground Truth” IP addresses
– January-2010 database, based on DNS & ISP collaboration
– In the results, city range considered at 100km range
Heatmap – Median distance between databases
85
216
IP hits
Country Match
G b t
Geobytes
67 3%
67.3%
80 1%
80.1%
HostIP.Info
28.1%
89.0%
IP2Location
100%
76.0%
IPligence
100%
76%
Netacuity
67.9%
96.9%
Spotter
54.1%
---
City Match
10.1K wrongly
26 5%in
26.5%
located
Washington DC
17.9%
13.3%
0.7%
20.5K wrongly
79.1%
located in
Washington
27.8%DC
CDF- distances between databases
Comparison with CAIDA
85
Database
Database Correlation
216
134
134
IPligence- MaxMind (CAIDA)
21
Evaluating GeoLocation databases
Database Anomalies - Disagreement Between Databases
Evaluating GeoLocation databases
DIMES
Database Anomalies - Disagreement Between Databases
Global Crossing (ASN 3549)
160-nodes PoP (w/Singletons)
Verizon/MCI/UUNET (ASN 703)
10-nodes PoP (w/Singletons)
Database Anomalies –
False Location Replies
Improving Active Measurements
Spotter
` Geolocation tool, active measurements based
` Our findings (Tech report):
◦ Failed to locate a third of the IP addresses
◦ Its range of convergence was considerably higher than other databases
` Idea: Leverage DIMES infrastructure to improve Spotter
◦ More vantage points, in locations possibly closer to the target
◦ Larger dataset
◦ Conclusions from PoP study help analyze incorrect location assignments
` Goal: study the limits of active measurement based geolocation
Evaluating GeoLocation databases
Agreement Between Databases – By Majority Vote
Qwest as an example
•
•
•
•
•
70 PoPs were discovered by the algorithm
MaxMind assigned the PoPs to 55 different locations
HostIP.Info assigned the PoPs to 46 different locations
IP2Location assigned the PoPs to 35 different locations
IPligence located the PoPs in only one distinct location;
– All the PoPs were placed in Denver, where Qwest HQ are located.
– Out of 20291 Qwest entries in IPligence, 20252 are located in Denver.
• MaxMind had the same problem as IPligence in their May-2009 DB, but it
was fixed in July-2009 DB.
Agreement Between Databases – By
Majority Vote
CDF of Database
Location Deviation
From PoP Median.
Long tail.
22
DIMES
Summary
What Topology Can Tell US
DIMES
Many bad news:
• Ground truth has bias
• Coherency ≠ Accuracy
– BUT: incoherency ⇒ inaccuracy
• Database correlation
– Majority vote is tricky
Most results appear in arXiv:1005.5674, May 2010 & JSAC Dec 2011
133
Background (cont.)
• [Dhamdhere and Dovrolis , CoNext 2010]
– a new Internet model that captures the Internet
transition from a hierarchyy of transit pproviders
to a flatter interconnection of peers.
Background
• Internet economics is changing rapidly
• [Gill, et al. PAM 2008]
– Large content providers (CPs) bypass many
tier-1 ISPs by pushing their networks closer to
the users
– The Internet becomes flatter.
• [Labovitz et al., SIGCOMM 2010]
– Traffic from CPs is increasing in time
– Most of it is routed outside the core
Methodology
• Collect one month of traceroutes every 3
months
• IP to AS mapping.
pp g AS route to topology
p gy
– When crossing IXP ignore it A-X-B ⇒ A-B
– Mapping error do not change the trends
• DIMES has a large vantage point churn
– Use iPlane data instead
– DIMES shows, in most cases, the same trend
but with larger topologies and more noise
23
Our Work
• A longitudinal study over 5 years
• Observe the change in the Internet
ecosystem using topology instead of traffic
• Focus on content providers and tier-1 ISPs
ASes Measured
Degree
Content Providers:
A clear increase
in the degree
MSN
Google
Yahoo!
Amazon
Facebook
Contend Provides
Tier-1 ISPs
• Google
• AT&T
• Yahoo!
• Level 3
• MSN
• Sprint
• Facebook
• Qwest
• Amazon
• Global Crossing
CPs Neighbor Degree
Amazon and
Facebook:
Move from reliance on
tier-1 to more diverse
neighbors
CP Neighbor Country
Tier-1 ISPs:
Mostly decreasing
Tier-1 Neighbors
2007
2008
2009
2010
April
Veteran CPs already exploited their US
growth potential
24
New CPs grow in the US
Degree increase:
•Lose low degree
customers
•Other
O e customers
cus o e s
increase degree (?)
Clustering Coefficient
CP Neighbor Type
CPs:
CC mostly decreasing
New CPs diversify their neighbor type
Decrease the percentage of LTP
k-Pruning Analysis
Tier-1:
Mostly increasing (losing small stub customers)
k-Pruning Analysis
k-Pruning Analysis
1-shell
25
1-shell
k-Pruning Analysis
k-Pruning Analysis
1-shell
k-Pruning Analysis
1-shell
k-Pruning Analysis
1-shell
1-shell
2-shell
2-shell
3-shell = core
k-Pruning Analysis
k-Pruning Analysis
1-shell
1-shell
2-shell
2-shell
3-shell
3-shell = core
4-shell = core
26
Core Make-up
Shell Index
In DIMES:
MSN is below Yahoo! and
Google
Content/Access/
Hosting Provider
Large Transit
Provider
Small Transit
Provider
Enterprise
Customer
[Dhamdhere and Dovrolis, IMC’08]
Not only LTPs,
Not noticed by [Carmi et al., PNAS 2007]
Centrality
• The veteran CP are in the core (or
close)
•The new CPs advance very fast
towards the core
All tier-1 ISPs are in the
core
How Do CPs Connect?
• Betweenness centrality
1. Use shortest paths on the undirected AS graph
2. Examine all traceroute containing the AS
•
•
Calculate the portion of traceroute where the AS is
in the middle of the AS level route
More accurate than using ToR + valley free SPR
– The two methods gave similar results
CPs tend to use IXPs
• PageRank
Plotted with DIMES data
CPs Betweenness Centrality
Tier-1 Betweenness Centrality
Note the scale
Google and Yahoo!
main AS serve as a
transit to sibling ASes:
In 14% and 22% of the
traces, respectively.
27
AS10310
AS15169
• Sprint & Level3 are
slowly decreasing
• The rest are stable
Summary
PageRank
Note the scale
• CPs increase their connectivity
• Diversify their connections:
– From mostly tier
tier-11 to tier-2
tier 2 and others
– From US centric to global
• Tier-1 centrality is decreasing
• Indeed we see a flatter Internet
28