Download Interconnection Networks

Lecture 22
Interconnection Networks
Computer Architecture
COE 501
Network Media
Twisted Pair:
Copper, 1mm think, twisted to avoid
attenna effect (telephone)
Coaxial Cable:
Fiber Optics
Transmitter
– L.E.D
– Laser Diode
light
source
Used by cable companies:
high BW, good noise
immunity
Light: 3 parts
are cable, light
source, light
detector.
Total internal
Multimode
reflection
light disperse
Receiver
– Photodiode (LED), Single
mode sinle
wave (laser)
Silica
Plastic Covering
Braided outer conductor
Insulator
Copper core
Air
Shared vs. Switched Media
• Shared media: nodes share a single interconnection
medium (e.g., Ethernet)
–
–
–
–
Only one message sent at a time
Inexpensive : one medium used by all processors
Limited bandwidth : medium becomes the bottleneck
Needs arbitration
• Switched media : allow direct communication
between source and destination nodes (e.g., ATM)
–
–
–
–
–
Multiple users at a time
More expensive : need to replicated medium
Higher total bandwidth
No arbitration
Added latency to go through the switch
Ethernet Protocol - Shared
• Ethernet is an example of a shared media - similar
to a bus
• The ethernet nodes must coordinate so that only
one message is sent at a time.
– Nodes “listens” to the network to ensure it is not being used.
– If the network is not being used, the node tries to send the
message.
– If another node tries to send a message at the same time, a
collision occurs
– Both nodes detect the collision and try to resend their
messages after a random amount of time
– Subsequent collisons result in exponentially increasing random
time between attempts to resend data
– What are advantages and disadvantages of this method?
Switch Topology
• Switched media have a topology that indicate
how nodes are connected
• Topology determines
–
–
–
–
Degree: number of links from a node
Diameter: max number of links crossed between nodes
Average distance: number of links to random destination
Bisection: minimum number of links that separate the
network into two halves
– Bisection bandwidth: link bandwidth x bisection
• Warning: Three-dimensional drawings must
be mapped onto chips and boards which are
essentially two-dimensional media
– Elegant when sketched on the blackboard may look
awkward when actually constructed
Common Topologies
Type
Degree Diameter Ave Dist
N/3
Bisection
1D mesh
2
N-1
2D mesh
4
2(N1/2 - 1) 2N1/2 / 3
N1/2
3D mesh
6
3(N1/3 - 1) 3N1/3 / 3
N2/3
nD mesh
2n
n(N1/n - 1) nN1/n / 3
N(n-1) / n
Ring
2
N/2
N/4
2
2D torus
4
N1/2
N1/2 / 2
2N1/2
Hypercube
Log2N n=Log2N
n/2
N/2
2D Tree
3
2Log2N
~2Log2 N 1
Crossbar
N-1
1
1
N = number of nodes, n = dimension
1
N2/2
Butterfly or Omega Network
N/2
Butterfly
°
°
°
• All paths equal length
• Unique path from any
input to any output
N/2
Butterfly
°
°
°
• Try to avoid conflicts
Multistage Fat Tree
• A multistage fat tree (CM-5) avoids congestion at
the root node
• Randomly assign packets to different paths on
way up to spread the load
• Increase degree, decrease congestion
Example Networks
(Figure 7.19, pg 591)
Name
nCube/ten
iPSC/2
MP-1216
Delta
CM-5
CS-2
Paragon
T3D
Number
1-1024
16-128
32-512
540
32-2048
32-1024
4-1024
16-1024
Topology
10-cube
7-cube
2D grid
2D grid
fat tree
fat tree
2D grid
3D Torus
Bits
Clock
1 10 MHz
1 16 MHz
1 25 MHz
16 40 MHz
4 40 MHz
8 70 MHz
16 100 MHz
16 150 MHz
Link
1.2
2
3
40
20
50
200
300
Bis. BW
640
345
1,300
640
10,240
50,000
6,400
19,200
MBytes/second
No standard topology!
Year
1987
1988
1989
1991
1991
1992
1992
1993
Connection-Based vs.
Connectionless
• Connection-based system:
– Telephone: operator sets up connection between the caller and the
receiver
– Once the connection is established, the line is held until it
completes
– Share transmission lines over long distances by using switches to
multiplex several conversations on the same lines
» “Time division multiplexing” divide B/W transmission line into
a fixed number of slots, with each slot assigned to a
conversation
– Often referred to as circuit-swtiching
– Problem: lines busy based on number of conversations, not
amount of information sent
– Advantage: reserved bandwidth - call not interrupted once started
Connection-Based vs.
Connectionless
• Connectionless system
– Postal service : package routed to destination based on
address
– Each message is divided into packets, with an address
per packet
– Packets can each take different routes through the
network
– Often referred to as packet-switching
– Disadvantage : additional information sent with each
message
Store and Forward vs. Cut-Through
• Store-and-forward policy: each switch waits for the
full packet to arrive in the switch before it is sent on
to the next switch
• Cut-through routing or worm hole routing: switch
examines the header, decides where to send the
message, and then starts forwarding it immediately
– In worm hole routing, when the head of the message is blocked the
message stays strung out over the network, potentially blocking
other messages (needs only buffer the size of the packet that is sent
between switches).
– Cut through routing lets the tail continue when the head is blocked,
storing the whole message into a single switch. (Requires a buffer
large enough to hold the largest packet).
Packet Formats
• Fields: Destination, Checksum(C), Length(L), Type(T)
• Data/Header Sizes in bytes:
– CM5: (4 to 20)/4, Ethernet: (0 to 1500)/26, ATM: 48/5
Congestion Control
• Packet switched networks do not reserve bandwidth; this
leads to contention
• Solution: prevent packets from entering until contention
is reduced (e.g., entrance lights on interstate)
• Options:
– Packet discarding: If a packet arrives at a switch and there is no room in
the buffer, the packet is discarded, SW detects and resends
– Flow control: between pairs of receivers and senders; use feedback to
tell the sender when it is allowed to send the next packet
» Back-pressure: separate wires to tell to stop
» Window: give the original sender the right to send N packets before
getting permission to send more (TCP uses)
– Choke packets: Each packet received by busy switch in warning state
sent back to the source via choke packet. Source reduces traffic to that
destination by a fixed % (ATM Forum)
Example Architectures
Interconnect
MPP
LAN
WAN
Example
Topology
CM-5
“Fat” tree
Ethernet
Line, Bus
Connection based?
Data Transfer Size
No
Variable:
4 to 20B
No
At source:
Flow
control
via back
pressure
No
Variable:
0 to 1500B
N/A
At source:
Listen for
E-net idle
ATM
Variable,
constructed
from multistage
switches
Yes
Fixed:
48B
Yes
Rate based
via choke
packets
Store & Forward?
Congestion control
Routing
• Deterministic—follows a prespecified route
– mesh: dimension-order routing
» (x1, y1) -> (x2, y2)
» first Dx = x2 - x1,
» then Dy = y2 - y1,
– hypercube: edge-cube routing
» X = xox1x2 . . .xn -> Y = yoy1y2 . . .yn
» R = X xor Y
» Traverse dimensions of differing
address in order
• Adaptive—route based on
network state (e.g., contention)
110
010
111
011
100
000
001
101
Practical Issues
Interconnection
Example
Standard
Fault Tolerance?
Hot Insert?
MPP
CM-5
No
No
No
LAN
Ethernet
Yes
Yes
Yes
WAN
ATM
Yes
Yes
Yes
• Standards: required for WAN, LAN!
• Fault Tolerance: Can nodes fail and still deliver
messages to other nodes? required for WAN, LAN!
• Hot Insert: If the interconnection can survive a failure,
can it also continue operation while a new node is
added to the interconnection? required for WAN, LAN!
Protocols
• Internetworking: allows computers on independent
and incompatible networks to communicate reliably
and efficiently;
– Enabling technologies: SW standards that allow reliable
communications without reliable networks
– Hierarchy of layers, giving each layer responsibility for portion of
overall communications task, called protocol families or protocol
suites
• Transmission Control Protocol/Internet Protocol
(TCP/IP)
– This protocol family is the basis of the Internet
– IP : layer above interconnection network - routes datagram to
destination machine
– TCP : layer above IP - guaranees reliable, in-order delivery and
prevents corruption of data
TCP/IP packet
• Application sends message
• TCP breaks into 64KB
segements, adds 20B
header
• IP adds 20B header, sends
to network
• If Ethernet, broken into
1500B packets with
headers, trailers
• Header, trailers have length
field, destination, window
number, version, ...
Ethernet
IP Header
TCP Header
IP Data
TCP data
(< 64KB)
Protocol
• Key to protocol families is that communication occurs
logically at the same level of the protocol, called peer-topeer, but is implemented via services at the lower level
• Danger is each level increases latency
Connecting Networks
• Bridges: connect LANs together, passing
traffic from one side to another depending on
the addresses in the packet.
– operate at the Ethernet protocol level
– usually simpler and cheaper than routers
• Routers or Gateways: these devices connect
LANs to WANs or WANs to WANs and resolve
incompatible addressing.
– Generally slower than bridges, they operate at the
internetworking protocol (IP) level
– Routers divide the interconnect into separate smaller
subnets, which simplifies manageability and improves
security
Summary
Interconnection Networks
• Shared vs. Switched media
• Switch topology indicates how nodes are connected
• Switching network can be either connection-based
(circuit switched) or connectionless (packet switched)
• In packet-switched networks congestion control is
required
• Protocols allow different computers on different
networks to communicate reliably