Download Issues of Interconnection Networks

Outline of Contents
① Issues, Criteria and Topics.
② Motivations behind This Line of Research
③ Survey of Typical Networks
④ My Research Work and Contributions
1
① Issues / Criteria / Topics
Shanghai Jiao Tong University 2011





Topology and Analysis
Routing and Communication
Mapping and Simulation
Algorithm and Computation
VLSI Design and Construction
2
① Issues / Criteria / Topics
Shanghai Jiao Tong University 2011

Topology and Analysis
 degree,diameter,average distance, bisection
bandwidth,connectivity,symmetry,recursiveness,
scalability,Hamiltonian path




Routing and Communication
Mapping and Simulation
Algorithm and Computation
VLSI Design and Construction
3
① Issues / Criteria / Topics
Shanghai Jiao Tong University 2011


Topology and Analysis
Routing and Communication
 Pattern: routing,broadcasting,multicasting,gossip
 Evaluation:Easy routing,deadlock-free,delay,traffic
density,
 Control Strategy: centralized/distributed,
deterministic/adaptive,minimal/non-minimal,
 Switching: circuit/packet, wormhole, virtual cut-through



Mapping and Simulation
Algorithm and Computation
VLSI Design and Construction
4
① Issues / Criteria / Topics
Shanghai Jiao Tong University 2011



Topology and Analysis
Routing and Communication
Mapping and Simulation
 To compare the computing power
embedding
G
H
 Smallest possible dilation and congestion


Algorithm and Computation
VLSI Design and Construction
5
① Issues / Criteria / Topics
Shanghai Jiao Tong University 2011
How to embed a ring into a line:
1
2
8
7
3
6
4
5
1
2
3
4
5
6
7
8
Dilation=7, congestion=2
1
2
3
4
5
6
7
8
8
1
7
2
6
3
5
4
Dilation=2, congestion=2
6
① Issues / Criteria / Topics
Shanghai Jiao Tong University 2011




Topology and Analysis
Routing and Communication
Mapping and Simulation
Algorithm and Computation
 PRAM model is unpractical.
 Basic algorithms:
sorting,searching,permutation,matrix
multiplication,bit reversal, graph
algorithm,iteration method,symbolic computing.

VLSI Design and Construction
7
① Issues / Criteria / Topics
Shanghai Jiao Tong University 2011





Topology and Analysis
Routing and Communication
Mapping and Simulation
Algorithm and Computation
VLSI Design and Construction
 Wafer-scale integration
 Layout design: area and wire length,wire
area,crossing number,node cost and modularity
 Applicable to the board design
8
② Motivations
Shanghai Jiao Tong University 2011



A branch of combinatorics
Structures for building multiprocessor
and multicomputer
More than structural:
 ASIC:implementing special parallel algorithm
on VLSI/Wafer scale.
 P2P overlay topologies
 Data Center Networking
 SoC or NoC for Many-core
9
② Big Names
Shanghai Jiao Tong University 2011
 F.
Thomson Leighton (Theory) http://theory.lcs.mit.edu/~ftl/
 Lionel M. Ni (wormhole) http://www.cps.msu.edu/~ni/
 Arnold L. Rosenberg (Butterfly) http://www.cs.umass.edu/~rsnbrg/
 Burkhard Monien (Embedding) http://www.uni-paderborn.de/fachbereich/AG/monien/
 Laxmi N. Bhuyan (Multi-stage) http://www.cs.tamu.edu/faculty/bhuyan/
 Kenneth E. Batcher (Bitonic) http://nimitz.mcs.kent.edu/~batcher/index.html
 Ivan Stojmenovic (Honeycomb) http://www.csi.uottawa.ca/~ivan/
 Ke Qiu (Star and Pancake) http://dragon.acadiau.ca/~kqiu/home.html
 Wei Zhao (Routing) http://www.cs.tamu.edu/faculty/zhao/
 S. Lennart Johnsson (Hypercube) http://www.cs.uh.edu/~johnsson/
 Satoshi Fujita(gossip) http://www.se.hiroshima-u.ac.jp/~fujita/
 William J. Dally (k-ary n-cube) http://www.ai.mit.edu/people/billd/
 S. Yalamanchili (Engineering ct) http://users.ece.gatech.edu/~sudha/
 C.E. Leiserson (Parallel Algorithms) http://supertech.lcs.mit.edu/~cel/
 Kai Hwang (Benchmarking) http://ceng.usc.edu/~kaihwang/
10
② A Special Journal on Interconnection Networks
Shanghai Jiao Tong University 2011
Among the editorial board:
D Frank Hsu (Fordham University)
Bruce M Maggs (Carnegie Mellon )
Jean-Claude Bermond (CNRS/INRIA/UNSA)
Tse-Yun Feng (Penn State University)
Yoji Kajitani (Tokyo Institute of Tech)
F Tom Leighton (MIT)
Guo-Jie Li (Chinese Academy of Sciences)
Burkhard Monien (University of Paderborn)
Howard Jay Siegel (Purdue University)
Tom Stern (Columbia University)
Website: http://journals.worldscinet.com/join
11
③ Survey of Network Topology
The following discussion of the properties of interconnection
networks is based on a collection of nodes that
communicate via links. In an actual system the nodes can
be either processors, memories, or switches. Two nodes
are neighbors if there is a link connecting them.
12
③ Network Parameters
Shanghai Jiao Tong University 2011

Channel width:
 Number of wires that are used per channel (i.e. the number of bits
that can be transmitted simultaneously on one channel).

Channel direction:
 The direction in which the messages can be transmitted.
 Unidirectional : can send messages in just one direction
 Bi-directional : support two-way communication over the same
channel.

Bisectional width (BW):
 It is defined as the number of channels that are cut when the
network is divided into two equal parts.
13
③ Network Parameters
Shanghai Jiao Tong University 2011

Node degree ():
Number of channels connecting a node to its neighbors.
In practice the degree of a topology has an effect on cost, since the
more links a node has the more logic it takes to implement the
connections.

Network diameter (D):
Maximum distance between any two nodes in the network.

Cost Effectiveness:  x D,
 Problem of Dense graph: Given the fixed degree and diameter, how
to design a graph which can contain as many nodes as possible?

Average distance:

=

i , jV
d (i, j )
2
n
14
③ Network Parameters
Shanghai Jiao Tong University 2011



Number of links (l):
The total number of channels in the network.
Symmetry:
A network is said to be symmetric if it looks the same
from every node. (E.g. Hypercube, crossbar)
Recursiveness
A large-size network consists of two or many small-size
networks of the same kind, such as tree and hypercube.
This property renders the network suitable for solving
divide-and-conquer algorithm
15
③ Crossbar switch
Shanghai Jiao Tong University 2011
P0 –> M1
P1 –> M3
P2 –> M2
P3 –> M0
Crossbar switch
N*N switches (N processors N memory modules). The switch
configures itself dynamically to connect a processor to a memory
module. No contention -- Supports N! permutations. Costly, hard to
scale, wastes switches for most patterns.
16
③ Crossbar switch
Shanghai Jiao Tong University 2011




Easy for broadcasting and also for any permutation. As long
as each processor wants to communicate with a different
memory there will be no contention -- Supports N!
permutations.
If two or more processors need to access the same memory,
however, one will be blocked until the switch reconfigures
itself.
A crossbar has a short diameter - information needs to pass
through only one switching element on a path from one
edge to another.
Poor scalability -- If there are N processors and N
memories, there are N2 interior switches. Adding another
processor or memory means adding another N interior
nodes.
17
③ Multistage Network
P1
P2
P2
Stage k
Stage 3
PP-1
Stage 2
P3
Stage 1
Shanghai Jiao Tong University 2011
P1
P3
PP-1
Systems built with these topologies have
processors on one edge of the network, memories
or processors on another edge, and a series of
switching elements at the interior nodes.
18
③ Multistage Network : Omega Network
Shanghai Jiao Tong University 2011
p processors and log2 p stages
Each stage consists of a perfect shuffle
19
③ Multistage Network : Butterfly
Shanghai Jiao Tong University 2011
ButterFly Network
20
③ Multistage Network
Shanghai Jiao Tong University 2011




Built from small (e.g., 2X2 crossbar) switch nodes, with a
regular interconnection pattern.
In order to send information from one edge to another, the
interior switches are configured to form a path that connects
nodes on the edges.
The information then goes from the sending node, through
one or more switches, and out to the receiving node.
The size and number of interior nodes contributes to the
path length for each communication, and there is often a
``setup time'' involved when a message arrives at an interior
node and the switch decides how to configure itself in order
to pass the message through.
21
③ Multistage Network: Problems
Shanghai Jiao Tong University 2011



Generalized MINs: for in Inputs and jn outputs, use
i×j switches at n stages.
Bens networks: with 2logn-1 stages, it becomes a
nonblocking network, allowing all permutations.
Fault tolerance: form all switches at a stage as a
ring.
22
③ Line, Ring and Fully Connected
Shanghai Jiao Tong University 2011
Linear Array: the simplest topology. Fully Connected: Direct
connection between every pair of processors, Highest cost, Similar to
crossbar in some properties. Ring: Each node in the ring is connected to
only two other nodes.Chordal Ring: A compromise between Ring and Fully
Connected Network.
Fully connected: Degree = N-1, Diameter = 1, BW = (N/2)2, Links = N*(N-1)/2,
Symmetric. Ring: Diameter = N/2, Degree=2, BW = 2, Symmetric
23
③ Mesh and Torus
Shanghai Jiao Tong University 2011
Mesh Topology
2D Torus
3D Torus
2D torus: Meshes with ``wraparound'' connections, e.g. the
node at the top of the grid has an ``up'' link that connects to
the node at the bottom of the grid (also left to right).
Mesh: Deg = 2,3,4, Diameter = 2*Sqrt(N), Bisect= Sqrt(N), Easy to
build, scalable .
k-ary n-cube: Generalization of mesh-like Networks
24
③ Mesh and Torus: Problems
Shanghai Jiao Tong University 2011



Diameter of Asymmetric Networks: linear array, mesh,
tree, shuffle exchange.
Torus and Midmiew: how to use the wraparound links
to obtain the optimal diameter?
Layout of torus: Reduce the physical wire length.
25
③ Hypercube
Shanghai Jiao Tong University 2011
Can embed Hamiltonian cycle, mesh, tree, etc. Low latency, high
bandwidth, but costly (high number of links). Hard to build (layout
the chip and wires). Hard to scale up (As degree increases, number
of I/O ports increases). Solution: CCC
For dimension D, Degree = D, Diameter = D, Bisect = 2(d-1) Nodes =
2D, Links = D*2(D-1)
26
③ Hypercube : assign node IDs
Shanghai Jiao Tong University 2011
011
The nodes are numbered so that two nodes are adjacent if and
only if the binary representations of their IDs differ by one bit. For
example, nodes 011 and 010 are immediate neighbors.
27
③ Hypercube : Properties
Shanghai Jiao Tong University 2011







Hamiltonian Cycle
2n-node hypercube contains I×J mesh, where I×J = 2n
Contains 2 binary tree with 2n-1－1 nodes
Optimal in fault-tolerance
Variants of hypercube:folded hypercube, hierarchical
hypercube,incomplete hypercube,CCC, shuffle-change.
Y. Saad and M.H. Schultz, Topological Properties of
Hypercube, IEEE TC, July 1988.
Generalized Hypercube, IEEE TC, April 1984.
28
③ CCC: Solution to Hypercube :
Shanghai Jiao Tong University 2011
110
100
111
101
010
000
011
001
(000,2)
(000,1)
(000,0)
1
2
F.P. Preparata, et al, The Cube-Connected Cycles: A Versatile
Network for Parallel Computation, CACM, May 1981,
G. Chen, et al, Tight Layouts of CCC, IEEE TPDS, Feb. 2000.
G. Chen, et at, Layout of CCC without Long Wires, Computer
Journal, in 2005.
29
③ Tree and Star
Shanghai Jiao Tong University 2011
Tree and Star
Tree: Degree = 1 (leaves), 2 (root), 3 (interior nodes),
Diameter = 2logN, Bisect = 1.
Tree/Star bottleneck: Expect at root
30
③ Tree
Shanghai Jiao Tong University 2011


The tree has nodes of degree 1 (leaves), 2 (root), 3 (interior
nodes). So it is not symmetric.
Short diameter:
 For depth k, the number of nodes is 2k - 1, and the diameter is
2k (or ~ 2 log N, at the same order as Hypercube).
 For example, a processor with 262,144 nodes would have
diameter 512 in a mesh but only 36 in a tree.

Bisection bandwidth is 1. It suffers from a serious bottleneck
-- Solution : Fat Tree
31
③ Fat Tree
Shanghai Jiao Tong University 2011
Expand link bandwidth at each higher level
B
B
B/2
B/4
B
B/8
Fat-Tree
Fat-tree layout
32
③ Fat Tree
Shanghai Jiao Tong University 2011



Leiserson's fat-trees [Lei85] – built in CM-5.
The Fat Tree network provides uniform bandwidth between
any two end-points on a net. It does this by doubling the
number of "up" paths as one goes "up" the tree from a
processor in a leaf to the root node.
Given a fat tree of this type with h levels of switches,
 Number of processor nodes = 2h
 Number of switch nodes = 2h- 1
 Greatest distance between processor nodes = 2h
33
③ Packet Routing in Fat Tree
Shanghai Jiao Tong University 2011
8 port
four paths
4 port
PE1
PE2
PE3
PE4
PE5
PE6
PE7
PE8
PE9
PE10
PE11
PE12
PE13
PE14
PE15
PE16
Note that a message going from PE2 to PE5 may choose any
one of four paths from the lower left router to the one at the
root. This means that all four PE's attached to the lower left
router have a path available to them to reach another node.
34
③ Comparison -Shanghai Jiao Tong University 2011
Degree
(d)
(N node, n=log 2 (N))
Diameter
(D)
Bisection
(B)
No. of Links
(L)
1D mesh
2
N-1
1
N-1
2D mesh
4
2(N1/2 - 1)
N1/2
2N- N1/2
N-1
2
2
N-1
Ring
2
N/2
2
N
2D torus
4
2(N1/2 / 2)
N1/2
2N
Hypercube
n
n
N/2
n N/2
Complete connected
N-1
1
(N/2)2
N(N-1)/2
star
35
③ Example MPP Networks
Shanghai Jiao Tong University 2011
Name
Number
nCube/ten 1-1024
iPSC/2
16-128
MP-1216 32-512
Delta
540
CM-5
32-2048
CS-2
32-1024
Paragon 4-1024
T3D
16-1024
Topology Bits
Clock
10-cube
1 10 MHz
7-cube
1 16 MHz
2D Mesh
1 25 MHz
2D Mesh
16 40 MHz
fat tree
4 40 MHz
fat tree
8 70 MHz
2D mesh 16 100 MHz
3D Torus 16 150 MHz
Link Bisect.
1.2
640
2
345
3
1,300
40
640
20
10,240
50
50,000
200
6,400
300
19,200
Year
1987
1988
1989
1991
1991
1992
1992
1993
MBytes/second
No standard MPP topology!
36
③ Scalable Topology
Shanghai Jiao Tong University 2011

Scalability
 Refers to the increase in the complexity of
communication (time) as more nodes are added.
 In a highly scalable topology more nodes can be added
without severely increasing the amount of logic required
to implement the topology and without increasing the
diameter.
 Example : Doubling the number of nodes in a
hypercube increases the degree by only 1 link per
node, and likewise increases the diameter by only 1
path. An opposite example is linear array.
37
③ Hypercube Problems
Shanghai Jiao Tong University 2011

Embedding the links:
 For a 64K node (d=16) hypercube machine, there would be
512K (16x64K/2) links.
 With current technology, it is difficult to scale a hypercube
beyond about 4K nodes, with about 24K links.

Operation cost:
 As the number of dimensions increases, the nodes must do
more work to keep up with the incident message traffic.
 In fact, because most processors can handle only one I/O
transaction at a time, many hypercube algorithms operate on
the principle of serializing processing by dimension. i.e. all
pairs of nodes in one dimension communicate, then the next
dimension, etc.  See Ascend/Descend Algorithms
38
③ Symmetric Topology
Shanghai Jiao Tong University 2011

Symmetric
 Rings, fully connected networks, and hypercubes
are all node symmetric.
 Trees and stars are not. A tree has three different
types of nodes, namely a root node, interior nodes,
and leaf nodes, each with a different degree. A star
has a distinguished node in the center which is
connected to every other node.
 When a topology is node asymmetric a distinguished
node can become a communications bottleneck.
 Importance of Symmetry: Node symmetry renders
identical software at every node;edge symmetry
avoids hot traffic spot in the network.
39
Routing algorithms—XY Routing
Shanghai Jiao Tong University 2011




P Sx,Sy source processor
P Dx,Dy destination
processor
Shortest path –
“manhattan distance”,
abs(Sx-Dx) + abs(Sy-Dy)
Algorithm:
1.Reduce distance along X
dimension until 0
2.Reduce distance along Y
dimension until at P Dy,Dx
P 0,0 :source
P 3,2 :destination
P 0,0
P 3,2
40
③ Routing algorithms—XY Routing
Shanghai Jiao Tong University 2011
M[S,D]
 In a mesh, S
 More Information
I
M[S,D]
D
Begin
S(0,0)
receive(M);
If D=I accept(M)
Elseif D • x >I • x
sendright(M)
Elseif D • x <I • x sendleft(M)
Elseif D • y >I • y
senddown(M)
Elseif D • y >I • y sendup(M)
Endif;
D(3,2)
end
41
E-Cube Routing
Shanghai Jiao Tong University 2011

E-Cube Routing
Ps source processor
Pd destination processor


Shortest path -- hamming
distance between Ps and Pd
Routing taken in the following
manner:
1. Exclusive-or  the source and
destination processor numbers
2. Going from the Least
Significant Bit (lsb) to Most
Significant Bit (MSB). Each
position a “1” exists in the
result of the exclusive, an edge
is taken.
P000 source
 P110 destination
110
42
④ My Work and contributions
 Novel Interconnection Networks:
Shuffle-Ring, Shuffle-Cube, Wall Mesh.
 Layout Design: MIDIMEW, CCC.
 Routing in special Networks:
Shuffle-Exchange, Unidirectional Networks
43
④ New Topology: Shuffle-Ring
Shanghai Jiao Tong University 2011
Definition: anan-1…a1a0 is connected to
shuffle
an-1an-2…a0an
ring
anan-1…a1 ( a0 ±1）
Properties:
Simplicity
Constant degree
Keep all hypercube properties
Compact layout
•G. Chen et al, Shuffle-Ring: Overcoming the Increasing Degree of
Hypercube, Proc. 2nd Int. Symp. on High-Performance Computer
Architecture(HPCA-2), California, Feb. 1996, 130-138.
•G. Chen el al, Shuffle-Ring: A New Constant-degree Network, International
Journal of Foundations of Computer Science, March 1998, 77-98
44
④ New Topology: Shuffle-Cube
Shanghai Jiao Tong University 2011
Definition: anan-1…a1a0 is connected to
shuffle
an-1an-2…a0an
Cube
anan-1…ai … a0 (i<k)
0
1
8
9
2
3
a
b
4
5
c
d
6
7
e
f
Properties:
Constant degree
Keep all hypercube properties
Compact layout
•G. Chen, et al, CTSN: A New Fault-Tolerant Network, Proceedings of 13th
International Conference on Parallel and Distributed Computing
Systems(PDCS'2000), Las Vegas, Nevada, August 2000, 517-522.
45
④ New Topology: Wall Mesh
Shanghai Jiao Tong University 2011
Definition: ( x, y ) is connected to
( x ±1, y ) horizontally
and ( x, y ±1 ) vertically.
Properties:
 Constant degree=3 or 4,
 Equivalent to mesh in Computing Power
 Logarithmic diameter
 Easy layout
•G. Chen,et al, The Wall Mesh, Computer
Architecture'97: Selected Papers of the
2nd Australasian Conference, Springer,
1997, 217-230.
•陈贵海等， 墙式网孔， 计算机学报， 2000
年4月， 374-381页。
46
④ New Topology: Isomorphism
Shanghai Jiao Tong University 2011
•G. Chen et al, Comments on ``A New Family of Cayley Graph
Interconnection Networks of Constant Degree Four'', IEEE Transactions
on Parallel and Distributed Systems, December 1997, 1299-1300.
47
④ Layout Design: MIDIMEW
Shanghai Jiao Tong University 2011
•G. Chen, et al, Laying Out Midimew Networks with Constant Dilation,
Lecture Notes in Computer Science(854), September 1994, 773-784.
•G. Chen et al, Optimal Layouts of Midimew Networks, IEEE Transactions
on Parallel and Distributed Systems, September 1996, 954-961.
48
④ Layout Design: Cube-Connected Cycles
Shanghai Jiao Tong University 2011
110
100
111
101
010
000
011
001
(000,2)
(000,1)
(000,0)
G. Chen, et al, A Compact Layout of Cube-Connected Cycles, Proc. of 4th
Int. Conf. on High Performance Computing, India, Dec. 1997, 422-427.
G. Chen, et al, Tight Layouts of CCC, IEEE Transactions on Parallel and
Distributed Systems, Feb. 2000, 182-191.
G. Chen, et at, Layout of CCC without Long Wires, Computer Journal,
Nov. 2001, Vol. 44, No. 5.
49
④ Routing: Shuffle-Exchange Networks
Shanghai Jiao Tong University 2011
Definition:
anan-1…a1a0 is connected to
shuffle
an-1an-2…a0an
Exchange
an-1an-2…a1a0
•G. Chen, et al, Shortest-Path Routing in
Shuffle-Exchange Networks, Lectures in
Operations Research, August 1998, 142-153.
•陈贵海等，洗牌交换网中的最短路由算法，
计算机学报，2001年1月。
•G. Chen, et al, An Algorithm for Optimal
Routing in Shuffle-Exchange Networks, to
appear in IEEE Transactions on Computers.
50
④ Routing: Unidirectional Networks
Shanghai Jiao Tong University 2011
G. Chen, at al, A Distance-Vector Routing Protocol for Networks with
Unidirectional Links, Computer Communications, Feb. 2000, 418-424.
•G. Chen, et al, An Improved Routing Protocol for Networks with
Unidirectional Links, ICPP, Spain, Sept. 2001.
51
Questions?
Shanghai Jiao Tong University 2011
Shanghai Jiao Tong University 2011
Assignment 1
1) Prove any graph/network has a longest path?
2) Choose one of the following
2.1) Write the routing algorithm for complete binary tree?
(Suppose nodes are numbered from top to down, from left to right, and
beginning from root.)
2.2) Prove that hypercube has optimal fault tolerance.
Send your solutions to TA via Email before Nov.17, 2012.
53
Shanghai Jiao Tong University 2011
Reading materials
1) Y. Saad and M. H. Schultz, Topological Properties of Hypbercube, IEEE
Transactions on Computers，Vol. 24, No. 5
2) L.N. Bhuyan and D.P. Agrawal, Generalized Hypercube and Hyperbus
Structures for a Computer Network, IEEE Transactions on Computers, Vol.
33, No. 5, 1984
3) C. E. Lerserson, Fat-tree: Universal network for Hardware-Efficient
Supercomputing, IEEE Transactions on Computers, pp. 892-901, 1994
54