Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Outline of Contents ① Issues, Criteria and Topics. ② Motivations behind This Line of Research ③ Survey of Typical Networks ④ My Research Work and Contributions 1 ① Issues / Criteria / Topics Shanghai Jiao Tong University 2011 Topology and Analysis Routing and Communication Mapping and Simulation Algorithm and Computation VLSI Design and Construction 2 ① Issues / Criteria / Topics Shanghai Jiao Tong University 2011 Topology and Analysis degree,diameter,average distance, bisection bandwidth,connectivity,symmetry,recursiveness, scalability,Hamiltonian path Routing and Communication Mapping and Simulation Algorithm and Computation VLSI Design and Construction 3 ① Issues / Criteria / Topics Shanghai Jiao Tong University 2011 Topology and Analysis Routing and Communication Pattern: routing,broadcasting,multicasting,gossip Evaluation:Easy routing,deadlock-free,delay,traffic density, Control Strategy: centralized/distributed, deterministic/adaptive,minimal/non-minimal, Switching: circuit/packet, wormhole, virtual cut-through Mapping and Simulation Algorithm and Computation VLSI Design and Construction 4 ① Issues / Criteria / Topics Shanghai Jiao Tong University 2011 Topology and Analysis Routing and Communication Mapping and Simulation To compare the computing power embedding G H Smallest possible dilation and congestion Algorithm and Computation VLSI Design and Construction 5 ① Issues / Criteria / Topics Shanghai Jiao Tong University 2011 How to embed a ring into a line: 1 2 8 7 3 6 4 5 1 2 3 4 5 6 7 8 Dilation=7, congestion=2 1 2 3 4 5 6 7 8 8 1 7 2 6 3 5 4 Dilation=2, congestion=2 6 ① Issues / Criteria / Topics Shanghai Jiao Tong University 2011 Topology and Analysis Routing and Communication Mapping and Simulation Algorithm and Computation PRAM model is unpractical. Basic algorithms: sorting,searching,permutation,matrix multiplication,bit reversal, graph algorithm,iteration method,symbolic computing. VLSI Design and Construction 7 ① Issues / Criteria / Topics Shanghai Jiao Tong University 2011 Topology and Analysis Routing and Communication Mapping and Simulation Algorithm and Computation VLSI Design and Construction Wafer-scale integration Layout design: area and wire length,wire area,crossing number,node cost and modularity Applicable to the board design 8 ② Motivations Shanghai Jiao Tong University 2011 A branch of combinatorics Structures for building multiprocessor and multicomputer More than structural: ASIC:implementing special parallel algorithm on VLSI/Wafer scale. P2P overlay topologies Data Center Networking SoC or NoC for Many-core 9 ② Big Names Shanghai Jiao Tong University 2011 F. Thomson Leighton (Theory) http://theory.lcs.mit.edu/~ftl/ Lionel M. Ni (wormhole) http://www.cps.msu.edu/~ni/ Arnold L. Rosenberg (Butterfly) http://www.cs.umass.edu/~rsnbrg/ Burkhard Monien (Embedding) http://www.uni-paderborn.de/fachbereich/AG/monien/ Laxmi N. Bhuyan (Multi-stage) http://www.cs.tamu.edu/faculty/bhuyan/ Kenneth E. Batcher (Bitonic) http://nimitz.mcs.kent.edu/~batcher/index.html Ivan Stojmenovic (Honeycomb) http://www.csi.uottawa.ca/~ivan/ Ke Qiu (Star and Pancake) http://dragon.acadiau.ca/~kqiu/home.html Wei Zhao (Routing) http://www.cs.tamu.edu/faculty/zhao/ S. Lennart Johnsson (Hypercube) http://www.cs.uh.edu/~johnsson/ Satoshi Fujita(gossip) http://www.se.hiroshima-u.ac.jp/~fujita/ William J. Dally (k-ary n-cube) http://www.ai.mit.edu/people/billd/ S. Yalamanchili (Engineering ct) http://users.ece.gatech.edu/~sudha/ C.E. Leiserson (Parallel Algorithms) http://supertech.lcs.mit.edu/~cel/ Kai Hwang (Benchmarking) http://ceng.usc.edu/~kaihwang/ 10 ② A Special Journal on Interconnection Networks Shanghai Jiao Tong University 2011 Among the editorial board: D Frank Hsu (Fordham University) Bruce M Maggs (Carnegie Mellon ) Jean-Claude Bermond (CNRS/INRIA/UNSA) Tse-Yun Feng (Penn State University) Yoji Kajitani (Tokyo Institute of Tech) F Tom Leighton (MIT) Guo-Jie Li (Chinese Academy of Sciences) Burkhard Monien (University of Paderborn) Howard Jay Siegel (Purdue University) Tom Stern (Columbia University) Website: http://journals.worldscinet.com/join 11 ③ Survey of Network Topology The following discussion of the properties of interconnection networks is based on a collection of nodes that communicate via links. In an actual system the nodes can be either processors, memories, or switches. Two nodes are neighbors if there is a link connecting them. 12 ③ Network Parameters Shanghai Jiao Tong University 2011 Channel width: Number of wires that are used per channel (i.e. the number of bits that can be transmitted simultaneously on one channel). Channel direction: The direction in which the messages can be transmitted. Unidirectional : can send messages in just one direction Bi-directional : support two-way communication over the same channel. Bisectional width (BW): It is defined as the number of channels that are cut when the network is divided into two equal parts. 13 ③ Network Parameters Shanghai Jiao Tong University 2011 Node degree (): Number of channels connecting a node to its neighbors. In practice the degree of a topology has an effect on cost, since the more links a node has the more logic it takes to implement the connections. Network diameter (D): Maximum distance between any two nodes in the network. Cost Effectiveness: x D, Problem of Dense graph: Given the fixed degree and diameter, how to design a graph which can contain as many nodes as possible? Average distance: = i , jV d (i, j ) 2 n 14 ③ Network Parameters Shanghai Jiao Tong University 2011 Number of links (l): The total number of channels in the network. Symmetry: A network is said to be symmetric if it looks the same from every node. (E.g. Hypercube, crossbar) Recursiveness A large-size network consists of two or many small-size networks of the same kind, such as tree and hypercube. This property renders the network suitable for solving divide-and-conquer algorithm 15 ③ Crossbar switch Shanghai Jiao Tong University 2011 P0 –> M1 P1 –> M3 P2 –> M2 P3 –> M0 Crossbar switch N*N switches (N processors N memory modules). The switch configures itself dynamically to connect a processor to a memory module. No contention -- Supports N! permutations. Costly, hard to scale, wastes switches for most patterns. 16 ③ Crossbar switch Shanghai Jiao Tong University 2011 Easy for broadcasting and also for any permutation. As long as each processor wants to communicate with a different memory there will be no contention -- Supports N! permutations. If two or more processors need to access the same memory, however, one will be blocked until the switch reconfigures itself. A crossbar has a short diameter - information needs to pass through only one switching element on a path from one edge to another. Poor scalability -- If there are N processors and N memories, there are N2 interior switches. Adding another processor or memory means adding another N interior nodes. 17 ③ Multistage Network P1 P2 P2 Stage k Stage 3 PP-1 Stage 2 P3 Stage 1 Shanghai Jiao Tong University 2011 P1 P3 PP-1 Systems built with these topologies have processors on one edge of the network, memories or processors on another edge, and a series of switching elements at the interior nodes. 18 ③ Multistage Network : Omega Network Shanghai Jiao Tong University 2011 p processors and log2 p stages Each stage consists of a perfect shuffle 19 ③ Multistage Network : Butterfly Shanghai Jiao Tong University 2011 ButterFly Network 20 ③ Multistage Network Shanghai Jiao Tong University 2011 Built from small (e.g., 2X2 crossbar) switch nodes, with a regular interconnection pattern. In order to send information from one edge to another, the interior switches are configured to form a path that connects nodes on the edges. The information then goes from the sending node, through one or more switches, and out to the receiving node. The size and number of interior nodes contributes to the path length for each communication, and there is often a ``setup time'' involved when a message arrives at an interior node and the switch decides how to configure itself in order to pass the message through. 21 ③ Multistage Network: Problems Shanghai Jiao Tong University 2011 Generalized MINs: for in Inputs and jn outputs, use i×j switches at n stages. Bens networks: with 2logn-1 stages, it becomes a nonblocking network, allowing all permutations. Fault tolerance: form all switches at a stage as a ring. 22 ③ Line, Ring and Fully Connected Shanghai Jiao Tong University 2011 Linear Array: the simplest topology. Fully Connected: Direct connection between every pair of processors, Highest cost, Similar to crossbar in some properties. Ring: Each node in the ring is connected to only two other nodes.Chordal Ring: A compromise between Ring and Fully Connected Network. Fully connected: Degree = N-1, Diameter = 1, BW = (N/2)2, Links = N*(N-1)/2, Symmetric. Ring: Diameter = N/2, Degree=2, BW = 2, Symmetric 23 ③ Mesh and Torus Shanghai Jiao Tong University 2011 Mesh Topology 2D Torus 3D Torus 2D torus: Meshes with ``wraparound'' connections, e.g. the node at the top of the grid has an ``up'' link that connects to the node at the bottom of the grid (also left to right). Mesh: Deg = 2,3,4, Diameter = 2*Sqrt(N), Bisect= Sqrt(N), Easy to build, scalable . k-ary n-cube: Generalization of mesh-like Networks 24 ③ Mesh and Torus: Problems Shanghai Jiao Tong University 2011 Diameter of Asymmetric Networks: linear array, mesh, tree, shuffle exchange. Torus and Midmiew: how to use the wraparound links to obtain the optimal diameter? Layout of torus: Reduce the physical wire length. 25 ③ Hypercube Shanghai Jiao Tong University 2011 Can embed Hamiltonian cycle, mesh, tree, etc. Low latency, high bandwidth, but costly (high number of links). Hard to build (layout the chip and wires). Hard to scale up (As degree increases, number of I/O ports increases). Solution: CCC For dimension D, Degree = D, Diameter = D, Bisect = 2(d-1) Nodes = 2D, Links = D*2(D-1) 26 ③ Hypercube : assign node IDs Shanghai Jiao Tong University 2011 011 The nodes are numbered so that two nodes are adjacent if and only if the binary representations of their IDs differ by one bit. For example, nodes 011 and 010 are immediate neighbors. 27 ③ Hypercube : Properties Shanghai Jiao Tong University 2011 Hamiltonian Cycle 2n-node hypercube contains I×J mesh, where I×J = 2n Contains 2 binary tree with 2n-1-1 nodes Optimal in fault-tolerance Variants of hypercube:folded hypercube, hierarchical hypercube,incomplete hypercube,CCC, shuffle-change. Y. Saad and M.H. Schultz, Topological Properties of Hypercube, IEEE TC, July 1988. Generalized Hypercube, IEEE TC, April 1984. 28 ③ CCC: Solution to Hypercube : Shanghai Jiao Tong University 2011 110 100 111 101 010 000 011 001 (000,2) (000,1) (000,0) 1 2 F.P. Preparata, et al, The Cube-Connected Cycles: A Versatile Network for Parallel Computation, CACM, May 1981, G. Chen, et al, Tight Layouts of CCC, IEEE TPDS, Feb. 2000. G. Chen, et at, Layout of CCC without Long Wires, Computer Journal, in 2005. 29 ③ Tree and Star Shanghai Jiao Tong University 2011 Tree and Star Tree: Degree = 1 (leaves), 2 (root), 3 (interior nodes), Diameter = 2logN, Bisect = 1. Tree/Star bottleneck: Expect at root 30 ③ Tree Shanghai Jiao Tong University 2011 The tree has nodes of degree 1 (leaves), 2 (root), 3 (interior nodes). So it is not symmetric. Short diameter: For depth k, the number of nodes is 2k - 1, and the diameter is 2k (or ~ 2 log N, at the same order as Hypercube). For example, a processor with 262,144 nodes would have diameter 512 in a mesh but only 36 in a tree. Bisection bandwidth is 1. It suffers from a serious bottleneck -- Solution : Fat Tree 31 ③ Fat Tree Shanghai Jiao Tong University 2011 Expand link bandwidth at each higher level B B B/2 B/4 B B/8 Fat-Tree Fat-tree layout 32 ③ Fat Tree Shanghai Jiao Tong University 2011 Leiserson's fat-trees [Lei85] – built in CM-5. The Fat Tree network provides uniform bandwidth between any two end-points on a net. It does this by doubling the number of "up" paths as one goes "up" the tree from a processor in a leaf to the root node. Given a fat tree of this type with h levels of switches, Number of processor nodes = 2h Number of switch nodes = 2h- 1 Greatest distance between processor nodes = 2h 33 ③ Packet Routing in Fat Tree Shanghai Jiao Tong University 2011 8 port four paths 4 port PE1 PE2 PE3 PE4 PE5 PE6 PE7 PE8 PE9 PE10 PE11 PE12 PE13 PE14 PE15 PE16 Note that a message going from PE2 to PE5 may choose any one of four paths from the lower left router to the one at the root. This means that all four PE's attached to the lower left router have a path available to them to reach another node. 34 ③ Comparison -Shanghai Jiao Tong University 2011 Degree (d) (N node, n=log 2 (N)) Diameter (D) Bisection (B) No. of Links (L) 1D mesh 2 N-1 1 N-1 2D mesh 4 2(N1/2 - 1) N1/2 2N- N1/2 N-1 2 2 N-1 Ring 2 N/2 2 N 2D torus 4 2(N1/2 / 2) N1/2 2N Hypercube n n N/2 n N/2 Complete connected N-1 1 (N/2)2 N(N-1)/2 star 35 ③ Example MPP Networks Shanghai Jiao Tong University 2011 Name Number nCube/ten 1-1024 iPSC/2 16-128 MP-1216 32-512 Delta 540 CM-5 32-2048 CS-2 32-1024 Paragon 4-1024 T3D 16-1024 Topology Bits Clock 10-cube 1 10 MHz 7-cube 1 16 MHz 2D Mesh 1 25 MHz 2D Mesh 16 40 MHz fat tree 4 40 MHz fat tree 8 70 MHz 2D mesh 16 100 MHz 3D Torus 16 150 MHz Link Bisect. 1.2 640 2 345 3 1,300 40 640 20 10,240 50 50,000 200 6,400 300 19,200 Year 1987 1988 1989 1991 1991 1992 1992 1993 MBytes/second No standard MPP topology! 36 ③ Scalable Topology Shanghai Jiao Tong University 2011 Scalability Refers to the increase in the complexity of communication (time) as more nodes are added. In a highly scalable topology more nodes can be added without severely increasing the amount of logic required to implement the topology and without increasing the diameter. Example : Doubling the number of nodes in a hypercube increases the degree by only 1 link per node, and likewise increases the diameter by only 1 path. An opposite example is linear array. 37 ③ Hypercube Problems Shanghai Jiao Tong University 2011 Embedding the links: For a 64K node (d=16) hypercube machine, there would be 512K (16x64K/2) links. With current technology, it is difficult to scale a hypercube beyond about 4K nodes, with about 24K links. Operation cost: As the number of dimensions increases, the nodes must do more work to keep up with the incident message traffic. In fact, because most processors can handle only one I/O transaction at a time, many hypercube algorithms operate on the principle of serializing processing by dimension. i.e. all pairs of nodes in one dimension communicate, then the next dimension, etc. See Ascend/Descend Algorithms 38 ③ Symmetric Topology Shanghai Jiao Tong University 2011 Symmetric Rings, fully connected networks, and hypercubes are all node symmetric. Trees and stars are not. A tree has three different types of nodes, namely a root node, interior nodes, and leaf nodes, each with a different degree. A star has a distinguished node in the center which is connected to every other node. When a topology is node asymmetric a distinguished node can become a communications bottleneck. Importance of Symmetry: Node symmetry renders identical software at every node;edge symmetry avoids hot traffic spot in the network. 39 Routing algorithms—XY Routing Shanghai Jiao Tong University 2011 P Sx,Sy source processor P Dx,Dy destination processor Shortest path – “manhattan distance”, abs(Sx-Dx) + abs(Sy-Dy) Algorithm: 1.Reduce distance along X dimension until 0 2.Reduce distance along Y dimension until at P Dy,Dx P 0,0 :source P 3,2 :destination P 0,0 P 3,2 40 ③ Routing algorithms—XY Routing Shanghai Jiao Tong University 2011 M[S,D] In a mesh, S More Information I M[S,D] D Begin S(0,0) receive(M); If D=I accept(M) Elseif D • x >I • x sendright(M) Elseif D • x <I • x sendleft(M) Elseif D • y >I • y senddown(M) Elseif D • y >I • y sendup(M) Endif; D(3,2) end 41 E-Cube Routing Shanghai Jiao Tong University 2011 E-Cube Routing Ps source processor Pd destination processor Shortest path -- hamming distance between Ps and Pd Routing taken in the following manner: 1. Exclusive-or the source and destination processor numbers 2. Going from the Least Significant Bit (lsb) to Most Significant Bit (MSB). Each position a “1” exists in the result of the exclusive, an edge is taken. P000 source P110 destination 110 42 ④ My Work and contributions Novel Interconnection Networks: Shuffle-Ring, Shuffle-Cube, Wall Mesh. Layout Design: MIDIMEW, CCC. Routing in special Networks: Shuffle-Exchange, Unidirectional Networks 43 ④ New Topology: Shuffle-Ring Shanghai Jiao Tong University 2011 Definition: anan-1…a1a0 is connected to shuffle an-1an-2…a0an ring anan-1…a1 ( a0 ±1) Properties: Simplicity Constant degree Keep all hypercube properties Compact layout •G. Chen et al, Shuffle-Ring: Overcoming the Increasing Degree of Hypercube, Proc. 2nd Int. Symp. on High-Performance Computer Architecture(HPCA-2), California, Feb. 1996, 130-138. •G. Chen el al, Shuffle-Ring: A New Constant-degree Network, International Journal of Foundations of Computer Science, March 1998, 77-98 44 ④ New Topology: Shuffle-Cube Shanghai Jiao Tong University 2011 Definition: anan-1…a1a0 is connected to shuffle an-1an-2…a0an Cube anan-1…ai … a0 (i<k) 0 1 8 9 2 3 a b 4 5 c d 6 7 e f Properties: Constant degree Keep all hypercube properties Compact layout •G. Chen, et al, CTSN: A New Fault-Tolerant Network, Proceedings of 13th International Conference on Parallel and Distributed Computing Systems(PDCS'2000), Las Vegas, Nevada, August 2000, 517-522. 45 ④ New Topology: Wall Mesh Shanghai Jiao Tong University 2011 Definition: ( x, y ) is connected to ( x ±1, y ) horizontally and ( x, y ±1 ) vertically. Properties: Constant degree=3 or 4, Equivalent to mesh in Computing Power Logarithmic diameter Easy layout •G. Chen,et al, The Wall Mesh, Computer Architecture'97: Selected Papers of the 2nd Australasian Conference, Springer, 1997, 217-230. •陈贵海等, 墙式网孔, 计算机学报, 2000 年4月, 374-381页。 46 ④ New Topology: Isomorphism Shanghai Jiao Tong University 2011 •G. Chen et al, Comments on ``A New Family of Cayley Graph Interconnection Networks of Constant Degree Four'', IEEE Transactions on Parallel and Distributed Systems, December 1997, 1299-1300. 47 ④ Layout Design: MIDIMEW Shanghai Jiao Tong University 2011 •G. Chen, et al, Laying Out Midimew Networks with Constant Dilation, Lecture Notes in Computer Science(854), September 1994, 773-784. •G. Chen et al, Optimal Layouts of Midimew Networks, IEEE Transactions on Parallel and Distributed Systems, September 1996, 954-961. 48 ④ Layout Design: Cube-Connected Cycles Shanghai Jiao Tong University 2011 110 100 111 101 010 000 011 001 (000,2) (000,1) (000,0) G. Chen, et al, A Compact Layout of Cube-Connected Cycles, Proc. of 4th Int. Conf. on High Performance Computing, India, Dec. 1997, 422-427. G. Chen, et al, Tight Layouts of CCC, IEEE Transactions on Parallel and Distributed Systems, Feb. 2000, 182-191. G. Chen, et at, Layout of CCC without Long Wires, Computer Journal, Nov. 2001, Vol. 44, No. 5. 49 ④ Routing: Shuffle-Exchange Networks Shanghai Jiao Tong University 2011 Definition: anan-1…a1a0 is connected to shuffle an-1an-2…a0an Exchange an-1an-2…a1a0 •G. Chen, et al, Shortest-Path Routing in Shuffle-Exchange Networks, Lectures in Operations Research, August 1998, 142-153. •陈贵海等,洗牌交换网中的最短路由算法, 计算机学报,2001年1月。 •G. Chen, et al, An Algorithm for Optimal Routing in Shuffle-Exchange Networks, to appear in IEEE Transactions on Computers. 50 ④ Routing: Unidirectional Networks Shanghai Jiao Tong University 2011 G. Chen, at al, A Distance-Vector Routing Protocol for Networks with Unidirectional Links, Computer Communications, Feb. 2000, 418-424. •G. Chen, et al, An Improved Routing Protocol for Networks with Unidirectional Links, ICPP, Spain, Sept. 2001. 51 Questions? Shanghai Jiao Tong University 2011 Shanghai Jiao Tong University 2011 Assignment 1 1) Prove any graph/network has a longest path? 2) Choose one of the following 2.1) Write the routing algorithm for complete binary tree? (Suppose nodes are numbered from top to down, from left to right, and beginning from root.) 2.2) Prove that hypercube has optimal fault tolerance. Send your solutions to TA via Email before Nov.17, 2012. 53 Shanghai Jiao Tong University 2011 Reading materials 1) Y. Saad and M. H. Schultz, Topological Properties of Hypbercube, IEEE Transactions on Computers,Vol. 24, No. 5 2) L.N. Bhuyan and D.P. Agrawal, Generalized Hypercube and Hyperbus Structures for a Computer Network, IEEE Transactions on Computers, Vol. 33, No. 5, 1984 3) C. E. Lerserson, Fat-tree: Universal network for Hardware-Efficient Supercomputing, IEEE Transactions on Computers, pp. 892-901, 1994 54