Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Storage Area Network Baoquan Zhang Outline • What is a SAN? • Why SAN? • What is a SAN composed of? • SAN, NAS or DAS? Outline • What is a SAN? • Why SAN? • What is a SAN composed of? • SAN, NAS or DAS? SAN • Any high-performance network whose primary purpose is to enable storage devices to communicate with computer systems and with each other. * • A high-speed network, an extension to the storage bus, allows the establishment of direct connections between storage devices and processors (servers). ** • A network that provides access to consolidated, block level data storage. *** *www.snia.org **Khattar, Ravi Kumar, et al. Introduction to Storage Area Network, SAN. IBM Corporation, International Technical Support Organization, 1999. ***https://en.wikipedia.org/wiki/Storage_area_network Outline • What is a SAN? • Why SAN? • What is a SAN composed of? • SAN, NAS or DAS? Why SAN? • Industry Recognition : Three tiers architecture Presentation Desktop(PC, NC) Processing Application Servers Data Storage Storage Devices Why SAN? Client/Server Computing Limited distance of data transmitting SCSI: 1.5m~25m Poor scalability Adding Disk for each server Hard to share data information island • Extra resource of copying and transmitting data • Work with out-of-date data Clients Client Access LAN A Application Storage Devices Servers B … DAS Information Island Why SAN? Storage Area Network Clients Universal Storage Connectivity Good scalability Scale performance and capacity Relatively long distance data transmitting IP : Internet-based Long-distance FC: 15m~10km IB : 15m~10km No-copy data sharing Shared storage pool Client Access LAN Application Servers … Storage Area Network Storage Storage Model 1: Direct Access Storage Server • All storage stranded behind server • Proprietary access (vendor specific) • Storage sharing creates CPU overhead • Network burdened with disk I/O traffic • Limited scalability and low performance Storage Model 2: Fibre Channel SAN • Replaces parallel SCSI transport • SAN is DAS from servers’ perspective SAN Intrane t • Optimized for movement of data from server to disk or tape • Facilitates storage clustering & LAN-free backup • Typically does not use LAN protocols, relies on serial SCSI (SCSI-3) Server SAN SAN Server Interne t Server Storage Model 2: FC SAN Limitations • Creates a 3rd network (LAN, WAN, SAN) • Pre-Gigabit Ethernet bandwidth assumptions SAN Intrane t • Management nightmare Server • Limited interoperability • Minimal storage security SAN SAN Server • Creates “SAN Islands” Interne t Server Storage Model 3: IP-SAN • Best features of Fibre Channel & IP networks Data Video SUN NT 4.0 WIN 2000 LINUX • Ease of configuration and management • Servers used optimally • Support IP Quality of Service, Error detection & Prioritization Storage IP • Multiple server operating systems supported • Maintain IT infrastructure, security & interoperability Voice IP Network SN 5420 Fibre Channel Active Disk with OSD Capability As An Example of Intelligent Storage Devices IP Network Attached More Processing Power and Memory Storage Area Network • Server Architecture Based on SAN & NAS • Network Protocol (FC-AL and SSA) • Spatial Reuse • Multiple Links and Switch Based Multiple FC-AL Internet Connection SAN Host FC-AL FC-AL Internet Connection SAN Host Host Host SAN FC-AL FC-AL FC-AL Internet Connection Internet Connection Previous Research on SAN • Efficient Protocol Design for FC-AL and SSA • Emphasis on performance for future disks • Built detailed simulation models for both FC-AL and SSA • Supported by Seagate and IBM Storage Systems Division • Scalable Streaming Video Servers based on SAN • Co-funded a streaming video server company- Steaming21 • Many publications on streaming video servers and streaming video delivery over Internet Serial Storage Interfaces • FC-AL • SSA • FC-TORN • FC-AL3 • InfiniBand Serial Storage Interfaces • Fibre Channel • FC-AL • FC Switch • Serial Storage Architecture (SSA) • • • • • Buffer Insertion Ring Link-by-link flow control Fairness Algorithm Independent links: Spatial Reuse Fault tolerance against link failure FC-AL Features • Bandwidth: 100 MB/s • Connectivity: 126 devices • Connection Distance: 30m device to device (with copper) and 10km (with Fiber Optics) • Fault-Tolerance: CRC protected frames, dual port, hot plug connector • Distributed switch logic FC-AL Fairness Algorithm • • • • Based on an Access Window with a history variable ACCESS Default value of ACCESS is true When an L_Port wins the arbitration, set ACCESS to false Before opening a circuit, winner send out ARB(F0) to detect if other L_Ports are also arbitrating • If receive ARBx, other L_Ports are arbitrating • When relinquish the loop, the winner sends out: • ARB(F0) if other L_Ports are arbitrating, or • IDLE to trigger all L_Ports re-set ACCESS to true SSA Features • 2-in and 2-out links per node (with 20 MB/s per link) • Fairness Access • Fault-Tolerance: A multiple host configuration offers fault tolerance again host, link and adapter failures. • Number of attachments: 126 for SSA • Compact connectors: serial vs. parallel for SCSI • Transmission distance: 25 m (2.5km) between devices with copper cables (fiber optic) Spatial Reuse • What is spatial reuse? • Concurrent non-overlap transfers can utilize full link bandwidth • Why is it important? • Throughput can scale up with more links and non-overlap transfers • Achieved throughput could be as low as link bandwidth • Device/data sharing may reduce spatial reuse potential SSA SAT Fairness Algorithm • Based on token passing and quota • Forwarding frames have higher priority than originating frames • Holding a token allows a node to switch the priority between the originating and forwarding traffic. • Hold quota (a_quota): number of frames that can be originated when holding the SAT token. • Idle quota (b_quota): number of frames that can be originated since a node passed the SAT token last time and the channel is idle. In general, b_quota =4*a_quota. Fairness vs. Channel Utilization • How to define fairness? • How to improve channel utilization? • Starvation possible? • Fairness+throughput FC-TORN • B_RDY (credit) is used to control the number of frames potentially can be sent to a destination (disk). • SAT token based on one quota for each source (host or disk) to control the maximum number of frames sent by a source. • B_RDY and B_RDY’ are used to produce fairness from sources to a destination. Storage Area Network • Server Architecture Based on SAN & NAS • Network Protocol (FC-AL and SSA) • Spatial Reuse • Multiple Links and Switch Based Multiple FC-AL Internet Connection SAN Host FC-AL FC-AL Internet Connection SAN Host Host Host SAN FC-AL FC-AL FC-AL Internet Connection Internet Connection Outline • What is a SAN? • Why SAN? • What is a SAN composed of? • SAN, NAS or DAS? SAN Components Interconnects (The heart of a SAN) • Cable Server Host Connect the components with each other Adapter • Adapters Connect to devices and control the protocol Interconnects • Switches (Fabric) Interconnect devices, increase bandwidth, reduce congestion and provide aggregate throughput provide simple NameServer services. or Hub (Arbitrated Loop) Share bandwidth Storage Array Tapes Hard Disks SAN Components Fibre Channel SAN or FC SAN IP Network SAN or IP SAN InfiniBand SAN or IB SAN Note: It doesn't say that a SAN uses Fibre Channel or Ethernet or any other specific interconnect technology. A growing number of network technologies have architectural and physical properties that make them suitable for use in SANs. - See more at: http://www.snia.org/education/storage_networking_primer/san/what_san#sthash.9cPWdUBs.dpuf SAN Components Fibre Channel SAN or FC SAN IP Network SAN or IP SAN InfiniBand SAN or IB SAN Note: It doesn't say that a SAN uses Fibre Channel or Ethernet or any other specific interconnect technology. A growing number of network technologies have architectural and physical properties that make them suitable for use in SANs. - See more at: http://www.snia.org/education/storage_networking_primer/san/what_san#sthash.9cPWdUBs.dpuf SAN Components Fibre Channel Fibre Channel started in 1988, with ANSI standard approval in 1994, to simplify HPPI (High Performance Parallel Interface) system. FC is a high-speed network technology (commonly running at 2-, 4-, 8- and 16-gigabit per second rates) primarily used to connect computer data storage. (32-Gigabit, 128-Gigabit speeds in 2016) FC is the best design combining the I/O Channel with Networking. Networking pays most attention on handling the changes of configuration and loads as well as addressing data to proper destination. I/O Channel focuses on the performance, which means to move data with least latency by utilizing a rigorous and simple protocol. FC maintains the speed and low overhead of a channel while adding the flexibility (through connectivity) and the longer distances that are characteristic of a networking. SAN Components The competition of High-end Storage Technology FC SSA Throughput 531.25 Mb/s 640Mb/s Device amount Unlimited Up to 192 hot swappable hard disk per system Up to 32 separate RAID arrays per adaptor Distance 10km 10km( with 25 meters apart among arrays) Up layer Protocol ATM, IP, FICON, SCSI SCSI-3 Eventually the market chose FC over SSA (Serial Storage Architecture). SAN Components Fibre Channel Topologies: FC-P2P:Point to point The easiest configuration The easiest to administer High-speed interconnect between two nodes Possible Usage • Between Central Processing Units • From a workstation to a specialized graphics processor or simulation accelerator • From a file server to a disk array …… SAN Components Fibre Channel Topologies: FC-AL: Arbitrated Loop 1. First arbitrate to win control of the loop. 2. Establish a point-to-point (virtual) connection 3. two nodes consume all of the loop’s bandwidth until the data transfer operation is complete Advantages • Lower-cost alternative • Support of up to 126 devices is possible on a single loop. • …… However, by 2007, FC-AL had become rare in server-to-storage communication SAN Components Fibre Channel Topologies: FC-SW: Switched Fabric Increased bandwidth Increased number of devices scalable performance maximum of 16 million devices FC-SW topology is what we deploy in a SAN. High cost : Switch is the most costly hardware device. SAN Components Fibre Channel SAN Server Host FC Host Bus Adapter FC Host Bus Adapter A unique World Wide Name (WWN) Fibre or copper cable Cable Copper Fibre 15m 100 MB/s 10km 2000MB/s Fibre Channel Switches Fibre Channel Switches Fibre cable Directors No single point of failure (high availability) Switches smaller, fixed-configuration, less redundant devices SAN Components Fibre Channel Protocol Layers FC-4 FC-3 FC-2 FC-1 FC-0 SAN Components Fibre Channel Layers FC-0 Physical layer : describes the physical interface • • • an analog interface to transmitter a digital interface to the FC-1 layer the requirements for infrastructures Transport media Receiver hardware …… Example of options of FC-0 Plants SAN Components Fibre Channel Layers FC-1 Encode/Decode Layer: describes the means of encoding/decoding user data 8/10 bit encode/decode scheme 8b/10b encoding was proposed by Albert X. Widmer and Peter A. Franaszek of IBM Corporation in 1983. Minimize errors by equalizing the number of 1’s and 0’s transmitted and not allowing more than 4 consecutive bits of the same type in a row. Allows for distinguishing “Special Characters (K28.5)” and also provides for simplifying byte and word alignment. the evening out of 1’s and 0’s allows for the design of relatively inexpensive transmitter/receiver circuitry. SAN Components FC-1 Encode/Decode Layer: Encode Process FC-2 byte notation: 0xBC (Hexadecimal) FC-2 bit notation: 7 6 5 4 3 2 1 0 1 0 1 1 1 1 0 0 FC-1 un-encoded: H G F E D C B A 1 0 1 1 1 1 0 0 Z XX Z E D C FC-1 reordered for : K 1 1 1 (+Previous Running Disparity) 5B/6B (Negative) A B C D E i FC-1 encoded : 0 0 1 1 1 1 .y B A F G H 0 0 1 0 1 3B/4B F G H j 1 0 1 0 Variable K Z K K28.5 SAN Components Fibre Channel Layers FC-2: Framing Protocol/Flow Control data using frames flow control classes of service SAN Components Frames are the basic package used to encapsulate and transport the data. Two types of Frames Data Frame Link Control Frame A group of related Frames transmitted in one direction constitute a sequence. Exchanges are groups of related Sequences. Expiration Security Header Network Header User Data(Not used in Link Control Frame ) Association Header Designates the end of the Frame content Device Header and validity of the Frame’s content SoF: the “comma” and 3 bytes indicating the type of connection service Verify the data integrity of the FH and Payload SAN Components FC-2 controls the flow of Frames between ports so that receiver buffers are not overrun. Buffer is maintained by the Sequence Initiator (transmitter) and is used to throttle the transmission of Frames. There are two basic types of flow control. End to End Control in N_port to N_port communications The receiver responds to all valid Frames it receives with an ACK Frame. Buffer to Buffer Control in N_port talking to a Fabric or an N_port to N_port connection in a Point to Point topology Each side is responsible for maintaining its own BB_Credit_Count. SAN Components FC-2 provides up to 5 Classes of Service (CoS). The different CoS represent different levels of delivery guarantee, bandwidth and connectivity. Class 1 dedicated connection remain active until being closed. R_RDY on Connect Request only sustained, high throughput transactions SAN Components Class 2 control on a Frame by Frame Basis allows interleaving of Sequences over the single connection from multiple N_ports the ACK for every Frame. Also R_RDY. SAN Components Class 3 provides a connectionless service with no acknowledgment lack of ACK. Only R_RDY for link maintenance SAN Components Fibre Channel Layers FC-3: Common Services The FC-3 level is not currently fully defined. The term “common services” means a service that would utilize multiple N_ports working together on a single node. SAN Components Fibre Channel Layers FC-4: Upper Level Protocol Support The FC-4 level supports the mapping of Upper Level Protocols (ULP) onto Fibre Channel data structures. SCSI (Small Computer Systems Interface) IPI-3 (Intelligent Peripheral Interface-3) HiPPI (High Performance Parallel Interface) IP (Internet Protocol) - IEEE 802.2 (TCP/IP) data ATM/AAL5 (ATM adaptation layer for computer data) SBCCS (Single Byte Command Code Set) The way that FC serves as a transport for ULPs is by mapping the ULP messages(known as Information Units) into FC Sequences and/or Exchanges. SAN Components IP over FC Two kinds of Information Units IP datagram Moving between nodes on networks using the IP protocol stack ARP datagram ARP datagram is used during network configuration to map IP addresses to Media Access Control addresses (used for routing). A dedicated ARP server must be set up at a “well known” address SAN Components IP over FC IP Packets Network Header Split Frame Header Network Header The First Frame Optional Header Payload Frame Header Payload Additional Frame … Frame SAN Components SCSI over Fibre Channel (Predominate in FC SAN) Generally, FCP stands for Fibre Channel Protocol for SCSI.* The transport is accomplished by wrapping SCSI command, response, status and data blocks. SCSI Command *Norman, David. "Fibre Channel Technology for Storage Area Networks." SAN Components SCSI over Fibre Channel Read Example Receive Handle Initiator FCP_Port *Norman, David. "Fibre Channel Technology for Storage Area Networks." Target FCP_Port SAN Components Fibre Channel SAN or FC SAN IP Network SAN or IP SAN InfiniBand SAN or IB SAN Note: It doesn't say that a SAN uses Fibre Channel or Ethernet or any other specific interconnect technology. A growing number of network technologies have architectural and physical properties that make them suitable for use in SANs. - See more at: http://www.snia.org/education/storage_networking_primer/san/what_san#sthash.9cPWdUBs.dpuf SAN Components IP SAN An IP SAN is a Storage Area Network that uses the iSCSI protocol to transfer block-level data over a network, generally Ethernet. iSCSI HBAs iSCSI Node Names Cable Ethernet Network Switches Server Host iSCSI Host Bus Adapter Ethernet Network Switches SAN Components Fibre Channel SAN or FC SAN IP Network SAN or IP SAN InfiniBand SAN or IB SAN Note: It doesn't say that a SAN uses Fibre Channel or Ethernet or any other specific interconnect technology. A growing number of network technologies have architectural and physical properties that make them suitable for use in SANs. - See more at: http://www.snia.org/education/storage_networking_primer/san/what_san#sthash.9cPWdUBs.dpuf IB SAN InfiniBand Network Architecture InfiniBand is a network communications protocol that offers a switch-based fabric of point-to-point bi-directional serial links between processor nodes, as well as between processor nodes and input/output nodes, such as disks or storage. Higher throughput – 56Gb/s per server and storage connection, and soon 100Gb/s, compared to up-to 40Gb Ethernet and Fibre Channel Lower latency – RDMA zero-copy networking reduces OS overhead so data can move through the network quickly Enhanced scalability – InfiniBand can accommodate theoretically unlimited-sized flat networks based on the same switch components simply by adding additional switches Higher CPU efficiency – Data movement offloads the CPU InfiniBand Architecture EndNodes: Servers and Devices Link: copper and optical fibre* Switches: IBA Switches A private, protected channel directly between the nodes was established by switches. Adapters: Host Channel Adapter Data and message movement without CPU involvement with RDMA and Send/Receive offloads is performed by adapters. The adapters are connected on one end to the CPU over a PCI Express interface and to the InfiniBand subnet through InfiniBand network ports. Subnet Manager: Routing define and Subnet discovery 1X fibre link has two optical fibres, one for each direction InfiniBand Architecture IB Storage Stack InfiniBand Architecture IB Communication Stack A Consumer is a process with virtual address space. A Consumer can have more than one QP. A QP(Queue Pair) is a Virtual Interface. A OP includes a Send Q and Receive Q. QPs are the endpoints of Channel. A Channel Adapter has up to 2^24 QPs. QPs are independent with each other. IB Message Transfer Semantics Send/Receive Simply send and receive. RDMA Read/Write Directly Read and write to Virtual Memory InfiniBand Architecture IB Message Transfer Semantics: Send/Receive Step: 1. 2. 3. 4. Initiator put the message in the SND. The Message is sent to Target. Target receive the Message. Target put the Message in the RCV. InfiniBand Architecture IB Message Transfer Semantics: RDMA Step: 1. Application on initiator registers a buffer and puts the send request in SND. 2. Target receives the request and reads the data from initiator buffer directly. 3. Target returns a status to Initiator. InfiniBand Architecture Message InfiniBand Architecture is said to be message-oriented. A message can be any size ranging up to 2^31 bytes in size. The InfiniBand hardware automatically segments the outbound message into a number of packets. Complete IBA Packet Format 8 Bytes 40 Bytes 12 Bytes 28 bytes 4 Bytes Local Routing Header Global Routing Header Extended Transport Header Immediate Data Intra-subnet Inter-subnet Base Transport Header tells endnodes what to do with packets 0-4096 Bytes 4 Bytes Message Payload Invariant CRC 2 Bytes Variant CRC InfiniBand Architecture IB Verbs InfiniBand architecture does not define APIs, only provides the basis for specifying the APIs. A verb is a method by which an application requests an action from InfiniBand’s message transport service. Other organizations, such as the OpenFabrics Alliance, provide a complete set of APIs and software that implements the verbs to work seamlessly with the InfiniBand hardware. InfiniBand Architecture IB Up Layer Protocol The upper level protocols IPoIB : IP over IB SRP : SCSI RDMA Protocol SDP : Sockets Direct Protocol iSER : iSCSI Extensions for RDMA Linux InfiniBand software architecture InfiniBand Architecture SRP Protocol SCSI RDMA Protocol (SRP) was defined by the ANSI T10 committee to provide block storage capabilities for the InfiniBand architecture. SRP is a protocol that tunnels SCSI request packets over InfiniBand hardware Linux InfiniBand SRP Protocol architecture SAN Components (Summary) Bandwidth Latency FC SAN 100Mb(Copper) 20Gb(Fibre) 32Gband 128Gb(Coming) Dedicated to block I/O IP SAN IB SAN 100Mb or 1Gb(Ethernet) 10Gb(10GB Ethernet) 120Gb(12X) Direct connection Dedicated to block I/O Distance 15m(Copper) 20km(Fibre) Internet-based Longdistance 125m(12X) 10km(1X) Cost High Cheap Medium Outline • What is a SAN? • Why SAN? • What is a SAN composed of? • SAN, NAS or DAS? SAN, NAS or DAS? SAN More Efficient Block-Level data access NAS Convenient data sharing in homogenous File System DAS Easy implement and low cost Acknowledgement Professor David Du gives me numerous basic knowledge on Storage System and provides this interesting topic. During the preparation for the presentation, Dr. Fenggang Wu helps me review the slices and gives me significant references. Reference • www.snia.org • Khattar, Ravi Kumar, et al. Introduction to Storage Area Network, SAN. IBM Corporation, International Technical Support Organization, 1999. • https://en.wikipedia.org/wiki/Storage_area_network • https://en.wikipedia.org/wiki/Fibre_Channel#Fibre_Channel_topologies • http://www.networkworld.com/article/2174282/lan-wan/fibre-channel-will-come-with32-gigabit--128-gigabit-speeds-in-2016.html • https://www.pctechguide.com/interfaces/hard-disks-what-is-serial-storage-architecture • https://en.wikipedia.org/wiki/Fibre_Channel_point-to-point • Shanley, Tom, and Joe Winkles. InfiniBand Network Architecture. Addison-Wesley Professional, 2003. • IP SAN Fundamentals: An Introduction to IP SANs and iSCSI • Norman, David. "Fibre Channel Technology for Storage Area Networks.“ • Grun, Paul. "Introduction to infiniband for end users." White paper, InfiniBand Trade Association (2010). Thank you