Download Best Practice and Deployment of the Network for

Best Practice and Deployment of the Network
for iSCSI, NAS and DAS in the Data Center
Samir Sharma, Juniper Networks
Author: Samir Sharma, Juniper Networks
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA unless otherwise
noted.
Member companies and individual members may use this material in presentations
and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without
modification
The SNIA must be acknowledged as the source of any material used in the
body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee.
Neither the author nor the presenter is an attorney and nothing in this
presentation is intended to be, or should be construed as legal advice or an opinion
of counsel. If you need legal advice or a legal opinion please contact your attorney.
The information presented herein represents the author's personal opinion and
current understanding of the relevant issues involved. The author, the presenter,
and the SNIA do not assume any responsibility or liability for damages arising out of
any reliance on or use of this information.
NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
2
Abstract
Best Practice and Deployment of the Network for
iSCSI, NAS and DAS in the Data Center
Low Latency, high bandwidth, lossless, spanning tree, IO
Convergence, Layer 2, Layer 3, Storage Resource
Management ,Two Tier, Three Tier, and Flat… when
deploying iSCSI, NAS and DAS what should the network
topology be? How can network topology affect Large,
Medium and Small Scale network deployments? In this
presentation we will discuss best practice and
deployments of the network for iSCSI , NAS and DAS in
the Data Center.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
3
Agenda
1
Scope of this presentation?
2
Definitions – The Storage Puzzle
3
Benefits and where are they being deployed?
4
Lossless or Lossy … That is the Question…
5
Ideal and Non Ideal Ethernet Storage Network Deployments
6
Mapping the Deployment to what is Ideal?
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
4
Agenda
1
Scope of this presentation?
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
5
Scope of this presentation
?
Background
The pieces of the puzzle that make up Ethernet network
storage deployments.
Topologies
Ethernet Network topologies that are conducive
to storage deployments.
Summary and final thoughts.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
6
Agenda
2
Definitions – The Storage Puzzle
3
6
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
7
Definitions – The Storage Puzzle
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
8
File Level Storage
File Level storage : is the most common storage system that we find with
our hard-drives, NAS systems, etc. In this type of storage, the storage disk
is configured with a particular protocol (Like NFS, etc) and files are stored
and accessed from it as such, in bulk.
Advantages of File Level Storage System:
•
•
•
•
•
•
•
File level storage system is simple to implement and simple to use.
It stores files and folders and is visible as such, to both the systems storing the files and
the systems accessing it.
File level storage systems are generally inexpensive, when compared to block level
storage systems.
File level storage systems are more popular with NAS based storage systems –
Network Attached Storage.
They can be configured with common file level protocols like NTFS (Windows), NFS
(Linux), etc.
File level storage systems are well suited for bulk file storage.
The file level storage device itself can generally handle operations like access control,
integration with corporate directories, etc.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
9
Block Level Storage
Block level storage : is raw blocks (storage volumes) are created and each
block can be controlled like an individual hard drive. Generally, these
blocks are controlled by the Server based Operating Systems. Each block/
storage volume can be individually formatted with the required file system.
Advantages of Block level storage systems:
•
•
•
•
•
•
•
Block level storage systems offer a better performance/ speed than file level storage
systems.
Each block / storage volume can be treated as an independent disk drive and are
controlled by external Server OS.
Each block / storage volume can be formatted with the file system required by the
application (NFS / NTFS / SMB , etc).
Block level storage systems are very popular with SAN – Storage Area Networks.
Block level storage systems are more reliable, and their transport systems are very
efficient.
Block level storage can be used to store files and also provide the storage required for
special applications like Databases, VMFS (Virtual Machine File Systems), etc.
They can support external boot-up of the systems connected to them.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
10
Direct-attached Storage (DAS)
Direct-attached storage (DAS) refers to a digital storage
system directly attached to a server or workstation, without a storage
network in between.
Protocols used in DAS: ATA, SATA, eSATA, SCSI, SAS, and Fibre
Channel.
Pros: Cheep, Compact great for very small business.
Con: It is hard to share data or unused resources
with other servers.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
11
Network-attached storage (NAS)
Network-attached storage (NAS) is file-level computer data
storage connected to a computer network providing data access
to heterogeneous clients. It provides file-based storage.
Protocols used in NAS: uses file-based protocols NFS, SMB/CIFS, or
AFP. NAS units rarely limit clients to a single protocol.
Pros: Often a striped down OS and hardware is needed.
Cons: Typically not conducive for large scale deployments.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
12
Internet Small Computer System Interface (iSCSI)
Internet Small Computer System Interface (iSCSI)- is a transport for iSCSI
using TCP/IP (typically native 1GbE, 10GbE, 40GbE and 100GbE)
Pros:
•
•
•
Cost effect components, hard drives, servers, network cards, cables, switches, optics..
Leverages TCP/IP to guarantee lossless traffic and in order frame delivery.
Built in security, authentication with RADIUS servers and can leverage IPSec, MACSec.
Cons:
•
Rely on overlay protocols to get lossless and in order frame delivery. Causing
complexity and overhead in the network.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
13
Fibre Channel over Ethernet (FCoE)
Fibre Channel over Ethernet (FCoE) is an encapsulation of Fiber
Channel frames over Ethernet networks.
Why is this important?
FCoE will carry FC frames (eg SCSI commands) directly over Ethernet.
This is one of the key drivers that enables SAN LAN convergence.
Ethernet HD
FC Frame
Ethernet
FCS
FCoE Frame = Ethernet Frame that has an FC frame inside it.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
14
Agenda
3
Benefits and where are they being deployed?
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
15
Benefits of Ethernet Storage
COST:
In-order/ Lossless:
Transport ultimately more
economical than other
storage transports.
TCP/IP = In-order,
guaranteed delivery
DCB = Lossless, traffic
seperation
SPEED:
EQUIPMENT
Higher bandwidth
• 10GbE
• 40GbE
• 100GbE
More cost effective
• Storage equipment
• Servers
• Networking equipment
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
16
Where is it being deployed?
FCoE Converged Access
Large
End to end FCoE &
iSCSI
Medium Business
iSCSI
Small Business
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
17
Agenda
4
Lossless or Lossy … That is the Question…
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
18
Lossless or Lossy.. That is the Question….
Very Important Rule:
Thy shall never loose or drop a storage frame.
Lossless Network Topology:
No Frame drop between initiator and target.
Lossy Network Topology:
The network allows for frame drop within the
network and rely on overlay protocols to handle
frame drop (eg TCP/IP).
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
19
Realities of Traditional Ethernet in the Data Center
Possible Problem Areas
Frame loss
Network
Topologies
Network
Congestion
• Traditionally not meant to transport lossless traffic
• Forced to rely on protocols like TCP/IP to achieve a lossless topology
-> Resulting in protocol overhead/congestion due to retransmission.
(iSCSI used TCP/IP)
• Multi-tier
• Multiple device to manage
• Spanning Tree adds inefficiencies 50% of links blocked
• Heavily Oversubscribed 10:1, 5:1, 4:1 take your pick……
• Mix Traffic Types
• Video Running on Ethernet
• Voice Running on Ethernet
• Traditional Applications Running on Ethernet
• Now add Storage and Server Virtualization to the mix……
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
Storage over Ethernet
What do you need?
Relying TCP/IP for the Solution….
Lossless
Network
Topology
In order frame
delivery
• Need: No frame drops
between initiator and target
• Need: Frames must be in
order between initiator to
target
• Solution : If a frame is dropped
TCP/IP will retransmit
• Result: This adds latency and can
exacerbate congestion
• Solution : TCP/IP guarantees inorder delivery per session
Fibre Channel’s Solution….
Lossless Network
Topology/ In order
frame delivery
• Need: No frame drops
between initiator and target
• Uses a buffer credit mechanism to
create a lossless network.
• Eliminates frame drop and frame
retransmission
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
Solution : Map storage flows to PFC
Creates a lossless network PFC and
DCB
(dcbx allows the creation of arbitrary
application tlv’s based on higher level
protocol to easily capture and
configure for cifs,nfs etc …)
Lossless
Ethernet
Storage
Fabric
No need for retransmission
(Closest way to emulate FC Credit
Mechanism)
Mapping Different Flows on Separate Priority…
Result True Traffic Separation
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
Agenda
5
Ideal and Non Ideal Ethernet Storage Network Deployments
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
23
Realities of Ethernet in the Data Center
The challenges
Multi-tier legacy network
 Too slow
 Too expensive
 Too complex
N
Up to 50% of the
ports interconnect
switches,
not servers or
storage
Network Complexity
TCP/IP can’t scale
W
Up to 75% of traffic
E
S
Complexity
Unnecessary
layers add hops
and latency
Scale
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
Spanning Tree
disables up to 50%
of bandwidth
Non-Ideal Network Topology
Ethernet L3
Core Switch/Router
Ethernet L2/L3
Aggregation Switch
Ethernet L2/L3
Access Switch
POD 1
POD 2
POD 3
POD 4
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
25
Non-Ideal Network Topology
Ethernet L3
Core Switch/Router
Ethernet L2/L3
Aggregation Switch
Ethernet L2/L3
Access Switch
POD 1
POD 2
POD 3
POD 4
The Network gets in the way…
• 7 switches
• 10 chances to drop the frame
• 7 devices to manage….
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
Summary of issues..
Three Tier design (access, aggregation and core)
design flaws:
To many hops add latency.
Spanning Tree removes ~50% of the links thus increasing
congestion.
Lossy fabric rely on overlay protocols like TCP/IP this adding
to congestion.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
27
Simple solutions..
Three Tier design (access, aggregation and core)
design flaws:
To many hops add latency.
Solution Flatten the Network Topology
Spanning Tree removes ~50% of the links thus increasing
congestion.
Use overlay protocols like MC-LAG
Lossy fabric rely on overlay protocols like TCP/IP this adding
to congestion.
Create a lossless fabric by mapping all storage
flows to a lossless queue (use DCB)
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
28
Traditional Network Topology
Aggregation
Layer
DCB Enabled
Storage Array
Access Layer
DCB Enabled
Servers
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
29
Ideal Network Topology
One way……..
Storage Array
Spine
40GbE/100GbE
MC-LAG
Leaf
10GbE/40GbE
Servers Connected to Leaf/Access Layer
Use MC-LAG to remove the ills of Spanning Tree….
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
30
Ideal Network Topology
Desired way..
Flat Fabric
• Everything is connected to everything
(Pooling all resources together)
• Access is always 1 hop away.
• Removing the ills of Spanning Tree….
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
31
Agenda
6
Mapping the Deployment to what is Ideal?
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
32
Hadoop Network Topology
One way
Spine
MC-LAG
40GbE/100GbE
Leaf
10GbE/40GbE
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
33
Hadoop Network Topology
Desired Way….
Flat Fabric
Note
• No need for STP everything
is connect to everything else
one hop away.
• Best Latency, least amounts of hops.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
34
Mixing DAS, NAS, iSCSI and FCoE
FC SAN
DCB Enabled Fabric
FC SAN
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
35
Example: DCB Enabled Fabric/Network
Mixing DAS, NAS, iSCSI in a DCB Enabled Fabric
MC-LAG
Note
• Keep Traffic separated on different priorities
on all network devices.
Eg) Priority 1 = Tape Traffic (on all switches)
Priority 2 = iSCSI Traffic (on all switches)
Priority 3 = FCoE Traffic (on all switches)
Priority 4 = NAS Traffic (on all switches)
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
36
Example: DCB Enabled Fabric/Network
Mixing DAS, NAS, iSCSI in a DCB Enabled Fabric
DCB Enabled Fabric
Note
• Keep Traffic separated on different priorities
on network device.
Eg) Priority 1 = Tape Traffic (on all switches)
Priority 2 = iSCSI Traffic (on all switches)
Priority 3 = FCoE Traffic (on all switches)
Priority 4 = NAS Traffic (on all switches)
Flat Topology:
Easy to do operationally when there is only
one device to manage……
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
37
Remote office Data Replication
LOSSLESS
LOSSLESS
LOSSLESS
Data Center
Frames in-fight
Backup
Data Center
PAUSE
Things to think about…
• Does your WAN devices have enough buffer to handle frame
that are in-flight after pause is sent.
Campus
• Distance correlates directly to the amount of
buffer needed.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
38
Remote office Data Replication
LOSSLESS
LOSSY with
Protocol Overlay
LOSSLESS
Backup
Data Center
Data Center
TCP/IP
Note
• WAN Routers should be able buffer several flows
to account for frame drop and ensure retransmission.
Campus
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
39
Remote office Data Replication
LOSSLESS
LOSSY with
Protocol Overlay
LOSSLESS
Backup
Data Center
Data Center
MPLS/VPLS
Note
• WAN links are dedicated to just storage traffic
and no other traffic.
• Speed is pre-negotiated to ensure no packet drop.
Campus
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
40
Final Thoughts
Loop in the Network (Spanning Tree)
Flatten your fabric to remove loops, this allows better pooling of Ethernet
storage.
Or Use protocols like MC-LAG (Spin-Leaf) to remove loop.
Lossless Fabric/Networks
Map flows on the same priority on all network devices.
For long distances be mindful of the amount of buffer on the network
device. Else frame loss will be introduced.
Lossy Fabric/Networks
Use a protocol like TCP/IP to ensure proper frame delivery.
If you are brave and just want to rely on Ethernet. (Not suggested)
Dedicated to just storage traffic and no other traffic.
Pre-negotiated speed to ensure no packet drop.
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
41
Q&A / Feedback
Many thanks to the following individuals
for their contributions to this tutorial.
- SNIA Education Committee
Joseph White
Simon Gordon
Andy Ingram
Kishore Inampudi
Haruki Sonehara
Send any questions or comments on this
presentation to SNIA: [email protected]
Best Practice and Deployment of the Network for iSCSI, NAS and DAS in the Data Center
© 2012 Storage Networking Industry Association. All Rights Reserved.
42