Download Efficient Way of iSCSI based Remote Storage Service for

An Efficient Way of iSCSI based Remote Storage
Service for Mobie Device using Intermediate
Target
Daegeun Kim
Soo-Mok Jung
Myong-Soon Park
Internet Computing Lab.
Korea University,
An-Am Dong, Sung-Buk Gu,
Seoul, 136-701, Korea
+82-2-927-3797
Internet Computing Lab.
Korea University
Internet Computing Lab.
Korea University
[email protected]
myongsp@
ilab.korea.ac.kr
vicroot@
ilab.korea.ac.kr
Abstract
As mobile device prevails, requests about
various services using mobile device have
increased. Specially, requests of application
services that need large data space such as
multimedia, game, database [1] greatly
increased. However, mobile appliance has
difficulty in applying various services like
wire environment because the storage
capacity of the one is not enough. Therefore,
researches [5] which provide remote storage
service for mobile appliance using iSCSI are
being progressed to overcome storage space
limitation of mobile appliance.
But, when iSCSI is applied for Mobile
appliance, iSCSI I/O performance drops
rapidly if iSCSI client moves from server to
far away position. It is because iSCSI has
structural quality to be very sensitive at delay
time.
In this paper, we introduce intermediate
target server and localize iSCSI target to
improve the shortcoming that iSCSI
performance drops sharply according as
latency increases when mobile appliance
recedes from storage server.
Keywords
iSCSI, SCSI, Mobile Device, Remote Storage, Block
Level I/O
1. Introduction
The explosive growth of mobile appliance
market has made a lot of demands in mobilerelated services. Effort to apply wired
network environment services which need
large amount of storage space such as
multimedia, databases to mobile appliance
with wireless network environment was
performed. However, mobile appliance
should be small and light to support mobility,
so that it uses small size flash memory
instead of hard-disk having large data space.
In the case of PDA, it usually has memory
space of 32 ~ 64M and cell-phone or smart
phone permit smaller memory space.
Therefore, there was difficult in saving
multimedia data such as mpg, mp3, etc. and
installing large size software such as database
engine. Limited storage space of Mobile
appliance has been a barrier for applying
various services in wired environment. As the
result, necessity of remote storage service for
mobile appliance rose to overcome limited
storage space of Mobile device and store
large amount of data or provide various
application service [5] [8] [9].
Remote storage service is classified by two,
one is file-level I/O and the other is block-
level I/O. File-level I/O service is provided
by communication between server and client
file systems. In the case of file I/O, because
client’s I/O request is passed to device via
server file system, data share is possible
between other clients through locking
mechanism which server file system offers,
but file system overhead drops I/O
performance.
However, in block-level service, client’s I/O
request is directly passed to storage device
without server file system in the form of
block I/O command. So, it is not able to
provide data sharing service by itself, but I/O
performance is better than the one of filebase I/O service [9].
The bandwidth of wireless network for
mobility, which mobile device use, is usually
lower than the one of wired network. If the
purpose of remote storage service is to extend
individual storage space for mobile appliance
without data sharing, block-level I/O service
is more suitable. iSCSI is a standard protocol
that transports SCSI command which is a
representative block I/O through TCP/IP
network. However, iSCSI has a problem that
I/O performance drops sharply if network
latency increases between iSCSI initiator and
target.
In this paper, when iSCSI-based remote
storage service applies to mobile appliance,
we improve the problem that iSCSI
performance is fallen rapidly according as
mobile client recedes from storage server and
network latency increases. This paper
comprises as the follow. In chapter III, we
present SCSI architecture model and iSCSI
basic operation to describe the reason that
iSCSI performance drops when distance
between iSCSI initiator and target is long. In
chapter IV, we explain a research related
with improving iSCSI performance using
client’s local cache. In chapter V, we are
going to explain the proposed way to
improve
iSCSI
performance
using
intermediate server, and we talk about system
architecture and algorithm in chapter VI. In
chapter VII, we analyze the result of
simulation with ns2. And at last we conclude
it and describe our future works.
2. Background
In this chapter, we describe SCSI
architecture model-3 and iSCSI basic
operation process to explain the relation of
iSCSI Initiator-Target's
distance
and
performance, and then analyze iSCSI
performance test result by different delay
time.
3.1 SAM-3 ( SCSI Architecture Model
-3)
SAM-3 is a draft for SCSI standard leaded
by T10, a Technical Committee of
Accredited Standards Committee INCITS
(InterNational Committee for Information
Technology Standards). SAM-3 specifies the
SCSI Architecture Model. The purpose of the
architecture is to provide a common basis for
the coordination of SCSI standards and to
specify those aspects of SCSI I/O system
behavior that are independent of a particular
technology
and
common
to
all
implementations[7].
3.1.1 Linked SCSI Command Processing
Serial SCSI Commands are processed
sequentially in SAM-3[7]. Only after
previous command processing is completed,
next command is processed. This sequential
processing mechanism cause that SCSI
protocol is sensitive to network latency.
Figure 1 shows the sequence in which two
serial SCSI commands are processed between
SCSI initiator and target. SCSI initiator has a
waiting time from sending created command
to target until receiving the result. If the
distance of initiator and target is longer,
initiator’s waiting time is larger because
electronic wave delay of the network is
grown. Even if the change of waiting time
about one SCSI command is small,
subsequent all commands will be delayed by
previous response latency because SCSI
command process is sequential. If there are n
serial SCSI commands to be processed, time
lag of n x dt (time lag of response delay per
one SCSI command) occur until last
command is processed.
Figure 1. Linked SCSI command process
As a result, when the distance between SCSI
Initiator and Target is long in some degree,
response time of a SCSI command is delayed
and link utilization of SCSI protocol dropped
causing degradation of I/O performance.
Even in the case of iSCSI which embody
SCSI protocol over TCP/IP network,
described feature of SCSI makes iSCSI
performance limited by the distance of
initiator and target, although iSCSI has
advantage that distance of Initiator and
Target can become overlong than SCSI cable
or fiber channel.
3.2 iSCSI
The iSCSI (Internet Small Computer
System Interface) is an emerging standard
storage protocol that can transfer SCSI
command over IP network[5]. Since the
iSCSI protocol can make client access the
SCSI I/O devices of server host over an IP
Network, client can use the storage of
another host transparently without the need to
pass server host's file system[6].
In iSCSI layer which is on top of TCP layer,
common SCSI commands and data are
encapsulated in the form of iSCSI
PDU(Protocol Data Unit). The iSCSI PDU is
sent to TCP layer for the IP network transport.
Through the procedure, client who wants to
use storage of the remote host can use it.
Because the encapsulation and the
decapsulation of SCSI I/O commands over
TCP/IP enable storage user to access remote
storage device of the remote host directly [3].
Likewise, if we build remote storage system
for mobile appliance using iSCSI protocol,
mobile clients can use the storage of server
host directly like their own local storage. It
enables mobile appliance to overcome the
limitation of storage capacity as well as to
adapt various application services of wired
environment in need of mass scale data.
Different from the traditional remote storage
system that based on file unit I/O, iSCSI
protocol provides block unit I/O. So it can
make more efficient transmission throughput
than the traditional remote storage system
like CIFS and NFS does.
Another characteristic of iSCSI protocol is
that it operates on standard and commonly
used network component like Ethernet. Since
iSCSI protocol directly can be plugged in
Ethernet environment, it is easier to manage
than another storage data transmission
protocol as Fibre Channel. Moreover, iSCSI
can reduce costs to build storage system due
to the use of infra network without additional
adjustment. The iSCSI protocol
was defined as the standard SCSI
transmission protocol lately by IETF and a
lot of related researches are being processed
with the development of Gigabit Ethernet
technology.
3.2.1 iSCSI Basic Operation
Figure 2 shows exchanged control and data
packets’ sequence in Read and Write
Operation of iSCSI protocol [10].
Figure 2. iSCSI basic operation
Like figure 2, iSCSI exchanges control
packets (SCSI Command, Ready to Transmit,
SCSI Response) and data packet (SCSI Data)
to process one R/W operation. Three control
packets and one data packet are used in write
operation and spend 2 x RTT. It takes 2
control packets and 1 x RTT in case of to
read operation. Because control packet
including header information is no more than
48 byte, bandwidth waste becomes serious
when initiator recedes a little from target.
Therefore, distance of iSCSI initiator and
target, and network latency influence iSCSI
I/O performance.
3.2.2 ISCSI performance change by network
latency
Figure 3 show storage system constructed in
University of Minnesota for performance
experiment of iSCSI based storage service in
different network environment [6]. They
analyzed the performance change and feature
of iSCSI when storage is accessed in Gigbit
Ethernet(LAN) and through Campus
Network(WAN).
propagation delay influenced by distance of
physical network is main part of iSCSI
latency. In the case of mobile appliance,
which usually run I/O by small size block,
delay by the distance of physical network that
packet move is more important factor
degrading iSCSi performance than the delay
by bandwidth.
3. Related Work
The way using cache to improve disk I/O
performance has been already studied already
for a long time. Usually, researches, that use
part of memory space as buffer cache to
improve performance of local disk, were
most cases. However, researches that use
local or network cache to improve iSCSI's
performance as iSCSI has been a standard
recently and products supporting iSCSI
appear.
3.1
Figure
3.
environment
iSCSI
performance
experiment
Table 1. Change of delay by network and data size
Read(ms)
Configuration
1KB
4KB
16KB
64KB
GigE LAN
9.57
9.7
10.91
15.82
iSCSI
Campus LAN
11.37
12.45
14.33
24.2
iSCSI
Write(ms)
Configuration
1KB
4KB
16KB
64KB
GigE LAN
10.08
10.35
11.24
15.0
iSCSI
Campus LAN
12.8
12.5
14.62
22.98
iSCSI
Through the experiment result in Table 1,
we can know that the latency by network
distance and bandwidth between Initiator and
Target influences iSCSI performance. ISCSI
shows similar performance in small Gigabit
Ethernet environment such as when disk is
directly accessed through fiber channel, but
shows low performance in Campus LAN
(WAN) environment in which network traffic
is liable to variation and latency is big. That
is why delay time affects iSCSI's I/O
performance a lot. Specially, in case data size
is large, transmission delay influenced by
network bandwidth is much bigger than other
latency, and in case data size is small,
iCache
iCache is a research to improve iSCSI
performance using local cache of client
system. Initiator’s system has specific cache
space for iSCSI data and iSCSI block data is
cached to minimize network block I/O.
Therefore, iSCSI does not send I/O request
through network every time when disk I/O
happens, instead read cached block or send
blocks cached in LogDisk at once to server
for improving iSCSI performance. iCache's
buffer space consists of two hierarchical
caches which are NVRAM and LogDisk.
Data is stored sequentially in NVRAM and
when enough data is gathered, iCache
process moves data from NVRAM to
LogDisk. Blocks which are frequently
accessed, are kept in NVRAM of which
access speed is fast and iCache stores less
accessed data in logDisk. Also, destage
operation is achieved by kernel thread called
as LogDestage. In Destage Operation,
NVRAM’s data is moved to LogDisk or
LogDisk's data is sent to remote storage
using iSCSI protocol.Cache techniques used
in iCache are based on DCD technology [11]
proposed to improve Disk I/O performance.
Figure 4. iCache architecture
It is difficult to apply iCache for mobile
device which has lacking memory because
iCache need additional memory and harddisk space to embody local cache which
consists of NVRAM and LogDisk.
In iSCSI based remote storage service for
mobile node, when mobile node serviced in
A area moves to C area, distance between
iSCSI initiator and target is prolonged and
packet transfer time increases. In chapter 3,
we showed SCSI command is processed
sequentially in SAM-3 [7] and iSCSI basic
operation needs exchange of several control
packets to process one command. Because
small size control packet of iSCSI protocol
influence iSCSI response time, if the distance
between server and client is long, the link
utilization drops sharply and iSCSI
performance becomes low.
4. Proposed Idea
In this chapter, we introduce the way
localizing iSCSI target with intermediate
server called intermediate target to solve the
problem that iSCSI performance drops when
iSCSI mobile client recedes from storage
server. iSCSI response time for read
operation is minimized with nearest
intermediate target which prefetches read
block for client to use or response beforehand
packet to iSCSI initiator for written block.
Figure 5. iSCSI target localization with intermediate
server
In proposed architecture, we suppose
following 3 to simplify the problem.
- Suppose distributed storage server
environment. iSCSI intermediate target
server spread over wide areas. mobile
client is connected with nearest iSCSI
intermediate target through iSNS (Internet
Storage Name Server) [3]
- As transmission distance increases,
propagation delay of physical media and
sum of the queing delay of intermediate
routers increases.
- ISCSI latency includes propagation delay
by
distance,
queuing
delay
and
transmission delay by bandwidth, and
ignores processing delay of end nodes.
5.1. iSCSI Intermediate Target
Figure 6. SCSI response time reduction
We introduce intermediate server (iSCSI
Intermediate Target) to reduce packet transfer
delay time between server and client, and
heighten practical utilization of the network
bandwidth. Intermediate target prefetch next
block to be used by client for iSCSI read
operation and can give quick response for
iSCSI write operation. The nearest
intermediate server from mobile client is
selected by iSNS and client runs iSCSI
protocol with intermediate server which also
has iSCSI connection with remote storage
server. Therefore, response delay time of I/O
request shorten because client has iSCSI
connection with near intermediate server
instead of long-distance storage.
Figure 7. Direct communication between
iSCSI Initiator and Target
intermediate server and client and another is
connection with storage server. Target
module has an iSCSI session with mobile
client's iSCSI initiator and initiator module
has one with iSCSI target module of storage
server. Two modules perform same role such
as general iSCSI Target/Initiator module.
However, the first iSCSI connection between
mobile client and intermediate target is used
for I/O requests of client, and the latter is
used to prefetch next blocks used by client
from storage server or deliver write block to
storage server. Target/initiator module of
intermediate server is controlled by block
management module. ISCSI Buffer is
managed by FIFO way to leave blocks used
most recently.
Figure 8. Response by intermediate target
Figure 7 and figure 8 show response delay
times are different when iSCSI Initiator and
Target do direct communication each other
and when put iSCSI intermediate server. 3
control packets should be exchanged to
process a write command and two control
packets as well as read command. If the
distance between initiator and target is short,
iSCSi I/O response time is reduced because
control packet size is small but has same
propagation and queuing delay. If the end
nodes’ processing delay of iSCSI packet is
ignored, response delay time is reduced by
A/L ratio when introducing intermediate
server in figure 8.
5.2 System architecture
Figure 9 shows a module diagram of
Intermediate target system. Intermediate
Target system is consisted of iSCSI initiator,
target and block management module. iSCSI
intermediate server has two iSCSI
connections. One is connection between
Figure 9. Intermediate Target System
Figure 10 shows processing algorithm about
client I/O request by block management
module of intermediate server. In case of
read request, block management module
searches requested block in iSCSI buffer to
service the requested block with target
module. If the requested block exists in
iSCSI buffer, target module of intermediate
server reply to client. But when there is not
requested block in buffer, initiator module of
intermediate server which has a connection
with storage server, send client request to
storage server and receive the block to send
to client. At this time, block management
module send read request for next logical
address of the block that client required.
When intermediate server receive write I/O
request, intermediate server send reply
message to client and send the write block to
storage server in no time. Therefore, mobile
client's iSCSI initiator has an effect of iSCSI
connection with near storage server.
When Intermediate Target Receives I/O Request
i : Requested Block
{
if(Read Op)
{//Client's I/O Request is Read Operation
if(i exists in iSCSI Buffer)
{
Send i and Response to Initiator
}
else
{
Send Read Request of i to Target
}
Send Read Request of i+1 from Target
and Put at Head of iSCSI Buffer
//Prefetch next block from Storage Server
}
else (Write Op)
{//Client's I/O Request is Write Operation
Put i at Head of iSCSI Buffer
Send Response to Initiator
Send Write Request of i to Target
}
}
Figure 10. Block Management Algorithm
5. Simulation
In this chapter, we measured the change of
iSCSI performance by the distance between
iSCSI target and initiator, and the position of
intermediate target. In case of read operation,
we measured the change of iSCSI
performance by the hit ratio of prefetched
block. The result shows that iSCSI
performance is high when the distance
between initiator and target is short, and data
is accessed sequentially such as multimedia
data.
6.1 Simulation environment
In figure 11, the network of iSCSI initiator,
target and intermediate target is simulated
using network simulator 2.27. The bandwidth
of the link between node1 and node2 is
limited by 14400bps considering wireless
network(CDMA 2000 1x). Performance is
measured by changing the delay time from
1ms to 64ms to analyze the change of iSCSI
performance according to the distance
between iSCSI initiator and target under
supposition that delay is proportional to
distance. We decided data size by 512 byte
that is SCSI block size used in PDA of
Windows CE base which is one of
representative mobile appliance.
Figure 11. Network Environment of NS2
6.2 Experiment result and analysis
Figure 12 shows the difference of iSCSI
performance by data type when intermediate
target is used or not. In case of the ratio of
delay time A and B specified by 7:3, we can
see that the difference of iSCSI performance
is very small regardless of data type when
total delay is less 2ms. In simulation using
ns2, iSCSI throughput is influenced by
propagation delay which depend on physical
media, and transmission delay that depends
on data size and bandwidth. In case iSCSI
has short delay when whole distance of iSCSI
initiator from target is short, transmission
delay is much bigger than the difference of
propagation
delay
by
introducing
intermediate target, so it makes little
influence on iSCSI performance. However,
when iSCSI initiator is in the distance from
target, difference of propagation delay time is
much longer than transmission delay,
intermediate target gives great effect on
iSCSI throughput. According to figure 13, we
know that iSCSI performance difference
appeared most greatly when propagation
delay of iSCSI initiator and target is
prolonged. In the case of animation, text and
application data, each has improved
throughput by 30%, 27% and 17% when the
latency is 64ms. Performance difference
according to data type is because iSCSI
buffer hit rate for read block is different. We
suppose that the hit ratio of multimedia data
which is accessed sequentially, is 90 percent
and each hit ratios are 80% and 50% in the
case of text and application data.
Multimedia
Text
Application
No Intermediate
1600
1400
Throughput(KB/Sec)
1200
1000
800
600
400
200
0
1
2
4
8
Delay(ms)
16
32
64
Figure 12. Performance result by data type ( read
operation )
Figure 13 shows throughput change by
distance ratio of intermediate target for iSCSI
write operation. In case of write operation,
throughput does not change regardless of
data type and we experiment how the
performance of iSCSI write operation
changes according to intermediate target’s
position. Same as read operation, iSCSi write
operation
has
small
difference
of
performance when total distance is short, but
the difference is larger if the distance is
longer. And the result shows that
performance appears high when delay time of
iSCSI initiator and intermediate target is
smaller. iSCSI performance difference was
biggest at the point that total delay time is
64ms as the case of read operation, and in
case of each ratios, A:B = 5:5, A:B = 3:7,
A:B = 9:1, performance elevation of each
25%, 40%, 59% happened. Therefore, we can
know that proposed method shows high
performance when iSCSI intermediate target
is near to initiator.
A=5,B=5
A=3,B=7
A=1,B=9
No Intermediate
1600
Throughput(KB/Sec)
1400
1200
improve the problem that iSCSI performance
is fallen when applying iSCSI-based remote
storage service for mobile appliance. In
proposed method, localizing iSCSI target
improved low link utilization and iSCSI
performance degradation problem when
initiator recedes from target. Through
simulation result we know that iSCSI
performance is fallen rapidly if the distance
(latency) of initiator and target is long and
could see that iSCSI performance is high
intermediate target is near from Initiator.
In future work, we apply various prefetch
algorithms for data type or improve present
FIFO iSCSI buffer management algorithm to
heighten buffer hit rate. To accomplish above
future work, we have to research into how
data type is classified and the way sending
information, and other buffer management
algorithms.
8. Reference
[1] Susan J. Shepard, "Embedded Databases Can
Power Emerging World of Information to Go",
IEEE Distributed Systems Online.
[2] Block Device Driver Architecture,
http://msdn.microsoft.com/library/enus/wceddk40/html/_wceddk_system_architecture_
for_block_devices.asp
[3] Tom Clark, IP SANs : A Guide to iSCSI, iFCP,
and FCIP Protocols for Storage Area Networks,
p.153, Addison-Wesley, 2002.
[4] Xubin He, Qing Yang, Ming Zhang, "A caching
strategy to improve iSCSI performance", LCN'02,
LCN 2002. 27th Annual IEEE Conference on , 6-8
Nov. 2002 .
[5] Sura Park, Bo-Suk Moon, "Design and
Implementation of iSCSI-based Remote Storage
System for Mobile Appliance", Data Storage
System Research Group, p236-240, 2003.
[6] Yingping Lu and David H. C. Du, University of
Minnesota, "Performance Study of iSCSI-Based
Storage Subsystems", IEEE Communication
Magazine, Agust 2003.
1000
800
600
400
200
0
1
2
4
8
Delay(ms)
16
32
64
Figure 13. Throughput change by distance ratio of
intermediate server (write operation)
7. Conclusion & Future Work
In this paper, we describe efficient way
introducing iSCSI intermediate target to
[7] SAM-3 : Information Technology - SCSI
Architecture Model 3, Working Draft, T10 Project
1561-D, Revision7, 9 May, 2003.
[8] Sula Park, Bo-Seok Moon, and Myong-Soon Park,
"Design, Implementation, and Performance
Analysis of the Remote Storage System in Mobile
Environment", ICITA 2004
[9] Bosuk Moon, Myong-soon Park, "A Performance
Analysis of the CIFS, NBD, iSCSI on the Wireless
Network by using CDMA-2000 1x", STORAEG
2003, Data Storeage System Research Group
[10] Julian Satran, iSCSI Draft20,
http://www.ietf.org/internet-draft/draft-ietf-ipsiscsi-20.txt
[11] Y. Hu and Q. Yang, "DCD-disk caching disk: A
New Approach for Boosting I/O Performance,"
Proc of 23rd Annual Intl. Symposium on
Computer Architecture (ISCA'96), 1996.
[12] Kalman Z.Mesh, Julian Satran, "Design of the
iSCSI Protocol", the 11th NASA Goddard
Conference on MSS'03