Download Networking

Scalable Sharing of Network
Storage
Micah Beck, Research Assoc. Professor
Director, Logistical Computing &
Internetworking (LoCI) Lab
Computer Science Department
University of Tennessee
CERN
May 30, 2002
A Broad View of Networking
• A network is a facility for sharing a pool of
resources within a community.
• Sharing is enabled by
–
–
–
–
Common goals
Standard protocols
Low allocation overhead
Mutual protection
IP Routing as Scalable Sharing
of Bandwidth
• The Internet is a collection of links
connected by routers
• Internet Protocol (IP) datagrams flow from
sender to reciever along some path of links
• The choice of paths is made by routers
according to topological considerations
(tempered by policy)
Wide Area Network Services
Are Like the Network Itself
•
•
•
•
•
Intermittently inaccessible
Vulnerable to partition
Prone to corruption in transit
Unpredictable latencies/jitter
End-to-End: Never require a network
service to be bigger, better or more complex
than wide area access allows
Why Is It Called
“End-to-End”?
• If the end-points requires a service that is
stronger than the network can provide, they
must implement it on top of the network.
• The fundamental tools are
– Aggregation (retransmission, fragmentation)
– Timers
– Protocol state maintained at the end-points
How Can Storage Be Shared
Scalably?
• Don’t require it to be
–
–
–
–
Unbounded in size
Unbounded in duration
Perfectly available
Perfectly reliable
• Best effort network storage
• Implement these properties end-to-end
Sharing Storage Enables
Advanced Network Apps
•
•
•
•
•
•
Caching
Staging (Replication)
Localization (Rendezvous)
Distribution (Multicast)
Spooling (E-Mail)
Temporary State Management
(Checkpointing)
“Logistical” Networking
• Analogy to military or industrial logistics
• Distributing goods requires not only
highways but also warehouses (depots)
• Data transmission is the highway
• Storage servers are the depots
• This is not Storage Networking
• Enabling new users and applications
Dimensions in Communication
Routing (spatial)
Storage (temporal)
Logistical Networking
Logistical Networking
• Principle Investigators
– Micah Beck
– James S. Plank
• Graduate Students
– Erica Fuentes
– Xiang Li
• Funding
– DoE SciDAC
– NSF Next Gen SW
– NSF Internet Tech.
• Research Staff
–
–
–
–
–
–
–
–
–
Scott Atchley
Alex Bassi
Ying Ding
Hunter Hagewood
Sharmila Kancherla
Jeremy Millar
Terry Moore
Stephen Soltesz
Yong Zheng
Internet Backplane Protocol
(IBP)
• Servers that make
allocation of primitive
“byte arrays” available
to clients
• Byte arrays are not
blocks (more abstract)
– Network capabilities
(primitive security)
– Variable extents
• Byte arrays are not files
(weaker semantics)
– Size & duration are limited
– “Volatile” allocations
– Best effort reliability and
availability
– No directory structure,
accounting
– No caching, replication
• An end-to-end approach
to network storage
IBP Software Structure
• IBP Depots (servers) are daemons that serve
local storage to IBP clients.
• IBP clients only link to a protocol library.
• Clients talk to depots using TCP/IP.
– Alternatives are being investigated
• Design is for high-performance/scalability.
Allocation Attributes
• Permanent vs. Time-Limited
• Volatile vs. Stable
• Read/Write semantics:
–
–
–
–
Byte Array
Pipe
Circular Queue
Completely Destructive Write
Impressions of Networking for
High Energy Physics (?)
• Reliable throughput is the limiting factor
• Modes of sharing can be “modified”
• The use of specialized data movement
middleware is acceptable
• The data transfer network has very few
nodes and small user communities
• Little patience for fancy architectural ideas
IBP Enables Heterogeneity
• The tyranny of “TCP-friendly” IP
– Datagrams and usage policy must be
homogeneous end-to-end
– Non-standard IP policies in the middle can leak
out onto public networks
• Good fences make good neighbors
• IBP can terminate a flow and regenerate it
in different network, with localized control
Example:Tsunami
• Uses very aggressive UDP transfer with
TCP back-link for control
• Runs using Abilene less-than-best-effort
“Scavenger Service”
• Tsunami must not be routed onto a link
where Scavenger Service is not supported!
• Solution: do not route the packets, require
explicit buffering and retransmission
Scheduling Control Over
Sharing Without IP QoS
• Consider a high speed link that operates at
10Gb/s, connected to two types of links:
– 100Mb/s and 1Gb/s
• Jobs are 1TB transfers
– ~20 hours at 100Mb/s, ~2 hours at 1Gb/s
• Suspending a slow job to allow a fast job
through delays it by only 10%
Scheduled Sharing of Network
Links Is In Your Future!
• Ignoring buffer control won’t make the need for
scheduled sharing go away
• When was more control over time & space ever a
bad thing?
• Are we revisiting CERNET/UUCP/BITNET/DECnet?
A Short History of Internet Protocols at CERN
Ben Segal / CERN PDP-NS
April, 1995
• IBP is designed according to end-to-end principles
(see SIGCOMM ’02 paper)
The Alternative to
Interoperability In Buffering
• High performance data transfer is an
isolated service, with “job scheduling” and
“data staging” like at computation centers
• As more high performance links are built,
concatenation is based on “gateways”
rather than a scalable routing infrastructure
• Should networking solutions be built
without the benefit of network architecture?
Application Areas
•
•
•
•
•
•
Source routing
Bandwidth adaption
Reducing (BWdelay)
Reliable multicast
Content Distribution
Remote access to
structured data
• Managing
computation state
(NetSolve caching)
• Temporary storage
• Very large data sets
• Collaborative
computing &
visualization
Building on IBP
• Many applications assume file semantics
– Unbounded size & duration
– High reliability & availability
– Caching & replication
• In a layered architecture, these are
implemented through aggregation and
additional intelligence at the next level
The
Network
Storage
Stack
Applications
Logistical
Runtime System
Logistical Tools
L-Bone exNode
IBP (Internet
Backplane Protocol)
Access (OS)
Physical
ExNode vs inode
IBP Allocations
the network
local system
capabilities
exNode
inode
user
kernel
block addresses
disk blocks
ExNode Mobility
XML Serialization
The exNode serialization is a portable soft link
Sameple exNodes Services:
Fragmentation, Replication
1
2
3
Building a Logistical Operating
Environment
• Resource Discovery
– Logistical Backbone registry (LBone: Plank)
– Can also use DNS, could use MDS
• Programming Tools
– exNode library, primitive commands
– policy modules (replication, fragmentation)
– cool applications (SC demo: IBPster MP3)
Related Work
• Peer-to-Peer Storage
– Napster, Gnutella, etc
– OceanStore (UC
Berkeley)
• Storage Area
Networking
– Fiber Channel,
Ultraband, iSCSI
– Network Attached
Storage (CMU)
• Overlay Networking
– Caching
– Multicast
– Data Grids
• Multimedia Protocols
– Content Delivery
– Streaming
– Video on Demand
Logistical Networking:
E2E Storage and Bandwidth
•
•
•
•
•
•
•
Complex paths and buffers are exposed
Assumptions are appropriately weak
Resource management is passive
Spatial and temporal control
Applic. autonomy, network transparency
Resources in the hands of end users
Futures: more control, greater scope
Logistical Computing and
Internetworking Laboratory
http://loci.cs.utk.edu
• Documents and software downloads