Download Document

Reduced Communication Protocol
for Clusters
Clunix Inc.
Donghyun Kim
2000.9
Introduction
 Communication
Sub-system Performance is
decided by followings
• Transmission speed of physical network
• I/O handling capability
• Overheads of the communication protocol
 Communication
using traditional protocols
is the bottle-neck of parallel systems
• Myrinet with TCP/IP is not FAST.
• Small-granularity or communication-dense apps
show poor performance
Clunix Inc.
Introduction – cont’d
 A high
proportion of apps don’t need very
complicated communication functions
• By practice and theoretic analysis
Clunix Inc.
Overheads analysis of
traditional protocols
 Traditional
protocols overheads
• Time of context switching
• Time of data copying
 User
space – system space, adjacent protocol layers
• Time of data partitioning, re-constructing, data
analyzing
• Time of transmitting packet headers
• Time of routing, connection maintaining, traffic
controlling, error detecting, recovering, buffer
management
Clunix Inc.
Overheads analysis of
traditional protocols - cont’d

End-to-end latency L, bandwidth W modeling
• Assumptions : homogeneous, low network traffic
L  T(0) or T(1)
n max
W
T(n max )
(1)
m
T(n)  T0 (n)  2(τ   Ti (n))
(2)
i 1
T(n) : n-bytes transmission time
nmax : comm. subsystem max packet length
m : # of protocol layers
Ti(n) : i-th protocol layer processing time
(T0(n) : physical network transmission time)
Clunix Inc.
Overheads analysis of
traditional protocols - cont’d
n 
n i  n i 1  (  i 1   1) ρ i
πi 
Ti (n)  τ i 
T0 (n) 
n
ω0
1 i  m
(3)
n i 1  n i 1 

  Ti 1 (π i  ρ i )  Ti 1 (n i 1modπ i ) (4)
ω πi 
(5)
 : context switching time
 : memory bandwidth
0 : physical network transmission bandwidth
i : max packet length of i-th layer
I : packet header length of i-th layer
ni : data length of i-th layer
i : calling expense (routing,traffic control, error
detecting, buffer management, connection maintaining)
Clunix Inc.
Overheads analysis of
traditional protocols - cont’d
 Analytical
& testing results
Protocol
 Testing
Analytical
Testing
Layer
L(s)
W(Mbps)
L(s)
W(Mbps)
TCP
1350
8.5
1450
8.6
UDP
1110
9.5
1150
9.5
DLPI
450
10.0
650
10.0
conclusions
• Very large overhead using above IP protocol layer
• Memory-to-memory copying is not neglected
 If
transmission bandwidth is the same as memory
bandwidth, data copying(ni+1/) problem is bigger
Clunix Inc.
Design Strategies for RPC
• Support reliable, synchronous, asynchronous
communications
• Implement reliale broadcast and multicast basing
directly on the physical layer
• Lay the protocol below the IP layer
 Above
physical or datalink layer
• Avoid data copying AFAP
• If possible, avoid buffer management using
hardware buffering
• Run the protocol entirely in the user space
 In
the form of libraries
Clunix Inc.
Implementation of RCP
 OSI-DLPI
version
• Standard physical-device independent data link
layer interface
 Can
write uniform program on different machines
and network devices
 Myrinet
version
 Providing
user interface like the TCP-socket
Clunix Inc.
Implementation of RCP – cont’d
 RCP supports
unicast, broadcast, multicast
 RCP addressing
• Unique source/destination using hostname+port#
• Static address configuration
 Supports
 No
heterogeneous machines
connection maintaining, error detecting
• Assuming that underlying network is reliable
Clunix Inc.
Implementation of RCP – cont’d
 Sequencing
control, traffic control
• Sliding-window algorithm+selective retransmission
• Windows size is adjusted accoring to retransmission
frequency
 Fast-Adapt
and Slow-Recover algorithm
• Very efficient traffic control
 Data
partitioning and packaging algorithm
• Almost no data-copy, work in user-space
Clunix Inc.
RCP Tesing results
Bandwidth(W)
Lantency(L)
Clunix Inc.
Conclusions and future issues
 RCP design
considerations
• How to reduce the overheads
 Over-complicated
protocol processing
 Context
switching
 Overhead of data copying
• How to use the transmission control functions
supported by hardware
 To
 Future
reduce the protocol processing
Work
• To gurantee the quality of the communication.
Clunix Inc.