Download ppt File - users.cs.umn.edu

VIA and Its Extension To
TCP/IP Network
Yingping Lu ([email protected])
Based on Paper “Queue Pair IP, …”
by Philip Buonadonna
Outline
Motivation
VIA Overview
QP/IP Architecture
QP/IP Performance
Summary
Motivation
High performance computing, clustering applications
require high-throughput, low-latency communications
facility
Traditional TCP/IP is not designed for highthroughput, low-latency communications
Application software has not kept pace with the
increase of I/O speed




Memory copy
Checksum Computation
Interrupt
Context Switching
Typical Communication Data
Path
Bandwidth Comparison
Throughput (MB/s)
Bandwidth Comparison
1000
800
600
TCP/IP
400
VIA
200
0
6
25
2
51
24
0
1
48
0
2
96
0
4
92
1
8
84
3
16
Message Length (Bytes)
68
7
32
VIA Solution
VIA is a industry standard convened by
Microsoft, Compaq, Intel.
Key features of VIA:





Reduce memory copy (Zero-copy)
Direct user level access to NIC hardware
Eliminate OS kernel from critical path
Collapse ISO/OSI model
Offload CPU processing to intelligent NIC
VIA Architecture
VIA Components
Consumer


The end entity to use VIA function to communicate, can be
user-level or kernel
Use VIPL for programming
VI User Agent

Implements OS bypassing agent
Kernel Agent

Device driver, handle security and OS-related issues
VIA-capable NIC (Channel Adapter)

Implements VIA communications
Programming Abstraction
Queue Pairs

Components
 Send queue
 Receive queue
 Completion queue (status)

Data Movement Operations
 Send/Receive
 RDMA Read
 RDMA Write
Virtual Interface (Queue Pair)
Memory Access
Memory Registration



Memory must registered before use
System pins out the memory region
Nic use DMA to transfer data from memory to Nic
Memory Protection

Registered memory are associated with a VI
consumer and only valid to the VI consumer
Gather/Scatter list


Gather list: a list of registered source data buffers
(read)
Scatter List: a list of registered destination data
buffers (write)
Memory Model
Registered
Memory
Region
Virtual
Memory
Space
Page 1
Page n-1
Physical Memory
Page 0
Descriptor
A work queue element to be placed into
queue pair (send or receive queue)
Contains control segment and a list of
address segment
Specifies operation command, memory
address, size
Door Bell
An asynchronous
mechanism to notify VI
NIC of a new work
queue post
Door Bell can be a
register in NIC accessed
by both CPU and NIC
VIPL
Descriptor
1 VI NIC
0
Operation Example –
Send/Receive
Sender:

Consumer:
Receiver

 Register receive buffer
 Post a receive buffer in
the receive queue
 Register send buffer
 Post a Send work
queue element

Channel Adapter:
 Send out the data
and header, data are
retrieved directly
from consumer
memory
Consumer:

Channel Adapter:
 Receive packets from
sender
 Find out a receive
queue element in the
receive queue
 Move data directly to
the buffer specified in
the receive queue
element
Operation Example - RDMA
Write
Initiator

Consumer:
Receiver

 Register receiving buffer
address
 Send the address, R-key
 Register sending buffer
address
 Get receiver’s address
 Post a RDMA Write

Channel Adapter
 Send out data with
header(the operation,
receiving address), data
are retrieved directly
from sender buffer
Consumer
and length to initiator

Channel Adapter
 Receive data
 Check the validity of
address in RDMA header
 Move data directly to
the memory specified in
the RDMA header
Summary of VIA
Goal: low-latency, high-throughput by
offering direct access to NIC, Zero copy
Architecture components: consumer
(VIPL), UA, KA, VI-NIC
Main concepts: queue pairs, memory
pin, gather/scatter, descriptor, door bell
Operations: Send/Receive, RDMA Read,
RDMA Write
Why QP/IP
TCP/IP network is robust, ubiquitous
However, TCP/IP is not designed for highperformance, low-latency purpose
Queue Pair abstraction provides a way to
offload CPU processing, reduce the critical
data path, provide memory zero copy
The Integration of QP and IP may be able to
reduce the latency, improve the throughput
between end-end node applications
connected through TCP/IP network
Challenges to QP/IP
Provide a VIPL supporting QP/IP
Integration of connection setup
Handle message segmentation
Implement TCP/IP mechanism at NIC
Handle message boundary for TCP
Handle zero-copy in the event of packet
loss
QP/IP Architecture
QPIP Components
FSM:




Doorbell FSM
Sched/XMT FSM
RECV FSM
Mgmt FSM
Major Data Abstract



QPs
CQs
TCP Control Block (TCB)
QP/IP State Machines
QPIP Prototype
Three components

Application Library
 PostSend(), PostRecv(), Poll(), Wait()

Kernel driver
 Initialization
 Address mapping mechanism
 Interrupt service

Network interface firmware
 Implement TCP, UDP, IPV6 protocols
Application-Application RTT
Application Throughput & CPU
Utilization
Network Interface Processing
Cost
QPIP Based on NBD
NDB Client Throughput and
CPU Effectiveness
Summary
Integrate the QP concept from VIA with the
ubiquitous TCP/IP network
Provide low-latency, high throughput for SAN
QP/IP contains doorbell FSM, Sched/XMT FSM, RECV
FSM, Mgmt FSM. It also contains QPs, CQs, TCB data
structure.
Demonstrate comparable performance, much lower
CPU utilization with modest hardware.
The programmability also adds flexibility to adapt
with the evolvement of TCP/IP and scheduling
requirements.
Issues
How to integrate TOE in the mechanism?
How to effectively handle message boundary in
TCP to support upper level application, I.e.
iSCSI? How to handle segmentation?
How to support zero-copy in the case of packet
loss?
How to extend this into a WAN environment
(more unpredictability, fluctuation of latency,
available bandwidth, congestion, LFN)?
How to effectively support OSD communication?
Questions?