Download Building a Reliable Ethernet/IP Network: GSFC Prototype

Building a Reliable Onboard Network with Ethernet:
A GSFC Prototype
Jane Marquart
NASA/GSFC
Requirements
• Critical real-time data must be delivered
reliably onboard the spacecraft
(target is 5 milliseconds)
• Ethernet/IP Onboard Network Bus
8 June 2004
Fourth Space Internet Workshop
2
Protocol Options
TCP
TCP works but doesn’t support reliability
features required by typical flight software
UDP
UDP is a better fit, but requires reliability be
implemented by the user
UDP +
Reliability
8 June 2004
Where to implement reliability? Application
layer or data link layer?
Fourth Space Internet Workshop
3
Reliability Trade
• Application layer:
– Rx & tx packet latency through the stack is highly variable
and non-deterministic (regardless of direction)
– Latency up to several milliseconds on a MCP750 @ 233mhz
• Datalink layer:
– avoids IP stack latency, but;
– the issue is standardization and portability of a datalink layer
solution.
8 June 2004
Fourth Space Internet Workshop
4
IEEE 802.3 LLC
• LLC – “Logical Link Control” offers a IEEE standardized datalink
layer reliability protocol, adding a 3 byte header to the ethernet
frame
• Supplies 3 major types of service;
– Type 1 (unreliable packet exchange, same as plain ethernet)
– Type 2 (reliable, statefull, connection oriented; used by FDDI, Wireless,
Token Ring)
– Type 3 (reliable, stateless, connectionless)
• Type 3 selected
–
–
–
–
Stateless but reliable
Well-defined (and simple) state machine
Suitable for a variety of physical layers
Packet overhead is 3 bytes
8 June 2004
Fourth Space Internet Workshop
5
Data-link Layer Reliability: LLC
•
Ack/retry algorithm implemented within the NIC driver.
•
Latency & delay well defined since control is right at the hardware.
•
Widely supported, if not tolerated, by common OS IP stacks,
routers, etc.
•
Well defined & supported but ad-hoc mapping onto standard
ethernet framing.
•
Reliability protocol only useable between conformant drivers.
8 June 2004
Fourth Space Internet Workshop
6
PDU Formats
Standard ethernet frame w/ EthernetType >= 0x600
Ethernet frame header
Default
datalink
pdu
Ethernet
destination
(6 bytes)
Ethernet source
(6 bytes)
Packet data
Ethernet type
(2 bytes)
User data &fill if req’d
EthernetType < 0x600, set to frame length indicating a 802.2 LLC pdu
Reliable pdu
(data packet)
Ethernet frame header
LLC header
ethernet ethernet ethernet
DSAP SSAP Control
source
type
dest
(1
byte) (1 byte) (1 byte)
(6 bytes) (6 bytes) (2 bytes)
Destination SAP
Ethernet frame header
Reliable
pdu
(ack
packet)
Packet data
user data &
fill if req’d
Source SAP
LLC header
ethernet ethernet ethernet
DSAP SSAP Control
source
type
dest
(1
byte) (1 byte) (1 byte)
(6 bytes) (6 bytes) (2 bytes)
Packet data
No user data,
Fill to min. valid pkt length
ack pdu's swap DSAP & SSAP, along
with ethernet source/destination so reply
is sent to originating host.
8 June 2004
Fourth Space Internet Workshop
7
Ethernet/IP Multi-node Testbed
4 Mbps
SWITCH
MEDIA
CONVERTER
150kbits
50 kbits
200 kbits
MEDIA
CONVERTER
4 Mbps
"Comm Card"
Dell PC
Flight NIC
Serial
NIC Driver
Router
Serial
Ethernet
Hub
IP OS Stack
1 kbits
Channel Link Simulator
Net
Mgr
Serial
Router
CI
CFDP
Subsys
‘A’
Coldfire
150 kbits
Subsys
‘B’
PC
SB
TO
ITOS
GSE
CFDP
8 June 2004
Fourth Space Internet Workshop
8
NIC Driver Architecture
IP stack
TX packets
RX
packets
Async Pkt
Channel Queues
Packet
validation &
LLC
processing
SOIS Independent
Layer
Packet
retrieved
RX interrupt
RX
packets
Packet
Scheduler
TX done interrupt
SOIS dependent
Data-link layer
LLC
ack/error
packets
Packet
submitted
to NIC
TX packets
Flight NIC
Eth0
8 June 2004
Eth1
Fourth Space Internet Workshop
9
Use Case Examples
Pkt sent with
reliable QOS
nominal
Pkt ‘N’ sent with
reliable QOS
Driver returns
‘ACK’
Pkt
lost
Pkt N+1 sent
with reliable QOS
Driver returns
‘ACK’
ACK Time-out,
Resend pkt,
Log error
Pkt ‘N’ sent with
reliable QOS
ACK drop
ACK
lost
Receive ACK
Done
8 June 2004
Driver returns
‘ACK’; logs error
Receive ACK
Done
Receive ACK
Done
Pkt sent with
reliable QOS
Pkt
lost
ACK Time-out,
Log error
pkt drop
ACK Time-out,
Resend pkt
Pkt
lost
ACK Time-out,
Resend pkt ‘N’
Log error
Receive ACK
Done
Pkt sent with
reliable QOS
pkt lost
ACK Time- out,
Resend pkt,
Log error
Driver returns
‘ACK’
ACK Time- out,
Log error
Driver returns ‘ACK’
Discards redundant
pkt
Logs error
Pkt N+1 sent
with reliable QOS
ACK lost
ACK
lost
ACK
lost
Driver returns
‘ACK’
Driver returns ‘ACK’
Discards redundant
pkt
Logs error
Driver returns
‘ACK’
Receive ACK
Done
Fourth Space Internet Workshop
10
NIC/OS Transmit Path Measurements
User app.
opens socket
to send packet
(UDP Client)
Switch
Stack Processes Packet
Tx_Stack_to_
Staging_Buffer
Tx
DMA/PIO
NIC-NIC
NIC receives pkt.
Tx Stack_to_NIC
Tx App_to_NIC
Separate test with SmartBits Analyzer
Metrics:
Key:
= timestamp marker (software)
= logic analyzer timestamp
1.) Stack Process Time
2.) Driver copy from stack to Staging buffer
3.) DMA vs. Programmed I/O time
4.) Switch Latency
= start receive markers
8 June 2004
Fourth Space Internet Workshop
11
NIC/OS Receive Path Measurements
Packet
Recv’d
Interrupt
App receives pkt.
Rx DMA/PIO
Rx_receive_buffer
_to_stack
Stack Process Time
(TBD)
Rx_NIC_to_Stack
Key:
= timestamp marker (software)
= start receive marker
8 June 2004
Metrics:
1.) DMA vs. Programmed I/O
2.) Copy time from staging
buffer(receive_buffer) to stack
Fourth Space Internet Workshop
12
Preliminary Performance Results
With LLC
Without LLC
Min.
Width
(us)
Max
Width
(us)
Avg
Width
(us)
Std.
Dev
(us)
39.073
TBD
TBD
TBD
TBD
2.525
0.481
TBD
TBD
TBD
TBD
27.843
39.156
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
TBD
Min.
Width
(us)
Max
Width
(us)
Avg
Width
(us)
Std.
Dev
(us)
16.560
296.400
25.264
Rx_BUFF_to_STACK*
1.500
3.360
RX_NIC_TO_STACK*
18.481
299.581
Rx_NIC_to_BUFF
(PIO or DMA)*
TX_STACK_
PROCESSING**
TBD
TBD
TX_STACK_TO_
STAGING BUFFER*
1.260
14.880
2.523
2.346
TX_BUFFER_TO_NIC
(PIO or DMA)*
5.100
144.600
17.256
23.849
TX_STACK_TO_NIC*
7.740
159.421
20.064
26.148
TBD
* Numbers are for Programmed I/O (which can be up to ~4 times slower than DMA), using TCP (ftp), MTU sizes
vary between 64-1500 bytes
** Numbers are for Programmed I/O, UDP, MTU sizes between 64-1500
8 June 2004
Fourth Space Internet Workshop
13
Status
• LLC working in lab (2 millisecond timeout)
• Additional performance testing includes:
– LLC
•
•
Bi-directional benchmark
Worst case footprint of LLC from CPU & network throughput cost
– Fixed packet size (minimum and maximum)
– Linux OS (should be worse due to virtual memory model)
8 June 2004
Fourth Space Internet Workshop
14
Conclusion
Preliminary tests indicate that:
the onboard network architecture, using LLC
for reliability, will meet the 5 millisecond
requirement for packet transfer
8 June 2004
Fourth Space Internet Workshop
15
The Team
Software
Hardware
Greg Menke
Art Ferrer
Freemon Johnson
Alan Cudmore
Jonathan Wilmot
Jane Marquart
Mike Lin
Scott Edfors
For more information contact: Jane Marquart at
[email protected]
8 June 2004
Fourth Space Internet Workshop
16