Download QNX Corporate Sales Presentation

Designing High-Performance Network Elements
Using Multiprocessing Technology
and Adaptive Partitioning
Peter van der Veen
QNX Software Systems
Typical Hardware Architecture
Low-speed bus
High-speed interconnect
Chassis
Line card
Network
Network
Network
Control card
...
Network
Network
2
Typical Netcom System Software Constraints
 Many millions of
lines of code
Device
Driver
Device
Driver
Kernel
(RTOS)
Device
Driver
TCP/IP
Stack
SS7
Stack
Application
Application
Filesystem
 Tens to hundreds
of S/W
components
 Hundreds to
thousands of
processors and
threads
 Strict availability
requirements
3
Software Architecture
ROUTE MANAGER
Thread A
Thread B
FILE SYSTEM
Thread C
ETHERNET DRIVER
Thread D
Thread E
QNX NEUTRINO REALTIME SCHEDULER (OS)
Thread A
Thread B
Thread C
PRIORITY
Thread D
Thread E
Thread B
CPU
CPU
CPU
CPU
CACHE
CACHE
CACHE
CACHE
HIGH-BANDWIDTH CPU BUS
MEMORY
 Multiple processors
sharing common hardware
Common memory bus and
address space
Access to all peripheral devices
and interrupts
OS manages tasks running on
processors – true concurrency
 Transparent to application
programs
 No incremental hardware
 No application software
changes needed
4
Symmetric Multiprocessing
SMP Memory Organization
Apps
"A"
e600 core0
Apps "A"
The OS kernel resides at
physical memory address
0, addressable by both
cores
Shared
memory
MMU
OS
Apps "A"
OS
e600 core1
Apps
"B"
Shared
memory
MMU
Apps "B"
Apps "B"
Shared
memory
OS
The MMU relocates
applications and shared
memory appropriately
OS
Physical
memory
6
Making the Most of SMP



Concurrency … divide and conquer
►
Write software components using threads
►
Remove serializations from dataflow
Caches … keep them hot
►
Minimize writes to globally shared data
►
Process data on the same processor where possible
Scheduling … get your ducks in a row
►
Take advantage of the OS scheduler
►
Use diagnostic tools to adjust runmasks and priorities
7
SMP Optimizing Tools


System Profiler
►
Provide a timeline view of
activity in the system
►
Identify resource contention
and serialization
►
Analyze SMP scheduling
thrashing
►
Visualize distributed message
passing
CPU Performance Counters
►
Count operations such as
cache misses
►
Statistically sample based on
significant events
8
Adaptive Partitioning
Introducing Adaptive Partitioning

What is Adaptive Partitioning?
►
Adaptive partitioning is a new QNX product that extends the Neutrino RTOS
► Allows you to build secure compartments or “partitions” around a set of
applications or threads
► Partitions enforce CPU guarantees for applications, controlled by easy to
use budgets


Why is it Adaptive?
►
Patent-pending design ensures all available CPU cycles are given to
partitions that need processing time – no CPU cycles wasted
►
Provides performance advantage by permitting full processor utilization to
accommodate spikes in demand
Easy to get started
►
No changes to how designers work today
 POSIX programming model for the same, familiar design, programming
& debugging techniques
► No code changes are required to implement partitions
10
Understanding “Adaptive”
Management
Interfaces
(CLI, SNMP)
Routing &
Forwarding
Reconfiguration
Processing
Load
Scenarios
Topology
Change
70%
10%
5%
System Restart
20%
90%
80%
Steady State
5%
0%
Maintenance
5%
10% 10%
Idle Time
95%
20%
40%
60%
80% 100%
11
Defining Partitions
Management
QNX Neutrino
micro-kernel
Management
Interface
5%
Routing &
Forwarding
Routing &
Forwarding
75%
Maintenance
Maintenance
20%
Given the processing scenarios, choose a partitioning approach and
appropriate partition budgets
12
Understanding “Adaptive” Partitioning
Adaptive: Budgets enforced
Maintenance
Adaptive:
can
when Applications
CPU is loaded
CPU Time wasted when use free CPU time if available
partitions do not consume their from other partitions
5%
20%
budget. Applications
cannot 75%
benefit from available time.
QNX Neutrino
Microkernel
Reconfigure
Management
Interface
Routing &
Forwarding
75%
5%
20%
75%
Reconfigure 5%
20%
10%
Topology 5%
Change
75%
10% 10%
Steady State 5%
75%
10% 10%
Restart 5%
0%
75%
20%
40%
20%
60%
Static
80%
100%
Topology 5%
Change
85%
80%
Steady State
10% 10%
95%
Restart 5%
0%
10%
20%
40%
60%
80%
100%
Adaptive
13
Uses for Adaptive Partitioning
Security Threats

Embedded systems are becoming network connected
►
►
Untrusted interfaces and network threats
Untrusted add on software

If appropriate measures are not included by design, your product’s security
and availability can be compromised

Rogue software can launch denial of service (DOS) attack and starve core
applications of CPU time
►

Need to ensure untrusted, add-on software can be contained to guard against attacks
Distributed DOS attacks can busy your system with network processing
QNX Neutrino
Microkernel
File
System
Device
Drivers
Core
Core stealing CPU
Rogue add-on
time
Application
Application
Core
Networking
Networking stack hogging CPU time
Application
Add-On
Add-On
15
Partitioning to Contain Threats

Create OS enforced partitions to ensure critical system
resources are protected
►
►

Ensure CPU available for core functions
Partition inheritance ensures applications get CPU time for OS
services (such as drivers, file systems, networking)
Contain threats and protect core applications
►
Limit impact of rogue applications
QNX Neutrino
micro-kernel
File
System
Device
Drivers
Core
Application
Core
Application
Networking
Rogue
add-on
thwarted
Consuming CPU
Time
Networking
Add-On
Core
Application
Add-On
16
How Adaptive Partitioning Works
Partition Accounting



What does “30% CPU Budget” mean?
►
CPU usage is calculated over a sliding window.
►
Partition budget guaranteed percentage of cpu time, balanced over sliding window
►
Partition usage CPU time executed, during last sliding window, expressed as percentage
Accuracy
►
Counting ticks is not enough. “Micro-billing” is used to track actual CPU utilization even when
threads don’t use their whole timeslice
►
Micro- and nano-second resolution
►
Threads are billed based on real usage, not statistics
“windowsize” is configurable as an argument to kernel at boot
►
Tradeoff maximum READY-state latency with accuracy of CPU budgeting

►
100ms window -> 1% accuracy or better.
Internal arithmetic accurate to 0.5% or better
QNX Neutrino
Microkernel
User Interface
30%
T= -100ms
Route
Calculation
40%
Data
Acquisition
Diagnostics
30%
T= now
18
Behavior During Normal Load
Blocked
Ready
6
6
7
11
8
4
6
10
9
CPU Budget
Available
Running
CPU Budget
Available

Hard real-time, priority based scheduler under normal load

Running thread selected as highest priority READY thread

No delay on scheduling if adaptive partition has budget
19
Behavior During Overload
Blocked
Ready
6
6
11
8
6
7
4
10
9
CPU Budget
Available
Runs before
higher priority
CPU Budget
Exceeded
Ready – No Budget

Partition budgets are enforced when the CPU is fully loaded

Highest priority READY thread in partition with budget runs

No delay on scheduling if partition has budget
20
Behavior with Free CPU Time
Blocked
Running
Blocked
Ready
6
6
6
6
10
7
11
8
9
CPU Budget
Exceeded
4
8
10
CPU Budget
Exceeded
CPU Budget
Available

If no partitions with remaining budget have READY threads, highest priority
READY thread is selected to run from other partitions

This allows “free” time to be given based upon priority
►
“Free” time is still accounted and may have to be paid back (for example, if partition 3
becomes ready within 1 averaging window)
21
Partition Inheritance

When a server process does work requested by a client, the time is
“billed” to the client

Prevents runaway client processes from monopolizing system services
such as device drivers and server processes

Ensures fair CPU scheduling

Allows you to create servers and assign server budgets independent of
number of clients

Builds on Neutrino micro-kernel and client-server, message passing
architecture
QNX Neutrino
Microkernel
Threads
Threads
File System
Application
Inheritance: File System operation
uses application’s budget
22
Borrowed Time: Critical Threads
Ready
Blocked
6
6
7
8
11
30
6
11
Critical
Thread
4
Running
CPU Budget
Available
CPU Budget
Exceeded

Critical threads still run (based on priority) even if partition has no budget

Critical threads provide deterministic scheduling even in overload

Critical threads are given critical budget and can go into short-term debt
►
Critical time is accounted and has to be repaid
►
Exceeding critical budget is considered an error and causes notification/action
23
Adaptive Partition APIs and Utilities

Control of Adaptive Partitioning Scheduler is done through a kernel
API
►
API is restricted to privileged processes (root)
► Must be called from within default (system) partition
► Partitions are created with budget (normal and possibly critical)

“aps” system utility provided
“aps” utility part of adaptive partitioning package
► Can be used to create and modify partitions
► Also provides usage stats over time
► Use “on” to launch processes into partitions
►

Boot script syntax extended
►
Define partitions within the build file
► Launch processes into specific partitions

Partition configuration completely dynamic
►
Can create partitions, modify budgets at runtime
► Averaging window can also be changed at runtime
24
Getting Started with Adaptive Partitioning
Step 1
Install Adaptive
Partitioning
Step 2
Step 3
Step 4
Build Image
Define
Partitions and
Budgets
Launch
Applications In
Partitions
POSIX
PROGRAMMING
ALLOWED
CODE
CHANGES
25
Summary

SMP is a key enabler for enhancing scalability

SMP delivers measurable performance gains in real-world
applications

QNX provides transparent support for SMP systems

Adaptive partitioning can be used to increase your systems
security and availability

Adaptive partitioning is easy to apply to existing designs and
implementations

Adaptive partition helps you integrate complex systems to
improve time to market
26