Download Design Flow for Embedded System Device Driver Development and

Design Flow for Embedded System Device Driver
Development and Verification
Ross Dickson
Virtutech, Inc
[email protected]
Jason Andrews
Cadence
[email protected]
Jakob Engblom
Virtutech, AB
[email protected]
Context and Motivation
•
Bugs in device drivers are common
– The majority of embedded system bugs are attributed to software
• Possibly because software is less expensive to fix
•
Cadence has proven utility of constrained random testing in the
Incisive Software Extensions to provoke bugs
Virtutech has proven the efficiency of the Simics Virtual Platform to
nonintrusively isolate and identify bugs
A combined environment provides the embedded device driver
writer with an improved environment to provoke, isolate, and
identify bugs in device drivers for embedded systems
A driver for a hardware accelerator for the Rule 30 Cellular
Atomata on a virtual Freescale MPC8641D SoC running Linux
version 2.6.23 is used as a running example
•
•
•
2
Copyright @ 2009 Virtutech, Cadence
Software Setup
Virtual MPC8641D Board
Virtual MPC8641D Board
MPC8641D SoC
Test program
exercising
device driver
Busybox
Other
program
Device
driver under
test
Virtual serial
console
UART
Linux 2.6.23 kernel
CPU cores 0 and 1
Ethernet
Virtual
network
MemCtrl
Timers
RAM
MPIC
PCI
Device model
under test
PCIe
Simics
Host operating system
Host computer
3
Copyright @ 2009 Virtutech, Cadence
Verification Planning
•
Determine the items that must be exercised and
monitored
– Software
• the software functions to be called
• their input parameters, and their outputs
• Variables and data structures to be monitored
– Hardware
• Registers modified
• Interrupts triggered
•
Examples:
– Software: device driver functions
• open and write
– Software: kernel functions
• interrupt handlers.
4
Copyright @ 2009 Virtutech, Cadence
Rule 30 Verification Plan
Create
CreateVerification
VerificationPlan
Plan
Describe
Software to be
Exercised
• Done using Word
• Saved as XML file
5
Copyright @ 2009 Virtutech, Cadence
Introduction to Coverage Driven Verification
Automatic
Generation
Constrained
Randomization
+
Coverage
Measurement
=
Coverage-Driven
Verification
Metrics
Define Verification Goals and Let Machines do the Work
6
Copyright @ 2009 Virtutech, Cadence
Directed Testing
An engineer writes directed test for each item in Test Plan:
Design
B1
B1
B1
B1
B2
B1
B3
It’s work determine and follow the paths
It’s work to verify the goal was reached
Poor coverage of non-goal scenarios
7
Copyright @ 2009 Virtutech, Cadence
Redo if design
changes
…so directed tests are not enough
• Tedious to write and maintain
– Tens of thousands of lines of test code
– Reuse to get to state quickly causes poor new coverage
– Design spec changes cause delays
• Difficult to think of all required verification cases
– Many different ways to reach a certain state
• Can't implement all identified tests in test plan within
project schedule
8
Copyright @ 2009 Virtutech, Cadence
Where did CDV Come From?
• Automated
Constrained
Random
Stimulus
Can the same
technology
be used
for Generation
embedded software?
• Coverage Collection (Functional, Assertion, Code)
• Can
Independent
& Data Checking
it be usedAssertion
by an embedded
software verification engineer?
Automated
Generation
Automated
Test Scenario
9
Scoreboard
Scoreboard
Data
Data Check
Check
Coverage
seed
23098432
38u48932
23432239
17821961
10932893
20395483
18902904
23843298
23432432
24324322
55252255
09273822
13814791
4098e092
23432424
24242355
25262622
26452454
24524522
Self
Self Checking
Checking
Monitors
Monitors
BFM
test
sw
BFM
test
sw
Coverage
Self
Self Checking
Checking
Monitors
Monitors
Stimulus
Stimulus
stimulus
stimulus
Linux
Linux
Scenarios
Scenarios
Scenarios
Scenarios
Device
Device
Driver
Driver
Copyright @ 2009 Virtutech, Cadence
Software Setup with Coverage Driven
Verification
Set test
program
parameters
Virtual MPC8641D Board
Virtual MPC8641D Board
MPC8641D SoC
Test program
exercising
device driver
Busybox
Other
program
Coverage
Driven
Verification
…
Device
driver under
test
Virtual serial
console
UART
Linux 2.6.23 kernel
CPU cores 0 and 1
Ethernet
Virtual
network
MemCtrl
Observe
hardware/
software
interface
Verification
Planning &
Management
Timers
RAM
MPIC
PCI
e
Tests
Helper
Device model
under test
PCIe
Change
hardware
parameters
vPlan
Simics
ISX
Simics
Host operating system
Host computer
10
Copyright @ 2009 Virtutech, Cadence
Rule 30 Driver Execution and Coverage
Analysis
Run
Runsequences
sequencesand
andcollect
collect
coverage
coverage
Generated
Generatedsequences
sequenceswith
with
random
data
random data
Achieved
AchievedCoverage
Coverage
11
Copyright @ 2009 Virtutech, Cadence
Rule 30 Driver Regression and Results
Automated
AutomatedExecution
Executionand
andAnalysis
Analysis
• Reads XML file
• Analyzes Results
12
Copyright @ 2009 Virtutech, Cadence
Rule 30 Results
•
Coverage
– Ability to measure hardware model parameters
• time_to_result for algorithm completion time to make sure
Linux driver is timing independent
– Ability to measure generated stimulus
• Length of rule30 line
– Ability to measure coverage on software
• Mode of device open() system call: read, write, read/write
• Return values from system calls
– Ability to cross all of the above and find out, “Did I test short
line length with fast hardware response time?”
13
Copyright @ 2009 Virtutech, Cadence
Rule 30 Results
•
Bug Hunting: hitting corner cases not thought writing
manual tests
– Operations out of order
– Parameters outside of expected values
– Memory corruption: “serial8250: too much work for irq42”
•
Learning
– Understanding driver behavior
– When does a read block waiting for results
– How interrupts work
14
Copyright @ 2009 Virtutech, Cadence
Repeatability and Reverse Debugging
•
Repeat any run trivially
– No need to rerun and hope
for bug to reoccur
•
On
On hardware,
hardware, only
only
some
some runs
runs
reproduce
reproduce an
an error
error
Stop & go back in time
– Instead of rerunning program
from start
– Breakpoints & watchpoints
backwards in time
– Investigate exactly what
happened this time
•
This control and reliable
repeatability is very powerful
for parallel code!
On
On virtual
virtual hardware,
hardware,
debugging
debugging is
is much
much
easier
easier
15
Copyright @ 2009 Virtutech, Cadence
Thank You
What Virtutech Does
•
Provider of Simics: a high-performance, high fidelity, full system
simulator
– High Performance – fast enough to run real software loads (typically
100’s of MIPS, up to multiple GIPS)
– High Fidelity – run full production software, including firmware, device
drivers, hypervisor, RTOS/OS, application software
– Full System – simulate entire systems, not just processor cores, or
SoCs, or boards
• Complete machines, networks, backplanes
•
17
The true value of Simics is through enablement of process change:
Virtualized Software Development
Copyright @ 2009 Virtutech, Cadence
Traditional Product/Project Development
Pre-Silicon
Testing
Post-Silicon
Testing
System
Integration
start time and duration
Hardware
Development
es
p
y
t
oto
r
P
Product
Specification
Board Bring-up
18
Software
Development
Platform
Development
Production,
Manufacture
Marketing /
Feature
Validation
System
Test
Application
Development
Copyright @ 2009 Virtutech, Cadence
End User
Documentation
& Translation
Agility, Higher Quality, Faster time to Market
True Iterative
Development
Production,
Manufacture
System
Bring-up
Hardware
Development
Marketing /
Feature
Validation
Software
Development
Product
Specification
System
Test
End User
Documentation
& Translation
19
Copyright @ 2009 Virtutech, Cadence
What is a Virtual Platform?
•
•
•
•
A piece of software
Running on a regular PC,
server, or workstation
Functionally identical to a
particular hardware
Runs the same software as
the physical hardware system
Virtual HW
20
Copyright @ 2009 Virtutech, Cadence