Download Effective Test Planning: Preparing for Success

Effective Test Planning:
Scope, Estimates, and Schedule
Presented By:
Shaun Bradshaw
[email protected]
Agenda
• Why Plan?
• What’s in a test plan?
• Define
– Objectives
– Scope
– Priorities
• Estimate work effort
• Develop schedule
Slide 2
Software Peace of Mind in Action
Why Plan?
Planning is
imperfect
• Involves
assumptions &
estimates
• Too many
variables to
control
Slide 3
But…
• Allows us to
think about what
we need
• Determine our
abilities &
constraints
• Develop
roadmap for
activities
Software Peace of Mind in Action
And…
• We are told to
do it
What’s in a Test Plan?
• Most test plan templates available are derived
from IEEE standards
• The template presented here is a minimalist
version of that template
• Generally, a test plan includes:
–
–
–
–
–
–
Objectives
Scope
Risks & Contingencies
Focus / Priorities
Work Effort Estimation
Go/No Go Criteria
Slide 4
Software Peace of Mind in Action
Sections of the Document
Slide 5
Software Peace of Mind in Action
Sections of the Document
Slide 6
Software Peace of Mind in Action
Sections of the Document
Slide 7
Software Peace of Mind in Action
Sections of the Document
Slide 8
Software Peace of Mind in Action
Sections of the Document
Slide 9
Software Peace of Mind in Action
Test Objectives
• Test Objectives should be based on the needs of
the business and established organizational
processes & goals
– Determine short-term and long-term business needs
– Is the business most concerned about qualitative or
quantitative objectives
– How can our test objectives help the business achieve
its objectives
– Test plan objectives should tie into business
objectives
Slide 10
Software Peace of Mind in Action
Test Objectives
Let’s identify test plan objectives related to each of
the following business objectives
Increase customer satisfaction with the application
Improve market reputation and industry recognition
Reduce the number of user calls to the help desk
Ensure key functionality is delivered on time
Slide 11
Software Peace of Mind in Action
Scope
• Identify the scope and how the test effort will
allow the business to meet its objectives
– Determine functionality, processes, and systems to be
validated
– Establish types of testing necessary to perform the
validation
– Identify supporting systems that should be in scope –
if not in scope then document it
– Read available documentation – identify potential
anomalies
– Examine existing applications & processes
– Get involved EARLY!!
Slide 12
Software Peace of Mind in Action
Risk
• Risk is the probability of an undesirable event
occurring and its subsequent impact to a project
Slide 13
Software Peace of Mind in Action
Types of Risk
Technical Risk
Environmental Risk
• Do available resources have
the appropriate technical or
business knowledge to
effectively perform testing?
• Is there an adequate test
environment? Are there
appropriate controls and
access restrictions?
Cost Risk
Schedule Risk
• Do sufficient funds exist to
cover the costs necessary
to complete the testing?
• Is there sufficient time to
complete testing activities?
Slide 14
Software Peace of Mind in Action
Contingency Planning
•
Because risks tend to have a domino effect,
develop contingency plans for each
•
Contingency should align with the
probability and cost of the risk
•
Contingency should address the
impact of the risk
Slide 15
Software Peace of Mind in Action
Prioritization
• Prioritization of application components assists
in determining what functionality has
precedence when organizing the test effort.
• Creates a systematic approach for identifying
what to test first, what can be minimally
tested, and what can be eliminated from
testing, if necessary
• Determine criticality and impact of application
components based on discussions with SMEs
Slide 16
Software Peace of Mind in Action
Prioritization
Consider the following factors for prioritization
• Customer Needs
• Customer Use
• Monetary Value
• Interdependencies
• Historical Fail Rates
Slide 17
Software Peace of Mind in Action
Risk/Impact Analysis
• Use a Prioritization Table to determine which
components have the highest priority
– Set Risk value (probability of failure)
– Set Impact value (impact of failure)
– Calculate Total and sort highest to lowest
Component
Risk
Impact
Total
Priority
Register
3
5
15
2
Search
2
2
4
4
Add to Cart
2
5
10
3
Checkout
4
5
20
1
Slide 18
Software Peace of Mind in Action
Estimating the Work Effort
• Producing estimates using a defined process
provides greater confidence in its potential
accuracy
– Is an “accurate estimate” an oxymoron?
– Think about The Price Is Right
• Following a process for estimation allows us to
review and identify information necessary to
complete the project – it is NOT wasted time
• The problem is not in doing an estimate, but in
the assumption that an estimate is automatically
accurate and therefore a promise!!
Slide 19
Software Peace of Mind in Action
Estimating Size
Estimate size based on amount of functionality in each
component
• Review business rules
• Identify logical branches
• Consider inputs and outputs
• Provides direct correlation to the number of
tests necessary to validate the component
• Use existing documentation, conversations, and
historical data to create estimate
Slide 20
Software Peace of Mind in Action
Estimating Complexity
Some items to consider when determining complexity:
• Real-time vs. batch processing
• Number & types of interfaces/integrated systems
• User interface
• Data model
• OS / platform / technology
• Data quality
The more complex an application, the longer it will
take to analyze, create, and execute the tests.
Slide 21
Software Peace of Mind in Action
Estimating Reliability
Use historical analysis to evaluate the likelihood
that the development process produces complete,
stable, high-quality applications.
Determine reliability based on:
• Defect rates on previous, similar projects
• Responsiveness of developers in correcting
defects in previous projects. Did
corrections result in additional defects?
Slide 22
Software Peace of Mind in Action
Estimation Example
Step 1
Estimate size by counting the number of business rules,
logical branches, inputs and outputs
Component
BusRule
LogBranch
Inputs
Outputs
Totals
Register
5
4
6
4
19
Search
15
25
8
6
54
Add to Cart
13
8
3
25
49
Checkout
54
36
22
8
120
Slide 23
Software Peace of Mind in Action
Estimation Example
Step 2
Estimate complexity & reliability for each component
using a 5-point scale of 0.25, 0.5, 1.0, 1.5, and 2.0.
Note that the scales for complexity and reliability are reversed,
i.e. a 0.25 for complexity indicates very low complexity, but a
0.25 for reliability indicates a highly reliable component.
Component
Complexity
Component
Reliability
Register
.5
Register
.5
Search
1.5
Search
.5
1.5
Add to Cart
1
Add to Cart
Checkout
1
Checkout
Slide 24
Software Peace of Mind in Action
1
Estimation Example
Step 3
Based on historical information or industry standards,
determine:
•
•
•
•
Slide 25
Average TC Analysis & Creation Time 1.5 hours
Average TC Execution Time
1.0 hours
Average Defect Rate
35%
Productive Hours/Day
6.5 hours
Software Peace of Mind in Action
Estimation Example
Step 4
Calculate component estimates
Calculated Factor (CF)
= Size * Complexity * Reliability
Analysis & Creation Time (ACT)
= CF * Avg TC Analysis & Creation Time ÷ Productive Hrs/Day
Execution Time (ET)
= CF * Avg Time to Execute a TC ÷ Productive Hrs/Day
Re-execution Time (RT)
= ET * Avg Defect Rate
Component
ACT
+ ET
+ RT
= Total
Register
1.25
0.75
0.25
2.25 days
Search
9.50
6.50
2.25
18.25 days
Add to Cart
17.00
11.50
4.00
32.50 days
Checkout
27.75
18.50
6.50
52.75 days
Test Schedule
• Develop a work schedule that assigns testing
resources and timeframes to each test activity
and identifies relationships/dependencies
between them
– Schedules help leads determine if the project is
adequately staffed
– Remember to match the experience of your resources
to the level of work necessary, but look for
opportunities to challenge them and cultivate their
skills
Slide 27
Software Peace of Mind in Action
Contingency Time
Time considerations when developing the test schedule
• Holidays
• Special events
• Environment downtime
• Environment setup
• Environment/resource sharing
• Application response time
• Defect correction time
• Team location
Slide 28
Software Peace of Mind in Action
Primary Test Activities
• There are 5 primary test activities
–
–
–
–
–
Test
Test
Test
Test
Test
planning
analysis
creation
execution (includes defect tracking)
summary
• Each of these activities may be combined or
duplicated for each type of test performed
• Each activity should have associated deliverables
identified in the schedule
Slide 29
Software Peace of Mind in Action
Sample Gantt Chart
Slide 30
Software Peace of Mind in Action
Conclusion
• Careful planning lays the groundwork for a
successful test effort by providing opportunities to:
– Learn the scope of the project and the testing needs
– Research and analyze, early in the project, the business
needs and environment
– Identify the relative size, complexity, and reliability of the
application
Slide 31
Software Peace of Mind in Action
Questions
Slide 32
Software Peace of Mind in Action