Download SEPG2006

SEPG Conference Amsterdam 2006
Developing Metrics for agile projects
compatible with CMMI
Graham Collins, UCL
[email protected]
Research supported by Deutsche Bank
Introduction
UCL’s research and projects
 The problems and requests
 EV (Earned Value)
 Agile practices
 Combining EV and agile is it possible?
 Developing suitable metrics compatible with
CMMI
 What works, and what reduces the overheads
 Is this approach likely to lead to CMMI Level 5?
 Key changes to projects

Background - Requests







We would like to use earned value
‘We would like to predict the outcome of project end date,
cost and value’
As a project manager I cannot be there all the time
What is the best way to measure progress with earned
value?
‘As developers we would like to experiment with some
agile approaches’
Your team will be working on other projects as well
We are hoping to achieve metrics suitable for CMMI level
3 and higher…
The
Project
Style
Quadrant
TheDeath
Death March
March Project
Style
Quadrant
high
Kamikaze
Mission
Impossible
Suicide
Ugly
Happiness
low
low
Chance of success
high
Edward Yourdon, Death March:The complete Software Developer’s guide to
surviving ‘Mission Impossible’ projects, 1997 Prentice Hall
Earned Value - Definition
‘The value of the useful work done at any given
point in a project. The value of completed work
expressed in terms of the budget assigned to that
work. A measure of project progress. Note: The
budget may be expressed in cost or labour hours’
APM (Association of Project Management) BoK 2006
Earned Value – Graphical Representation
Planned
Actual
Cost
or
value
Earned Value
Time
Planned (BCWS = Budgeted Cost of Work Scheduled)
Actual (ACWP = Actual Cost of Work Performed)
Earned Value (BCWP = Budgeted Cost of Work Performed)
CPI = Cost Performance Index = BCWP/ACWP (or
Earned/Actual)
SPI = Schedule Performance Index = BCWP/BCWS (or
Earned/Planned) note this is SPIc i.e. cost.
Use of Variance
Variance (Monthly)
1.20
1.10
1.00
Value
0.90
0.80
cpi
0.70
spi
0.60
0.50
0.40
0.30
1
2
3
4
Months
5
6
spic is
used in
this
situation
Earned Value compared to Agile Process Planning
Earned Value
Agile Development
Based on predictive planning
Estimates effort, cost and completion date
End-to-end value tracking
Adaptive planning
Iteration to iteration tracking
Predication of the next iterations effort
Schedule most of the activities
Adaptation to unpredictable events is problematic.
Changes may require the planned to be revised or
baselined
Near the beginning, it is not always possible to
schedule.
Time based iterations allow initial estimate of
duration which can be revised through the adaptive
driven build-feedback cycle
Estimates based on past performance
Estimates are based on progress being made
(velocity)
Change rates often low
Unpredictable change the norm
Small variations in early measurements of cost and
time at the start the project give wide variation in
forward predications
Unknown team development rates
Some organisations do not chart progress for an
initial period
Progress is tracked immediately
Earned value well established
Prioritization of the value of user stories
No earned value approaches in methods
CMMI- Process Areas
Category
Process Area
Project Management
Project Planning
Project Monitoring and Control
Quantitative Project Management
Support
Process and Product Quality
Assurance
Causal Analysis & Resolution
‘Project planning is a necessity for success,
Yet it still ranks on most surveys within the top three
or four problem areas leading to failure.’
Tony Ciorra, Planner’s Progress, Project, APM May 2006
CMMI Comparative Advantages
Continuous Representation
Staged Representation
Grants explicit freedom to select the order of
improvement that best meets the organization’s
business objectives and mitigates the
organisation’s areas of risk
Enables organisations to have a predefined
path
Enables increased visibility of the capability
achieved in each individual process area
Focuses on a set of processes that provide an
organization with a specific capability that is
characterized by each maturity level
Provides a capability-level rating that is used
primarily for improvement in an organisation and
is rarely communicated externally
Provides a maturity-level rating that is often
used in internal management communication,
statements external to the organization, and
during acquisitions as a means to qualify
bidders
Allows improvements of different processes to
be performed at different rates
Summarizes process-improvement results in a
simple form – a single maturity-level number
Reflects a newer approach that does not yet
have the data to demonstrate its ties to return on
investment
Builds on a relatively long history of use that
includes case studies and data that
demonstrate proved return on investment
Agile Manifesto
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items
on the left more
Several agile projects have
achieved CMMI level 3,
example David Anderson,
Stretching Agile to fit CMMI
Level 3, Agile Conference 2005
The Agile Principles
www.agilealliance.com
Our highest priority is to satisfy the customer
through early and continuous delivery of
valuable software
Agile processes promote sustainable
development
Welcome changing requirements, even
late in development. Agile processes harness
change for the customer competitive
advantage
The sponsors, developer, and users should
be able to maintain a constant pace
indefinitely
Deliver working software frequently, from a
couple of weeks to a couple of months, with
a preference to the shorter time scale
Continuous attention to technical
excellence and good design enhances
agility
Business people and developers must
work together daily throughout the project
Simplicity - the art of maximizing the
amount of work done – is essential
Build projects around motivated
individuals. Give them the environment and
support they need, and trust them to get the
job done.
The best architectures, requirements, and
designs emerge from self-organizing teams
The most efficient and effective method of
conveying information to and within a
development team is face-to-face
conversation
At regular intervals the team reflects on
how to become more effective, then tunes
and adjusts its behaviour accordingly
Working software is the primary measure
of progress
Iterative Development (Bittner-Spence)
1.
2.
3.
4.
5.
6.
Agree with the team the objectives for the iteration,
including evaluation criteria, timescales, and constraints
Agree on a plan for how the team will achieve the
objectives
Execute the plan
Assess the achievements of the team against the initial
set of objectives and evaluation criteria
Assess the impact of the iteration’s results on the project
as a whole
Start the next iteration.
What is iterative development? Part 3: The management perspective 15 May
2005
www-128.ibm.com/developerworks/rational/library/may05/bittnerspence/index.html
Fundamental shift in measurement
100%
Progress ( %
complete
measured in
scenarios
0%
Iteration 1
coded
2
3
tested
4
Tested &
Passed
Developer Perspective





Developers are less interested in the business value,
benefits realization and return on investment
They work on a small number of requirements or change
requests from their list of outstanding work
They anticipate a decreasing number of requirements and
change requests as the product is developed
Outstanding requirements and change requests is termed
the product backlog
The developer will therefore be aware of progress via
work completed, product backlog and new work allocated.
User Satisfaction driving Development
Release
User satisfaction
Release
planning
Iteration
User
satisfaction
Iteration
planning
Development
Increment
Iteration Planning (Goal identification, story selection, task
estimation, team commitment)
Project teams need to adopt some attributes
What is the purpose?
Community of
practice
Project team
To develop members’
capabilities; to build and
exchange knowledge
To accomplish a
specified task
What holds it together?
Passion, commitment,
and identification with
the group’s expertise
The project’s
milestones and
goals
Adapted from: Communities of Practice: The organizational Frontier, Etienne C.
Wenger and William M. Snyder, Harvard Business Review p139-145 Jan-Feb 2000
Rate of work - velocity
Velocity
50
45
40
Story Points
35
30
25
20
15
10
5
24
/0
2/
06
17
/0
2/
06
10
/0
2/
06
03
/0
2/
06
27
/0
1/
06
20
/0
1/
06
13
/0
1/
06
06
/0
1/
06
0
Velocity, gives an indication of the average rate of work
and also a comparison of planned against delivered, each
iteration
story points
weeks
mR
24/02/2006
20
17/02/2006
10/02/2006
03/02/2006
27/01/2006
20/01/2006
13/01/2006
06/01/2006
story points
Individuals and Moving Range (XmR) Charts
Velocity
60
50
UNPLx
40
30
UCLR
10
0
Control Limits for XmR Charts
k sequential measurements provide k-1 =r (two-point) moving range
values
ith moving range = mRi = │Xi+1 – X i │where integer i is 1 ≤ i ≤ k – 1
__
i=r
Individuals average moving range =mR = 1 ∑ mRi
r i=1
_
__ _
__
Upper Natural Process Limit =UNPLx= X + 3mR = X + 2.660mR
d2
_
i=k
Centerline = CLx = X = 1 ∑ Xi (average of individual values)
k i=1
_
__ _
__
Lower Natural Process Limit =LNPLx= X - 3mR = X - 2.660mR
d2
___
Centerline or average moving range =UCLR = mR
__
__
Upper Control Limit for moving range =UCLR= D4mR = 3.268mR
__
Sigma for individual values = sigmax (σ) = mR
d2
When n=2 d2 =1.128and D4 =3.268 (from Dispersion and Bias factor tables)
‘Under Control’
Velocity
60
UNPLX
story points
50
40
30
20
LNPLX
10
24/02/2006
17/02/2006
10/02/2006
03/02/2006
27/01/2006
20/01/2006
13/01/2006
06/01/2006
0
weeks
story points
Velocity measures of work rate are useful in that estimates of
the next iteration can be planned in a rolling process
The use of σ variation is supportive in this aim
Automated colour coding (Red Amber Green) can be used to
show condition requirements
‘Weighted average’
Story
points
Velocity
30
25
weighted average
20
15
Weighted
points
0
1
0
5
2
10
18
3
54
14
4
56
24
5
120
27
6
162
14.66667
3.5
19.14286
UNPLx
35
story points
count
centreline (CLx)
10
5
0
1
2
3
4
5
6
iterations
50
45
40
35
30
25
20
15
10
5
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
24/02/2006
17/02/2006
10/02/2006
03/02/2006
27/01/2006
20/01/2006
13/01/2006
06/01/2006
Story Points
‘Burn-down’
Story Points Remaing
350
300
250
200
150
100
50
0
With the appropriate metrics we can improve
Acceptance Test Data
100
90
80
ATs
70
60
Acceptance Tests
50
Failing ATs
40
Passing ATs
30
20
10
0
1
2 3 4
5 6
7 8
Iteration
9 10 11 12
Use of Multipliers
Iterations
Completed
Low Multiplier
High Multiplier
1
0.6
1.60
2
0.8
1.25
3
0.85
1.15
4 or more
0.90
1.10
Multipliers for estimating velocity based on number of iterations completed from Cohn 2006
Charts and Metrics


Velocity and Burn-down
Cumulative acceptance tests
Inventory
Failing
Passing

Cumulative Issue Charts
Backlog - Active issues (which show inventory line)
Resolved issues
Closed issues

Earned Value
EV progress charts
Performance via cpi and spi
Cpi and spi combined with control charts
Earned Value



EV can be applied to estimates of agile projects
- this is complex if more stories are added as the work
progresses
EV may need to be shown to senior managers
- who are used to EV figures, or comparison to other
projects where EV figures have been tracked
EV estimates can be accurate
- story points tend to remain static in an iteration when the
process is understood by managers and developers.
When additional stories are added, stories with lower
business priority level may be dropped to compensate
and keep the work load (story points) similar.
Business Value
More importantly business value (or contribution) should be
considered and evaluated
Often units of measure such as story points can be valued
as 0.5 or 1.0 units
The key issue in agile project management is to continually
assess with the client the most important work that should
be done.
Story
number
‘Business
Value’
Story
Points
‘Points
earned’
Planned
(developer
hours)
Actual
(logged
hours)
EV
(earned
value)
1
3
10
10
100
120
100
2
2
8
8
60
60
60
3
2
4
4
60
80
60
4
1
2
0
20
0
0
total
8
24
22
240
260
220
Conclusions







EV can be combined with agile reporting
The most useful approach to achieve process improvement is
via understanding of process control, the basis for CMMI
The use of acceptance tests is the most useful approach for
both methods and is the basis for metrics outlined
EV remains a viable approach where story points work load is
maintained
EV is useful for reporting to senior managers and at a
programme and project level, but even with major scope
changes, re-planning can give some useful insights
The process used in agile project management is critical, that
this should be adhered to in the team. It needs to be explained
to managers that the team are trying to achieve maximum
business value per iteration, and that the final goal and plans
evolve during the process
Tracking suitable metrics and understanding variation is the key
to better estimation and process improvement – CMMI
assessments can help achieve this goal.
Key Project Changes
1. Pair reporting - valuing individuals and team,
moving towards self-determining teams
2. Acceptance testing - working software
3. Ensuring Business Value – continual prioritisation,
estimation and understanding there is a cost to
development
4. Measuring progress over shorter time periods meeting the needs of process improvement CMMI,
velocity tracking in agile methods, and better EV
planning
Individuals and interactions
Working software
over processes and tools
over comprehensive documentation
Customer collaboration
over contract negotiation
Responding to change
over following a plan