Download Project Oversight Program - PMI-SVC

Building an Effective
Oversight Program
Cindy Blehm, PMP
July 23, 2008
®
Agenda


Why Build an Oversight Program?
Objectives and Goals for Developing an Oversight
Program





Expected Benefits and Critical Success Factors
Defining Process Scope and Technical Scope
Examples of Oversight Program Development
Walk Through of Common Oversight Program
Deliverables
Next Steps


Training
Enterprise Oversight
Background: Project Oversight

According to the OCIO Independent Information
Technology Project Oversight Framework (ITPOF),
project oversight is defined as “an independent
review and analysis … to determine if the project is
on track to be completed within the estimated
schedule and cost, and will provide the functionality
required by the sponsoring business entity. Project
oversight identifies and quantifies any issues and
risks affecting these project components.”

The OCIO ITPOF is primarily concerned with:



Ensuring that independent oversight is in place at project
start up for reportable projects
Identifying issues/risks and ensure they are acted upon by
all project teams
Providing an independent assessment of project health
Background: Objectives

In order to meet these objectives, many departments
are beginning to implement centers of excellence for
providing and monitoring oversight of reportable IT
projects in order to:






Respond to an audit of oversight or project management
practices;
Assess the Department’s current oversight-related practices,
including standardizing the acquisition of consulting services
required to perform IPOC and IV&V services;
Define and formally establish independent project oversight;
Bring oversight in-house for low and medium criticality
projects;
Develop and implement standards and processes for
preparing IPORs and IV&V reports; and
Develop and implement a standard model for oversight
consulting services.
Background



CDCR Project Oversight Program (POP)
DMV Enterprise Oversight Program (EOP)
FTB Project Oversight Group (POG)
Development and Execution of an Oversight Program is a
continuous service improvement program focused on achieving
the following program objectives:



Improve the maturity and execution of project oversight for low,
medium and high criticality projects reportable to OCIO;
Improve the effectiveness of oversight in project management
processes; and
Standardize processes across multiple types of project categories to
improve consistency in the execution and reporting of the oversight
function.
Summary

Establish a center of excellence for:




Managing oversight contracts within the
department
Assigning internal oversight staff to low and
medium criticality projects
Establishing standards, processes and procedures
to guide oversight in their activities
Providing auditing and tracking of the oversight
function to ensure consistency and continued
improvement.
Defining Process Scope




A Project Oversight Program should encompass oversight for all
nine of the Project Management Institute’s (PMI) Project
Management Body of Knowledge (PMBOK) process management
areas addressing both the review of the planning document as
well as the execution of the plan.
It is important to understand that in order for oversight
consultants and staff to retain their “independent” status, the
oversight boundaries must be clearly defined.
Review, assessment, evaluation, guidance and
recommendations on project management are all clearly within
process scope of an Oversight Program. However, development
of those plans and the actual execution of project management
activities are not within the boundaries.
The following chart represents the PMBOK process areas and
the corresponding oversight activities that are in scope and out
of process scope for an Oversight Program.
Scope: Integration Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Integration Management – requires each project and
product process to be appropriately aligned and
connected with other processes to facilitate their
coordination. Oversight responsibilities in this
process area include:
Integration Management – The scope of project oversight
does NOT include the following for this process area:







Assessment of Project Charter
Assessment of preliminary project scope statement
Assessment of the Project Management Plan
Assessment of the Change Control Plan
Assessment of how the project monitors and controls
project work
Assessment of the project’s integrated change control
process
Assessment of project close out activities



Direct and Manage project execution
Development of any of the integration plans (project
charter, scope statement, PMP, or Change Control
Plan)
Definition of scope
Scope: Scope Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Scope Management - includes the processes required to
ensure that the project includes all the work required,
and only the work required, to complete the project
successfully. Oversight responsibilities in this area
include:
Scope Management – The scope of project oversight does
NOT include the following for this process area::
•
•
•
•
Review, assessment, guidance and recommendations
for scope planning, scope definition, scope
verification and scope change control activities
Review and assessment of the Work Breakdown
Structure (WBS)
Review and assessment of the Scope Management
Plan
Review and assessment of the tactical implementation
and execution of the Scope Management Plan
•
•
Development of the WBS or Scope Management plan
Creation of the Scope Change Control process and/or
Scope Management methodology or processes
Scope: Time Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Time Management – includes the processes required to
accomplish timely completion of the project.
Oversight responsibilities in this area include:
Time Management – The scope of project oversight does
NOT include the following for this process area:
•
•
•
Review, assessment, guidance and recommendations
for time activity definition, activity sequencing,
resource estimating, activity duration estimating, and
schedule control
Review and assessment of the project schedule and the
processes and methodologies supporting updates to the
schedule
Creation of the project schedule or supporting
processes
Scope: Cost Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Cost Management – includes the processes involved in
planning, estimating, budgeting and controlling costs
so that the project can be completed within the
approved budget. Oversight responsibilities in this
area include:
Cost Management – The scope of project oversight does
NOT include the following for this process area:
•
•
•
•
•
Review, assessment, guidance and recommendations
on cost estimating, cost budgeting and cost control
processes
Review and assessment of Cost Management Plan
Review and assessment of the tactical implementation
and execution of the Cost Management Plan
Review and assessment of the process for creating the
cost baseline for the project
•
•
Creation of the Cost Management Plan
Creation of Cost Management Methodologies and/or
processes
Creation of the cost baseline
Scope: Human Resources
Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Human Resources Management – includes the processes
that organize and manage the project team. Oversight
responsibilities in this area include:
Human Resources Management – The scope of project
oversight does NOT include the following for this
process area:
•
•
•
•
•
Review, assessment, guidance and recommendations
on
human
resources
planning,
acquisition,
development and management of the project team
Review and assessment of the Staffing Management
Plan
Review and assessment of the tactical implementation
and execution of the Staffing Management Plan
Performing actual staff acquisition
Development of the Staff Management plan or
supporting staffing management methodologies
Scope: Communications Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Communications Management – employs the processes
required to ensure timely and appropriate generation,
collection, distribution, storage, retrieval and ultimate
disposition of project information.
Oversight
responsibilities in this area include:
Communications Management – The scope of project
oversight does NOT include the following for this
process area:
•
•
•
•
Review, assessment, guidance and recommendations
on communication planning, information distribution,
performance reporting and administrative closure
Review and assessment of the Communications
Management Plan
Review and assessment of the tactical implementation
and execution of the Communications Plan
Review and assessment of project status reports,
variance
assessment,
and
project
analysis
responsibilities
•
•
Creation of the Communication Management Plan
Actual execution of project communication
Scope: Quality Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Quality Management - include all of the activities of the
performing organization that determine quality
policies, objectives, and responsibilities so that the
project will satisfy the needs for which it was
undertaken. Oversight responsibilities in this area
include:
Quality Management – The scope of project oversight
does NOT include the following for this process area:
•
•
•
Review, assessment, guidance and recommendations
on quality planning, quality assurance and quality
control processes and procedures on the project
Review and assessment of the Quality Management
Plan
Review and assessment of the tactical implementation
and execution of the Quality Management Plan
•
•
Creation of the Quality Management Plan
Creation of the Quality Management methodology to
be used on the project
Scope: Risk Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Risk Management – includes the processes involved with
conducting risk management planning, identification,
analysis, responses and monitoring and control on a
project. Oversight responsibilities in this area include:
Risk Management – The scope of project oversight does
NOT include the following for this process area:
•
•
•
Review, assessment, guidance and recommendations
on risk management planning, risk identification, risk
qualification and quantification, risk response
planning and risk monitoring and control
Review and assessment of the Risk Management Plan
Review an assessment of the tactical implementation
and execution of the risk management plan
•
•
•
Creation of Risk Management Plans
Creation of Risk Mitigation Strategies
Creation of Risk Management methodology
Scope: Procurement Management
Within the Scope of an Oversight Program
Outside the Scope of an Oversight Program
Procurement Management – includes the processes to
purchase or acquire the products, services or results
needed from outside the project team to perform the
work. Oversight responsibilities in this area include:
Procurement Management – The scope of project
oversight does NOT include the following for this
process area:
•
•
•
•
•
Review, assessment, evaluation, guidance and
recommendations
on
procurement
planning,
solicitation planning
Review and assessment of the Procurement Plan
Review and assessment of the tactical implementation
and execution of the Procurement Plan
Review and assessment of SOWs
•
•
Review, assessment, evaluation, guidance and
recommendations on the actual solicitations, source
selection,
contract
negotiations
or
contract
administration
Creation of the Procurement Plan
Creation of SOWs
Defining Technical Scope




A Project Oversight Program should encompass the full Software
Development Life Cycle (SDLC) components associated with
IEEE Standard 1012-2004 addressing both the review of the
technical documentation as well as the actual technical product.
It is important to understand that in order for oversight
consultants and staff to retain their “independent” status, the
oversight boundaries must be clearly defined.
Review, assessment, evaluation, guidance and
recommendations on the technical aspects of a project are all
clearly within the scope of an Oversight Program. However,
development of those plans and the actual execution of
technical activities are not within the boundaries.
The following represents the SDLC processes that are in
technical scope for an Oversight Program.
Technical Scope








Requirements: Database, Interface, Hardware, Software, and Architectural
Requirements including Network, Security, Programming, and Disaster Recovery
Detailed Design including: Database, Interface, System, Hardware, and
Software Designs
Build including Entity Relationship Diagrams and Data Dictionary
Configuration Management
Testing including: Test and Evaluation Master Plan for all testing components,
Test scripts and test cases, Test results for Integration, System, Load, and User
Acceptance testing, Test Summary Report
Data Conversion
Training and Implementation including Training Plan, Training Materials,
Knowledge Transfer Materials, Technical System Documentation, Cutover
Strategy Document, Customer Documentation, Implementation Plan, Disaster
Recovery Plan, Business Continuity and Resumption Plan
Traceability analysis for the following documents:




Requirements from various high level sources such as Feasibility Study Reports (FSRs)
or the Request For Proposal (RFP) compared to detailed sources such as the SyRS
SyRS to DDS
DDS to Unit Test Cases
SyRS to System Test Cases
Critical Success Factors














Ensure regular meetings with the project managers in order to facilitate notification
of project changes
Monitor progress of issue/risk mitigation actions
Require artifacts to substantiate project status findings
Set up minimum requirements for notification of project start-up and changes in
order to allow for the contract process timeframes
Implement regular reporting between the oversight program and Executive
Management
Improve the current oversight SOW and RFO process
Procure consultants, formally establish oversight, develop standards, and train
Create and obtain executive management approval of an oversight program charter
Establish escalation and resolution processes for addressing project risks identified by
independent oversight
Develop and implement a standard model for oversight consulting
Develop and implement standards and processes for preparing oversight reports
Train oversight staff
Implement an oversight training and expectation program for oversight customers
Develop and establish quality assurance processes
Benefits of an Oversight Program


The biggest benefit of an Oversight Program is that project
managers will be better informed, expectations for content of
deliverables will be better defined, and project teams will be
more aware of what project oversight expects when reviewing
project documentation and process execution.
The department will reap the benefits of having standardized
statement of work templates for various types of oversight
contracts, which will improve the consistency between oversight
vendors on various projects.


It is difficult to hold project teams accountable for compliance to
standards or best practices, or to project processes, when clear
expectations have not been defined or implemented.
The Oversight program seeks to establish a standard baseline
by which all projects manage with regards to the level of
oversight provided.
Example: CDCR POP
POP Goals
POP Objectives
Implement a standard model for project oversight
Develop standards and procedures for the oversight of
technical and project management processes.
Develop a standard Statement Of Work (SOW) for low,
medium and high projects across basic categories
Improve risk assessment and risk control processes for
risks identified by project oversight
Improve communication channels between the oversight
program and executive management
Implement monthly meetings with the Oversight Program
Manager and Executive Management to discuss
goals/objectives, risks/concerns and outcomes of oversight
Implement audit and tracking mechanisms for risk and
issue management
Example: CDCR POP
POP Goals
POP Objectives
Develop and implement quality assurance processes



Establish criteria for evaluation of project
management process assessments based on ITPOF,
PMI and IEEE standards for compliance to standards
and best practices
Implement training for all internal oversight staff in
the core competency areas (cost, schedule, risk,
quality and change management)
Establish criteria for evaluation of technical process
assessments based on ITPOF, IEEE and best practices
Oversight Program Deliverables
Deliverable Name
Deliverable Description
Project Oversight Process Assessment
A new project oversight model that assesses the baseline of the department’s
existing project oversight model, processes and procedures. The new model
should base its recommendations on the comparison of the department’s
current model to other successful project entities surveyed, and
recommended improvements to the oversight processes, practices, and
principles by including them in the new oversight model.
Program Oversight Charter
Documents the goals and expectations that will guide the Department in the
implementation of an oversight model. This document will be used to
educate the department staff in the purpose of the Oversight Program,
document expectations for roles and responsibilities, identify channels of
communication, and gain executive management support and approval for
implementation, and any subsequent changes that arise during the transition
period.
Oversight Program Deliverables
Deliverable Name
Deliverable Description
Recommended Project Oversight
Consulting Deliverables
A model that enables CDCR to obtain project oversight consulting services from
outside consultants as required to comply with the ITPOF and SAM. The
model identifies standardized deliverables to include in a Statement of Work
(SOW) for multiple types of project categories.
Project Management Principles and
Procedures
Develops guidelines which define the process for performing project management
oversight throughout the entire project life cycle, and addresses the
principles of performing oversight and the guidelines to be applied to levels
of oversight to provide the most effective value to the project.
Oversight Program Deliverables
Deliverable Name
Deliverable Description
Technical Oversight Principles and
Procedures
Develops guidelines which define the process for performing technical oversight
throughout the entire project life cycle, and addresses the principles of
performing IV&V and the guidelines to be applied to levels of oversight to
provide the most effective value to the project.
IPOR Preparation Guidelines
Processes and procedures describe how an oversight analyst should complete an
IPOR, including attachments F and G for all levels of criticality projects.
Oversight Contract Management
Process
Processes and procedures describe monitoring and overseeing the services of an
IPOC and IV&V consultant.
Approach & Methodology







Project Oversight Process Assessment
Development of Program Oversight Charter
Recommended Project Oversight Consulting
Deliverables
Project Management Oversight Principles and
Procedures
Technical Oversight Principles and Procedures
IPOR and IV&V Report Preparation Guidelines
Contract Management Process Guidelines
Estimated Timeline
Quarter 1
1
2
3
Quarter 2
4
5
6
Quarter 3
7
8
9
Quarter 4
10 11 12
Quarter 5
13 14 15
Quarter 6
16 17 18
Quarter 7
19 20 21
Quarter 8
22 23 24
Project Oversight Process
Assessment
Project Oversight Program
Charter
Recommended Project
Oversight Consulting
Deliverables
Project Management
Oversight Principles and
Procedures
Technical Oversight
Principles and Procedures
IPOR and IV&V Report
Guidelines
Contract Management
Process Guidelines
Basic Staff Training
Development of Oversight
Office
Project Management Fundamentals Training
Knowledge Transfer Activities
Develop Communication Plan, Risk
and Issue Management Plan, Staffing
Management Plan and Quality Plan
Develop Quality
Assurance
Processes &
Procedures
Project Oversight Training
Develop Control Pairing up external PMs
Process:
and oversight
Reporting, Metrics consultants w ith State
and Compliance
staff
Assignment of
Low and Medium
criticality projects
to internal
oversight
Project Oversight Process Assessment

Develop the Oversight Baseline







Oversight
Oversight
Oversight
Oversight
Determination
Procurement
Process
Reporting
Best Practices and Proven, Successful Projects
Current Process Observations
Recommendations for Improvement
Oversight Process Assessment
Methodology and Approach
Design To-Be
Process
Recommend
Areas for
Improvement
Map & Analyze
As-Is Process
▪ Interview
stakeholders and
assess current
process
▪ Analyze current
process/procedure
▪ Identify Roles &
Responsibilities
▪ Identify
disconnects &
value add
processes
▪ Benchmark
current processes
▪ Develop
observations
against the “as-is”
model
▪ Prioritize the
observations
▪ Develop
recommendations
for improvement
▪ Create project
management and
technical
methodologies,
templates and
procedures
▪ Design To-Be
Processes
▪ Prioritize Gaps
▪ Validate To-Be
Processes
Implement
Reengineered
Processes
▪ Develop
Implementation
Plan
▪ Develop
Transition Plan
▪ Initiate Training
Programs
▪ Solicit Executive
Support
▪ Implement
Transition Plan
Improve
Continuously
▪ Initiate on-going
measurement
▪ Review
performance
against target
▪ Improve process
continuously
Development of the Program
Oversight Charter

Define the Purpose of the Oversight Program



Benefits, Goals and Objectives, Critical
Success Factors, Assumptions and Constraints
Program Organization




Background, Program Scope, Organization Scope,
Process Scope and Technical Scope
Roles and Responsibilities
Timelines
Deliverables
Program Approach and Methodology
Oversight Roles & Responsibilities
External Reporting Agencies
(OCIO, Agency and State
Legislature)
Project Oversight Program
(POP)
Oversight
Stakeholders
Department Executive
Management
Oversight Management
Team
IPOCs
(internal & external)
IV&V
(internal & external)
Project Stakeholders
PMO Management &
Project Managers
Business & Technical PMs
Sponsors & Project Team
Vendor Project Managers
Recommended Oversight
Deliverables


Recommended Content for the Oversight
SOW and Standard Template
Determination of Project Criticality and
Budget




Determining Project Criticality: Project Size, PM
Experience, Team Experience, Project Type
Determining the Type of Oversight needed
Determining the Oversight Budget
Recommended Deliverables to be Included in
the SOW For Low-Medium-High criticality projects/IPOC and IV&V
Medium - IPOC - PMP
Deliverable Title:
Project Management Plan (PMP) Assessment
Deliverable
Description:
Review, assess, document findings and provide recommendations for improvement on the
Project Management Plan for the project
Value-Add to Project
/ Department
The Project Management Plan is one of the most critical plans associated with the project,
especially a medium criticality project. IPOC will review this plan and ensure that it
adequately lays the foundation for managing a project of the size and complexity of the one
in question, and most importantly, lays a roadmap for how the project will manage the daily
activities and processes.
During the procurement phase, the Project Management Plan should focus on how the
procurement activities will be managed. At this stage in the project, the Project
Management Plan can contain high-level subsections for each of the other process areas
that touch on how issues, risks, communication, quality and staffing will be managed during
the procurement phase. Specifically, the PMP for a medium criticality project at the
procurement phase must address how the procurement activities will be managed in the
following areas:

Requirements Management

Communication Management (including a project org chart)

Issue and Risk Management

Change Control / Configuration Management

Quality Management

Staffing Management
The PMP will need a major revision once the project starts and the vendor comes on board.
The revised PMP should focus on how the actual project will be managed, and for a
medium criticality project separate sub-documents should be created for each of the project
management process areas. At that time, IPOC will re-evaluate the newly focused PMP.
The Project Management Plan review by IPOC is a critical component of the IPOC contract. A
formal Anomaly Report is required for this plan.
Medium - IPOC - PMP
Frequency:
Once during
Procurement and
once during
Analysis and
Planning
Periodic update review
to ensure
consistency
Timeframe:
When available
Phases affected:
Procurement
Planning &
Analysis
Medium Criticality
Deliverables
IPOC Services
IV&V for
Custom
Developed
Software
IV&V for
COTS
projects
IV&V for
network
impleme
ntation
Combined
IPOC/IV&V
services
X
X
X
X
X
Procurement Phase
Solicitation Documentation Assessment
Evaluation Plan Assessment
Requirements Assessment
Traceability Assessment
Project Management Plan Assessment
X
X
Project Schedule Assessment
X
X
Planning & Analysis Phase
System Requirements Specification
Assessment
X
X
X
X
Traceability Assessment
X
X
X
X
Fit-Gap Assessment
X
Project Charter Assessment
X
Project Management Plan Assessment
X
X
X
X
X
X
X
Medium Criticality
Deliverables
Project Schedule Assessment
IPOC
Services
X
IV&V for Custom
Developed
Software
X
IV&V for
COTS projects
IV&V for
network
implementatio
n
X
X
Issue Management Plan Assessment
Combined
IPOC/IV&V
services
X
X
Communication Management Plan
Assessment
X
Quality Management Plan Assessment
X
Staff Management Plan Assessment
X
Configuration Management Plan
Assessment
X
X
X
X
X
X
X
X
X
X
Change Control Plan Assessment
X
X
Requirements Management Plan
Assessment
X
X
X
X
X
Risk Management Plan Assessment
X
Test Management Plan Assessment
X
X
X
X
X
Implementation Management Plan
Assessment
X
X
X
X
X
X
Medium Criticality
Deliverables
Business Process Re-engineering
Plan Assessment
IPOC
Services
IV&V for Custom
Developed
Software
IV&V for
COTS
projects
IV&V for
network
implementati
on
X
Combined
IPOC/IV&V
services
X
Data Conversion Plan
X
X
X
X
Data Security Plan
X
X
X
X
System Development Plan
X
X
Site Visit Evaluations
X
X
Training Plan
X
X
X
X
Knowledge Transfer Plan
Project Phase Reports
X
X
X
X
X
X
X
Architecture Design Assessment
X
X
X
X
Detailed Design Specification
Assessment
X
X
X
X
Design Phase
Interface Design Specification
Assessment
Medium Criticality
Deliverables
IV&V for
COTS projects
IV&V for
network
implementatio
n
X
X
X
X
Test Scripts and Test Cases
Assessment
X
X
X
X
Test Results for Unit, Integration,
System, Load and User Acceptance
Testing Assessment
X
X
X
X
Traceability Assessment
X
X
X
X
System Performance Assessment
X
X
X
X
X
X
X
X
Traceability Assessment
IPOC
Services
IV&V for Custom
Developed
Software
Combined
IPOC/IV&V
services
Software Estimate Review
Testing Phase
Network Performance Assessment
Rollout and Project Closeout Phase
Disaster Recovery Plan Assessment
Business Continuity and Resumption
Plan Assessment
End User Documentation Assessment
Medium Criticality
Deliverables
IPOC
Servic
es
IV&V for Custom
Developed
Software
IV&V for
COTS
projects
IV&V for
network
impleme
ntation
Combined
IPOC/IV
&V
service
s
X
X
X
X
X
X
X
X
Cutover Strategy Document
Assessment
Help Desk Manual Assessment
Technical Maintenance Manual
Assessment
Implementation Readiness Assessment
X
Data Conversion Monitoring
Assessment
Project Closure Assessment
X
Final Oversight Summary Report
Assessment
X
X
X
X
X
X
General
Oversight Management Plan
X
Software Verification and Validation
Plan
Monthly Status Report
X
Independent Project Oversight Report
(IPOR)
X
X
X
X
X
X
X
X
X
X
X
PM Oversight Principles & Procedures

Project Management Oversight Principles




COTS/MOTS, Application Development, Network
Infrastructure
Project Management Documentation
Procedures
Project Management Execution Procedures
Anomaly Report Procedures
3.3.3 Project Management Plan
The Project Management Plan (PMP) is considered the blueprint for how the project will be
managed, monitored and controlled throughout the project life cycle. Depending on the
size and complexity of the project, the Project Management Plan may contain all the
applicable sub-plans within one document, or may simply reference associated sub-plans
in stand-alone documents. Together, the Project Management Plan and all associated
sub-plans lay the foundation for the general management, schedule management,
change control, requirements management, scope management, communication
management, resource management, quality management, procurement management,
risk management and issues management for the project.
Based on the recommendations in 4-1 Recommended Model for a Project Oversight Statement of
Work (SOW), the determination of which oversight services should review the PMP varies
based on the criticality rating of the project. For low criticality projects, Sac IT has
recommended that only IPOC and combined IPOC/IV&V contracts review the Project
Management Plan. For medium and high criticality projects, Sac IT has recommended
that both IPOC and IV&V review the Project Management Plan in order to provide the
foundational assessment of how the project management and technical deliverables and
processes will be managed by the project team. A formal anomaly report is required for
all Project Management Plan reviews.
During the assessment of the PMP, the oversight consultant should be looking for the following
general elements:

Clarity of process: The plan(s) should clearly communicate the processes to be
followed in each of the project management process areas described above. The depth
of process description should be applicable for the type of project being performed and
should include roles and responsibilities and specific tasks to be performed according to
the identified process flow.

Internal & External Consistency: Consistency both within the document itself and
within each of the ancillary sub-plans is especially important on projects where a
separate sub-plan is created for each major process area. The oversight consultant
needs to ensure that cross referencing is consistent between documents (e.g. if the
Project Management Plan refers to the Issue Management Plan for more detail on
process X, the oversight consultant needs to ensure that the Issue Management Plan
contains the detail for process X).

Specificity to the Project: The processes defined in the project plans should contain
sufficient detail regarding the type of project being implemented. For example, on a
high-criticality project with a large (100+ person) team and actively engaged external
stakeholders, the issue management process needs to include how issues will be rolled
up and down throughout the organization. A single, flat model (where all issues are
raised to a single entity) would most likely not be a realistic approach. The oversight
consultant should expect to see a detailed process describing how issues are escalated to
a team lead, and then up to perhaps another management level. The oversight
consultant needs to look for a tactical implementation description for each process area
rather than a theoretical overview of the topic.

Appropriateness to Project Phase: The Project Management Plan for
the procurement phase of a medium or high criticality project will differ
significantly from the Project Management Plan during implementation.
The focus on the former should revolve around the actual solicitation
process and the mechanisms in place for managing, monitoring and
controlling such areas as schedule, issue, risk and communication
management during the procurement process. Once the vendor has been
chosen, the implementation Project Management Plan should focus on how
the actual project will be managed, monitored and controlled.
Identification of Frequency of Revisions: The Project Management
Plan is a living, breathing document that should be revised throughout the
project as appropriate. On long-term projects, unknown complexities,
legislative changes, inadequate processes and other factors could require
changes to be made to the project management plan. The oversight
vendors should ensure that the PMP is evaluated on a regular basis to
ensure that the daily execution of processes maps to the processes
identified in the PMP.
The oversight consultant should refer to the IEEE Standard for Software Project
Management Plans (IEEE Std 1058) for expected content and should
modify expectations based on the type of project. For example, the
standard includes a section on subcontractor management; however, if the
project is being developed in-house, this section may not be applicable.
IEEE Std 1058 recommends the following content areas be included in the
Project Management Plan:

Overview: includes project summary, purpose, scope and objectives for
the project, assumptions, constraints, deliverables, schedule and budget

References and Definitions

Project Organization: includes external interfaces, internal structure and
roles and responsibilities

Managerial Process plans: includes start-up plan (estimation plan,
staffing plan, resource acquisition plan, training plan), work plan (work
activities, schedule allocation, resource allocation and budget allocation),
control plan (requirements control plan, schedule control plan, budget
control plan, quality control plan, reporting plan, metrics collection plan),
risk management plan and closeout plan

Technical process plans: includes process model, methods, tools and
techniques, infrastructure plan and product acceptance plan

Supporting Process plans: includes configuration management plan,
verification and validation plan, documentation plan, quality assurance
plan, reviews and audits, problem resolution plan, subcontractor
management plan, and process improvement plans.

The OCIO ITPOF does not specifically identify the Project Management Plan as a
deliverable, but indirectly refers to several elements of the PMP in the list
of requirements.
Low
√
Med
High
√
√
√
OCIO Project Oversight Requirements
Deliverables
Development and maintenance of a project
organization chart

Org Chart
Formal staff planning, including organization
chart, written roles and responsibilities,
plans for staff acquisition, schedule for
arrival and departure of specific staff,
and staff training plans

Staff Management Plan
Development and maintenance of a project
work plan including identification of
activities, milestones and schedule


Schedule Management Plan
Work Plan
√
√
Detailed project planning with all activities
(tasks), milestones, dates and estimated
hours by task loaded to project
management software; lowest level of
tasks of short duration with measurable
outcomes


Schedule Management Plan
Work Plan
√
√
√
Tracking and reporting (within status reporting
process) of work plan activities, resource
utilization, schedule and milestone
completion status


Schedule Management Plan
Work Plan
√
√
√
Development and maintenance of project cost
estimates and supporting data for each
cost category

Cost Management Plan
√
√
Change control/approval for key specification
documents (e.g. contracts, requirements
specifications and other contract
deliverables) and software products



Change Control Plan
Requirements Management Plan
Scope Management Plan
Low
Med
√
√
High
OCIO Project Oversight Requirements
Deliverables
√
Formal configuration control, including written
configuration management plan covering
change control/approval for key specifications
documents (e.g. contracts, requirement
specifications and/or contract deliverables) and
software products and specific staff roles and
responsibilities for configuration management.
Change Control Plan
Requirements Management Plan
Scope Management Plan
Configuration
√
Formal tracking of issues/problems and their
resolution, including assignment of specific
staff responsibility for issue resolution and
specific deadlines for completion of resolution
activities
Issue Management Plan
√
Planning in compliance with formal standards
or system development lifecycle (SDLC)
methodology
Project Management Plan and associated sub plans
√
Formal continuous risk management, including
development of a written risk management
plan, identification, analysis, mitigation and
escalation of risks in accordance with
DOF/TOSU Guidelines, and regular
management team review of risks and
mitigation progress
Risk Management Plan
√
Formal communications management,
including a written project communications
plan. Regular status reporting to key
stakeholders, including progress against
timeline and budget; risk management results
and status; issue management results and
status; written escalation policy for issues and
risks. Regular stakeholder involvement in
major project decisions, issue resolution and
risk mitigation
Communications Plan
Project Management Plan
Oversight Analysis
Specific Areas of Interest
Does the Project Management
Plan (PMP) exist?
A PMP should be created during both the procurement and implementation
phases for high and medium criticality projects. During the procurement
phase, the content of the PMP should be focused on the procurement related
activities and management of those activities and the implementation PMP
should focus on the actual project implementation management.
Type of Project: If the project is being co-managed by a solution vendor and
CDCR, the oversight consultant should ensure that all aspects of the project
are governed by a project management plan. If CDCR has requested the
vendor to produce a PMP as a deliverable, the oversight consultant should
ensure that either the solution vendor PMP includes CDCR roles and
responsibilities and processes and procedures, or that CDCR develops a
separate PMP to govern their work.
Criticality of Project: Projects of all criticality should have a Project
Management Plan, however, projects of low criticality may include the content
for the sub plans within the PMP, and may describe the processes and
procedures at a higher-level than a medium or high criticality project.
Does the PMP, or documents
referenced within the PMP,
describe the process for
acquisition of non-personnel
resources needed to support
the Project, such as equipment,
computer hardware/software,
and training, and which
organization is responsible for
these acquisitions?
In some projects, this criteria may not be applicable.
Type of Project: There are no additional considerations based on whether
the project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: No additional considerations for criticality type.
Project Management Plan
Oversight Analysis
Specific Areas of Interest
Does the PMP include generic plan
information; a description of any
constraints/assumptions on which
the Project is based; a list of
Project deliverables; a summary of
the Project’s schedule and budget;
and the methods for updating,
reviewing and disseminating the
PMP?
The oversight consultant should ensure that the criteria listed are consistent with the
FSR, SPR or other OCIO approved documentation for the project.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: No additional considerations for criticality type.
Does the PMP, or documents
referenced within the PMP, provide
a complete list of all documents
and other sources of information
referenced within it?
The oversight consultant should ensure that all referenced documentation is externally
consistent with the PMP. Sac IT recommends that for all source documentation, a
reference owner and document date and version be referenced for auditing purposes.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: No additional considerations for criticality type.
Does the PMP, or documents
referenced within the PMP,
describe all responsibilities and
boundaries between the Project,
stakeholders, and external
Agencies or Entities?
Clearly defined roles and responsibilities are a critical aspect of a successful project.
Any project that has multiple, actively engaged stakeholder entities, needs to ensure
that the boundaries are clearly defined. In some cases, this may necessitate a more
detailed process to be defined. For example, on a low criticality project with only
internal stakeholders, it may be sufficient to simply state that status reports are due by
COB Friday. However, on a high criticality project comprised of 150+ team members
divided into 10 different sub teams consisting of both vendor and state staff, the
process would need to explicitly state how many status reports are required, who is
responsible for rolling up information to the next level, who is responsible for
reconciling differences reported between the vendor and state, etc.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: No additional considerations for criticality type.
Project Management Plan
Oversight Analysis
Specific Areas of Interest
Does the PMP, or documents referenced
within the PMP, contain plans
describing supporting processes
including: configuration management,
verification and validation, software
documentation, quality assurance,
reviews and audits, problem
resolution, and contractor
management?
Ensure that each sub plan or sub section within the PMP describes the tactical
processes and procedures that will apply to the daily execution of the
project management process areas. Ensure both internal consistency
within the PMP itself, as well as external consistency to the sub-plans
referenced.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: No additional considerations for criticality type.
Does the PMP, or documents referenced
with the PMP, specify the risk
management, schedule management,
scope management, issue
management, cost management,
communication management, and
staffing management processes?
Ensure that each sub plan or sub section within the PMP describes the tactical
processes and procedures that will apply to the daily execution of the
project management process areas.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: No additional considerations for criticality type.
Does the PMP or documents referenced
within the PMP provide an
organization chart depicting the
Project reporting relationships to
include all personnel involved in the
project?
It’s important to understand the distinction between a report-to org chart and a
project org chart. On some projects, they are one in the same. However,
on many State projects, the report-to structure does not fully communicate
how the project will be managed and controlled. For example, the risk
manager may not be defined in the department organization report-to org
chart; however, this is an important role to understand in terms of how the
project is managed and in some cases the risk manager may report project
risks to one entity and be resource managed by another.
Type of Project: If the project involves a vendor, it is important to represent the
reporting structure from a project perspective between the State and the
vendor. There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: No additional considerations for criticality type.
Technical Oversight Principles &
Procedures

Technical Oversight Principles




COTS/MOTS, Application Development,
Network Infrastructure
Technical Oversight Documentation
Procedures
Technical Oversight Execution
Procedures
Anomaly Report Procedures
3.3.6 Test and Evaluation Master Plan
The Test and Evaluation Master Plan is designed to describe the scope, approach, resources, and
schedule of all testing activities. The plan must identify the items to be tested, the
features to be tested, the types of testing to be performed, the personnel responsible for
testing, the resources and schedule required to complete testing, and the risks associated
with the plan. A well-written test plan can help facilitate communication by providing a
common frame of reference to describe the expectations for the operational functionality
of the system.
The Test Plan should describe the overall approach for all types of testing that will be conducted
throughout the project, including integration testing, functional testing, system testing,
load and stress testing, performance testing, regression testing, conversion testing (if
applicable), security testing, availability testing and user acceptance testing. NOTE: Unit
testing of code is often included in the development process and therefore not explicitly
included in the Test Plan. For each type of testing, the plan should discuss the
techniques and tools that will be used to test the set of designated features.

Integration and System Testing. Integration testing is when individual system,
solutions or software modules are combined and tested as a group. Integration testing
can also be conducted at the solution level where components of the solution are
combined and tested as a whole. System testing is performed on the entire system in
the context of system requirements and includes elements such as security, recovery and
reliability. IV&V should evaluate the plans, requirements, environment, tools, and
procedures used for integration testing of system modules and will evaluate the level of
automation and the availability of the system test environment. IV&V should verify that
an appropriate level of test coverage is achieved by the test process, that test results are
verified, that the correct code configuration has been tested, and that the tests are
appropriately documented, including formal logging of errors found in testing. IV&V
should verify that the test organization has an appropriate level of independence from
the development organization.

User Acceptance Testing. A project is not successful unless project stakeholders and
key users have had a chance to experience the product or result, and acknowledge
acceptance of the product or system. Acceptance Testing must include the appropriate
audience or testers, and expose all aspects of a given solution. Acceptance testing plans
must be complete, and considered well in advance of when testing begins to allow for
appropriate scheduling. IV&V will ensure that all testing plans are comprehensive and
that all factors have been considered.

Load Testing. Load testing is the process of creating demand on a system or device and
measuring its response. Load testing is usually done with goals set in accordance with
expected usage as relating to the requirements specification. Stress testing is a variation
of load testing wherein the load placed on the system is raised beyond normal usage
patterns, in order to test the system’s response at unusually high or peak loads. Stress
testing is usually designed to generate an error condition. The purpose of stress testing
is for determination of failure behavior and system limits for capacity management.
The approach for testing should be described in sufficient detail to permit identification of the
major testing tasks and the timeframe estimated to perform each phase of testing.
Roles and responsibilities for each type of testing should be identified in terms of the
activities required during each phase and if, external stakeholders are responsible for
testing, the approximate time commitment expected for each role.
Although IV&V will be conducting independent testing for verification and
validation purposes, the project is still required to perform their own
testing and quality assurance to ensure that the product or system being
created or implemented meets the original requirements and needs of the
business. Sections 3.3.6 Test and Evaluation Master Plan, Section 3.3.7
Test Scripts and Test Cases, Section 3.3.8 Test Results for Integration,
System, Load and User Acceptance Testing and Section 3.3.9 Test
Summary Report, are all focused on the assessment that IV&V will perform
on the planning and execution that the project will conduct in terms of
testing activities. Independent testing performed by IV&V is addressed in
Section 3.4 Technical Process Execution Procedures. IV&V should review
the Test Plan to ensure that adequate planning for all types of testing
occurs early in the project in order to build testing activities into the
lifecycle of the project. IV&V should ensure that all appropriate types of
testing are addressed and are scalable for the size, complexity and
criticality of the project. When reviewing the test plan, traceability should
be addressed at a high-level to ensure that all the features and
functionality identified in the requirements are acknowledged and planned
for within the test plan. NOTE: low-level traceability will be performed
once the test cases and test scripts are created in a subsequent
deliverable.
According to IEEE Standard 829 Standard for Software Test Documentation, a Test
Plan should include the following content:

Test Plan Identifier. Consists of a unique number for the test plan

Introduction. Includes a summary of the software items and software
features to be tested as well as references to existing project
documentation that will have a bearing on the Test Plan

Test Items. Includes a list of documents that will serve as the basis for
defining correct operation (i.e. SyRs, SRS, design documentation, etc.),
program modules to be tested, the job control procedures (if applicable),
user procedures and operator procedures

Features to be Tested. Includes a list of the features to be tested

Features not to be Tested. Includes a list of the features not included in
the testing.

Approach. Describes the approach of testing to be conducted on the
project and the general methodology for accomplishing each type of
testing. The techniques that will be used to judge the comprehensiveness
of the testing effort should be included as well as the techniques for
adhering to requirements traceability. Types of testing could include:
conversion testing, job stream testing, interface testing, system testing,
security testing, recovery testing, performance testing, regression testing,
comprehensiveness and constraints










Item pass/fail criteria. Describes the criteria used to determine success or failure for
each item being tested
Suspension criteria & resumption requirements. Describe any conditions that
would prevent testing activities from continuing on the project as well as the criteria that
must be met to resume testing.
Test deliverables. Identifies the deliverables the project will create within the testing
phase (i.e. test design specifications, test case specifications, test procedure
specifications, test item transmittal reports, test logs, test incident reports, test summary
reports, etc.)
Testing tasks. Identifies the set of tasks necessary to prepare for and perform the
testing including all inter-task dependencies and any special skills required or external
stakeholders needed for testing.
Environmental needs. Specifies both the necessary and desired properties of the test
environment including physical characteristics (hardware, software, communications,
etc.) and any other software or supplies needed to support the test.
Responsibilities. Includes the groups responsible for managing, designing, preparing,
executing, witnessing, checking and resolving.
Staffing and Training Needs. Specifies test staffing needs by skill level and training
options for providing necessary skills
Schedule. Includes test milestones identified in the software project schedule as well as
all item transmittal events. NOTE: a link to a testing schedule can be provided.
Risks and Contingencies. Identifies the high-risk assumptions of the test plan and
contingency plans for each risk.
Approvals. Specifies the names and titles of all persons who must approve this plan.
IV&V should ensure, when appropriate, that the Test Plan encompasses both black box and
white box testing. Black box testing involves creating test cases from an external
perspective in order to test the expected functional behavior. When performing black
box testing, no knowledge of the test object’s internal structure is needed. White box
testing on the other hand uses an internal perspective of the system to design test cases
based on the internal structure of the code and architecture. For example, in custom
application development projects, both black box and white box testing should occur
because the internal workings of the system are known to the project team. In
comparison, COTS projects would only involve black box testing of the desired
functionality.
Good test management must provide complete and proper reporting of all pertinent information
and the test plan should outline the strategy for how this information will be
communicated to the appropriate stakeholders. Ongoing real-time status, measurement
of goals, and results should be made available, in all the appropriate formats, to all
relevant team members on the project. Reporting techniques should also include both
static and dynamic capabilities to allow stakeholders to be kept up to date in the fast
paced and constantly changing world of testing. Additionally, the outputs of test
management should be integrated into the other project processes such as change
management, configuration management and requirements management. IV&V should
assess the Test Plan to ensure complete coverage of testing throughout all phases of the
life cycle and appropriate incorporation of testing into the other project processes. The
plan should address the collaboration between the different process roles and activities,
maximum communication of important information and integrated tooling to support
effective testing. Without this coordination, quality will be reduced from missed or
misunderstood requirements, untested code, missed defects and/or a lack of information
about the actual software quality level.
IEEE Standard 1012 Standard for Software Verification and Validation states that
IV&V should assess the Test Plan to “verify that the scope, strategy,
resources, and schedule of the (component, integration, system,
acceptance) testing process have been completely and accurately
specified, that all items to be tested and all required tasks to be performed
have been defined, and to ensure that all personnel and resources
necessary to perform the testing have been identified.”
The table on the next page depicts the specific criteria and components that IV&V
should look for when assessing a Test and Evaluation Master Plan. If there
are specific criteria that apply to a low, medium or high criticality project,
or to a specific type category of project, Sac IT has annotated the
distinction from the normal assessment.
Test and Evaluation Master Plan
Oversight Analysis
Specific Areas of Interest
Does a Master Test Plan exist?
Regardless of the type of project, some level of testing must occur on all projects.
The strategy, processes, roles and responsibilities and most importantly, the
acceptance criteria must be defined prior to initiating testing. Most projects, whether
custom application development, network infrastructure or COTS/MOTS will
undergo integration testing, system testing, performance testing, load and stress
testing and user acceptance testing. Depending upon the type of project, security
and data conversion testing may also be included. On large, complex projects,
separate testing plans for each of the above types of testing may be created and the
Master Test Plan might simply contain the overall strategy involved in testing. The
location of these documents is not as important as ensuring that the appropriate
content and planning for testing has been identified.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: There are no additional considerations based on the
criticality of project.
Does the test plan list the site, test
environments, personnel,
participating organizations,
security issues, orientation plans,
hardware, software and test tools
and other information/ material,
needed to conduct the testing?
Different types of testing will involve different environment setups, personnel, tools,
etc. and should be explicitly defined for each type of testing conducted. For
example, integration testing may occur within the department by the project team
using a set of servers that mirror the development environment. User acceptance
testing, however, may occur within each of the counties (for a county-wide system
implementation) and would involve a completely different subset of stakeholders.
IV&V should ensure that each environment is sufficient to adequately simulate the
production environment given the constraints and complexity of the system.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: There are no additional considerations based on the
criticality of project.
Test and Evaluation Master Plan
Oversight Analysis
Specific Areas of Interest
Does the Test Plan include a
process for defect management or
reference another plan that
addresses how defects will be
discovered, reported,
communicated and mitigated?
Defect Management is an important aspect of testing and must be scalable to the size
and complexity of the project. Defect management should involve a discussion on
traceability to ensure that the defects uncovered can be traced back to in-scope
requirements for the system. Defects (and their resolution) should be tracked by
unique identifiers and communicated to all appropriate stakeholders in an efficient
manner. For example, if the navigational flow within an application changes as a
result of an identified defect, the training team should be notified of the new
implementation in order to determine if changes need to be made to their training
materials. IV&V should assess the defect management strategy, tools and
techniques identified to ensure that the process is effective and scalable for the
project.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: There are no additional considerations based on the criticality
of project.
Has a regression test strategy been
defined for re-testing (for problem
resolution, enhancements in
system design, or change requests)
and is a configuration control
methodology referenced or
included?
IV&V should ensure that the Test Plan includes the strategy for re-testing the system
after new versions of the code are incorporated due to defect resolution or new
development. IV&V should refer back to the projects Configuration Management Plan
to ensure consistency with how versioning will occur and how the changes will be
integrated into the new release. IV&V should pay particular attention to the scalability
of the approach defined for configuration management of the code versions and the
overall regression testing strategy to ensure the project is adequately preparing for
the number of stakeholders involved, the potential number of different versions that
will be in existence at one time, the logistics of merging changes into current releases,
the acceptance strategy for each version and that rollback procedures are defined.
Type of Project: There are no additional considerations based on whether the
project is a COTS/MOTS implementation, custom development or network
infrastructure project.
Criticality of Project: There are no additional considerations based on the criticality
of project.
Test and Evaluation Master Plan
Oversight Analysis
Specific Areas of Interest
Have the test elements (software
items, hardware items, manual
operations and other necessary
systems) been defined for each type
of testing in the acceptance test
procedure?
Each type of testing should explicitly describe the items being tested and the
acceptance criteria for each item.
Type of Project: There are no additional considerations based on whether the project
is a COTS/MOTS implementation, custom development or network infrastructure
project.
Criticality of Project: There are no additional considerations based on the criticality of
project.
Are acceptance test procedures for
checking the internal and external
data formats, interface protocols,
frequency of data exchange at each
interface included in the
acceptance/system qualification test
procedure? Do the test procedures
include operator data entry errors?
If there are any interfaces from the new system to other systems (new, modified or
existing), the Test Plan should indicate a strategy for testing these interfaces.
Sometimes, interface testing involves obtaining data and staff from other agencies and
therefore must be well-planned and scheduled to avoid conflicts and problems. If
external stakeholders are involved in any type of testing, IV&V should ensure that the
roles and responsibilities are very clearly defined, that inputs, outputs and
dependencies between the two organizations are adequately determined and that highlevel testing schedules are included in the plan in order for both organizations to
allocate resources effectively.
Type of Project: There are no additional considerations based on whether the project
is a COTS/MOTS implementation, custom development or network infrastructure
project.
Criticality of Project: There are no additional considerations based on the criticality of
project.
Does the Test Plan include a testing
strategy for both positive and
negative testing?
Both positive testing (testing the expected functionality) and negative testing (testing for
illegal inputs, data outside the boundaries) should be addressed in the Test Plan.
Type of Project: There are no additional considerations based on whether the project
is a COTS/MOTS implementation, custom development or network infrastructure
project.
Criticality of Project: There are no additional considerations based on the criticality of
project.
Test and Evaluation Master Plan
Oversight Analysis
Specific Areas of Interest
Are resources, management
activities, personnel and
participating organizations defined
in the Test Plan?
Testing responsibilities are often spread among various different groups to perform the
different types of testing. It is critical that the Test Plan identify each set of
stakeholders responsible for testing and clearly identify their roles and responsibilities.
Type of Project: There are no additional considerations based on whether the project
is a COTS/MOTS implementation, custom development or network infrastructure
project.
Criticality of Project: There are no additional considerations based on the criticality
of project.
Does the acceptance/system
qualification test procedure provide
a detailed acceptance test
schedule?
A high-level testing schedule should be included or referenced to show when each
type of testing starts, the dependencies involved and the resources assigned.
Type of Project: There are no additional considerations based on whether the project
is a COTS/MOTS implementation, custom development or network infrastructure
project.
Criticality of Project: There are no additional considerations based on the criticality
of project.
Have the approval procedures been
defined in the acceptance/system
qualification test procedure?
Because testing is so pervasive over the entire project, the list of reviewers and
approvers of the test plan may involve additional participants. For example,
developers on custom application development projects should be included in the
review of the test plans so that they are aware of the criteria for acceptance and so
that they know how the testers plan to “break” their code.
Type of Project: There are no additional considerations based on whether the project
is a COTS/MOTS implementation, custom development or network infrastructure
project.
Criticality of Project: There are no additional considerations based on the criticality
of project.
Test and Evaluation Master Plan
Oversight Analysis
Specific Areas of Interest
Does the Test Plan address
traceability back to the
requirements?
The Test Plan should identify the strategy for ensuring traceability in all areas of testing. IV&V
should assess this strategy to ensure that it is scalable and inclusive. IV&V should also
ensure high-level traceability exists in the types of testing covered under the Testing Plan.
Features and high-level functionality identified in the SRS and/or SyRS should be included as
test items within the Test Plan.
Type of Project: There are no additional considerations based on whether the project is a
COTS/MOTS implementation, custom development or network infrastructure project.
Criticality of Project: There are no additional considerations based on the criticality of project.
Does the System Testing
include recovery and
reliability testing
strategy?
Recovery testing tests how well the system recovers from failures such as system crashes and/or
situations where the system has hung and is no longer operational. The architecture of the
system should be resilient enough to handle system recovery issues such as data integrity.
Reliability testing involves how the system handles failure of a component of the solution (i.e.
a single server in a cluster fails, causing the other servers to pick up the load). IV&V should
ensure that the Test Plan includes a strategy for recovery and reliability and that the strategy
is scalable for the complexity of project.
Type of Project: There are no additional considerations based on whether the project is a
COTS/MOTS implementation, custom development or network infrastructure project.
Criticality of Project: There are no additional considerations based on the criticality of project.
Does the Test Plan contain a
strategy for load and
stress testing?
Load testing is the process of creating demand on a system or device and measuring its response.
Load testing is usually done with goals set in accordance with expected usage as relating to
the requirements specification. Stress testing is a variation of load testing wherein the load
placed on the system is raised beyond normal usage patterns in order to test the system’s
response at unusually high or peak loads. Stress testing is usually designed to generate an
error condition to determine failure behavior and system limits for capacity management.
IV&V should ensure that both load and stress testing occur, especially in custom application
development and network infrastructure projects.
Type of Project: For COTS/MOTS projects, the load and stress limits for the product should be
provided by the vendor and simply verified and tested by the department during
implementation. There are no additional considerations based on whether the project is a
COTS/MOTS implementation, custom development or network infrastructure project.
Criticality of Project: There are no additional considerations based on the criticality of project.
IPOR and IV&V Report
Preparation Guidelines




Methodology and Approach
ITPOF Framework Guidelines
IEEE-1012 Standard for Verification and
Validation Guidelines
Risk Management and Escalation
Oversight Provider Information
Oversight Leader:
Field 4
Phone Number:
Field 6
Organization:Field 5
Email:
Field 7
The Oversight Provider Information section of the IPOR is locked for editing, and the spelling/grammar checking
feature is not available when the file is locked. This section is very straight-forward and rarely changes from report
to report unless staffing changes occur within the oversight team. Below is a list of the specific field names found
in the Oversight Provider Information section and suggestions for completing the fields.
Field #
Description
Comments or Suggestions
4
The Oversight Leader is the
person who has the primary
responsibility for the oversight
information and who is the
first point of contact for OCIO
In a team environment for external oversight, this is
usually the overall project lead for account
manager/executive for the team.
5
The Organization providing
the oversight service
For internal oversight, this would be CDCR, PMO.
For external oversight, it would be the legal name
of the vendor providing the service.
6
The Phone Number
Include the area code, extension, and an alternate
phone number for the Oversight Leader.
7
The Email Address
Include an email address for the Oversight Leader.
Contract Management Process
Guidelines


Methodology and Approach
Contract Management Activities





Service Delivery Management
Relationship Management
Contract Administration
Management Guidelines for Oversight of
Consultants
Assessment Components of Anomaly Reports
Training and Staff Readiness




Development of Training Materials
Staff Training
Knowledge Transfer Activities
Customer Education Materials