Download Audit Report - Default Style

Appendix 5e
Final Internal Audit Report
IT Project Management
December 2006
Report 5e Page 1 of 15
Contents
Page
Executive Summary
3
Observations and Recommendations
8
Appendix 1 – Audit Framework
13
Appendix 2 - Staff Interviewed
14
Statement of Responsibility
15
IT Project Management
2006/2007
Report 5e Page 2 of 15
Audit Ref: 709
Executive Summary
Introduction & Background
1.
This audit forms part of the 2006/2007 Internal Audit Plan, which has been
approved by the Mayor and the Audit Panel. The plan entails a review of the
systems and controls operating over Project Management within the Technology
Group.
2.
This audit has been carried out in conjunction with the Authority’s Technology
Programme Business Development Review, which is being led by the Squares &
Business Development Group who aim to improve the overall programme
management of ICT and to ensure all projects are delivered to time, cost and quality
as per the outline scope of the work that was agreed by Mayor’s Management
Board (MMB) on 3 April 2006. The audit work performed aims to support the
Technology Programme Business Development Review process to seek
improvements where required.
3.
The Technology Group is part of the Corporate Services directorate. The group was
restructured approximately one year ago and now compromises of three teams;
Projects & Consultancy team, Development team and Operations team. The
Projects & Consultancy team is responsible for business analysis (of the Authority’s
business requirements), account management (liaising with customers – i.e. IT
users) and project management (of projects to develop, buy-in and implement new
systems). The Project & Consultancy team is headed up by the Projects &
Consultancy Manager, who reports to the Head of Technology Group.
4.
It should be noted that the various projects reviewed during the course of this audit
were managed and controlled under the Authority’s former Project Management
Framework, as the new Quality Framework had not yet been applied to any projects
at the time of our work. The aim was to establish the extent of compliance and
control effectiveness in the management of these IT projects. The results of the
audit have demonstrated that whilst some procedures and controls operate as
intended, some control weaknesses have been identified. Many of these have either
arisen as a result of inconsistency in the application of the process, non compliance
or a lack of sufficient controls being inherent within the former framework. Some of
the issues raised within this report have already been reflected in the new Quality
Management Framework and others will form part of the ongoing development of
best practice in this area.
5.
Random samples of prime source documents pertaining to the financial years 2003
to date, have been selected and tested to evaluate the effectiveness of the controls
in operation. Our audit work was performed during the period of August to October
2006. A summary of the findings is contained within the following paragraphs.
Mayoral/Senior Management Approval
6.
All major projects should be subject to sufficient approval in line with the Authority’s
Decision Making Framework and Scheme of Delegation. Approval should be clearly
documented on the relevant approval form (MAF or DAF as appropriate) and the
form should clearly state what resources will be allocated to the project. This should
Report 5e Page 3 of 15
be approved by senior management to ensure that there is a dedicated resource for
all IT projects.
7.
For all projects there should be a completed Project Initiation Document (PID) that
identifies the perceived need (i.e. why is the project required), the objectives of the
project (i.e. what will the project achieve and how the need will be met) and a
summary of the key information such as stakeholders and risks.
8.
Audit reviewed a sample of six different projects managed by the Technology Group
that had taken place over the last few years. It was confirmed that each project has
been sufficiently scoped in a standard Project Initiation Document with clearly
defined objectives and a summary overview of the project, which had been agreed
by the Project Board and Executive in each case reviewed.
9.
There was evidence that initial project planning was undertaken to a sufficient level,
with an initial budget requirement being set out at the beginning of the project in
each case reviewed. However it is recognised that there has been inconsistency in
the past in the way project plans were documented.
10.
All IT projects should be included in the Strategic IT Plan to ensure that there is
some structure and prioritisation of project work, so that key projects are given
priority. Five of the six projects included in our sample were identified in the 3-year
IT Strategy for April 03 - April 06. One project, which related to GLAAS, was an
abortive project that was restarted with a revised scope.
No recommendations have been raised following our review of this area.
Risk Identification, Management & Monitoring
11.
For each of the six projects reviewed, there was sufficient documentary evidence to
show that associated project risks had been identified, assessed and prioritised.
However it was noted that in one case there was insufficient evidence within the risk
log that mitigating strategies had been devised to guard against identified risks. It
was also noted that for GLAAS there was no evidence of the risk register being
continually reviewed and updated throughout the life off the project. However, it was
noted that whilst the risk logs may not have been updated, there was evidence
obtained from the Project Board minutes of meetings held, to confirm that risks had
been identified and that the Project Board was monitoring them.
12.
The Project Board should consider the risk register at project meetings as a
standard item, to ensure that any change in the risk profile is picked up and
discussed. For the majority of projects it was evident that the risk register was
regularly reviewed and updated as and where necessary.
13.
Any risks arising from a project that may have a wider corporate impact should be
fed upwards to directorate registers and the corporate risk register, as appropriate.
The extent to which this occurs was measured for each of the projects in our
sample, however it was considered that the level of risk specified in the project
registers (which are not necessarily project orientated) was generally not relevant to
the directorate and corporate risk registers, which record broader, higher level risks.
Therefore it was not considered necessary for the majority of project risks to be
reflected in the directorate/corporate registers.
14.
It was noted that the Finance System project (which did have a corporate impact)
had been reflected in the Finance & Performance Risk Register.
Report 5e Page 4 of 15
One recommendation has been made following our review of this area.
Compliance with Project Management Methodology
15.
There is a standard methodology for IT projects conducted at the GLA. The
Authority has a Corporate Project Management Methodology that is based on
PRINCE 2 and the Technology group has an additional quality standard for projects
that should be adhered to. The Project Management Standard requires that all
Technology Group projects should incorporate a specified list of quality documents,
including:

Project Brief

PID

Project Plan

Implementation Plan

Quality Plan

Work Packages

Project Change Control logs

End Project Report & Lessons Learned
16.
A general review of the working papers for each of the six projects in our sample
demonstrated that they had adhered to accepted project management conventions.
However, not all of the standard documents listed above were incorporated in every
project, although it should be noted that the current Project Management Quality
Standard has been introduced post-completion of most of those projects.
17.
It was noted that there are exceptions where key areas have been omitted, for
instance, concerning the completion and closedown of projects, which has been
illustrated later in our report as part of paragraphs 28 to 31 (Measurement of Value
For Money & Organisational Benefits).
18.
Given that no projects have taken place using the new methodology, it was not
possible to test compliance with this methodology. It is notable that there was not
such a defined standard for project management previously, and the new quality
standard should provide a framework that ensures that the standard of
documentation of projects is more consistent and comprehensive in future.
No recommendations have been made following our review of this area.
Project Delivery Timescales
19.
A Project Plan should be produced at the commencement of the project, setting out
timescales and deadlines for key milestones. There should be regular progress
review of the project by the Project Board to measure progress against timescales.
Project Plans were being held to support each project contained in our sample.
20.
Progress against the plan is monitored by means of the Project Programme Report.
This is monitored by the Strategy Group on a regular basis. There is a programme
report for all IT projects which contains information on milestones and shows
originally planned and actual dates.
Report 5e Page 5 of 15
21.
The most recent programme report that was initially provided was from November
2005 (this was the last programme report before the Strategy Board was reformed).
Another programme report post reformation of the Strategy Board was
subsequently provided, which was from February 2006. It was apparent that there
were mechanisms in place for monitoring the progress of projects against
milestones, although for all of the sampled projects there generally appeared to be
some slippage on original timescales.
No recommendations have been raised following our review of this area.
Project Costs
22.
The budget for each IT project should be allocated prior to the commencement of
the project and approved by senior management. Expenditure on the project should
be regularly monitored against budget. For all projects in our sample the original
budget was set at the outset, but whilst we recognise that there is review of budgets
in place, there was a lack of documentary evidence available to support the review
in particular where the projects ran over the year.
23.
In the past, monthly meetings were held between Finance and the Project &
Consultancy team regarding project budget monitoring. However no formal minutes
of such meetings have been retained. These meetings were for the purposes of
addressing specific issues arising rather than being formal governance meetings.
24.
For the projects sampled, there was documentary evidence at the time of the audit
to show that expenditure was within the allocated budget and it was noted that the
management of the finances is a standing item at each board meeting and finance
information is regularly reported through the project management reporting
hierarchy. It was recognised that the systems of control could be improved by
ensuring that the Technology Group maintains comprehensive post project
completion analysis of ‘total budget’ variances (i.e. sum of annual budgets for multiyear projects) and also ensuring expenditure against budget over the whole
duration of the projects is monitored. There was also a lack of sufficient
management information available in this respect.
One recommendation has been raised following our review of this area.
Quality Management
25.
The Technology Group has introduced a new quality framework for projects, which
should be utilised for all future projects undertaken. In order to ensure that the
framework is complied with, there should be periodic review to confirm that IT
projects are carried out and documented to a sufficient level of quality, in
accordance with the Quality Management Framework.
26.
It was noted that all key documents are retained electronically and signed off by the
Project Board in accordance with Prince 2 methodology.
27.
The Project & Consultancy Manager has informed us, that a review of compliance
with the new quality framework will be taking place in January 2007, six months
after it has been introduced.
One recommendation has been raised following our review of this area.
Report 5e Page 6 of 15
Measurement of Value for Money & Organisational Benefits
28.
It was verified that all of the sampled six projects were instigated in response to a
business need and the anticipated benefit of the project to the Authority was
realised in meeting that need. In the case of some projects such as the London
Development Database (LDD), the benefit is wider than just the GLA as this will
provide benefits across the GLA family and London Boroughs.
29.
All projects should be properly closed down upon completion and an end of project
report should be produced that evaluates the success of the project (in terms of
whether it provides VFM) and highlights any areas for improvement. Audit testing
showed that only one of the six projects reviewed maintained evidence of formal
closedown and this was the Finance project. Of the remaining five:

GLAAS had not been completed at the time of the audit

Consult was terminated, and the specific reasons were observed within a MAF

Blackberry was a pilot project and very small and didn't warrant a formal
closedown

For LDD it was agreed that the project closedown would take place after 6
months had elapsed since the completion of the project, so the closedown report
was due at the time of this audit, and we have been informed that this report is
being completed.

For GIS there was no reason why a closedown project had not been produced.
30.
Formal closedown of projects seems to be a problem generally. This is indicative of
the fact that IT staff are constantly moving on to the next project before being able
to properly wrap up current projects. A consequence is that it is difficult to evaluate
the success of projects accurately when they are not properly closed. It is
acknowledged that the Authority with the instigation of the new quality framework is
attaching a higher priority to this area.
31.
At the end of a project, as well as at the beginning, key stakeholders should be
interviewed to ensure that any learning points are noted and discussed in order to
aid continuous improvement in project management. The only one of the six
projects with documented lessons learned was the Finance System project.
One recommendation has been made following our review of this area.
Report 5e Page 7 of 15
Audit Opinion
Substantial Assurance
Evaluation Opinion: While there is a basically sound system, there are areas of weakness
which put some of the system objectives at risk,
Testing Opinion: There is evidence that the level of non-compliance with some of the
controls may put some of the system objectives at risk.
Observations and Recommendations
In order to assist management in using our reports:
We categorise our opinions according to our assessment of the controls in place and the
level of compliance with these controls
Full
Assurance
There is a sound system of control designed to achieve the system
objectives
and the controls are being consistently applied.
Substantial
Assurance
While there is a basically sound system, there are areas of weakness
which put some of the system objectives at risk,
and/or there is evidence that the level of non-compliance with some of the
controls may put some of the system objectives at risk.
Limited
Assurance
Weaknesses in the system of controls are such as to put the system objectives
at risk,
and /or the level of non-compliance puts the system objectives at risk.
No
Assurance
Control is generally weak, leaving the system open to significant error or
abuse,
and/or significant non-compliance with basic controls leaves the system
open to error or abuse.
b) We categorise our recommendations according to their level of priority.
Priority 1
Major issues for the attention of senior management.
Priority 2
Other recommendations for local management action.
Priority 3
Minor matters.
Report 5e Page 8 of 15
Risk Identification, Management & Monitoring
1. Mitigating Controls
Recommendation
(Priority 3)
Rationale
It is recommended that the project risk
registers should be consistently updated
with mitigating strategies and should be
reviewed by the Project Board as a standard
item.
Risk Management is an essential element
of good project management. It is
important that all project risks are identified
and controlled effectively.
Audit testing identified one project where
there
was
insufficient
documentary
evidence within the risk log to support the
mitigating strategies agreed. However, it
was noted that the Project Board was
reviewing the risks, although the 0project
was later abandoned because risks were
realised.
If adequate controls are not put in place to
mitigate risks then there may potentially be
adverse consequences that effect the
achievement of project objectives.
Management response: Project & Consultancy Manager
Agreed
Consideration of risk register is a standing item on all Project Board agendas.
Implementation Date: Has already been implemented.
Report 5e Page 9 of 15
Project Costs
2. Budget Monitoring & Analysis
Recommendation
(Priority 2)
Rationale
It is recommended that a clear analysis of
actual spend against budget is maintained
for all projects, monitored on an ongoing
basis and reviewed at the end of project.
This should cover the entire budget for the
duration of the project, not just the annual
budget.
Effective budget monitoring relies on the
production of timely and accurate
management information. A meaningful
evaluation of the success of a project must
include a measure of how the project
progressed against timescale and budget.
Whilst there was evidence of budget
monitoring having taken place on individual
projects, there was a lack of audit trail
clearly showing changes to the budget and
what the final position against budget was
in instances where budgets rolled over a
year.
Without proper evaluation of the project
performance in terms of budget, it is not
possible to properly gauge the success of
the project. There is a risk that inefficiency
and wastage will not be highlighted and
investigated.
Management response: Executive Director of Finance &
Performance
Agreed
The Authority’s new financial systems provides the ability to monitor budgets and actual
spend over five years. Given the small size of the Authority’s capital programme,
development of this element of the system has had a lower priority against revenue
account and Ebis developments. Work is now in progress to establish project monitoring
from 2006/07 onwards.
Implementation Date: January 2007
Report 5e Page 10 of 15
Quality Management
3. Compliance with Quality Framework
Recommendation
(Priority 2)
Rationale
It is recommended that a periodic review of
project
working
papers
should
be
undertaken to ensure that business
consultants are managing and documenting
projects in the appropriate manner, in
accordance with the framework.
It is noted that the Technology Group has
recently introduced a comprehensive new
quality framework for project management,
which incorporates Prince 2 methodology.
None of the projects in our sample were
undertaken using the new framework but
they were reviewed against compliance
with the old framework and good practice.
It was noted that certain key stages of the
projects had not been documented
consistently, in particular the project
closedown and post implementation review
processes.
Without sufficient compliance review it may
be difficult to enforce and maintain
compliance with the quality management
framework.
Management response: Project & Consultancy Manager
Agreed
All members of the Technology Group Projects and Consultancy Team (including the
Team manager) already have a performance review objective to deliver the Quality
Management Framework. The corporate performance reviews take place in June with a 6month review point in January.
The Project boards sign off key project documents such as the PID, in accordance with
Prince2. Project wok is authorised in accordance with GLA procedures for approval and
procurement.
The project support office will undertake spot checks of project documentation in the
course of the year. The Manager of Projects and Consultancy will audit quality
management framework documents in advance of the performance review points and any
issues relating to delivery of the system will be raised in the review meetings.
Technology projects may have a long (multi-year) delivery cycle. Hence 6 monthly
reviews are deemed to be appropriate.
Implementation Date: First review point is January 2007
Report 5e Page 11 of 15
Measurement of Value For Money
4. Project Closedown & Lessons Learned
Recommendation
(Priority 2)
Rationale
It is recommended that formal retrospective
close down should be completed for all
projects. Management should also ensure
that formal close down procedures are
consistently applied for each future project.
It is a requirement of the Project Quality
Management
Framework
that
all
completed projects have an End Project
Report and Lessons Learned documented.
There was insufficient evidence to support
the close down of the GIS project.
However, the remaining projects reviewed
have either been formally closed down or
there was sufficient documentary evidence
to support the reason for formal close
down not taking place.
It should be noted that none of the
sampled projects were undertaken to the
current quality framework, and although
the
former
Project
Management
Framework required formal closedown of
projects, it is acknowledged that a higher
priority is now being attached to this area
with the instigation of the new quality
framework.
If projects are not properly closed down
then there is potential for the Project
Managers to be continually involved in
ongoing work on the business area after
the objectives of the project have been
achieved. In addition it is difficult to
properly evaluate the success of the
project if there is not a definitive cut-off
point.
Management response: Project & Consultancy Manager
Agreed
End Project and Lessons Learned reports are required documents in the new Quality
Management Framework. Compliance will be monitored as outlined in Recommendation
3.
Implementation Date: Immediate effect with first review in January 2007
Report 5e Page 12 of 15
Appendix 1 – Audit Framework
Audit Objectives
The audit was designed to ensure that management has implemented adequate and
effective controls within the Technology Group to ensure the effective delivery of IT
projects to time, cost, and quality.
Audit Approach and Methodology
The audit approach was developed with reference to an assessment of risks and
management controls operating within each area of the scope.
The following procedures were adopted:
 identification of the role and objectives of each area;
 identification of risks within the systems, and controls in existence to allow the control
objectives to be achieved; and
 evaluation and testing of controls within the systems.
From these procedures we have identified weaknesses in the systems of control,
produced specific proposals to improve the control environment and have drawn an overall
conclusion on the design and operation of the system.
Areas Covered
Audit work was undertaken to cover controls in the following areas:
 Mayor / Senior Management Approval
 Risk Identification, Management and Monitoring
 Compliance with IT Project Management Methodology
 Organisational Benefits and User Requirements
 Project Delivery Timescales
 Project Costs
 Quality Requirements and Standards
 Measurement of Value for Money
Report 5e Page 13 of 15
Appendix 2 - Staff Interviewed
We would like to thank all staff that provided assistance during the course of this audit, and
in particular:
Project & Consultancy Manager
Business Consultants (x2)
Report 5e Page 14 of 15
Statement of Responsibility
We take responsibility for this report, which is prepared on the basis of the limitations set
out below.
The matters raised in this report are only those, which came to our attention during the
course of our internal audit work and are not necessarily a comprehensive statement of all
the weaknesses that exist or all improvements that might be made. Recommendations for
improvements should be assessed by you for their full impact before they are
implemented. The performance of internal audit work is not and should not be taken as a
substitute for management’s responsibilities for the application of sound management
practices. We emphasise that the responsibility for a sound system of internal controls
and the prevention and detection of fraud and other irregularities rests with management
and work performed by internal audit should not be relied upon to identify all strengths and
weaknesses in internal controls, nor relied upon to identify all circumstances of fraud or
irregularity. Auditors, in conducting their work, are required to have regards to the
possibility of fraud or irregularities. Even sound systems of internal control can only
provide reasonable and not absolute assurance and may not be proof against collusive
fraud. Internal audit procedures are designed to focus on areas as identified by
management as being of greatest risk and significance and as such we rely on
management to provide us full access to their accounting records and transactions for the
purposes of our audit work and to ensure the authenticity of these documents. Effective
and timely implementation of our recommendations by management is important for the
maintenance of a reliable internal control system.
Deloitte & Touche Public Sector Internal Audit Limited
St Albans
November 2006
In this document references to Deloitte are references to Deloitte & Touche Public Sector
Internal Audit Limited.
Deloitte & Touche Public Sector Internal Audit Limited is a subsidiary of Deloitte & Touche
LLP, which is the United Kingdom member firm of Deloitte Touche Tohmatsu. Deloitte
Touche Tohmatsu is a Swiss Verein (association), and, as neither such, neither Deloitte
Touche Tohmatsu nor any of it member firms has any liability for each other’s acts or
omissions. Each of the member firms is a separate and independent legal entity operating
under the names “Deloitte”, “Deloitte & Touche”, “Deloitte Touche Tohmatsu”, or other
related names. Services are provided by the member firms or their subsidiaries or
affiliates and not by the Deloitte Touche Tohmatsu Verein.
©2006 Deloitte & Touche Public Sector Internal Audit Limited. All rights reserved.
Deloitte & Touche Public Sector Internal Audit Limited is registered in England and Wales
with registered number 4585162. Registered office: Stonecutter Court, 1 Stonecutter
Street, London EC4A 4TR, United Kingdom
Report 5e Page 15 of 15