Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Appendix 5e Final Internal Audit Report IT Project Management December 2006 Report 5e Page 1 of 15 Contents Page Executive Summary 3 Observations and Recommendations 8 Appendix 1 – Audit Framework 13 Appendix 2 - Staff Interviewed 14 Statement of Responsibility 15 IT Project Management 2006/2007 Report 5e Page 2 of 15 Audit Ref: 709 Executive Summary Introduction & Background 1. This audit forms part of the 2006/2007 Internal Audit Plan, which has been approved by the Mayor and the Audit Panel. The plan entails a review of the systems and controls operating over Project Management within the Technology Group. 2. This audit has been carried out in conjunction with the Authority’s Technology Programme Business Development Review, which is being led by the Squares & Business Development Group who aim to improve the overall programme management of ICT and to ensure all projects are delivered to time, cost and quality as per the outline scope of the work that was agreed by Mayor’s Management Board (MMB) on 3 April 2006. The audit work performed aims to support the Technology Programme Business Development Review process to seek improvements where required. 3. The Technology Group is part of the Corporate Services directorate. The group was restructured approximately one year ago and now compromises of three teams; Projects & Consultancy team, Development team and Operations team. The Projects & Consultancy team is responsible for business analysis (of the Authority’s business requirements), account management (liaising with customers – i.e. IT users) and project management (of projects to develop, buy-in and implement new systems). The Project & Consultancy team is headed up by the Projects & Consultancy Manager, who reports to the Head of Technology Group. 4. It should be noted that the various projects reviewed during the course of this audit were managed and controlled under the Authority’s former Project Management Framework, as the new Quality Framework had not yet been applied to any projects at the time of our work. The aim was to establish the extent of compliance and control effectiveness in the management of these IT projects. The results of the audit have demonstrated that whilst some procedures and controls operate as intended, some control weaknesses have been identified. Many of these have either arisen as a result of inconsistency in the application of the process, non compliance or a lack of sufficient controls being inherent within the former framework. Some of the issues raised within this report have already been reflected in the new Quality Management Framework and others will form part of the ongoing development of best practice in this area. 5. Random samples of prime source documents pertaining to the financial years 2003 to date, have been selected and tested to evaluate the effectiveness of the controls in operation. Our audit work was performed during the period of August to October 2006. A summary of the findings is contained within the following paragraphs. Mayoral/Senior Management Approval 6. All major projects should be subject to sufficient approval in line with the Authority’s Decision Making Framework and Scheme of Delegation. Approval should be clearly documented on the relevant approval form (MAF or DAF as appropriate) and the form should clearly state what resources will be allocated to the project. This should Report 5e Page 3 of 15 be approved by senior management to ensure that there is a dedicated resource for all IT projects. 7. For all projects there should be a completed Project Initiation Document (PID) that identifies the perceived need (i.e. why is the project required), the objectives of the project (i.e. what will the project achieve and how the need will be met) and a summary of the key information such as stakeholders and risks. 8. Audit reviewed a sample of six different projects managed by the Technology Group that had taken place over the last few years. It was confirmed that each project has been sufficiently scoped in a standard Project Initiation Document with clearly defined objectives and a summary overview of the project, which had been agreed by the Project Board and Executive in each case reviewed. 9. There was evidence that initial project planning was undertaken to a sufficient level, with an initial budget requirement being set out at the beginning of the project in each case reviewed. However it is recognised that there has been inconsistency in the past in the way project plans were documented. 10. All IT projects should be included in the Strategic IT Plan to ensure that there is some structure and prioritisation of project work, so that key projects are given priority. Five of the six projects included in our sample were identified in the 3-year IT Strategy for April 03 - April 06. One project, which related to GLAAS, was an abortive project that was restarted with a revised scope. No recommendations have been raised following our review of this area. Risk Identification, Management & Monitoring 11. For each of the six projects reviewed, there was sufficient documentary evidence to show that associated project risks had been identified, assessed and prioritised. However it was noted that in one case there was insufficient evidence within the risk log that mitigating strategies had been devised to guard against identified risks. It was also noted that for GLAAS there was no evidence of the risk register being continually reviewed and updated throughout the life off the project. However, it was noted that whilst the risk logs may not have been updated, there was evidence obtained from the Project Board minutes of meetings held, to confirm that risks had been identified and that the Project Board was monitoring them. 12. The Project Board should consider the risk register at project meetings as a standard item, to ensure that any change in the risk profile is picked up and discussed. For the majority of projects it was evident that the risk register was regularly reviewed and updated as and where necessary. 13. Any risks arising from a project that may have a wider corporate impact should be fed upwards to directorate registers and the corporate risk register, as appropriate. The extent to which this occurs was measured for each of the projects in our sample, however it was considered that the level of risk specified in the project registers (which are not necessarily project orientated) was generally not relevant to the directorate and corporate risk registers, which record broader, higher level risks. Therefore it was not considered necessary for the majority of project risks to be reflected in the directorate/corporate registers. 14. It was noted that the Finance System project (which did have a corporate impact) had been reflected in the Finance & Performance Risk Register. Report 5e Page 4 of 15 One recommendation has been made following our review of this area. Compliance with Project Management Methodology 15. There is a standard methodology for IT projects conducted at the GLA. The Authority has a Corporate Project Management Methodology that is based on PRINCE 2 and the Technology group has an additional quality standard for projects that should be adhered to. The Project Management Standard requires that all Technology Group projects should incorporate a specified list of quality documents, including: Project Brief PID Project Plan Implementation Plan Quality Plan Work Packages Project Change Control logs End Project Report & Lessons Learned 16. A general review of the working papers for each of the six projects in our sample demonstrated that they had adhered to accepted project management conventions. However, not all of the standard documents listed above were incorporated in every project, although it should be noted that the current Project Management Quality Standard has been introduced post-completion of most of those projects. 17. It was noted that there are exceptions where key areas have been omitted, for instance, concerning the completion and closedown of projects, which has been illustrated later in our report as part of paragraphs 28 to 31 (Measurement of Value For Money & Organisational Benefits). 18. Given that no projects have taken place using the new methodology, it was not possible to test compliance with this methodology. It is notable that there was not such a defined standard for project management previously, and the new quality standard should provide a framework that ensures that the standard of documentation of projects is more consistent and comprehensive in future. No recommendations have been made following our review of this area. Project Delivery Timescales 19. A Project Plan should be produced at the commencement of the project, setting out timescales and deadlines for key milestones. There should be regular progress review of the project by the Project Board to measure progress against timescales. Project Plans were being held to support each project contained in our sample. 20. Progress against the plan is monitored by means of the Project Programme Report. This is monitored by the Strategy Group on a regular basis. There is a programme report for all IT projects which contains information on milestones and shows originally planned and actual dates. Report 5e Page 5 of 15 21. The most recent programme report that was initially provided was from November 2005 (this was the last programme report before the Strategy Board was reformed). Another programme report post reformation of the Strategy Board was subsequently provided, which was from February 2006. It was apparent that there were mechanisms in place for monitoring the progress of projects against milestones, although for all of the sampled projects there generally appeared to be some slippage on original timescales. No recommendations have been raised following our review of this area. Project Costs 22. The budget for each IT project should be allocated prior to the commencement of the project and approved by senior management. Expenditure on the project should be regularly monitored against budget. For all projects in our sample the original budget was set at the outset, but whilst we recognise that there is review of budgets in place, there was a lack of documentary evidence available to support the review in particular where the projects ran over the year. 23. In the past, monthly meetings were held between Finance and the Project & Consultancy team regarding project budget monitoring. However no formal minutes of such meetings have been retained. These meetings were for the purposes of addressing specific issues arising rather than being formal governance meetings. 24. For the projects sampled, there was documentary evidence at the time of the audit to show that expenditure was within the allocated budget and it was noted that the management of the finances is a standing item at each board meeting and finance information is regularly reported through the project management reporting hierarchy. It was recognised that the systems of control could be improved by ensuring that the Technology Group maintains comprehensive post project completion analysis of ‘total budget’ variances (i.e. sum of annual budgets for multiyear projects) and also ensuring expenditure against budget over the whole duration of the projects is monitored. There was also a lack of sufficient management information available in this respect. One recommendation has been raised following our review of this area. Quality Management 25. The Technology Group has introduced a new quality framework for projects, which should be utilised for all future projects undertaken. In order to ensure that the framework is complied with, there should be periodic review to confirm that IT projects are carried out and documented to a sufficient level of quality, in accordance with the Quality Management Framework. 26. It was noted that all key documents are retained electronically and signed off by the Project Board in accordance with Prince 2 methodology. 27. The Project & Consultancy Manager has informed us, that a review of compliance with the new quality framework will be taking place in January 2007, six months after it has been introduced. One recommendation has been raised following our review of this area. Report 5e Page 6 of 15 Measurement of Value for Money & Organisational Benefits 28. It was verified that all of the sampled six projects were instigated in response to a business need and the anticipated benefit of the project to the Authority was realised in meeting that need. In the case of some projects such as the London Development Database (LDD), the benefit is wider than just the GLA as this will provide benefits across the GLA family and London Boroughs. 29. All projects should be properly closed down upon completion and an end of project report should be produced that evaluates the success of the project (in terms of whether it provides VFM) and highlights any areas for improvement. Audit testing showed that only one of the six projects reviewed maintained evidence of formal closedown and this was the Finance project. Of the remaining five: GLAAS had not been completed at the time of the audit Consult was terminated, and the specific reasons were observed within a MAF Blackberry was a pilot project and very small and didn't warrant a formal closedown For LDD it was agreed that the project closedown would take place after 6 months had elapsed since the completion of the project, so the closedown report was due at the time of this audit, and we have been informed that this report is being completed. For GIS there was no reason why a closedown project had not been produced. 30. Formal closedown of projects seems to be a problem generally. This is indicative of the fact that IT staff are constantly moving on to the next project before being able to properly wrap up current projects. A consequence is that it is difficult to evaluate the success of projects accurately when they are not properly closed. It is acknowledged that the Authority with the instigation of the new quality framework is attaching a higher priority to this area. 31. At the end of a project, as well as at the beginning, key stakeholders should be interviewed to ensure that any learning points are noted and discussed in order to aid continuous improvement in project management. The only one of the six projects with documented lessons learned was the Finance System project. One recommendation has been made following our review of this area. Report 5e Page 7 of 15 Audit Opinion Substantial Assurance Evaluation Opinion: While there is a basically sound system, there are areas of weakness which put some of the system objectives at risk, Testing Opinion: There is evidence that the level of non-compliance with some of the controls may put some of the system objectives at risk. Observations and Recommendations In order to assist management in using our reports: We categorise our opinions according to our assessment of the controls in place and the level of compliance with these controls Full Assurance There is a sound system of control designed to achieve the system objectives and the controls are being consistently applied. Substantial Assurance While there is a basically sound system, there are areas of weakness which put some of the system objectives at risk, and/or there is evidence that the level of non-compliance with some of the controls may put some of the system objectives at risk. Limited Assurance Weaknesses in the system of controls are such as to put the system objectives at risk, and /or the level of non-compliance puts the system objectives at risk. No Assurance Control is generally weak, leaving the system open to significant error or abuse, and/or significant non-compliance with basic controls leaves the system open to error or abuse. b) We categorise our recommendations according to their level of priority. Priority 1 Major issues for the attention of senior management. Priority 2 Other recommendations for local management action. Priority 3 Minor matters. Report 5e Page 8 of 15 Risk Identification, Management & Monitoring 1. Mitigating Controls Recommendation (Priority 3) Rationale It is recommended that the project risk registers should be consistently updated with mitigating strategies and should be reviewed by the Project Board as a standard item. Risk Management is an essential element of good project management. It is important that all project risks are identified and controlled effectively. Audit testing identified one project where there was insufficient documentary evidence within the risk log to support the mitigating strategies agreed. However, it was noted that the Project Board was reviewing the risks, although the 0project was later abandoned because risks were realised. If adequate controls are not put in place to mitigate risks then there may potentially be adverse consequences that effect the achievement of project objectives. Management response: Project & Consultancy Manager Agreed Consideration of risk register is a standing item on all Project Board agendas. Implementation Date: Has already been implemented. Report 5e Page 9 of 15 Project Costs 2. Budget Monitoring & Analysis Recommendation (Priority 2) Rationale It is recommended that a clear analysis of actual spend against budget is maintained for all projects, monitored on an ongoing basis and reviewed at the end of project. This should cover the entire budget for the duration of the project, not just the annual budget. Effective budget monitoring relies on the production of timely and accurate management information. A meaningful evaluation of the success of a project must include a measure of how the project progressed against timescale and budget. Whilst there was evidence of budget monitoring having taken place on individual projects, there was a lack of audit trail clearly showing changes to the budget and what the final position against budget was in instances where budgets rolled over a year. Without proper evaluation of the project performance in terms of budget, it is not possible to properly gauge the success of the project. There is a risk that inefficiency and wastage will not be highlighted and investigated. Management response: Executive Director of Finance & Performance Agreed The Authority’s new financial systems provides the ability to monitor budgets and actual spend over five years. Given the small size of the Authority’s capital programme, development of this element of the system has had a lower priority against revenue account and Ebis developments. Work is now in progress to establish project monitoring from 2006/07 onwards. Implementation Date: January 2007 Report 5e Page 10 of 15 Quality Management 3. Compliance with Quality Framework Recommendation (Priority 2) Rationale It is recommended that a periodic review of project working papers should be undertaken to ensure that business consultants are managing and documenting projects in the appropriate manner, in accordance with the framework. It is noted that the Technology Group has recently introduced a comprehensive new quality framework for project management, which incorporates Prince 2 methodology. None of the projects in our sample were undertaken using the new framework but they were reviewed against compliance with the old framework and good practice. It was noted that certain key stages of the projects had not been documented consistently, in particular the project closedown and post implementation review processes. Without sufficient compliance review it may be difficult to enforce and maintain compliance with the quality management framework. Management response: Project & Consultancy Manager Agreed All members of the Technology Group Projects and Consultancy Team (including the Team manager) already have a performance review objective to deliver the Quality Management Framework. The corporate performance reviews take place in June with a 6month review point in January. The Project boards sign off key project documents such as the PID, in accordance with Prince2. Project wok is authorised in accordance with GLA procedures for approval and procurement. The project support office will undertake spot checks of project documentation in the course of the year. The Manager of Projects and Consultancy will audit quality management framework documents in advance of the performance review points and any issues relating to delivery of the system will be raised in the review meetings. Technology projects may have a long (multi-year) delivery cycle. Hence 6 monthly reviews are deemed to be appropriate. Implementation Date: First review point is January 2007 Report 5e Page 11 of 15 Measurement of Value For Money 4. Project Closedown & Lessons Learned Recommendation (Priority 2) Rationale It is recommended that formal retrospective close down should be completed for all projects. Management should also ensure that formal close down procedures are consistently applied for each future project. It is a requirement of the Project Quality Management Framework that all completed projects have an End Project Report and Lessons Learned documented. There was insufficient evidence to support the close down of the GIS project. However, the remaining projects reviewed have either been formally closed down or there was sufficient documentary evidence to support the reason for formal close down not taking place. It should be noted that none of the sampled projects were undertaken to the current quality framework, and although the former Project Management Framework required formal closedown of projects, it is acknowledged that a higher priority is now being attached to this area with the instigation of the new quality framework. If projects are not properly closed down then there is potential for the Project Managers to be continually involved in ongoing work on the business area after the objectives of the project have been achieved. In addition it is difficult to properly evaluate the success of the project if there is not a definitive cut-off point. Management response: Project & Consultancy Manager Agreed End Project and Lessons Learned reports are required documents in the new Quality Management Framework. Compliance will be monitored as outlined in Recommendation 3. Implementation Date: Immediate effect with first review in January 2007 Report 5e Page 12 of 15 Appendix 1 – Audit Framework Audit Objectives The audit was designed to ensure that management has implemented adequate and effective controls within the Technology Group to ensure the effective delivery of IT projects to time, cost, and quality. Audit Approach and Methodology The audit approach was developed with reference to an assessment of risks and management controls operating within each area of the scope. The following procedures were adopted: identification of the role and objectives of each area; identification of risks within the systems, and controls in existence to allow the control objectives to be achieved; and evaluation and testing of controls within the systems. From these procedures we have identified weaknesses in the systems of control, produced specific proposals to improve the control environment and have drawn an overall conclusion on the design and operation of the system. Areas Covered Audit work was undertaken to cover controls in the following areas: Mayor / Senior Management Approval Risk Identification, Management and Monitoring Compliance with IT Project Management Methodology Organisational Benefits and User Requirements Project Delivery Timescales Project Costs Quality Requirements and Standards Measurement of Value for Money Report 5e Page 13 of 15 Appendix 2 - Staff Interviewed We would like to thank all staff that provided assistance during the course of this audit, and in particular: Project & Consultancy Manager Business Consultants (x2) Report 5e Page 14 of 15 Statement of Responsibility We take responsibility for this report, which is prepared on the basis of the limitations set out below. The matters raised in this report are only those, which came to our attention during the course of our internal audit work and are not necessarily a comprehensive statement of all the weaknesses that exist or all improvements that might be made. Recommendations for improvements should be assessed by you for their full impact before they are implemented. The performance of internal audit work is not and should not be taken as a substitute for management’s responsibilities for the application of sound management practices. We emphasise that the responsibility for a sound system of internal controls and the prevention and detection of fraud and other irregularities rests with management and work performed by internal audit should not be relied upon to identify all strengths and weaknesses in internal controls, nor relied upon to identify all circumstances of fraud or irregularity. Auditors, in conducting their work, are required to have regards to the possibility of fraud or irregularities. Even sound systems of internal control can only provide reasonable and not absolute assurance and may not be proof against collusive fraud. Internal audit procedures are designed to focus on areas as identified by management as being of greatest risk and significance and as such we rely on management to provide us full access to their accounting records and transactions for the purposes of our audit work and to ensure the authenticity of these documents. Effective and timely implementation of our recommendations by management is important for the maintenance of a reliable internal control system. Deloitte & Touche Public Sector Internal Audit Limited St Albans November 2006 In this document references to Deloitte are references to Deloitte & Touche Public Sector Internal Audit Limited. Deloitte & Touche Public Sector Internal Audit Limited is a subsidiary of Deloitte & Touche LLP, which is the United Kingdom member firm of Deloitte Touche Tohmatsu. Deloitte Touche Tohmatsu is a Swiss Verein (association), and, as neither such, neither Deloitte Touche Tohmatsu nor any of it member firms has any liability for each other’s acts or omissions. Each of the member firms is a separate and independent legal entity operating under the names “Deloitte”, “Deloitte & Touche”, “Deloitte Touche Tohmatsu”, or other related names. Services are provided by the member firms or their subsidiaries or affiliates and not by the Deloitte Touche Tohmatsu Verein. ©2006 Deloitte & Touche Public Sector Internal Audit Limited. All rights reserved. Deloitte & Touche Public Sector Internal Audit Limited is registered in England and Wales with registered number 4585162. Registered office: Stonecutter Court, 1 Stonecutter Street, London EC4A 4TR, United Kingdom Report 5e Page 15 of 15